scsi vs ide performance on fsync's
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Quoted 2 lines · Linus Torvalds
Well, it's entirely possible that the mid-level SCSI layer is doing something horribly stupid.
Well it's in good company as FreeBSD 4.2 on the same hardware returns the same result (including IDE timings that were too fast). My timepeg analysis showed that the SCSI disk was consuming the time, not any of the SCSI layers.
Quoted 5 lines
On the other hand, it's also entirely possible that IDE is just a lot better than what the SCSI-bigots tend to claim. It's not all that surprising, considering that the PC industry has pushed untold billions of dollars into improving IDE, with SCSI as nary a consideration. The above may just simply be the Truth, with a capital T.
What exactly do you think fsync() and fdatasync() should do? If they need to wait for dirty buffers to get flushed to the disk oxide then multiple reported IDE results to this thread are defying physics.
Doug Gilbert
Quoted 11 lines · Douglas Gilbert
Quoted 6 lines
On the other hand, it's also entirely possible that IDE is just a lot better than what the SCSI-bigots tend to claim. It's not all that surprising, considering that the PC industry has pushed untold billions of dollars into improving IDE, with SCSI as nary a consideration. The above may just simply be the Truth, with a capital T.
What exactly do you think
fsync()andfdatasync()should do? If they need to wait for dirty buffers to get flushed to the disk oxide then multiple reported IDE results to this thread are defying physics.
Well, it's fairly hard for the kernel to do much about that - it's almost certainly just IDE doing write buffering on the disk itself. No OS involved.
The kernel VFS and controller layers certainly wait for the disk to tell us that the data has been written, there's no question about that. But it's also not at all unlikely that the disk itself just lies.
I don't know if there is any way to turn of a write buffer on an IDE disk.
I do remember that there were some reports of filesystem corruption with some version of Windows that turned off the machine at shutdown (using software power-off as supported by most modern motherboards), and shut down so fast that the drives had not actually written out all data. Whether the reports were true or not I do not know, but I think we can take for granted that write buffers exist.
Now, if you really care about your data integrity with a write-buffering disk, I suspect that you'd better have an UPS. At which point write buffering is a valid optimization, as long as you trust the harddisk itself not to crash even if the OS were to crash.
Of course, whether you should even trust the harddisk is another question.
Linus
Quoted 1 line
I don't know if there is any way to turn of a write buffer on an IDE disk.
hdparm has an option of this nature, but it makes no difference (as I reported). It's worth noting that even turning off UDMA to the disk on my machine doesn't help the situation - although it does slow things down a little, it's not "slow enough" to indicate that the drive is behaving properly. Might be worth running the test on some of my other machines, with their diverse collection of IDE controllers (mostly non-UDMA) and disks.
Quoted 1 line
Of course, whether you should even trust the harddisk is another question.
I think this result in itself would lead me *not* to trust the hard disk, especially an IDE one. Has anybody tried running this test with a recent IBM DeskStar - one of the ones that is the same mech as the equivalent UltraStar but with IDE controller? I only have SCSI and laptop IBMs here - all my desktop IDE drives are Seagate. However I do have one SCSI Seagate, which might be worth firing up for the occasion...
--------------------------------------------------------------
from: Jonathan "Chromatix" Morton mail: [email protected] (not for attachments) big-mail: [email protected] uni-mail: [email protected]
The key to knowledge is not to rely on people to teach you it.
Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/
-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r- y+
-----END GEEK CODE BLOCK-----
Quoted 5 lines · Jonathan Morton
Pathological shutdown pattern: assuming scatter-gather is not allowed (for IDE), and a 20ms full-stroke seek, write sectors at alternately opposite ends of the disk, working inwards until the buffer is full. 512-byte sectors, 2MB of them, is 4000 writes * 20ms = around 80 seconds
i don't understand why the disk couldn't elevator in this case and be done in 20ms + rotational.
Quoted 6 lines
Quoted 1 line
Of course, whether you should even trust the harddisk is another question.
I think this result in itself would lead me *not* to trust the hard disk, especially an IDE one. Has anybody tried running this test with a recent IBM DeskStar - one of the ones that is the same mech as the equivalent UltraStar but with IDE controller?
i assume you meant to time the xlog.c program? (or did i miss another program on the thread?)
i've an IBM-DJSA-210 (travelstar 10GB, 5411rpm) which appears to do *something* with the write cache flag -- it gets 0.10s elapsed real time in default config; and gets 2.91s if i do "hdparm -W 0".
ditto for an IBM-DTLA-307015 (deskstar 15GB 7200rpm) -- varies from .15s with write-cache to 1.8s without.
and an IBM-DTLA-307075 (deskstar 75GB 7200rpm) varies from .03s to 1.67s.
of course 1.8s is nowhere near enough time for 200 writes to complete.
so who knows what that flag is doing.
-dean
Quoted 7 lines · dean gaudet
i assume you meant to time the
xlog.cprogram? (or did i miss another program on the thread?)i've an IBM-DJSA-210 (travelstar 10GB, 5411rpm) which appears to do *something* with the write cache flag -- it gets 0.10s elapsed real time in default config; and gets 2.91s if i do "hdparm -W 0".
Quoted 6 lines
ditto for an IBM-DTLA-307015 (deskstar 15GB 7200rpm) -- varies from .15s with write-cache to 1.8s without.
and an IBM-DTLA-307075 (deskstar 75GB 7200rpm) varies from .03s to 1.67s.
of course 1.8s is nowhere near enough time for 200 writes to complete.
hi, not enough sleep, can't do math. 1.67s is exactly the ballpark you'd expect for 200 writes to a correctly functioning 7200rpm disk. and the travelstar appears to be doing the right thing as well.
-dean
Quoted 2 lines
Quoted 2 lines
i assume you meant to time the
xlog.cprogram? (or did i miss another program on the thread?)
Yes.
Quoted 7 lines
Quoted 7 lines
i've an IBM-DJSA-210 (travelstar 10GB, 5411rpm) which appears to do *something* with the write cache flag -- it gets 0.10s elapsed real time in default config; and gets 2.91s if i do "hdparm -W 0".
ditto for an IBM-DTLA-307015 (deskstar 15GB 7200rpm) -- varies from .15s with write-cache to 1.8s without.
Quoted 7 lines
Quoted 3 lines
and an IBM-DTLA-307075 (deskstar 75GB 7200rpm) varies from .03s to 1.67s.
of course 1.8s is nowhere near enough time for 200 writes to complete.
hi, not enough sleep, can't do math. 1.67s is exactly the ballpark you'd expect for 200 writes to a correctly functioning 7200rpm disk. and the travelstar appears to be doing the right thing as well.
I was just about to point that out. :) I ran the program with 2000 packets in order to magnify the difference.
So, it appears that the IBM IDE drives are doing the "right thing" when write-caching is switched off, but the Seagate drive (at least the one I'm using) appears not to turn the write-caching off at all. I want to try this out with some other drives, including a Seagate SCSI drive and a different Seagate IDE drive (attached to a non-UDMA controller), and perhaps a couple of older drives which I just happen to have lying around (particularly a Maxtor and an old TravelStar with very little cache). That'll have to wait until later, though - university work beckons. :(
--------------------------------------------------------------
from: Jonathan "Chromatix" Morton mail: [email protected] (not for attachments) big-mail: [email protected] uni-mail: [email protected]
The key to knowledge is not to rely on people to teach you it.
Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/
-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r- y+
-----END GEEK CODE BLOCK-----
Quoted 3 lines · Linus Torvalds
Well, it's fairly hard for the kernel to do much about that - it's almost certainly just IDE doing write buffering on the disk itself. No OS involved.
I am pushing for WC to be defaulted in the off state, but as you know I have a bigger fight than caching on my hands...
Quoted 1 line
I don't know if there is any way to turn of a write buffer on an IDE disk.
You want a forced set of commands to kill caching at init?
Andre Hedrick
Linux ATA Development
ASL Kernel Development
-----------------------------------------------------------------------------
ASL, Inc. Toll free: 1-877-ASL-3535 1757 Houret Court Fax: 1-408-941-2071 Milpitas, CA 95035 Web: www.aslab.com
Quoted 2 lines
Quoted 1 line
I don't know if there is any way to turn of a write buffer on an IDE disk.
You want a forced set of commands to kill caching at init?
Wrong model
You want a write barrier. Write buffering (at least for short intervals) in the drive is very sensible. The kernel needs to able to send drivers a write barrier which will not be completed with outstanding commands before the barrier.
Quoted 10 lines · Alan Cox
Quoted 3 lines
Quoted 2 lines
I don't know if there is any way to turn of a write buffer on an IDE disk.
You want a forced set of commands to kill caching at init?
Wrong model
You want a write barrier. Write buffering (at least for short intervals) in the drive is very sensible. The kernel needs to able to send drivers a write barrier which will not be completed with outstanding commands before the barrier.
Agreed.
Write buffering is incredibly useful on a disk - for all the same reasons that an OS wants to do it. The disk can use write buffering to speed up writes a lot - not just lower the _perceived_ latency by the OS, but to actually improve performance too.
But Alan is right - we needs a "sync" command or something. I don't know if IDE has one (it already might, for all I know).
Linus
Hi,
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Alan Cox
You want a write barrier. Write buffering (at least for short intervals) in the drive is very sensible. The kernel needs to able to send drivers a write barrier which will not be completed with outstanding commands before the barrier.
But Alan is right - we needs a "sync" command or something. I don't know if IDE has one (it already might, for all I know).
Sync and barrier are very different models. With barriers we can enforce some elemnt of write ordering without actually waiting for the IOs to complete; with sync, we're explicitly asking to be told when the data has become persistant. We can make use of both of these.
SCSI certainly lets us do both of these operations independently. IDE has the sync/flush command afaik, but I'm not sure whether the IDE tagged command stuff has the equivalent of SCSI's ordered tag bits. Andre?
--Stephen
Quoted 18 lines · Stephen C. Tweedie
Hi,
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Alan Cox
You want a write barrier. Write buffering (at least for short intervals) in the drive is very sensible. The kernel needs to able to send drivers a write barrier which will not be completed with outstanding commands before the barrier.
But Alan is right - we needs a "sync" command or something. I don't know if IDE has one (it already might, for all I know).
Sync and barrier are very different models. With barriers we can enforce some elemnt of write ordering without actually waiting for the IOs to complete; with sync, we're explicitly asking to be told when the data has become persistant. We can make use of both of these.
Quoted 4 lines
SCSI certainly lets us do both of these operations independently. IDE has the sync/flush command afaik, but I'm not sure whether the IDE tagged command stuff has the equivalent of SCSI's ordered tag bits. Andre?
ATA-TCQ suxs to put is plain and simple. It really requires a special host and only the HPT366 series works. It is similar but not clear as to the nature. We are debating the usage of it now in T13.
Cheers,
Andre Hedrick
Linux ATA Development
Quoted 4 lines · Stephen C. Tweedie
SCSI certainly lets us do both of these operations independently. IDE has the sync/flush command afaik, but I'm not sure whether the IDE tagged command stuff has the equivalent of SCSI's ordered tag bits. Andre?
IDE has no concept of ordered tags...
Signature · 1 line
Jens Axboe
Write caching is the culprit for the performance diff:
On IDE:
time xlog /blah.dat fsync
0.000u 0.190s 0:01.72 11.0% 0+0k 0+0io 91pf+0w
# hdparm -W 0 /dev/hda
/dev/hda:
setting drive write-caching to 0 (off)
# time xlog /blah.dat fsync 0.000u 0.220s 0:50.60 0.4% 0+0k 0+0io 91pf+0w # hdparm -W 1 /dev/hda
/dev/hda:
setting drive write-caching to 1 (on)
# time xlog /blah.dat fsync 0.010u 0.230s 0:01.88 12.7% 0+0k 0+0io 91pf+0w
On my SCSI setup:
# time xlog /usr5/blah.dat fsync 0.020u 0.230s 0:30.48 0.8% 0+0k 0+0io 91pf+0w
________________________________________
Michael D. Black Principal Engineer
[email protected] 321-676-2923,x203
http://www.csihq.com Computer Science Innovations
http://www.csihq.com/~mike My home page
FAX 321-676-2355
----- Original Message -----
From: "Andre Hedrick" <[email protected]>
To: "Linus Torvalds" <[email protected]>
Cc: "Douglas Gilbert" <[email protected]>; <[email protected]>
Sent: Tuesday, March 06, 2001 2:12 AM
Subject: Re: scsi vs ide performance on fsync's
Quoted 3 lines · Linus Torvalds
Well, it's fairly hard for the kernel to do much about that - it's almost certainly just IDE doing write buffering on the disk itself. No OS involved.
I am pushing for WC to be defaulted in the off state, but as you know I have a bigger fight than caching on my hands...
Quoted 1 line
I don't know if there is any way to turn of a write buffer on an IDE disk.
You want a forced set of commands to kill caching at init?
Andre Hedrick
Linux ATA Development
ASL Kernel Development
----------------------------------------------------------------------------
-
ASL, Inc. Toll free: 1-877-ASL-3535 1757 Houret Court Fax: 1-408-941-2071 Milpitas, CA 95035 Web: www.aslab.com
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Ahh, now we're getting somewhere.
IDE:
jeremy:~# time ./xlog file.out fsync
real 0m33.739s user 0m0.010s sys 0m0.120s
so now this corresponds to the performance we're seeing on SCSI.
So I guess what I'm wondering now is can or should anything be done about this on the SCSI side?
Thanks
-jeremy
Quoted 71 lines · Mike Black
Write caching is the culprit for the performance diff:
On IDE:
time xlog /blah.dat fsync
0.000u 0.190s 0:01.72 11.0% 0+0k 0+0io 91pf+0w# hdparm -W 0 /dev/hda
/dev/hda:
setting drive write-caching to 0 (off)# time xlog /blah.dat fsync 0.000u 0.220s 0:50.60 0.4% 0+0k 0+0io 91pf+0w # hdparm -W 1 /dev/hda
/dev/hda:
setting drive write-caching to 1 (on)# time xlog /blah.dat fsync 0.010u 0.230s 0:01.88 12.7% 0+0k 0+0io 91pf+0wOn my SCSI setup:
# time xlog /usr5/blah.dat fsync 0.020u 0.230s 0:30.48 0.8% 0+0k 0+0io 91pf+0w________________________________________
Michael D. Black Principal Engineer
[email protected] 321-676-2923,x203
http://www.csihq.com Computer Science Innovations
http://www.csihq.com/~mike My home page
FAX 321-676-2355
----- Original Message -----
From: "Andre Hedrick" <[email protected]>
To: "Linus Torvalds" <[email protected]>
Cc: "Douglas Gilbert" <[email protected]>; <[email protected]>
Sent: Tuesday, March 06, 2001 2:12 AM
Subject: Re: scsi vs ide performance on fsync'sQuoted 3 lines · Linus Torvalds
Well, it's fairly hard for the kernel to do much about that - it's almost certainly just IDE doing write buffering on the disk itself. No OS involved.
I am pushing for WC to be defaulted in the off state, but as you know I have a bigger fight than caching on my hands...
Quoted 1 line
I don't know if there is any way to turn of a write buffer on an IDE disk.
You want a forced set of commands to kill caching at init?
Andre Hedrick
Linux ATA Development
ASL Kernel Development
----------------------------------------------------------------------------
-ASL, Inc. Toll free: 1-877-ASL-3535 1757 Houret Court Fax: 1-408-941-2071 Milpitas, CA 95035 Web: www.aslab.com-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
this is my sig.
Quoted 3 lines · Mike Black
Quoted 2 lines
Write caching is the culprit for the performance diff:
Indeed, and my during-the-boring-lecture benchmark on my 18Gb IBM TravelStar bears this out. I was confused earlier by the fact that one of my Seagate drives blatently ignores the no-write-caching request I sent it. :P
Quoted 6 lines · Jeremy Hansen
Ahh, now we're getting somewhere.
so now this corresponds to the performance we're seeing on SCSI.
So I guess what I'm wondering now is can or should anything be done about this on the SCSI side?
Maybe, it depends on your perspective. In my personal opinion, the IDE behaviour is incorrect and some way of dealing with it (while still retaining the benefits of write-caching for normal applications) would be highly desirable. However, some applications may like or partially rely on that behaviour, to gain better on-disk data consistency while not suffering too much in performance (eg. the transaction database mentioned by at least one poster).
The way to make all parties happy is to fix the IDE driver (or drives!) and make sure an *alternative* syscall is available which flushes the buffers asynchronously, as per the current IDE behaviour. It shouldn't be too hard to make the SCSI driver use that behaviour in the alternative syscall (which may already exist, I don't know Linux well enough to say).
May this be a warning to all hardware manufacturers who "tweak" their hardware to gain better benchmark results without actually increasing performance - you *will* be found out!
--------------------------------------------------------------
from: Jonathan "Chromatix" Morton mail: [email protected] (not for attachments) big-mail: [email protected] uni-mail: [email protected]
The key to knowledge is not to rely on people to teach you it.
Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/
-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r- y+
-----END GEEK CODE BLOCK-----
So in the meantime as this gets worked out on a lower level, we've decided to take the fsync() out of berkeley db for mysql transaction logs and mount the filesystem -o sync.
Can anyone perhaps tell me why this may be a bad idea?
Thanks
-jeremy
Quoted 95 lines · Jeremy Hansen
Ahh, now we're getting somewhere.
IDE:
jeremy:~# time ./xlog file.out fsyncreal 0m33.739s user 0m0.010s sys 0m0.120sso now this corresponds to the performance we're seeing on SCSI.
So I guess what I'm wondering now is can or should anything be done about this on the SCSI side?
Thanks
-jeremyQuoted 71 lines · Mike Black
Write caching is the culprit for the performance diff:
On IDE:
time xlog /blah.dat fsync
0.000u 0.190s 0:01.72 11.0% 0+0k 0+0io 91pf+0w# hdparm -W 0 /dev/hda
/dev/hda:
setting drive write-caching to 0 (off)# time xlog /blah.dat fsync 0.000u 0.220s 0:50.60 0.4% 0+0k 0+0io 91pf+0w # hdparm -W 1 /dev/hda
/dev/hda:
setting drive write-caching to 1 (on)# time xlog /blah.dat fsync 0.010u 0.230s 0:01.88 12.7% 0+0k 0+0io 91pf+0wOn my SCSI setup:
# time xlog /usr5/blah.dat fsync 0.020u 0.230s 0:30.48 0.8% 0+0k 0+0io 91pf+0w________________________________________
Michael D. Black Principal Engineer
[email protected] 321-676-2923,x203
http://www.csihq.com Computer Science Innovations
http://www.csihq.com/~mike My home page
FAX 321-676-2355
----- Original Message -----
From: "Andre Hedrick" <[email protected]>
To: "Linus Torvalds" <[email protected]>
Cc: "Douglas Gilbert" <[email protected]>; <[email protected]>
Sent: Tuesday, March 06, 2001 2:12 AM
Subject: Re: scsi vs ide performance on fsync'sQuoted 3 lines · Linus Torvalds
Well, it's fairly hard for the kernel to do much about that - it's almost certainly just IDE doing write buffering on the disk itself. No OS involved.
I am pushing for WC to be defaulted in the off state, but as you know I have a bigger fight than caching on my hands...
Quoted 1 line
I don't know if there is any way to turn of a write buffer on an IDE disk.
You want a forced set of commands to kill caching at init?
Andre Hedrick
Linux ATA Development
ASL Kernel Development
----------------------------------------------------------------------------
-ASL, Inc. Toll free: 1-877-ASL-3535 1757 Houret Court Fax: 1-408-941-2071 Milpitas, CA 95035 Web: www.aslab.com-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
this is my sig.
Quoted 6 lines · Jeremy Hansen
So in the meantime as this gets worked out on a lower level, we've decided to take the
fsync()out of berkeley db for mysql transaction logs and mount the filesystem -o sync.Can anyone perhaps tell me why this may be a bad idea?
Two reasons:
- it doesn't help. The disk will _still_ do write buffering. It's the DISK, not the OS. It doesn't matter what you do.
- your performance will suck.
Use fsync(). That's what it's there for.
Tell people who don't have an UPS to disable write caching. If they have one (of the many, apparently) IDE disks that refuse to disable it, tell them to either get an UPS, or to switch to another disk.
Linus
Hi,
Quoted 11 lines · Linus Torvalds
Quoted 6 lines · Jeremy Hansen
So in the meantime as this gets worked out on a lower level, we've decided to take the
fsync()out of berkeley db for mysql transaction logs and mount the filesystem -o sync.Can anyone perhaps tell me why this may be a bad idea?
- it doesn't help. The disk will _still_ do write buffering. It's the DISK, not the OS. It doesn't matter what you do.
- your performance will suck.
Added to which, "-o sync" only enables sync metadata updates. It still doesn't force an fsync on data writes.
--Stephen