Skip to contentSkip to conversation

scsi vs ide performance on fsync's

DISCUSSION Douglas Gilbert 10 participants 18 messages Mar 6, 2001 – Mar 12, 2001

lore
Share & tools
lore
#0 lore raw

Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Quoted 2 lines · Linus Torvalds

Well, it's entirely possible that the mid-level SCSI layer is doing something horribly stupid.

Well it's in good company as FreeBSD 4.2 on the same hardware returns the same result (including IDE timings that were too fast). My timepeg analysis showed that the SCSI disk was consuming the time, not any of the SCSI layers.

Quoted 5 lines

On the other hand, it's also entirely possible that IDE is just a lot better than what the SCSI-bigots tend to claim. It's not all that surprising, considering that the PC industry has pushed untold billions of dollars into improving IDE, with SCSI as nary a consideration. The above may just simply be the Truth, with a capital T.

What exactly do you think fsync() and fdatasync() should do? If they need to wait for dirty buffers to get flushed to the disk oxide then multiple reported IDE results to this thread are defying physics.

Doug Gilbert

#1 lore raw
Quoted 11 lines · Douglas Gilbert
Quoted 6 lines

On the other hand, it's also entirely possible that IDE is just a lot better than what the SCSI-bigots tend to claim. It's not all that surprising, considering that the PC industry has pushed untold billions of dollars into improving IDE, with SCSI as nary a consideration. The above may just simply be the Truth, with a capital T.

What exactly do you think fsync() and fdatasync() should do? If they need to wait for dirty buffers to get flushed to the disk oxide then multiple reported IDE results to this thread are defying physics.

Well, it's fairly hard for the kernel to do much about that - it's almost certainly just IDE doing write buffering on the disk itself. No OS involved.

The kernel VFS and controller layers certainly wait for the disk to tell us that the data has been written, there's no question about that. But it's also not at all unlikely that the disk itself just lies.

I don't know if there is any way to turn of a write buffer on an IDE disk.

I do remember that there were some reports of filesystem corruption with some version of Windows that turned off the machine at shutdown (using software power-off as supported by most modern motherboards), and shut down so fast that the drives had not actually written out all data. Whether the reports were true or not I do not know, but I think we can take for granted that write buffers exist.

Now, if you really care about your data integrity with a write-buffering disk, I suspect that you'd better have an UPS. At which point write buffering is a valid optimization, as long as you trust the harddisk itself not to crash even if the OS were to crash.

Of course, whether you should even trust the harddisk is another question.

Linus

#2 lore raw
Quoted 1 line

I don't know if there is any way to turn of a write buffer on an IDE disk.

hdparm has an option of this nature, but it makes no difference (as I reported). It's worth noting that even turning off UDMA to the disk on my machine doesn't help the situation - although it does slow things down a little, it's not "slow enough" to indicate that the drive is behaving properly. Might be worth running the test on some of my other machines, with their diverse collection of IDE controllers (mostly non-UDMA) and disks.

Quoted 1 line

Of course, whether you should even trust the harddisk is another question.

I think this result in itself would lead me *not* to trust the hard disk, especially an IDE one. Has anybody tried running this test with a recent IBM DeskStar - one of the ones that is the same mech as the equivalent UltraStar but with IDE controller? I only have SCSI and laptop IBMs here - all my desktop IDE drives are Seagate. However I do have one SCSI Seagate, which might be worth firing up for the occasion...

--------------------------------------------------------------

from:     Jonathan "Chromatix" Morton
mail:     [email protected]  (not for attachments)
big-mail: [email protected]
uni-mail: [email protected]

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r- y+
-----END GEEK CODE BLOCK-----

#3 lore raw
Quoted 5 lines · Jonathan Morton

Pathological shutdown pattern: assuming scatter-gather is not allowed (for IDE), and a 20ms full-stroke seek, write sectors at alternately opposite ends of the disk, working inwards until the buffer is full. 512-byte sectors, 2MB of them, is 4000 writes * 20ms = around 80 seconds

i don't understand why the disk couldn't elevator in this case and be done in 20ms + rotational.

Quoted 6 lines
Quoted 1 line

Of course, whether you should even trust the harddisk is another question.

I think this result in itself would lead me *not* to trust the hard disk, especially an IDE one. Has anybody tried running this test with a recent IBM DeskStar - one of the ones that is the same mech as the equivalent UltraStar but with IDE controller?

i assume you meant to time the xlog.c program? (or did i miss another program on the thread?)

i've an IBM-DJSA-210 (travelstar 10GB, 5411rpm) which appears to do *something* with the write cache flag -- it gets 0.10s elapsed real time in default config; and gets 2.91s if i do "hdparm -W 0".

ditto for an IBM-DTLA-307015 (deskstar 15GB 7200rpm) -- varies from .15s with write-cache to 1.8s without.

and an IBM-DTLA-307075 (deskstar 75GB 7200rpm) varies from .03s to 1.67s.

of course 1.8s is nowhere near enough time for 200 writes to complete.

so who knows what that flag is doing.

-dean

#4 lore raw
Quoted 7 lines · dean gaudet

i assume you meant to time the xlog.c program? (or did i miss another program on the thread?)

i've an IBM-DJSA-210 (travelstar 10GB, 5411rpm) which appears to do *something* with the write cache flag -- it gets 0.10s elapsed real time in default config; and gets 2.91s if i do "hdparm -W 0".

Quoted 6 lines

ditto for an IBM-DTLA-307015 (deskstar 15GB 7200rpm) -- varies from .15s with write-cache to 1.8s without.

and an IBM-DTLA-307075 (deskstar 75GB 7200rpm) varies from .03s to 1.67s.

of course 1.8s is nowhere near enough time for 200 writes to complete.

hi, not enough sleep, can't do math. 1.67s is exactly the ballpark you'd expect for 200 writes to a correctly functioning 7200rpm disk. and the travelstar appears to be doing the right thing as well.

-dean

#5 lore raw
Quoted 2 lines
Quoted 2 lines

i assume you meant to time the xlog.c program? (or did i miss another program on the thread?)

Yes.

Quoted 7 lines
Quoted 7 lines

i've an IBM-DJSA-210 (travelstar 10GB, 5411rpm) which appears to do *something* with the write cache flag -- it gets 0.10s elapsed real time in default config; and gets 2.91s if i do "hdparm -W 0".

ditto for an IBM-DTLA-307015 (deskstar 15GB 7200rpm) -- varies from .15s with write-cache to 1.8s without.

Quoted 7 lines
Quoted 3 lines

and an IBM-DTLA-307075 (deskstar 75GB 7200rpm) varies from .03s to 1.67s.

of course 1.8s is nowhere near enough time for 200 writes to complete.

hi, not enough sleep, can't do math. 1.67s is exactly the ballpark you'd expect for 200 writes to a correctly functioning 7200rpm disk. and the travelstar appears to be doing the right thing as well.

I was just about to point that out. :) I ran the program with 2000 packets in order to magnify the difference.

So, it appears that the IBM IDE drives are doing the "right thing" when write-caching is switched off, but the Seagate drive (at least the one I'm using) appears not to turn the write-caching off at all. I want to try this out with some other drives, including a Seagate SCSI drive and a different Seagate IDE drive (attached to a non-UDMA controller), and perhaps a couple of older drives which I just happen to have lying around (particularly a Maxtor and an old TravelStar with very little cache). That'll have to wait until later, though - university work beckons. :(

--------------------------------------------------------------

from:     Jonathan "Chromatix" Morton
mail:     [email protected]  (not for attachments)
big-mail: [email protected]
uni-mail: [email protected]

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r- y+
-----END GEEK CODE BLOCK-----

#6 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

Well, it's fairly hard for the kernel to do much about that - it's almost certainly just IDE doing write buffering on the disk itself. No OS involved.

I am pushing for WC to be defaulted in the off state, but as you know I have a bigger fight than caching on my hands...

Quoted 1 line

I don't know if there is any way to turn of a write buffer on an IDE disk.

You want a forced set of commands to kill caching at init?

Andre Hedrick
Linux ATA Development
ASL Kernel Development
-----------------------------------------------------------------------------

ASL, Inc.                                     Toll free: 1-877-ASL-3535
1757 Houret Court                             Fax: 1-408-941-2071
Milpitas, CA 95035                            Web: www.aslab.com
#7 lore raw
Quoted 2 lines
Quoted 1 line

I don't know if there is any way to turn of a write buffer on an IDE disk.

You want a forced set of commands to kill caching at init?

Wrong model

You want a write barrier. Write buffering (at least for short intervals) in the drive is very sensible. The kernel needs to able to send drivers a write barrier which will not be completed with outstanding commands before the barrier.

#8 lore raw
Quoted 10 lines · Alan Cox
Quoted 3 lines
Quoted 2 lines

I don't know if there is any way to turn of a write buffer on an IDE disk.

You want a forced set of commands to kill caching at init?

Wrong model

You want a write barrier. Write buffering (at least for short intervals) in the drive is very sensible. The kernel needs to able to send drivers a write barrier which will not be completed with outstanding commands before the barrier.

Agreed.

Write buffering is incredibly useful on a disk - for all the same reasons that an OS wants to do it. The disk can use write buffering to speed up writes a lot - not just lower the _perceived_ latency by the OS, but to actually improve performance too.

But Alan is right - we needs a "sync" command or something. I don't know if IDE has one (it already might, for all I know).

Linus

#9 lore raw
In reply to Alan Cox

Hi,

Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Alan Cox

You want a write barrier. Write buffering (at least for short intervals) in the drive is very sensible. The kernel needs to able to send drivers a write barrier which will not be completed with outstanding commands before the barrier.

But Alan is right - we needs a "sync" command or something. I don't know if IDE has one (it already might, for all I know).

Sync and barrier are very different models. With barriers we can enforce some elemnt of write ordering without actually waiting for the IOs to complete; with sync, we're explicitly asking to be told when the data has become persistant. We can make use of both of these.

SCSI certainly lets us do both of these operations independently. IDE has the sync/flush command afaik, but I'm not sure whether the IDE tagged command stuff has the equivalent of SCSI's ordered tag bits. Andre?

--Stephen

#10 lore raw
Quoted 18 lines · Stephen C. Tweedie

Hi,

Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Alan Cox

You want a write barrier. Write buffering (at least for short intervals) in the drive is very sensible. The kernel needs to able to send drivers a write barrier which will not be completed with outstanding commands before the barrier.

But Alan is right - we needs a "sync" command or something. I don't know if IDE has one (it already might, for all I know).

Sync and barrier are very different models. With barriers we can enforce some elemnt of write ordering without actually waiting for the IOs to complete; with sync, we're explicitly asking to be told when the data has become persistant. We can make use of both of these.

Quoted 4 lines

SCSI certainly lets us do both of these operations independently. IDE has the sync/flush command afaik, but I'm not sure whether the IDE tagged command stuff has the equivalent of SCSI's ordered tag bits. Andre?

ATA-TCQ suxs to put is plain and simple. It really requires a special host and only the HPT366 series works. It is similar but not clear as to the nature. We are debating the usage of it now in T13.

Cheers,

Andre Hedrick
Linux ATA Development

#11 lore raw
In reply to Alan Cox
Quoted 4 lines · Stephen C. Tweedie

SCSI certainly lets us do both of these operations independently. IDE has the sync/flush command afaik, but I'm not sure whether the IDE tagged command stuff has the equivalent of SCSI's ordered tag bits. Andre?

IDE has no concept of ordered tags...

Signature · 1 line
Jens Axboe
#12 lore raw
In reply to Andre Hedrick

Write caching is the culprit for the performance diff:

On IDE:
time xlog /blah.dat fsync
0.000u 0.190s 0:01.72 11.0% 0+0k 0+0io 91pf+0w

# hdparm -W 0 /dev/hda

/dev/hda:
setting drive write-caching to 0 (off)

# time xlog /blah.dat fsync
0.000u 0.220s 0:50.60 0.4%      0+0k 0+0io 91pf+0w
# hdparm -W 1 /dev/hda

/dev/hda:
setting drive write-caching to 1 (on)

# time xlog /blah.dat fsync
0.010u 0.230s 0:01.88 12.7%     0+0k 0+0io 91pf+0w

On my SCSI setup:

# time xlog /usr5/blah.dat fsync
0.020u 0.230s 0:30.48 0.8%      0+0k 0+0io 91pf+0w

________________________________________
Michael D. Black Principal Engineer
[email protected] 321-676-2923,x203
http://www.csihq.com Computer Science Innovations
http://www.csihq.com/~mike My home page
FAX 321-676-2355
----- Original Message -----
From: "Andre Hedrick" <[email protected]>
To: "Linus Torvalds" <[email protected]>
Cc: "Douglas Gilbert" <[email protected]>; <[email protected]>
Sent: Tuesday, March 06, 2001 2:12 AM
Subject: Re: scsi vs ide performance on fsync's

Quoted 3 lines · Linus Torvalds

Well, it's fairly hard for the kernel to do much about that - it's almost certainly just IDE doing write buffering on the disk itself. No OS involved.

I am pushing for WC to be defaulted in the off state, but as you know I have a bigger fight than caching on my hands...

Quoted 1 line

I don't know if there is any way to turn of a write buffer on an IDE disk.

You want a forced set of commands to kill caching at init?

Andre Hedrick
Linux ATA Development
ASL Kernel Development
----------------------------------------------------------------------------
-

ASL, Inc.                                     Toll free: 1-877-ASL-3535
1757 Houret Court                             Fax: 1-408-941-2071
Milpitas, CA 95035                            Web: www.aslab.com

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#13 lore raw

Ahh, now we're getting somewhere.

IDE:

jeremy:~# time ./xlog file.out fsync
real    0m33.739s
user    0m0.010s
sys     0m0.120s

so now this corresponds to the performance we're seeing on SCSI.

So I guess what I'm wondering now is can or should anything be done about this on the SCSI side?

Thanks
-jeremy

Quoted 71 lines · Mike Black

Write caching is the culprit for the performance diff:

On IDE:
time xlog /blah.dat fsync
0.000u 0.190s 0:01.72 11.0% 0+0k 0+0io 91pf+0w

# hdparm -W 0 /dev/hda

/dev/hda:
setting drive write-caching to 0 (off)

# time xlog /blah.dat fsync
0.000u 0.220s 0:50.60 0.4%      0+0k 0+0io 91pf+0w
# hdparm -W 1 /dev/hda

/dev/hda:
setting drive write-caching to 1 (on)

# time xlog /blah.dat fsync
0.010u 0.230s 0:01.88 12.7%     0+0k 0+0io 91pf+0w

On my SCSI setup:

# time xlog /usr5/blah.dat fsync
0.020u 0.230s 0:30.48 0.8%      0+0k 0+0io 91pf+0w

________________________________________
Michael D. Black Principal Engineer
[email protected] 321-676-2923,x203
http://www.csihq.com Computer Science Innovations
http://www.csihq.com/~mike My home page
FAX 321-676-2355
----- Original Message -----
From: "Andre Hedrick" <[email protected]>
To: "Linus Torvalds" <[email protected]>
Cc: "Douglas Gilbert" <[email protected]>; <[email protected]>
Sent: Tuesday, March 06, 2001 2:12 AM
Subject: Re: scsi vs ide performance on fsync's

Quoted 3 lines · Linus Torvalds

Well, it's fairly hard for the kernel to do much about that - it's almost certainly just IDE doing write buffering on the disk itself. No OS involved.

I am pushing for WC to be defaulted in the off state, but as you know I have a bigger fight than caching on my hands...

Quoted 1 line

I don't know if there is any way to turn of a write buffer on an IDE disk.

You want a forced set of commands to kill caching at init?

Andre Hedrick
Linux ATA Development
ASL Kernel Development
----------------------------------------------------------------------------
-

ASL, Inc.                                     Toll free: 1-877-ASL-3535
1757 Houret Court                             Fax: 1-408-941-2071
Milpitas, CA 95035                            Web: www.aslab.com

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 1 line
this is my sig.
#14 lore raw
Quoted 3 lines · Mike Black
Quoted 2 lines

Write caching is the culprit for the performance diff:

Indeed, and my during-the-boring-lecture benchmark on my 18Gb IBM TravelStar bears this out. I was confused earlier by the fact that one of my Seagate drives blatently ignores the no-write-caching request I sent it. :P

Quoted 6 lines · Jeremy Hansen

Ahh, now we're getting somewhere.

so now this corresponds to the performance we're seeing on SCSI.

So I guess what I'm wondering now is can or should anything be done about this on the SCSI side?

Maybe, it depends on your perspective. In my personal opinion, the IDE behaviour is incorrect and some way of dealing with it (while still retaining the benefits of write-caching for normal applications) would be highly desirable. However, some applications may like or partially rely on that behaviour, to gain better on-disk data consistency while not suffering too much in performance (eg. the transaction database mentioned by at least one poster).

The way to make all parties happy is to fix the IDE driver (or drives!) and make sure an *alternative* syscall is available which flushes the buffers asynchronously, as per the current IDE behaviour. It shouldn't be too hard to make the SCSI driver use that behaviour in the alternative syscall (which may already exist, I don't know Linux well enough to say).

May this be a warning to all hardware manufacturers who "tweak" their hardware to gain better benchmark results without actually increasing performance - you *will* be found out!

--------------------------------------------------------------

from:     Jonathan "Chromatix" Morton
mail:     [email protected]  (not for attachments)
big-mail: [email protected]
uni-mail: [email protected]

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r- y+
-----END GEEK CODE BLOCK-----

#15 lore raw
In reply to Jeremy Hansen

So in the meantime as this gets worked out on a lower level, we've decided to take the fsync() out of berkeley db for mysql transaction logs and mount the filesystem -o sync.

Can anyone perhaps tell me why this may be a bad idea?

Thanks
-jeremy

Quoted 95 lines · Jeremy Hansen

Ahh, now we're getting somewhere.

IDE:

jeremy:~# time ./xlog file.out fsync
real    0m33.739s
user    0m0.010s
sys     0m0.120s

so now this corresponds to the performance we're seeing on SCSI.

So I guess what I'm wondering now is can or should anything be done about this on the SCSI side?

Thanks
-jeremy

Quoted 71 lines · Mike Black

Write caching is the culprit for the performance diff:

On IDE:
time xlog /blah.dat fsync
0.000u 0.190s 0:01.72 11.0% 0+0k 0+0io 91pf+0w

# hdparm -W 0 /dev/hda

/dev/hda:
setting drive write-caching to 0 (off)

# time xlog /blah.dat fsync
0.000u 0.220s 0:50.60 0.4%      0+0k 0+0io 91pf+0w
# hdparm -W 1 /dev/hda

/dev/hda:
setting drive write-caching to 1 (on)

# time xlog /blah.dat fsync
0.010u 0.230s 0:01.88 12.7%     0+0k 0+0io 91pf+0w

On my SCSI setup:

# time xlog /usr5/blah.dat fsync
0.020u 0.230s 0:30.48 0.8%      0+0k 0+0io 91pf+0w

________________________________________
Michael D. Black Principal Engineer
[email protected] 321-676-2923,x203
http://www.csihq.com Computer Science Innovations
http://www.csihq.com/~mike My home page
FAX 321-676-2355
----- Original Message -----
From: "Andre Hedrick" <[email protected]>
To: "Linus Torvalds" <[email protected]>
Cc: "Douglas Gilbert" <[email protected]>; <[email protected]>
Sent: Tuesday, March 06, 2001 2:12 AM
Subject: Re: scsi vs ide performance on fsync's

Quoted 3 lines · Linus Torvalds

Well, it's fairly hard for the kernel to do much about that - it's almost certainly just IDE doing write buffering on the disk itself. No OS involved.

I am pushing for WC to be defaulted in the off state, but as you know I have a bigger fight than caching on my hands...

Quoted 1 line

I don't know if there is any way to turn of a write buffer on an IDE disk.

You want a forced set of commands to kill caching at init?

Andre Hedrick
Linux ATA Development
ASL Kernel Development
----------------------------------------------------------------------------
-

ASL, Inc.                                     Toll free: 1-877-ASL-3535
1757 Houret Court                             Fax: 1-408-941-2071
Milpitas, CA 95035                            Web: www.aslab.com

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 1 line
this is my sig.
#16 lore raw
Quoted 6 lines · Jeremy Hansen

So in the meantime as this gets worked out on a lower level, we've decided to take the fsync() out of berkeley db for mysql transaction logs and mount the filesystem -o sync.

Can anyone perhaps tell me why this may be a bad idea?

Two reasons:
- it doesn't help. The disk will _still_ do write buffering. It's the DISK, not the OS. It doesn't matter what you do.
- your performance will suck.

Use fsync(). That's what it's there for.

Tell people who don't have an UPS to disable write caching. If they have one (of the many, apparently) IDE disks that refuse to disable it, tell them to either get an UPS, or to switch to another disk.

Linus

#17 lore raw
In reply to Jeremy Hansen

Hi,

Quoted 11 lines · Linus Torvalds
Quoted 6 lines · Jeremy Hansen

So in the meantime as this gets worked out on a lower level, we've decided to take the fsync() out of berkeley db for mysql transaction logs and mount the filesystem -o sync.

Can anyone perhaps tell me why this may be a bad idea?

- it doesn't help. The disk will _still_ do write buffering. It's the DISK, not the OS. It doesn't matter what you do.
- your performance will suck.

Added to which, "-o sync" only enables sync metadata updates. It still doesn't force an fsync on data writes.

--Stephen