Disk hardware caching, performance, and journalling
Hi,
I made a couple of discoveries today which were surprising to me:
1. Disk hardware caching defaults to ON. (hdparm -W1 /dev/hda)
2. It makes a *big* difference in write performance.
I had always thought that the default was off. I also always assumed that a small cache behind a large (OS) cache would make no difference.
Here are my results with bonnie under kernel 2.4.14 on a reiserfs with a maxtor Diamond max+ 60GB udma100 drive:
Write caching on:
-------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
256 12618 97.1 38027 36.3 9647 6.9 11250 73.6 31832
12.1 200.9 1.2
Write caching off:
-------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
256 9917 76.3 12280 11.2 5159 3.5 9934 65.3 33056
14.1 203.9 1.4
Note that block writes are over 3 times faster with caching on.
So what are the implications here for journalling? Do I have to turn off caching and suffer a huge performance hit?
-Steve Bergman
Quoted 2 lines · Steve Bergman
Note that block writes are over 3 times faster with caching on.
With a large linear write, linux typically feeds requests into the disk like this:
write 248 sectors
write 248 sectors
...
write 248 sectors
write 8 sectors
write 248 sectors
...
Now, 248+8 sectors is 128 kbytes. A track is, say, 300 kbytes.
With writebehind the disk can write that entire track in pretty much a single spin. But if we're waiting on the result of each request we'll lose revolutions. In synchronous mode it's going to take three or four spins to write a track.
Quoted 2 lines
So what are the implications here for journalling? Do I have to turn off caching and suffer a huge performance hit?
In theory, yes. In my opinion, no. For ext3, at least. Caching isn't bad per-se. It's reordering which can break the journalling constraints. But given that the journal is, we hope, a strictly ascending and (we really hope) contiguous chunk of blocks, it's quite unlikely that the disk will decide to write them in an unexpected order. This is especially true if the journal was created when the disk was relatively unfragmented.
And if the disk _does_ write them in the wrong order, it has to be specifically the journal commit block which was written prior to some data blocks. And you need to lose power (not just crash) prior to the data blocks hitting disk. It's a very small time window containing an improbable occurrence.
Now that's all just vigorous handwaving, and may be wrong, and yes, we really need a way of propagating barriers down to the request queue. But I've not seen a whisker of a report which indicates that write reordering has caused on-recovery corruption.
-
Quoted 7 lines · Andrew Morton
In theory, yes. In my opinion, no. For ext3, at least. Caching isn't bad per-se. It's reordering which can break the journalling constraints. But given that the journal is, we hope, a strictly ascending and (we really hope) contiguous chunk of blocks, it's quite unlikely that the disk will decide to write them in an unexpected order. This is especially true if the journal was created when the disk was relatively unfragmented.
When the journal resides on multiple disks or disks different from the actual data (think LVM or RAID), all bets are off. You need synchronous write operations in these cases, I think.
Signature · 3 lines
Florian Weimer [email protected] University of Stuttgart http://cert.uni-stuttgart.de/ RUS-CERT +49-711-685-5973/fax +49-711-685-5898
Quoted 2 lines
So what are the implications here for journalling? Do I have to turn off caching and suffer a huge performance hit?
why does everyone get freaked out about disk caches? afaikt, there's only an O(50ms) window at each catastrophic power failure:
trivial for any reasonable rate of failures...
On Sat, Nov 24, 2001 at 11:08:17AM -0800, Andrew Morton wrote:
| With writebehind the disk can write that entire track in pretty | much a single spin. But if we're waiting on the result of each | request we'll lose revolutions. In synchronous mode it's going | to take three or four spins to write a track. | | > So what are the implications here for journalling? Do I have to turn | > off caching and suffer a huge performance hit? | | In theory, yes. In my opinion, no. For ext3, at least. Caching | isn't bad per-se. It's reordering which can break the journalling | constraints. But given that the journal is, we hope, a strictly | ascending and (we really hope) contiguous chunk of blocks, it's | quite unlikely that the disk will decide to write them in an | unexpected order. This is especially true if the journal was | created when the disk was relatively unfragmented. | | And if the disk _does_ write them in the wrong order, it has | to be specifically the journal commit block which was written | prior to some data blocks. And you need to lose power (not | just crash) prior to the data blocks hitting disk. It's a | very small time window containing an improbable occurrence. | | Now that's all just vigorous handwaving, and may be wrong, | and yes, we really need a way of propagating barriers down | to the request queue. But I've not seen a whisker of a report | which indicates that write reordering has caused on-recovery | corruption.
What if (and maybe it is so, or maybe not) the write cache does write-back (or write-behind) zones per track? That is, when it gets a write request either when the cache is not dirty, or is in the same _physical_ track that everything that is in there dirty goes into, it gets queued. Then when some time passes, or a request goes to a new track, it will write that track now and move on. IWSTM this gives you the advantage of "one spin" writes for sequential data, and still keeps the order right for journaled data. Could they be doing this?
Signature · 4 lines
----------------------------------------------------------------- | Phil Howard - KA9WGN | Dallas | http://linuxhomepage.com/ | | [email protected] | Texas, USA | http://phil.ipal.org/ | -----------------------------------------------------------------
On Sat, Nov 24, 2001 at 12:36:18PM -0600, Steve Bergman wrote:
1. Disk hardware caching defaults to ON. (hdparm -W1 /dev/hda)
2. It makes a *big* difference in write performance.
I depends on the drive, my IDE drives do default to on, my SCSI drives do not.
The difference in write performance doesn't seem to be a problem other that in contrived situations (eg. streaming 5G of data to disk takes the same amount of time either way, but untar something then 'sync' is faster with the drive caching).
It also depends of your filesystems to some extent and the operations being performed [1].
So what are the implications here for journalling? Do I have to turn off caching and suffer a huge performance hit?
Yes. I do this on workstations and it doesn't seem to hurt in practice (only in benchmarks).
I can't comment on your bonnie++ results and I have no idea how well they reflect reality (I assume to a large extent they try to though).
--cw
[1] XFS rm -rf some_large_dir bites with drive-caching off for example.
On Sat, Nov 24, 2001 at 11:08:17AM -0800, Andrew Morton wrote:
In theory, yes. In my opinion, no. For ext3, at least. Caching isn't bad per-se. It's reordering which can break the journalling constraints.
Some disks[1] most definately do reorder; I've actually been able to demonstrate this in some circumstances (it wasn't trivial to produce and required several attempts).
--cw
[1] SCSI, which we know does and will reorder writes when a barrier isn't specificed, it appears IDE drives can do the same but lack of hot-swap makes testing tedious :)
On Sat, Nov 24, 2001 at 02:39:44PM -0500, Mark Hahn wrote:
why does everyone get freaked out about disk caches? afaikt, there's only an O(50ms) window at each catastrophic power failure:
trivial for any reasonable rate of failures...
If your disks are busy all the time (eg. a large mail server) then you will trivially hit this and it will be a problem.
--cw
I think if you have a large mail server and zero power protection, you've got much larger problems to worry about than write-behind caching on your disk drives... my servers have never (in my memory) experienced a catastrophic power failure, because they're too easy to avoid.
----- Original Message -----
From: "Chris Wedgwood" <[email protected]>
To: "Mark Hahn" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, November 25, 2001 2:23 AM
Subject: Re: Disk hardware caching, performance, and journalling
Quoted 20 lines · Mark Hahn
why does everyone get freaked out about disk caches? afaikt, there's only an O(50ms) window at each catastrophic power failure:
trivial for any reasonable rate of failures...If your disks are busy all the time (eg. a large mail server) then you will trivially hit this and it will be a problem.
--cw
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 2 lines · In article <00e001c175fa$90d02b40$6caaa8c0@kevin> you wrote:
disk drives... my servers have never (in my memory) experienced a catastrophic power failure, because they're too easy to avoid.
Well, even UPS can fail. But battery protected RAID Controllers are a MUST, anyway.
Greetings
Bernd
On Sun, Nov 25, 2001 at 02:45:57PM -0700, Kevin P. Fleming wrote:
I think if you have a large mail server and zero power protection, you've got much larger problems to worry about than write-behind caching on your disk drives... my servers have never (in my memory) experienced a catastrophic power failure, because they're too easy to avoid.
In the specific case of email; you want to make certain guarantees, and having data written to non-volatile storage is one of them.
As for power-failures, given enough time and enough hardware you will get them, even if your machines are dual or triple powered of diverse UPSs or -48V powered; it still is possible and _will eventually happen_ that something further down the line like the motherboard will fry or whatever.
People who assume that a small-window is small enough and decide that is 'good enough' are dangerous :)
--cw
Quoted 3 lines
I think if you have a large mail server and zero power protection, you've got much larger problems to worry about than write-behind caching on your disk drives... my servers have never (in my
...
Quoted 2 lines
In the specific case of email; you want to make certain guarantees, and having data written to non-volatile storage is one of them.
it's pitiful to pretend that this miniscule risk (50ms per catastrophic power failure) is all that stands between you and absolute stability of storage.
Quoted 2 lines
People who assume that a small-window is small enough and decide that is 'good enough' are dangerous :)
your religious pursuits are your own business.
the rest of the world will go on calculating probabilities of failure, rather than emoting. using raid, redundant sites, upses, etc, rather than obsessing on how terrible IDE disks are.
in summary: write caching on disks is *not* an impediment to robust systems.
I've omitted lkml from this reply since it has nothing to do with the kernel.