Skip to contentSkip to conversation

File system performance, hardware performance, ext3, 3ware RAID1, etc.

DISCUSSION Timothy Miller 5 participants 9 messages Feb 12, 2004 – Feb 13, 2004

lore
Share & tools
lore
#0 lore raw

I'm attempting to get some idea of how much overhead ext3 causes me on my workstation at home. Furthermore, I'm trying to determine what sort of advantage I'm really getting from my 3ware RAID controller (model 7000-2, configured for RAID1) over single disks.

Using iozone, I'm finding an upper bound for disk reads at about 40 megs/sec, which is okay, but no better than single disk. That's probably expected, since sequential reads can't come off the disk any faster than the disk spins. RAID1 would show its greatest benefit with RANDOM reads.

To determine the highest upper bound for sequential read throughput, I timed a dd of the first gigabyte from /dev/sga (the raw device) to /dev/null. This showed a throughput of 47meg/sec. This shows that ext3
isn't hurting reads.

For writes, iozone found an upper bound of about 10megs/sec, which is abysmal. Typically, I'd expect writes to be faster (on a single drive) than reads, because once the write is sent, you can forget about it. You don't have to wait around for something to come back, and that latency for reads can hurt performance. The OS can also buffer writes and reorder them in order to improve efficiency.

The 3ware has this write cache that you can turn on or off. With it off, it ensures that writes make it to the disks in order. With it on, it will reorder writes more efficiently. However, I noticed that the performance only went up to about 16meg/sec with the cache ON.

For comparison, I would like to estimate the maximum WRITE throughout for the raw device. But I'm not ready to dump zeros to my working partitions. I was thinking that I could do this with the SWAP partition. I could turn off swap and then dd TO the swap partition. Being on the inner tracks, it won't perform as well as the max for the drive, but it'll give me a lower bound for raw write throughput to compare against.

IMPORTANT QUESTION: Is there any metadata anywhere in the swap partition (when it's not in use) that I need to save before I fill it with zeros?

Also, what do I use as a source for zeros when writing with dd? "/dev/zero"?

What's the command? How about this:

    time dd if=/dev/zero of=/dev/sga3 bs=1024 count=1024

Will that do it? Should I use an offset to avoid any kind of header or metadata?

If anyone has numbers for what they get with WD1200JB drives, I'd love to compare.

Thanks!

#1 lore raw
Quoted 6 lines · Timothy Miller

For writes, iozone found an upper bound of about 10megs/sec, which is abysmal. Typically, I'd expect writes to be faster (on a single drive) than reads, because once the write is sent, you can forget about it. You don't have to wait around for something to come back, and that latency for reads can hurt performance. The OS can also buffer writes and reorder them in order to improve efficiency.

It depends on the disk too. Lots of disks (specially IDE) are far slower on writes than they are on reads.

Quoted 4 lines

The 3ware has this write cache that you can turn on or off. With it off, it ensures that writes make it to the disks in order. With it on, it will reorder writes more efficiently. However, I noticed that the performance only went up to about 16meg/sec with the cache ON.

I don't think that the FS type has much importance once the cache is ON.

Quoted 3 lines

IMPORTANT QUESTION: Is there any metadata anywhere in the swap partition (when it's not in use) that I need to save before I fill it with zeros?

No, there's nothing to save, but you'll have to rebuild the swap at the end of your tests : mkswap /dev/sga3
I too often use the swap partition for testing purpose or for temporary storage ; it's easy to do a swapoff; mke2fs on it :-)

Quoted 2 lines

Also, what do I use as a source for zeros when writing with dd? "/dev/zero"?

yes. but don't try to copy any file from the disk onto this partition, because head seeks will slow things down.

Quoted 2 lines

What's the command? How about this:

    time dd if=/dev/zero of=/dev/sga3 bs=1024 count=1024

do it like this, but use higher values, particularly for bs which is only 1kB here. Using something like bs=65536 and count=4096 will give you a 256 MB file.

Quoted 2 lines

Will that do it? Should I use an offset to avoid any kind of header or metadata?

not needed. Just ensure that you write to the right partition, and better check twice.

Regards
Willy

#2 lore raw

Thanks for your suggestions. They have aided me in my tests... read on.

Quoted 8 lines · Willy Tarreau
Quoted 1 line · Timothy Miller

For writes, iozone found an upper bound of about 10megs/sec, which is abysmal. Typically, I'd expect writes to be faster (on a single drive) than reads, because once the write is sent, you can forget about it. You don't have to wait around for something to come back, and that latency for reads can hurt performance. The OS can also buffer writes and reorder them in order to improve efficiency.

It depends on the disk too. Lots of disks (specially IDE) are far slower on writes than they are on reads.

I wonder why that is. What adds extra overhead to the writes? Verify?
Given the lousy quality of today's hard drives, I can see why that
might be necessary.

Quoted 6 lines
Quoted 3 lines

The 3ware has this write cache that you can turn on or off. With it off, it ensures that writes make it to the disks in order. With it on, it will reorder writes more efficiently. However, I noticed that the performance only went up to about 16meg/sec with the cache ON.

I don't think that the FS type has much importance once the cache is ON.

It might, but from my tests (see below), you're right.

Quoted 10 lines
Quoted 4 lines

What's the command? How about this:

   time dd if=/dev/zero of=/dev/sga3 bs=1024 count=1024

do it like this, but use higher values, particularly for bs which is only 1kB here. Using something like bs=65536 and count=4096 will give you a 256 MB file.

I mistyped. I did "bs=1024k count=1024".

Quoted 7 lines
Quoted 2 lines

Will that do it? Should I use an offset to avoid any kind of header or metadata?

not needed. Just ensure that you write to the right partition, and better check twice.

Done. Here are my results:

As I had mentioned, I did raw read performance by timing a dd from /dev/sda to /dev/null. That demonstrated a throughput of 47 megs/sec which is pretty close to the benchmarks that I find in reviews.

I was able to perform the write performance test last night. The swap partition was actually /dev/sda2. I think / and /boot are sda1 and sda3, but I forget the order. Either way, the swap partition is not worst case. In my test, I wrote 1 gigabyte in 1 meg blocks in 73.522 seconds which translates into 13.92 megs/sec. That's terrible -- worse because the 3ware's write cache is ON.

According to Tom's Hardware, raw write throughput for the WD1200JB varies from 39500 k/sec at the outer tracks to 14200 k/sec at the inner tracks. Something is seriously bottle-necking the performance through the 3ware.

For comparison, my wife's computer has the same model of drive as its primary IDE. This is a single drive, and the box runs Windows XP Pro with 1GB of RAM. The disk was defragmented this morning at 5AM, and I ran the test this morning at about 9AM. I ran SiSoft SANDRA, and these are the results:

Buffered Read 73 mb/s
Seqential Read 37 mb/s
Random read 6 mb/s

Buffered write    60 mb/s
Sequential write  39 mb/s
Random write      12 mb/s

Assuming that the "buffered" speeds are being buffered by the OS, we'll ignore those. I am therefore observing that the writes to a single drive are 3 to 4 times faster than they are through the RAID controller, even with the 3ware write cache ON.

Does that make any sense?

[Kernel version: Most recent Red Hat update, 2.4.20-something.
3ware 7000-2 with two 120GB WD drives in RAID 1 array.]

Thanks.

#3 lore raw
Quoted 6 lines · Timothy Miller

Assuming that the "buffered" speeds are being buffered by the OS, we'll ignore those. I am therefore observing that the writes to a single drive are 3 to 4 times faster than they are through the RAID controller, even with the 3ware write cache ON.

Does that make any sense?

Well, it reminds me a disk I had problems with several years ago. It had a few defects in the FAT area, which were relocated at the end. Performance was terrible since the head had to move constantly. It took ages to install Win31 on it, so it finally was returned to the vendor. But in your case it seems a little bit different since you experience slow writes anywhere on the medium. Would it be possible that your controller does something like read-modify-write because of too big chunk size ?

Regards,
Willy

#4 lore raw
Quoted 8 lines · Willy Tarreau
Quoted 3 lines · Timothy Miller

Assuming that the "buffered" speeds are being buffered by the OS, we'll ignore those. I am therefore observing that the writes to a single drive are 3 to 4 times faster than they are through the RAID controller, even with the 3ware write cache ON.

Does that make any sense?

Quoted 8 lines

Well, it reminds me a disk I had problems with several years ago. It had a few defects in the FAT area, which were relocated at the end. Performance was terrible since the head had to move constantly. It took ages to install Win31 on it, so it finally was returned to the vendor. But in your case it seems a little bit different since you experience slow writes anywhere on the medium. Would it be possible that your controller does something like read-modify-write because of too big chunk size ?

I'm getting 10-15 meg/sec even with the largest block sizes. With dd, I
set the block size to 1 megabyte, so there's no chance that the block
being written is too small or that the disk block is too big.

Also, it's a RAID1.

#5 lore raw
In reply to Willy Tarreau
Quoted 2 lines · Willy Tarreau

It depends on the disk too. Lots of disks (specially IDE) are far slower on writes than they are on reads.

This may be a function of the operating system or the filesystem, but it isn't necessarilly an artifact of the drives themselves. With both read and write caching enabled, random writes will always be faster than random reads from the drive perspective.

Even with queueing enabled (legacy TCQ or Native-SATA "NCQ"), your ability to reorder reads is limited in ATA to 32 tags, while the ability to reorder cached writes is limited only by buffer size and cache granularity... the absolute worst-case write performance should be the same as read performance.

--eric

Signature · 2 lines
Eric D. Mudama
[email protected]
#6 lore raw
Quoted 5 lines · Eric D. Mudama

This may be a function of the operating system or the filesystem, but it isn't necessarilly an artifact of the drives themselves. With both read and write caching enabled, random writes will always be faster than random reads from the drive perspective.

Well.. Yes and no.

Writes are fundamentally more schedulable, and that's a huge advantage for throughput, since latency really doesn't matter. Which means that you'll find a lot of loads that can much more easily get to platter speeds for writes.

On the other hand, reads are inherently easier on a pure hardware level, since read-ahead and track buffers are trivial ways to get to true platter speeds for a lot of reasonable loads.

And the software-visible 512-byte blocking factor just has to be _incredibly_ painful on a hardware level, and I'd be surprised if there aren't disks out there already where the actual real physical block-size is bigger. Which means that I would expect a lot of drives to internally do read-modify-write cycles for small writes.

And especially in a market where density is often more important than pure speed, I'd expect hw manufacturers to have a _huge_ bias towards big blocks on the platter, in order to avoid having to have lots of inter-sector gaps etc.

So in that kind of schenario, random writes would be clearly slower than random reads.

Quoted 2 lines

the absolute worst-case write performance should be the same as read performance.

That is only true if the disk block-size is smaller than the IO blocksize. Can somebody fill me in on what modern disks do, especially the high-density ones?

Linus

#7 lore raw
Quoted 10 lines

And the software-visible 512-byte blocking factor just has to be _incredibly_ painful on a hardware level, and I'd be surprised if there aren't disks out there already where the actual real physical block-size is bigger. Which means that I would expect a lot of drives to internally do read-modify-write cycles for small writes.

And especially in a market where density is often more important than pure speed, I'd expect hw manufacturers to have a _huge_ bias towards big blocks on the platter, in order to avoid having to have lots of inter-sector gaps etc.

Since drives don't generally do defect re-allocation on read, only on write, large blocks at the hardware level cause difficulties with defect management, so hardware 512 byte sectors are still common :-(.

John.

#8 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 2 lines

the absolute worst-case write performance should be the same as read performance.

That is only true if the disk block-size is smaller than the IO blocksize. Can somebody fill me in on what modern disks do, especially the high-density ones?

This is purely hypothetical, but perhaps write-precompensation or the signal intensity or shape in the head implies to write at "safer" frequencies ? we're speaking about hundreds of megahertz, and I've always wondered what the signal looks like when it reaches the head.

Cheers,
Willy