MD RAID1 performance very different from non-RAID partition

(Kernel: 2.6.18, x86_64)

Is it normal for an MD RAID1 partition with 1 active disk to perform differently from a non-RAID partition?

md0 : active raid1 sda2[0]

      8193024 blocks [2/1] [U_]

I'm building a search engine database onto this partition. All of the source data is cached into memory already (i.e., only writes should be hitting the disk).
If I mount the partition as /dev/md0, building the database consistently takes 18 minutes.
If I stop /dev/md0 and mount the partition as /dev/sda2, building the database consistently takes 31 minutes.

Why the difference?

The "fast" time seen when the partition is mounted as /dev/md0 actually creates a serious problem: the kernel apparently flushes dirty pages so aggressively that other processes attempting to write to the same partition during the database build become blocked for several minutes(!) at a time.

When mounted as /dev/sda2, that doesn't happen: other processes writing to the same partition are blocked for no more than a few seconds at a time.

I don't know if it's relevant, but the results from iostat when writing large chunks of data to RAID1 partitions seem somewhat curious, as if MD is telling the I/O layer "all done!" before it's actually finished writing the data out to the member disks. Note the unrealistically high kB_wrtn/s numbers on md0 in the following test. (And why does it show 50000 tps?)

# iostat -dk 1 md0 sda

# fgrep MemTotal /proc/meminfo
MemTotal:      2059784 kB
# cat /proc/sys/vm/dirty_ratio
40
# cat /proc/sys/vm/dirty_background_ratio
10
# dd if=/dev/zero of=/testpart/bigfile bs=1M count=400

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda              91.00         0.00     46592.00          0      46592
md0           48692.00         0.00    194768.00          0     194768

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda              99.00         0.00     50176.00          0      50176
md0               0.00         0.00         0.00          0          0

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda              96.00         0.00     49152.00          0      49152
md0               0.00         0.00         0.00          0          0

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda              86.00         0.00     44032.00          0      44032
md0               0.00         0.00         0.00          0          0

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda              96.00         0.00     48160.00          0      48160
md0           51636.00         0.00    206544.00          0     206544

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda              95.05         0.00     48665.35          0      49152
md0               0.00         0.00         0.00          0          0

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda              92.00         0.00     46596.00          0      46596
md0               0.00         0.00         0.00          0          0

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda              95.96         0.00     48614.14          0      48128
md0               0.00         0.00         0.00          0          0

...

Jordan Russell

Maybe it's because md doesn't support barriers whereas the disks supports them? In this case some filesystems, for example XFS, will work faster on raid1 because they can't force the flush to disk using barriers.

Just a guess...

regards,
iustin

Shouldn't it be the other way around? With a barrier the filesystem can enforce an order on the data written and can then continue writing data to the cache. More data is queued up for write. Without barriers the filesystem should do a sync at that point and have to wait for the write to fully finish. So less is put into cache.

Or not?

MfG
Goswin

I don't know in general, but XFS will simply not issue any sync at all if the block device doesn't support barriers. It's the syadmin's job to either ensure you have barriers or turn off write cache on disk (see the XFS faq, for example).

However, I never saw such behaviour from MD (i.e. claiming the write has completed while the disk underneath is still receiving data to write from Linux) so I'm not sure this is what happens here. In my experience, MD acknowledges a write only when it has been pushed to the drive (write cache enabled or not) and there is no buffer between MD and the drive.

regards,
iustin

It's an ext3 partition, so I guess that doesn't apply?

I tried remounting /dev/sda2 with the "barrier=0" option (which I assume disables barriers, looking at the source), though, just to see if it would make any difference, but it didn't; the database build still took 31 minutes.

Jordan Russell

Compare the read ahead settings.

MfG
Goswin

I'm not sure what you mean. There's a per-mount read ahead setting?

Jordan Russell

per device

compare
blockdev --getra /dev/sda2
and blockdev --getra /dev/md0

L.

Luca Berra -- [email protected]
       Communication Media & Services S.r.l.
/"\
\ /     ASCII RIBBON CAMPAIGN
 X        AGAINST HTML MAIL
/ \

Ah.

256 on /dev/sda2 with md0 stopped
256 on /dev/sda2 with md0 active
256 on /dev/md0 with md0 active

Jordan Russell