Speed drop /dev/sda -> /dev/sda1 -> /dev/vg0/test (3ware/LVM)

Hello,

I'm running Linux 2.6.0 with a 3ware 8506 controller in
hardware RAID5 mode. The RAID5 array is built of 7+1 200 MB SATA disks.

Now it appears that more "mappings" on the array have a bad influence on speed. /dev/sda is the fastest, /dev/sda1 is quite a bit slower, LVM on /dev/sda is slower yet and LVM on /dev/sda1 is the slowest.

Results of a dd write directly on the RAID5 whole device:

time -p sh -c "dd if=/dev/zero of=/dev/sda bs=4k count=100000; sync" 100000+0 records in
100000+0 records out
409600000 bytes transferred in 5.120484 seconds (79992437 bytes/sec) real 5.23
user 0.03
sys 0.98

Results on the first partition:

time -p sh -c "dd if=/dev/zero of=/dev/sda1 bs=4k count=100000; sync" 100000+0 records in
100000+0 records out
409600000 bytes transferred in 6.141125 seconds (66697878 bytes/sec) real 6.24
user 0.04
sys 2.06

LVM is created on /dev/sda or /dev/sda1 using the following commands:
pvcreate $DEVICE
vgcreate vg0 $DEVICE
lvcreate -L 100G -n test vg0

Results for a logical volume on /dev/sda:

time -p sh -c "dd if=/dev/zero of=/dev/vg0/test bs=4k count=100000; sync" 100000+0 records in
100000+0 records out
409600000 bytes transferred in 8.624306 seconds (47493676 bytes/sec) real 8.65
user 0.05
sys 1.03

time -p sh -c "dd if=/dev/zero of=/dev/vg0/test bs=4k count=100000; sync" Results for a logical volume on /dev/sda1:
100000+0 records in
100000+0 records out
409600000 bytes transferred in 9.223930 seconds (44406234 bytes/sec) real 9.27
user 0.07
sys 0.97

This is reproducable. I also tested this on a software RAID5 array, but with SW RAID5 it doesn't matter if I use the whole disk, a partition, or a logical volume; performance is always the same (slightly faster than HW RAID5; 100 MB/sec).

Any idea how to get the performance back ? How can I go about debugging this ?

Mike.

In article <[email protected]>,

For reference, I just took 2.4.24-pre2, applied the device-mapper patches to it, and re-ran the tests.

Base performance on /dev/sda is a bit slower (70 MB/sec vs 80 MB/sec raw writespeed), but speeds on /dev/sda1 and /dev/vg0/test are exactly the same.

So it doesn't look like a 3ware driver or LVM issue as those are basically the same between the 2.4 and 2.6 systems.

Mike.

When life hands you lemons, grab the salt and pass the tequila.

In article <[email protected]>,

Okay, I found a clue. It appears that by default, the blocksize of /dev/sda is 4096 and the blocksize of /dev/sda1 is 1024 on my system:

# blockdev --getbsz /dev/sda
4096
# blockdev --getbsz /dev/sda1
1024

On 2.4, setting the blocksize is sticky, on 2.6 it isn't. So I have to work around that to test with other blocksizes, for example:

# /usr/bin/time -p sh -c "blockdev --setbsz 4096 /dev/stdout; dd if=/dev/zero bs=4k count=100000; sync" > /dev/sda1
100000+0 records in
100000+0 records out
real 5.00
user 0.03
sys 0.89

409600000 bytes in 5 seconds =~ 80 MB/sec, which is quite good and the same throughput as through /dev/sda. So, it appears that the 3ware driver really really wants a 4K blocksize, otherwise performance drops at least 30-40%.

Now with LVM on top of it, performance drops again. Say I create a logical volume on /dev/sda1:

# pvcreate /dev/sda1
# vgcreate vg0 /dev/sda1
# lvcreate -L 100G -n test vg0

# blockdev --getbsz /dev/sda1
1024
# blockdev --getbsz /dev/vg0/test
4096

Performance:

# /usr/bin/time -p sh -c "dd if=/dev/zero of=/dev/vg0/test bs=4k count=100000; sync"
100000+0 records in
100000+0 records out
409600000 bytes transferred in 8.512663 seconds (48116553 bytes/sec)
real 8.54
user 0.04
sys 1.02

So, I tried a small patch in dm-table.c to set the blocksize of the underlying device to 4096:

# diff -u dm-table.c.b4 dm-table.c
--- dm-table.c.b4       2003-12-29 15:18:18.000000000 +0100
+++ dm-table.c  2003-12-30 13:08:06.000000000 +0100
@@ -359,8 +359,10 @@
        r = bd_claim(bdev, _claim_ptr);
        if (r)
                blkdev_put(bdev, BDEV_RAW);
-       else
+       else {
                d->bdev = bdev;
+               set_blocksize(bdev, 4096);
+       }
        return r;
 }

While it appears to work, it doesn't seem to make any difference:

# blockdev --getbsz /dev/sda1
4096
# blockdev --getbsz /dev/vg0/test
4096

Performance:

# /usr/bin/time -p sh -c "dd if=/dev/zero of=/dev/vg0/test bs=4k count=100000; sync"
100000+0 records in
100000+0 records out
409600000 bytes transferred in 8.528826 seconds (48025367 bytes/sec)
real 8.54
user 0.05
sys 0.98

Not so weird, since in dm-table.c the device is opened in RAW mode and raw devices do I/O at hardsect_size, right? So that would be 512 bytes, explaining the bad performance.

I'm hoping someone has read this far and can offer me a clue as how to fix this. Are somehow requests not merged right, and are small 512 byte requests submitted to the 3ware driver? Is the 3ware driver responsible for merging requests ? I really don't know anything about the block layer or bio's so I'm kind of lost here.

Mike.