LVM2 seems to chop performance by 33%

Hi

I didn't think lvm2 had such a big performance hit?

65Mb/s on the raid5 device
44Mb/s on the lv

Is this expected?

Kernel 2.6.6

representative dd's:

cu:/huge/editing/tmp# time dd if=/dev/video_vg/video_lv of=/dev/null bs=1024k count=4k
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 97.363340 seconds (44112777 bytes/sec)

real    1m37.437s
user    0m0.027s
sys     0m27.126s

yet:

cu:/huge/editing/tmp# time dd if=/dev/md0 of=/dev/null bs=1024k count=4k
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 65.715222 seconds (65357267 bytes/sec)

real    1m5.770s
user    0m0.026s
sys     0m27.524s

cu:/huge/editing/tmp# lvdisplay /dev/video_vg/video_lv
 --- Logical volume ---
 LV Name                /dev/video_vg/video_lv
 VG Name                video_vg
 LV UUID                hyCwfR-d7ZG-lptP-XJFK-vMUb-JWE8-LgQSbb
 LV Write Access        read/write
 LV Status              available
 # open                 2
 LV Size                935.00 GB
 Current LE             14960
 Segments               1
 Allocation             inherit
 Read ahead sectors     0
 Block device           253:0

Changing Read ahead makes no difference (actually it slows it down a touch with values of 10 + 100)

cu:/huge/editing/tmp# vgdisplay /dev/video_vg
 --- Volume group ---
 VG Name               video_vg
 System ID
 Format                lvm2
 Metadata Areas        1
 Metadata Sequence No  6
 VG Access             read/write
 VG Status             resizable
 MAX LV                0
 Cur LV                1
 Open LV               1
 Max PV                0
 Cur PV                1
 Act PV                1
 VG Size               935.00 GB
 PE Size               64.00 MB
 Total PE              14960
 Alloc PE / Size       14960 / 935.00 GB
 Free  PE / Size       0 / 0
 VG UUID               kFVbcB-UgLa-XQjy-mF7R-BT2D-sUaA-uGUmeK

cu:/huge/editing/tmp# pvdisplay /dev/md0
 --- Physical volume ---
 PV Name               /dev/md0
 VG Name               video_vg
 PV Size               935.00 GB / not usable 0
 Allocatable           yes (but full)
 PE Size (KByte)       65536
 Total PE              14960
 Free PE               0
 Allocated PE          14960
 PV UUID               pEWAKP-tbaH-DSXP-W2jF-Pg6H-2z35-k1adQj

David

While doing backups of my LVM2 drive, I've found that setting the DD block size to 4096 greatly improved my performance. Formerly, DDs on my 240G LVM were taking 23hrs to complete with a block size of 512 or 1024. The same volume now takes less than 2 hours with a block size of 4096. Large numbers did not seem to increase DD speed.

try bs=64k, even better...

Thanks

I'd tried that, but no real change. I started 1t 128k and also tried 64k, 256k :) (oh, and 1k)

dd if=/dev/video_vg/video_lv of=/dev/null bs=4k count=256k 262144+0 records in
262144+0 records out
1073741824 bytes transferred in 24.130318 seconds (44497624 bytes/sec)

dd if=/dev/md0 of=/dev/null bs=4k count=256k
262144+0 records in
262144+0 records out
1073741824 bytes transferred in 15.404947 seconds (69701105 bytes/sec)

David

PS wait 'til you see what I'm getting through Rieserfs on top of it! <sigh>

cu:/huge/editing/tmp# time dd if=dummy.deleteme of=/dev/null bs=4k count=256k
262144+0 records in
262144+0 records out
1073741824 bytes transferred in 31.627904 seconds (33949193 bytes/sec)
so throughput down by a factor of 2...
but 1 step at a time....

I did some tests a few months ago with bonnie++.. might offer some encouragement (please don't post this to slashdot.. ;)

http://spamaps.org/raidtests.php

There's a lot of data there, but if you look at the LVM stuff, you might notice that the concurrent performance (having 3 processes hammering the disks in different places instead of just one) was quite good when compared to flat out RAID5. I'll pay 5% performance for manageability any day. :-D

Hi,

Did you try to ajust the read ahead with blockdev on the LV itself?

# blockdev --setra XXXX /dev/sda  (raid5 device, just for comparisson)
# blockdev --setra XXXX /dev/vg/lv (lv)

Try some values for the readahead. On my system I use 2048.

Best regards

Miguel Cabe�a

No, I didn't
hmm... man blockdev
useful - new to me :) Not seen it on any googling to do with lvm.

I'll try this later when it's idle - thanks for the suggestion.

I do already have
Read ahead sectors 100
in the lvdisplay

yet blockdev --getra /dev/video_vg/video_lv reports 256 (and, to be sure --getss says 512byte sectors)

Why is this different?

David

Thanks to those that made suggestions.

In the end I used
blockdev --setra 4096 on all my devices (/dev/sda,b,c,d and the /dev/md0 and the /dev/video_vg/video_lv) and this doubled throughput.

I am reading multi-gigabyte video files so these parameters are not for everyone.

No-one ever replied as to why blockdev --setra / --getra is not the same as that displayed in lvdisplay And it's not documented that I can find. There's a comment: "Not used by device-mapper." And that means.....? It's ignored? not implemented yet? Good luck?

# lvdisplay /dev/video_vg/huge_lv
 --- Logical volume ---
 LV Name                /dev/video_vg/huge_lv
 VG Name                video_vg
 LV UUID                3kz7n9-97Rg-2LJw-J9ml-1BBS-jGs0-Onh4NI
 LV Write Access        read/write
 LV Status              available
 # open                 1
 LV Size                312.50 GB
 Current LE             5000
 Segments               1
 Allocation             inherit
 Read ahead sectors     120
 Block device           253:1

# blockdev --getra /dev/video_vg/huge_lv
4096

<sigh> Let this post be there for Google - the modern man-page for linux. (if you've got your fingers crossed!)

David

Any LVM read ahead setting is ignored by LVM2 i.e. not passed to the kernel.

If it's demonstrated to be useful it could still be implemented, but the current development priorities are still correctness and new features rather than performance.

Alasdair

[email protected]

Thanks Alasdair

It would be good to modify the docs to be clearer and maybe say "use blockdev --setra"

David