LVM & high system load
Hi,
I am newbie to LVM. I am experiencing high cpu load (system time > 90%) and
relatively poor IO perf. (my cpu is only PII 333) on LVM volumes. I have tested reiserfs and ext2fs and I only get about 2.5Mb/s with 100% CPU load,
mostly system time.
Due to some failures of vgscan (no volume found) and vgchange (core dumps,)
I had to upgrade my 2.4.3 kernel to LVM beta7 and also the system tools. Now
it seems that theses tools work but I still have very high loads, much higher than what I would have on a fs lying on a partition.
My logical volumes are not stripped.
Is this a known issue ?
Thanks in advance,
DL
Signature · 112 lines
Dominique Larchey LORIA, Nancy, France Tel. (Work) +33 (0) 3 83 59 20 13 ---------------------------- [root@charlie temp]# /usr/sbin/vgdisplay mmedia --- Volume group --- VG Name mmedia VG Access read/write VG Status available/resizable VG # 0 MAX LV 256 Cur LV 3 Open LV 3 MAX LV Size 255.99 GB Max PV 256 Cur PV 3 Act PV 3 VG Size 28.49 GB PE Size 4 MB Total PE 7294 Alloc PE / Size 5066 / 19.79 GB Free PE / Size 2228 / 8.7 GB VG UUID dF3isK-xdlk-YF7u-FwxQ-KnPQ-jgG6-qjt73X [root@charlie temp]# /usr/sbin/pvdisplay /dev/hda1 /dev/hda5 /dev/hda6 --- Physical volume --- PV Name /dev/hda1 VG Name mmedia PV Size 9.77 GB / NOT usable 1.37 MB [LVM: 130 KB] PV# 1 PV Status available Allocatable yes (but full) Cur LV 1 PE Size (KByte) 4096 Total PE 2500 Free PE 0 Allocated PE 2500 PV UUID 3aeAAl-88en-VZxJ-84gE-Uaxu-AWZm-zoA3jC --- Physical volume --- PV Name /dev/hda5 VG Name mmedia PV Size 9.77 GB / NOT usable 1.37 MB [LVM: 130 KB] PV# 2 PV Status available Allocatable yes (but full) Cur LV 3 PE Size (KByte) 4096 Total PE 2500 Free PE 0 Allocated PE 2500 PV UUID lUiRFu-JEhC-d5JS-Jycy-VFa8-2o2y-6I9V9S --- Physical volume --- PV Name /dev/hda6 VG Name mmedia PV Size 8.96 GB / NOT usable 1.73 MB [LVM: 129 KB] PV# 3 PV Status available Allocatable yes Cur LV 0 PE Size (KByte) 4096 Total PE 2294 Free PE 2228 Allocated PE 66 PV UUID GOf2OV-khJd-w1cJ-AEk3-RvyA-Wdly-p5OuWd [root@charlie temp]# /usr/sbin/lvdisplay /dev/mmedia/music /dev/mmedia/temp /dev/mmedia/test --- Logical volume --- LV Name /dev/mmedia/music VG Name mmedia LV Write Access read/write LV Status available LV # 2 # open 1 LV Size 9 GB Current LE 2304 Allocated LE 2304 Allocation next free Read ahead sectors 120 Block device 58:1 --- Logical volume --- LV Name /dev/mmedia/temp VG Name mmedia LV Write Access read/write LV Status available LV # 1 # open 1 LV Size 9.79 GB Current LE 2506 Allocated LE 2506 Allocation next free Read ahead sectors 120 Block device 58:0 --- Logical volume --- LV Name /dev/mmedia/test VG Name mmedia LV Write Access read/write LV Status available LV # 3 # open 1 LV Size 1 GB Current LE 256 Allocated LE 256 Allocation next free Read ahead sectors 120 Block device 58:2
Quoted 4 lines · Dominique LARCHEY-WENDLING
relatively poor IO perf. (my cpu is only PII 333) on LVM volumes. I have tested reiserfs and ext2fs and I only get about 2.5Mb/s with 100% CPU load,
mostly system time.
More over, doing a CPU intensive task (like kernel comp.) (not involving LVM, the
sources are not on a LV) cut the IO perf. of LVM fs by half.
So it seems that the LVM is eating a lot of CPU on my system and this is what is limiting fs perf.
DL
Signature · 3 lines
Dominique Larchey LORIA, Nancy, France Tel. (Work) +33 (0) 3 83 59 20 13
Quoted 2 lines · Dominique Larchey
I am newbie to LVM. I am experiencing high cpu load (system time > 90%) and relatively poor IO perf. (my cpu is only PII 333) on LVM volumes.
Note that I am running LVM on Pentium 90 w 16MB RAM and no problems...
Quoted 2 lines
I have tested reiserfs and ext2fs and I only get about 2.5Mb/s with 100% CPU load, mostly system time.
There is one other person that has reported very slow LVM performance. However, there is no obvious reason why they have performance issues, and others are unable to reproduce this problem.
Can you please try the following and report results:
time dd if=/dev/mmedia/music of=/dev/null bs=512 time dd if=/dev/mmedia/music of=/dev/null bs=65536 time dd if=/dev/hda1 of=/dev/null bs=512 time dd if=/dev/hda1 of=/dev/null bs=65536
Cheers, Andreas
Signature · 3 lines
Andreas Dilger Turbolinux filesystem development http://sourceforge.net/projects/ext2resize/ http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 10 lines · Andreas Dilger
There is one other person that has reported very slow LVM performance. However, there is no obvious reason why they have performance issues, and others are unable to reproduce this problem.
Can you please try the following and report results:
time dd if=/dev/mmedia/music of=/dev/null bs=512 time dd if=/dev/mmedia/music of=/dev/null bs=65536 time dd if=/dev/hda1 of=/dev/null bs=512 time dd if=/dev/hda1 of=/dev/null bs=65536
Here are the tests I have done ... I use the command line
dd if=/dev/sda of=/dev/null bs=1k count=500k
to fill up the buffer cache before any timing measure. I point out that /dev/mmedia/music resides on /dev/hda5
time dd if=/dev/mmedia/music of=/dev/null bs=512
count=200000 0m37.113s
time dd if=/dev/hda5 of=/dev/null bs=512
count=200000 0m35.437s
Ok I am ridiculous ... this has nothing to do with LVM. It has to do with DMA set off by default by linux 2.4. This is the reason of poor IO perf. and CPU eating. (as you might have understood, I usually do not use IDE drives)
After DMA is set on,
time dd if=/dev/mmedia/music of=/dev/null bs=512
count=200000 0m12.934s
time dd if=/dev/hda5 of=/dev/null bs=512
count=200000 0m12.110s
Sorry for anoying you with my stupidity ...
Perhaps poor perf. for the other person is also DMA related ?-)
Thanks anyway
DL
Signature · 3 lines
Dominique Larchey LORIA, Nancy, France Tel. (Work) +33 (0) 3 83 59 20 13
Quoted 6 lines · Dominique Larchey
I point out that
/dev/mmedia/musicresides on/dev/hda5time dd if=/dev/mmedia/music of=/dev/null bs=512
count=200000 0m37.113s
time dd if=/dev/hda5 of=/dev/null bs=512
count=200000 0m35.437s
LVM shows 4.7% overhead. This is not so great.
Quoted 10 lines
Ok I am ridiculous ... this has nothing to do with LVM. It has to do with DMA set off by default by linux 2.4. This is the reason of poor IO perf. and CPU eating.
After DMA is set on,
time dd if=/dev/mmedia/music of=/dev/null bs=512
count=200000 0m12.934s
time dd if=/dev/hda5 of=/dev/null bs=512
count=200000 0m12.110s
Good news - not an LVM problem. Bad news - LVM shows 6.8% overhead.
Quoted 1 line
Perhaps poor perf. for the other person is also DMA related ?-)
No, the other person has drastically different performance with LVM for 512-byte I/O compared to raw disk. For 64k-byte I/O performance on LVM and raw disk are the same. I _suppose_ that if I/O is done in small chunks that PIO mode will have more overhead. Maybe worth asking if CPU is high when he is doing 512-byte I/Os.
Cheers, Andreas
Signature · 3 lines
Andreas Dilger Turbolinux filesystem development http://sourceforge.net/projects/ext2resize/ http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 14 lines · Andreas Dilger
Quoted 6 lines · Dominique Larchey
I point out that
/dev/mmedia/musicresides on/dev/hda5time dd if=/dev/mmedia/music of=/dev/null bs=512
count=200000 0m37.113s
time dd if=/dev/hda5 of=/dev/null bs=512
count=200000 0m35.437sLVM shows 4.7% overhead. This is not so great.
Quoted 4 lines
Ok I am ridiculous ... this has nothing to do with LVM. It has to do with DMA set off by default by linux 2.4. This is the reason of poor IO perf. and CPU eating.
Quoted 8 lines
Quoted 6 lines
After DMA is set on,
time dd if=/dev/mmedia/music of=/dev/null bs=512
count=200000 0m12.934s
time dd if=/dev/hda5 of=/dev/null bs=512
count=200000 0m12.110sGood news - not an LVM problem. Bad news - LVM shows 6.8% overhead.
Did nothing else produce io load while the test on the LV was running?
Did you repeat tests in order to get statistic certainty?
Quoted 8 lines
Quoted 2 lines
Perhaps poor perf. for the other person is also DMA related ?-)
No, the other person has drastically different performance with LVM for 512-byte I/O compared to raw disk. For 64k-byte I/O performance on LVM and raw disk are the same. I _suppose_ that if I/O is done in small chunks that PIO mode will have more overhead. Maybe worth asking if CPU is high when he is doing 512-byte I/Os.
Yep.
Quoted 10 lines
Cheers, Andreas
Signature · 7 lines
Andreas Dilger Turbolinux filesystem development http://sourceforge.net/projects/ext2resize/ http://www-mddsp.enel.ucalgary.ca/People/adilger/ _______________________________________________ linux-lvm mailing list [email protected] http://lists.sistina.com/mailman/listinfo/linux-lvm
Signature · 15 lines
Regards,
Heinz -- The LVM Guy --
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
[email protected] +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-