LVM causing IO contention or slowdown?

linux-lvm Austin Gonyou 2 participants 4 messages Jan 18, 2002 – Jan 20, 2002

Share & tools

Austin Gonyou

author

2002-01-18 22:30 UTC #0 lore raw

Some disturbing things about LVM I've found. I'm not sure if this has anything to do with XFS or not, I'm still working on that part.

Anyway, I've got an AMI MegaRAID 1600(or so, Dell PERC2/DC, and a Dell PERC3/DC[also ami/lsi]). I conducted a test with dbench in the following manner:

1. create a 3 drive RAID0 on the PERC.
2. mkfs.xfs the new "drive" as the OS sees it. (/dev/sdxx)
3. mount /dev/sdxx /mnt/test
4. chown austin:austin /mnt/test
<login as austin>
5. cd /mnt/test
6. cp ~/dbench/client.txt .
7. dbench 200 (ended up at 29MB/s)
8. dbench 150 (ended up at 31MB/s)
<login as root>
9. umount /mnt/test
10. fdisk /dev/sdxx
11. 'T' for partition type
12. '1' for partition #
13. '8e' for LVM and 'w' to save partition config.
14. pvcreate /dev/sdxx;vgcreate -Ay testvg /dev/sdxx
15. lvcreate -Cy -L 109G -n testlv testvg
16. mkxfs.xfs /dev/testvg/testlv
17. mount /dev/testvg/testlv /mnt/test
18. chown austin:austin /mnt/test
<login as austin>
19. cp ~/dbench/client.txt .
20. dbench 200 (ended up at 54MB/s)
21. dbench 150 (ended up at 61MB/s)

The disk which was made into a pv you see up there is /dev/sdxx. That means it was a partition. I also set it to be contiguous. This could be very bad for the dbench kind of test. I'm not sure. Next I did something similar to the above, but I removed all partitions from the physical device and recreated the volume using just /dev/sdx. That said, I retested just now and got the following:

o Throughput 49.733 MB/sec (NB=62.1662 MB/sec 497.33 MBit/sec) 200 procs
o Throughput 43.5752 MB/sec (NB=54.469 MB/sec 435.752 MBit/sec) 150 procs (not sure about why this is)

and then on a whim...

o Throughput 37.4975 MB/sec (NB=46.8719 MB/sec 374.975 MBit/sec) 250 procs
and again...
o Throughput 43.4332 MB/sec (NB=54.2916 MB/sec 434.332 MBit/sec) 150 procs

Either way, these results are a far cry from what was shown above using Contiguous on a partition, instead of a whole disk. If there's a way to improve this, Please advise.

And then I removed the volume from LVM control, and tested again:

o Throughput 51.2596 MB/sec (NB=64.0745 MB/sec 512.596 MBit/sec) 200 procs
oThroughput 52.0488 MB/sec (NB=65.061 MB/sec 520.488 MBit/sec) 150 procs

Not as huge a jump as last night, but still rather far above what it is with LVM turned on.

Anyone know why this might be?

Quoted 6 lines · On Fri, 2002-01-18 at 17:08, [email protected] wrote:

I had visions of using XFS on LVM volumes for our product. I would like to start with some default LV sizes, and have the ability to shrink and grow them as needed to meet individual requirements. For this, I will need to use ext3fs. (and very stable LVM; still learning about it.)

Signature · 8 lines

Austin Gonyou
Systems Architect, CCNA
Coremetrics, Inc.
Phone: 512-698-7250
email: [email protected]

"It is the part of a good shepherd to shear his flock, not to skin it."
Latin Proverb

Austin Gonyou

author

2002-01-19 01:33 UTC #1 lore raw

As a follow up to this I've done some more testing, and will test the rest of this weekend, using AIM db benchmark.

What I've found is that when mounting with logbufs=8 on my Quad Xeon 2MB cache 6450 with 8 Ultra-2 Drives, 4 RAID0 volumes, two of 3 disks each and two of 1 disk each.

The larger Volumes, the more "costly" volumes, when mounted using logbufs=8, outperformed the same volume addressed when not under LVM control. Not only did it out perform itself, but either WITH or WITHOUT LVM controlling the volume, I nearly doubled my throughput to those drives.

Here is what I'm talking about:

#With LVM management
Throughput 109.715 MB/sec (NB=137.144 MB/sec 1097.15 MBit/sec) 200 procs

#Without LVM management
Throughput 100.803 MB/sec (NB=126.004 MB/sec 1008.03 MBit/sec) 200 procs

This only happened once though and I'm not sure exactly why. Still, the max I could get out of it for the same test more than once with LVM enabled was around 80-85 MB/s. Still a HUGE improvement over 43/44 MB/s.

Quoted 86 lines · Austin Gonyou

Some disturbing things about LVM I've found. I'm not sure if this has anything to do with XFS or not, I'm still working on that part.
Anyway, I've got an AMI MegaRAID 1600(or so, Dell PERC2/DC, and a Dell PERC3/DC[also ami/lsi]). I conducted a test with dbench in the following manner:
1. create a 3 drive RAID0 on the PERC.
2. mkfs.xfs the new "drive" as the OS sees it. (/dev/sdxx)
3. mount /dev/sdxx /mnt/test
4. chown austin:austin /mnt/test
<login as austin>
5. cd /mnt/test
6. cp ~/dbench/client.txt .
7. dbench 200 (ended up at 29MB/s)
8. dbench 150 (ended up at 31MB/s)
<login as root>
9. umount /mnt/test
10. fdisk /dev/sdxx
11. 'T' for partition type
12. '1' for partition #
13. '8e' for LVM and 'w' to save partition config.
14. pvcreate /dev/sdxx;vgcreate -Ay testvg /dev/sdxx
15. lvcreate -Cy -L 109G -n testlv testvg
16. mkxfs.xfs /dev/testvg/testlv
17. mount /dev/testvg/testlv /mnt/test
18. chown austin:austin /mnt/test
<login as austin>
19. cp ~/dbench/client.txt .
20. dbench 200 (ended up at 54MB/s)
21. dbench 150 (ended up at 61MB/s)
The disk which was made into a pv you see up there is /dev/sdxx. That means it was a partition. I also set it to be contiguous. This could be very bad for the dbench kind of test. I'm not sure. Next I did something similar to the above, but I removed all partitions from the physical device and recreated the volume using just /dev/sdx. That said, I retested just now and got the following:
o Throughput 49.733 MB/sec (NB=62.1662 MB/sec 497.33 MBit/sec) 200 procs
o Throughput 43.5752 MB/sec (NB=54.469 MB/sec 435.752 MBit/sec) 150 procs (not sure about why this is)
and then on a whim...
o Throughput 37.4975 MB/sec (NB=46.8719 MB/sec 374.975 MBit/sec) 250 procs
and again...
o Throughput 43.4332 MB/sec (NB=54.2916 MB/sec 434.332 MBit/sec) 150 procs
Either way, these results are a far cry from what was shown above using Contiguous on a partition, instead of a whole disk. If there's a way to improve this, Please advise.
And then I removed the volume from LVM control, and tested again:
o Throughput 51.2596 MB/sec (NB=64.0745 MB/sec 512.596 MBit/sec) 200 procs
oThroughput 52.0488 MB/sec (NB=65.061 MB/sec 520.488 MBit/sec) 150 procs
Not as huge a jump as last night, but still rather far above what it is with LVM turned on.
Anyone know why this might be?
Quoted 6 lines · On Fri, 2002-01-18 at 17:08, [email protected] wrote:
I had visions of using XFS on LVM volumes for our product. I would like to start with some default LV sizes, and have the ability to shrink and grow them as needed to meet individual requirements. For this, I will need to use ext3fs. (and very stable LVM; still learning about it.)
Signature · 8 lines
Austin Gonyou
Systems Architect, CCNA
Coremetrics, Inc.
Phone: 512-698-7250
email: [email protected]

"It is the part of a good shepherd to shear his flock, not to skin it."
Latin Proverb

Signature · 8 lines

Austin Gonyou
Systems Architect, CCNA
Coremetrics, Inc.
Phone: 512-698-7250
email: [email protected]

"It is the part of a good shepherd to shear his flock, not to skin it."
Latin Proverb

Stephen Lord

2002-01-20 08:13 UTC #2 lore raw

Quoted 19 lines · Austin Gonyou

As a follow up to this I've done some more testing, and will test the rest of this weekend, using AIM db benchmark.
What I've found is that when mounting with logbufs=8 on my Quad Xeon 2MB cache 6450 with 8 Ultra-2 Drives, 4 RAID0 volumes, two of 3 disks each and two of 1 disk each.
The larger Volumes, the more "costly" volumes, when mounted using logbufs=8, outperformed the same volume addressed when not under LVM control. Not only did it out perform itself, but either WITH or WITHOUT LVM controlling the volume, I nearly doubled my throughput to those drives.
Here is what I'm talking about:
#With LVM management
Throughput 109.715 MB/sec (NB=137.144 MB/sec 1097.15 MBit/sec) 200 procs

Quoted 9 lines

#Without LVM management
Throughput 100.803 MB/sec (NB=126.004 MB/sec 1008.03 MBit/sec) 200 procs
This only happened once though and I'm not sure exactly why. Still, the max I could get out of it for the same test more than once with LVM enabled was around 80-85 MB/s. Still a HUGE improvement over 43/44 MB/s.

Off topic for the lvm list, but....

The logbufs=8 parameter basically means you have 8 buffers capable of pushing transactions out to disk. If you have lots of threads going at once in xfs then the transactions tend to get throttled waiting for a buffer to do a log write into, so adding more is good.

There is a perl script in the cmd/xfsmisc directory called xfs_stats.pl if you run this it will format the output of the xfs /proc kernel statistics. You will see a parameter called xs_log_noiclogs near the bottom of the first column. This number means a transaction finished, but then had to wait for a log buffer before it could hand off its data.

The downside of increasing the number of logbuffers is the amount of data which can be lost after a crash (i.e. how many ops in the filesystem are effectively undone by recovery). However, looking at stats on my box, each log write contains about 5 transactions, so you can never loose much.

We have found adding more than 8 usually does not help, making them bigger would be a different story, but that is actually a very non-trivial change.

Steve

Austin Gonyou

author

2002-01-20 23:55 UTC #3 lore raw

Thanks for the info Steve. The biggest part about this was that just putting the HW RAID0 under LVM control caused a major slowdown. Even if it is from 100MB/s to 80MB/s. I really expect none, or just a small amount. e.g. from 100MB/s to 95MB/s, not 80MB/s or less.

It was just very relieving to see that even though I was using LVM, if I added logbufs=8 to my mount line for a given filesystem it certainly relieved MUCH contention. I just wish I knew why LVM was dogging it so bad.

Quoted 73 lines · Stephen Lord

Quoted 4 lines · Austin Gonyou
As a follow up to this I've done some more testing, and will test the rest of this weekend, using AIM db benchmark.
What I've found is that when mounting with logbufs=8 on my Quad Xeon
2MB
Quoted 15 lines
cache 6450 with 8 Ultra-2 Drives, 4 RAID0 volumes, two of 3 disks each and two of 1 disk each.
The larger Volumes, the more "costly" volumes, when mounted using logbufs=8, outperformed the same volume addressed when not under LVM control. Not only did it out perform itself, but either WITH or WITHOUT LVM controlling the volume, I nearly doubled my throughput to those drives.
Here is what I'm talking about:
#With LVM management
Throughput 109.715 MB/sec (NB=137.144 MB/sec 1097.15 MBit/sec) 200 procs
Quoted 8 lines
#Without LVM management
Throughput 100.803 MB/sec (NB=126.004 MB/sec 1008.03 MBit/sec) 200 procs
This only happened once though and I'm not sure exactly why. Still, the max I could get out of it for the same test more than once with LVM enabled was around 80-85 MB/s. Still a HUGE improvement over 43/44
MB/s.
Quoted 1 line
Off topic for the lvm list, but....
The logbufs=8 parameter basically means you have 8 buffers capable of pushing
transactions out to disk. If you have lots of threads going at once in xfs then the
transactions tend to get throttled waiting for a buffer to do a log write into, so
adding more is good.
There is a perl script in the cmd/xfsmisc directory called xfs_stats.pl if you
run this it will format the output of the xfs /proc kernel statistics. You will
see a parameter called xs_log_noiclogs near the bottom of the first column.
This number means a transaction finished, but then had to wait for a log buffer before it could hand off its data.
The downside of increasing the number of logbuffers is the amount of data
which can be lost after a crash (i.e. how many ops in the filesystem are effectively undone by recovery). However, looking at stats on my box, each log write contains about 5 transactions, so you can never loose much.
We have found adding more than 8 usually does not help, making them bigger would be a different story, but that is actually a very non-trivial
change.
Steve
_______________________________________________
linux-lvm mailing list
[email protected]
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Signature · 8 lines

Austin Gonyou
Systems Architect, CCNA
Coremetrics, Inc.
Phone: 512-698-7250
email: [email protected]

"It is the part of a good shepherd to shear his flock, not to skin it."
Latin Proverb