LVM2 scalability within volume group

Greetings,

I'm doing some evaluation of LVM2 on large systems, with lots of disk devices. I'm currently using LVM2.2.00.08, along with device-mapper.1.00.07. I plan to eventually upgrade to the lastest CVS trees for LVM2. I recall earlier mail that it's faster than what I'm using.

The first thing I noticed is that creating a VG with lots of PVs is a bad idea. Creating a volume group with one PV takes about 12 seconds elapsed time. Adding a new PV initially takes about 5 seconds. But this grows to 15 seconds when adding the 40th PV, 25 seconds adding the 60th PV. Adding the 200th PV takes about 6 minutes.

Activing this volume group (with 200 PVs) takes about 48 minutes.

I can be spreading PVs out among the available 99 VGs. The individual VGs seem to be independent of each other, so having large numbers of VGs with a few PVs performs OK.

But stil, would it make sense for adding PVs to VG to scale better? I'm guessing that the current lack of scaling is from the way redundant meta data is stored. Are redundant copies of meta data being updated on every PV within the VG whenever a new PV is added? Even so, why should 200 read/writes take so long?

Having redundant copies of meta data is a good thing. But how about allowing the adminstrator to set a limit on the degree of redundancy when a VG is created. You could limit a VG to having for example 10 redundant copies. Then adding more PVs beyond the 10th would encounter less overhead.

Am I missing something important?

Thanks!
Dave Olien

There'll be a VG-level option for this eventually; until then, use the pvcreate options to say how many copies of metadata you want on each PV. e.g. pvcreate --metadatacopies 0
[Careful use of the --restorefile option lets you reduce it on a PV already in the VG.]

For complex VGs you should increase the space set aside for metadata too:
--metadatasize

See the pvcreate man page.

Alasdair

[email protected]

Thanks! The pvcreate man page I have installed now (from LVM2.00.08) doesn't mention this option. I'll move to the cvs version and give it a try.

Dave

I finally tried reducing metadata rundancy, and re-ran my experiment with the single volume group containing 200 physical volumes. I constructed the new volume group with redundant metadata on only 6 PVs, instead of the default copy on every volume group. This helped a lot. Here's comparing the new configuration with the old configuration:

- Time to add a PV to a VG with large number of PV's:

elasped time (secs)
PV number: New Old
1st PV took 5 3
40th PV took 6 13
60th PV took 7 24
200th PV took 15 426

- Time to create a Logical Volume within that Volume group:
New Old
30 seconds 14 minutes

- Time to activate a volume group:
New Old
29 seconds 45 minutes

While this is a big improvement, 15 second still seems a long time for adding that 200th PV. Likewise 29 seconds to activate the VG is much better. But can these be made faster?

I did an strace on some of these commands. It seems that every command opens about 480 file descriptors. I Iooked at the /etc/lvm/.cache file. It looks like every device listed there is opened for every command. I wasn't able to reduce this .cache file very much, because even though I was using only 200 devices in one volume group, I still wanted to put the other 200 devices into other volume groups.

Can these user-level commands be made smarter in this regard?

Is this something that using the lvm(8) shell would help? On a large system, re-activing lots of large volume groups could take a while. Could the startup script for LVM benefit from run an lvm(8) script to do the startup work?

Yes, I've a list of performance enhancements waiting to be made - they aren't top priority yet though.

Yes.
(a) Additional internal state can be written to the cache file. [The index of which PVs are in which VGs; the index of which UUIDs were found on which devices.]
(b) Some disk reads are still duplicated and can be cached safely.

Yes, this is equivalent to point (a).
Either pipe the commands through it, or try linking against the library I checked into CVS today.
(configure --enable-cmdlib; brief docn in lvm2cmd.h)

Quick example prog below.

Alasdair

#include "lvm2cmd.h"

/* All output gets passed to this function line-by-line */
void test_log_fn(int level, const char *file, int line, const char *format)

{
	/* Extract and process output here instead of printing it */

        if (level != 4)
                return;
 
        printf("%s\n", format);
        return;
}

int main(int argc, char **argv)

{
        void *handle;
        int r;
 
        lvm2_log_fn(test_log_fn);
 
        handle = lvm2_init();
 
        lvm2_log_level(handle, 1);
        r = lvm2_run(handle, "vgs --noheadings vg1");

	/* More commands here */
 
        lvm2_exit(handle);
 
        return r;
}

Thanks for the feedback. I'll be experimenting more with this over the next few weeks. I'd love to try changes as they come available.

Dave Olien

Other speed-ups:
Add a filter to lvm.conf so it doesn't waste time looking at non-PVs. Invoke commands using the lvm shell (configure --enable-readline) so the
tools retain the maximum amount of internal state between commands. [I'll fix this to avoid the warning when piping]

Alasdair

[email protected]