Allocation Policy for Cloud Computing needed

Hi LVM list,

I'm experimenting with storage for many QEMU/KVM virtual machines in cloud computing. I've got many concurrent IO processes and 24 hard drives. I've tested the scalability with a single IO reader process per hard drive. Single drives scale best and have the best performance of cause, but we need mirroring and volume management. So I've created MD RAID-1 arrays and created on each a VG and two LVs. This gives me good overall performance (up to 2 GB/s, HBA limit: 2.2 GB/s).

Then, I've tested to put all my RAID-1 arrays into a single VG, because LV size should be adjustable over all hard drives. I've tried all allocation policies but none does what I want to achieve here. Yeah, that this isn't implemented fully is in the man page, ... .

I want to have an allocation which distributes the LVs equally over the PVs as long as space is left and LVs aren't resized. The goal is to minimize the number of concurrent IO processes per hard drive (striping is total crap in this situation).

I've tested LVM2 2.02.66 and kernel 3.0.15. Is something like that implemented in newer releases or is something like that intended to be implemented in near future? Or does someone want to implement this together with me?

Thanks, cheers,

Sebastian

I wrote a script to do this, with LVM 1 that just extended the lv in chunks over a range of PVs until it was full. while it worked on LVM it produced lots of backups in /etc. I felt with the LVM stripe function that this was not necessary, and could acheive the same thing by using large stripe size, as LVM allows concats of different stripeness it was not a problem extending the lv.

Hi LVM list,

I'm experimenting with storage for many QEMU/KVM virtual machines in cloud computing. I've got many concurrent IO processes and 24 hard drives. I've tested the scalability with a single IO reader process per hard drive. Single drives scale best and have the best performance of cause, but we need mirroring and volume management. So I've created MD RAID-1 arrays and created on each a VG and two LVs. This gives me good overall performance (up to 2 GB/s, HBA limit: 2.2 GB/s).

Then, I've tested to put all my RAID-1 arrays into a single VG, because LV size should be adjustable over all hard drives. I've tried all allocation policies but none does what I want to achieve here. Yeah, that this isn't implemented fully is in the man page, ... .

I want to have an allocation which distributes the LVs equally over the PVs as long as space is left and LVs aren't resized. The goal is to minimize the number of concurrent IO processes per hard drive (striping is total crap in this situation).

I've tested LVM2 2.02.66 and kernel 3.0.15. Is something like that implemented in newer releases or is something like that intended to be implemented in near future?

Or does someone want to implement this together with me?

Quoted 2 lines

Then, I've tested to put all my RAID-1 arrays into a single VG, because LV size should be adjustable over all hard drives.

Quoted 2 lines

I want to have an allocation which distributes the LVs equally over the PVs as long as space is left and LVs aren't resized.

You can get pretty complex in similar scripts, if you really want to... consider using
pvs -o vg_name,lv_name,pv_name,pvseg_start,pvseg_size,seg_pe_ranges
and explicitly listing not only the PVS, but even the PE ranges to your lvcreate commands...

Quoted 17 lines · Sebastian Riemer

Hi LVM list,

I'm experimenting with storage for many QEMU/KVM virtual machines in cloud computing. I've got many concurrent IO processes and 24 hard drives. I've tested the scalability with a single IO reader process per hard drive. Single drives scale best and have the best performance of cause, but we need mirroring and volume management. So I've created MD RAID-1 arrays and created on each a VG and two LVs. This gives me good overall performance (up to 2 GB/s, HBA
limit: 2.2 GB/s).

Then, I've tested to put all my RAID-1 arrays into a single VG, because LV size should be adjustable over all hard drives. I've tried all allocation policies but none does what I want to achieve here. Yeah, that this isn't implemented fully is in the man page, ... .

Quoted 8 lines

I want to have an allocation which distributes the LVs equally over the PVs as long as space is left and LVs aren't resized. The goal is to minimize the number of concurrent IO processes per hard drive (striping is total crap in this situation).

I've tested LVM2 2.02.66 and kernel 3.0.15. Is something like that implemented in newer releases or is something like that intended to be implemented in near future?

I don't know. Does not look like it, though.

Quoted 1 line

Or does someone want to implement this together with me?

I would certainly be here for discussions.

Though, as you always will be more flexible with scripts than with pre-implemented fixed algorithms, I probably would first check if I can solve it with some scripting.
[completely untested, but you get the idea]

#!/bin/bash
export LANG=C LC_ALL=C
name=$1 vg=$2 size_in_MiB=$3
PVS=$(vgs --nohead --unit m -o pv_name,pv_free -O -pv_free,pv_name

$vg | awk -v need=$size_in_MiB '{ print $1; sum += $2;
	if (sum >= need) exit; }')

lvcreate -n $name -L ${size_in_MiB}m $vg $PVS

(similar for lvextend)

Which basically implements this allocation policy:
use the pvs with most free space available, and no more than necessary.

If I understand you correctly, that would almost do what you asked for.

You can get pretty complex in similar scripts, if you really want to... consider using
pvs -o vg_name,lv_name,pv_name,pvseg_start,pvseg_size,seg_pe_ranges
and explicitly listing not only the PVS, but even the PE ranges to your lvcreate commands...

Lars

Since you're using RAID anyway, consider testing RAID 10, which will distribute IO across spindles.

For scripting, see Linux::LVM on CPAN. It gives you that information as a nice data structure. I welcome feature requests and patches. (Linux::LVM::Do coming soon for modifying rather than just querying LVM objects.)

Quoted 1 line

Or does someone want to implement this together with me?

I would certainly be here for discussions.

Though, as you always will be more flexible with scripts than with pre-implemented fixed algorithms, I probably would first check if I can solve it with some scripting.
[completely untested, but you get the idea]

#!/bin/bash
export LANG=C LC_ALL=C
name=$1 vg=$2 size_in_MiB=$3
PVS=$(vgs --nohead --unit m -o pv_name,pv_free -O -pv_free,pv_name $vg |

	awk -v need=$size_in_MiB '{ print $1; sum += $2;
	if (sum >= need) exit; }')

lvcreate -n $name -L ${size_in_MiB}m $vg $PVS

(similar for lvextend)

Which basically implements this allocation policy:
use the pvs with most free space available, and no more than necessary.

If I understand you correctly, that would almost do what you asked for.

You can get pretty complex in similar scripts, if you really want to... consider using
pvs -o vg_name,lv_name,pv_name,pvseg_start,pvseg_size,seg_pe_ranges
and explicitly listing not only the PVS, but even the PE ranges to your lvcreate commands...