advice sought on setting up new system

Hi there

I'm new to LVM2 and am considering using it on a production system, but would welcome advice from experienced users out there, before I set things up. Your advice and comments would be appreciated. Details follow.

Summary:

Primary function: mail system (pop3,smtp,http)
Scalable to 10's of thousands (and hopefully millions) of users ;-)
Applications: qmail, PostgreSQL, djbdns

Current system:

1GB RAM
4x U320 15k SCSI 72GB disks in hardware RAID1+0
All functions running on same host.

Future system to include external drive cage(s) and splitting of functions onto separate hosts (DB/mail/DNS). The external drive cage(s) to be used for mail storage. qmail uses maildir, as opposed to mbox delivery, so each message is a separate file... this means I require performance for a vastly greater number of smaller files as opposed to a single mbox file per user. Another consideration is a popular belief that mounting /var/qmail/queue with the sync option set, or manually invoking sync() improves the queue's robustness (that's another story).

At the moment the box has:

/       520MB           ext3
/boot   32MB            ext3
free    +/-130GB        LVM

There's a swap partition, but otherwise the rest is available. I'm considering something approximating the following.

/                       520MB           ext3
/boot                   32MB            ext3
/usr                    512MB           LVM/ext3
/var                    1GB             LVM/ext3
/var/lib/pgsql          512MB           LVM/ext3
/var/qmail/queue        512MB           LVM/ext3        sync
/home                   1GB             LVM/ext3
/home/mail              127GB+          LVM/ext3

All the file systems on LVs may require expansion/contraction as the system grows, so I'd like as flexible a setup as possible. The qmail queue should probably be larger, rather than smaller, so that expansion doesn't adversely affect performance. What about adding swap space?

What size should the various PEs be, linear/striped?... what other configurable LVM settings would suit this setup etc. I'm a complete newbie as far as LVM is concerned, so need valuable pointers.

The primary goal here is for flexibility and I/O performance - the hardware RAID takes care of availability/redundancy, together with backup media.

Thanks in advance
Dale

The location of the extents of a logical volume can have a great impact on its I/O performance:

1) LVM does its best to make the logical extents physically contiguous, but pvmove(8) can be used to improve this as the need arises.

2) Here's an often neglected fact: The performance of the outer cylinders of almost any hard drive (larger than a few GB) is usually about double that of the inner cylinders of the same drive. (This is due to the fact that there are about twice as many sectors per track on the outside of the platter as there are on the inside of the platter and thus data moves past the read/write heads about twice as fast on the outer cylinders as the inner cylinders.)

Therefore, to improve I/O performance, try to create often used logical volumes first as LVM tends to allocate lower physical extents first. Otherwise, use pvmove(8) to move the more frequently accessed extents to the fastest (lowest) physical extents of the fastest LVM physical volume. It is best to use (almost) whole disks as LVM physical volumes, since smaller physical volumes (partitions) would be either fast, average, or slow and not the full spectrum of fast through slow. The least accessed logical extents should be moved to the highest numbered (slow) physical extents of the slowest disks.

3) LVM striping can also be used to improve I/O performance over two or more spindles (disks). Avoid LVM striping over two physical volumes on the same spindle (disk), since this reduces performance as the system waits for seeks between these two physical volumes (for a simple sequential access).

Sincerely,

Ken Fuchs <[email protected]>

Thanks for the response Ken - I've been swamped, so appologies for the belated reply.

Surely under hardware raid this doesn't apply, as the location of the data is mirrored/striped across many disks, so the physical location of data is no longer obvious?

Now by using harware raid, theoretically there may well be an additional performance gain (increasing in proportion to the number of spindles of course).

My original primary concern is the performance impact the system will experience, particularly if I append logical extents which are non-contiguous to a LV. Say for example, the original setup was:

PEs(0-100) for LV /dev/mapper/v0-var-qmail-queue PEs (101-1000) for LV /dev/mapper/v0-home-mail

And then I find at a later stage that my queue is too small, so append PEs (1001-1100) to the original LV /dev/mapper/v0-var-qmail-queue.

Will the performance impact be noticeable, taking into consideration that each file written to the queue is synced, together with hardware raid and U320 SCSI disks?

I know, it's difficult to assume things up-front, but I don't want to rush into using LVM for the wrong reasons... on that note, with hardware raid and the setup I posted originally, what are the clear practical benefits of using LVM, other than reducing the possible future need to sym-link directories onto other file- systems?

I have LVM2/Linux-2.6.1 running on my laptop, but that's the only experience I have with LVM, so do not really know whether or not to go with it in a production environment, as I haven't the experience.

Anyone care to enlighten me further? I'm not entirely convinced that LVM is the right way to go yet. Why _should_ I use it? Will the benefits out-weigh the negative aspects?

Thanks.
Dale

The extent size should be chosen to be a convenient size for the system administrator. Typically 4-16M these days; it tends to get bigger as hard disks grow. So the extent size is going to be far too large for you to worry about files being broken across extent boundaries. Think about the percentage of ios that will incur this extra seek that you are concerned about.

- Joe

Hardware RAID just makes the physical location of the data just a little harder to determine. Hardware RAID simply transforms n disks of the same size into a larger virtual disk. Hardware RAID will almost certainly combine the outer (lowest) cylinders of all disks together to form the lowest cylinders of the virtual disk and similarly it will combine the inner (highest) cylinders of all disks to form the highest cylinders of the virtual disk. Thus, the lowest cylinders of the hardware RAID virtual disk will be about twice as fast as the highest cylinders of the virtual disk in exactly the same way it would apply to any individual disk in the array. Of course the hardware RAID performance will be much better than a single drive, depending on the actual application.

However, it is possible (though very unlikely) that a hardware RAID is set up in a way that eliminates the performance differential between lower and higher cylinders for example by matching lower cylinders of the odd drives with higher cylinders of the even drives. This is an unlikely set up, since the maximum performance of such a RAID would be the same as the average performance of a normal hardware RAID or only 67% of the maximum performance of a normal RAID.

Thus, for the typical hardware RAID, one would want to allocate low physical extents to logical volumes needing high performance and higher physical extents to logical volumes with lower performance demands.

Yes, there should be a significant performance increase depending on the application and how well its data accesses are distributed over all spindles in the RAID.

However, multiple simultaneous applications may cause more long seeks and in unlikely worst case scenarios, RAID can even perform worst than a like number of independent hard drives, if for example every access requires a (long) seek on all actuators in the RAID.

For sequential queue access, the physical extent gap should have no noticeable impact on performance. The performance impact on an application doing random access on a logical volume with a large physical extent gap could be noticeable.

Not all applications perform properly when symbolic links are used in this fashion.

Hard drives can be added and removed from the volume group(s) as needed to increase capacity and remove old drives. Logical volumes or parts of them can be resized smaller or larger and moved between and within physical volumes to facilitate removal of a drive or act on a performance consideration.

Sincerely,

Ken Fuchs <[email protected]>