LVM mirror questions
Hello,
What happens to an lvm mirror if the disk log is lost ?
I am having a hard time understanding Linux lvm ; I created a mirror for my
home system, two same-sized drives. I decided to go for a mirror because I
have too much space and I thought with the rash of failed hard drives I've had
recently, having a mirror would be ideal. I had already installed my linux
system, and I already had my /home filesystem in a single stripe/disk lv anyway,
all I had to do was to add the other disk and lvcovert. But it wouldn't work,
saying I needed to add space on yet a third device for the log. Either that,
or I could keep the mirror log in memory. Lvm would have to rebuild a 1TB
mirror every time the system was rebooted if the log was left in memory. That
doesn't sound efficient. So. Put the log on disk. I couldn't find an
answer in man pages, but if the log is lost, then is the whole mirror lost ? That would mean to have the benefits of a mirror, the log would have to be at least as reliable as the mirror itself (or inconsequential).
So, another mirror would have to be built of tiny size to hold the log, then lvconvert can be given something like -log disk /dev/vghuh/lvtiny, etc..
But then for the original mirror to be reliable, the second mirror would need a
third mirror for the 2nd mirror's log. Unless it is ok to leave the 2nd
mirrors log in memory because it is so small.
What does everyone else do ?
Thank you in advance,
Jason Harris
Quoted 1 line · Jay
What happens to an lvm mirror if the disk log is lost?
The mirror should continue to work with a core (memory) log.
Yours,
Petr.
Hi Petr,
Thanks for the reply. "Should continue to work.." leaves me with some doubt.
My intent for a mirror is for a datastore that has reliability at the expense
of space. In addition, when a large datastore is rebooted, there would be a
period of time when the data is being mirrored, which leaves a window of peril. After I migrate my data off of lvm and onto an md device, I will test lvm.
Test #1 will be start a huge write operation on a mirror that consists of 2
mirrors and 1 log, all on seperate pvs. While the write operation is
happening, I will unplug the log drive and see if lvm switches in midstream to a memory log without loss.
Test #2 will be to see if I can make lvm activate lvs or vgs in a certain order.
That way I could make the first 20 extents of every physical device part of a
vg/lv that houses mirror logs. These log lvs would be mirrored across all
physical devices. The log lvs would have memory logs, which would be ok since
they will be so small. The window of peril is reduced from hours to a few
seconds.
Anyways, thanks again for the answer!
Jason Harris
________________________________
From: Petr Rockai <[email protected]>
To: LVM general discussion and development <[email protected]>
Sent: Tue, October 5, 2010 1:02:55 AM
Subject: Re: [linux-lvm] LVM mirror questions
Quoted 1 line · Jay
What happens to an lvm mirror if the disk log is lost?
The mirror should continue to work with a core (memory) log.
Yours,
Petr.
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Please let us know the results of your mirror testing.
--
Ray Morris
[email protected]
Strongbox - The next generation in site security:
http://www.bettercgi.com/strongbox/
Throttlebox - Intelligent Bandwidth Control
http://www.bettercgi.com/throttlebox/
Strongbox / Throttlebox affiliate program:
http://www.bettercgi.com/affiliates/user/register.php
Quoted 40 lines · Jay
Hi Petr,
Thanks for the reply. "Should continue to work.." leaves me with some doubt.
My intent for a mirror is for a datastore that has reliability at
the expense
of space. In addition, when a large datastore is rebooted, there would be a
period of time when the data is being mirrored, which leaves a window of peril. After I migrate my data off of lvm and onto an md device, I will test lvm.Test #1 will be start a huge write operation on a mirror that consists of 2
mirrors and 1 log, all on seperate pvs. While the write operation is
happening, I will unplug the log drive and see if lvm switches in midstream to a memory log without loss.Test #2 will be to see if I can make lvm activate lvs or vgs in a certain order.
That way I could make the first 20 extents of every physical device
part of a
vg/lv that houses mirror logs. These log lvs would be mirrored across all
physical devices. The log lvs would have memory logs, which would be ok since
they will be so small. The window of peril is reduced from hours to a few
seconds.Anyways, thanks again for the answer!
Jason Harris
________________________________
From: Petr Rockai <[email protected]>
To: LVM general discussion and development <[email protected]>
Sent: Tue, October 5, 2010 1:02:55 AM
Subject: Re: [linux-lvm] LVM mirror questionsQuoted 1 line · Jay
What happens to an lvm mirror if the disk log is lost?
The mirror should continue to work with a core (memory) log.
Quoted 12 lines
Yours,
Petr._______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
------quoted attachment------
Quoted 5 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Hi again,
Quoted 6 lines · Jay
Thanks for the reply. "Should continue to work.." leaves me with some doubt. My intent for a mirror is for a datastore that has reliability at the expense of space. In addition, when a large datastore is rebooted, there would be a period of time when the data is being mirrored, which leaves a window of peril. After I migrate my data off of lvm and onto an md device, I will test lvm.
This depends on how you set up things. With the normal dmeventd monitoring enabled, the mirror, when experiencing a log failure, will discard the log and continue working with core log. Same for faulty images (it'll discard the image, and depending on how many images you have, continue as linear or with fewer images; with recent versions, you can say in /etc/lvm/lvm.conf that you want the images/logs replaced automatically from free PV space).
Quoted 4 lines
Test #1 will be start a huge write operation on a mirror that consists of 2 mirrors and 1 log, all on seperate pvs. While the write operation is happening, I will unplug the log drive and see if lvm switches in midstream to a memory log without loss.
If you have dmeventd running and the mirror volumes are monitored, that's exactly what should happen. There might be a hitch in the IO flow while the mirror is converted, but nothing should be lost. Please do let us know if you observe anything else!
Quoted 6 lines
Test #2 will be to see if I can make lvm activate lvs or vgs in a certain order. That way I could make the first 20 extents of every physical device part of a vg/lv that houses mirror logs. These log lvs would be mirrored across all physical devices. The log lvs would have memory logs, which would be ok since they will be so small. The window of peril is reduced from hours to a few seconds.
You can use --mirrorlog mirrored for that, you just need to play a bit with the options to achieve the layout you want. Also please note that having a log and a mirror image on the same drive may come with a non-trivial performance penalty, since log updates now need to seek all the way through the disk, on both images (regardless of how people think things SHOULD work, the obvious is not always the best).
Yours,
Petr.
PS: What happens when you don't have dmeventd running/monitoring enabled probably depends on your kernel version. I believe it used to work in a similar fashion without dmeventd (faulty log would be ignored), but I am not sure that mode of operation is supported in newer versions. Someone else might chime in knowing more about the kernel level details.
Quoted 10 lines · Jay
the other disk and lvcovert. But it wouldn't work, saying I needed to
add space on yet a third device for the log. Either that, or I could
keep the mirror log in memory. Lvm would have to rebuild a 1TB mirror
every time the system was rebooted if the log was left in memory. That
doesn't sound efficient. So. Put the log on disk. I couldn't find
an answer in man pages, but if the log is lost, then is the whole mirror
lost ? That would mean to have the benefits of a mirror, the log would
have to be at least as reliable as the mirror itself (or inconsequential).
Quoted 3 lines
So, another mirror would have to be built of tiny size to hold the log, then lvconvert can be given something like -log disk
/dev/vghuh/lvtiny, etc..
As you have noticed, LVM mirror support is basically broken or not fully implemented yet. The mirror log SHOULD be stored on both legs of the mirror in a block allocated automatically in the metadata area by lvm, but isn't. If you want to mirror, you should do so using mdadm and then use the md device as an lvm pv.
Phillip,
Thank you for your answer and advice. How unfortunate about lvm mirroring being
broken. I like your suggestion about how it SHOULD work. In fact, long ago in
a different age when I was working with rs6000s, I believe that is how reliable mirroring was accomplished with the ibm lvm. Either that or it was that the metadata/log block was stored on multiple disks with a quorum needed to be functional.
I will convert to md mirror device. If I had more skill, I would contribute
somehow to the development effort. As it is, I will lurk on the list to see I
can be helpful to someone else. Again, I appreciate your help.
Regards,
Jason Harris
________________________________
From: Phillip Susi <[email protected]>
To: LVM general discussion and development <[email protected]>
Cc: Jay <[email protected]>
Sent: Tue, October 5, 2010 8:40:46 AM
Subject: Re: [linux-lvm] LVM mirror questions
Quoted 10 lines · Jay
the other disk and lvcovert. But it wouldn't work, saying I needed to
add space on yet a third device for the log. Either that, or I could
keep the mirror log in memory. Lvm would have to rebuild a 1TB mirror
every time the system was rebooted if the log was left in memory. That
doesn't sound efficient. So. Put the log on disk. I couldn't find
an answer in man pages, but if the log is lost, then is the whole mirror
lost ? That would mean to have the benefits of a mirror, the log would
have to be at least as reliable as the mirror itself (or inconsequential).
Quoted 3 lines
So, another mirror would have to be built of tiny size to hold the log, then lvconvert can be given something like -log disk
/dev/vghuh/lvtiny, etc..
As you have noticed, LVM mirror support is basically broken or not fully implemented yet. The mirror log SHOULD be stored on both legs of the mirror in a block allocated automatically in the metadata area by lvm, but isn't. If you want to mirror, you should do so using mdadm and then use the md device as an lvm pv.
Quoted 4 lines · Jay
Thank you for your answer and advice. How unfortunate about lvm mirroring
being broken. I like your suggestion about how it SHOULD work. In fact,
long ago in a different age when I was working with rs6000s, I believe that is how reliable mirroring was accomplished with the ibm lvm. Either that or
Yes, AIX mirroring was great.
The current LVM strategy could work, just use the option to store the mirror log on the same PV as one of the mirrors. If you lose the PV without the log - no problem. If you lose the PV *with* the log, it reverts to an in-core log, which is not a problem with the single remaining leg. The drawback to this is performance. The main reason to put the mirror log on a 3rd device is so the 3 writes occasioned by every write to the LV can be done in parallel.
I don't know how LVM handles issues of PVs that are not permanently dead and come back on line again later. The md driver uses a generation count in the super block to determine which is newer. (How is that updated without loss of efficiency?)
I still use the md driver myself. The LVM mirror support just doesn't "feel" reliable yet.
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 7 lines · Stuart D. Gathman
The current LVM strategy could work, just use the option to store the mirror log on the same PV as one of the mirrors. If you lose the PV without the log - no problem. If you lose the PV *with* the log, it reverts to an in-core log, which is not a problem with the single remaining leg. The drawback to this is performance. The main reason to put the mirror log on a 3rd device is so the 3 writes occasioned by every write to the LV can be done in parallel.
Yes, (write) performance is a concern. Otherwise, what you describe is perfectly valid and achievable with LVM.
Quoted 4 lines
I don't know how LVM handles issues of PVs that are not permanently dead and come back on line again later. The md driver uses a generation count in the super block to determine which is newer. (How is that updated without loss of efficiency?)
LVM has a generation counter as well. It's only updated on metadata writes though, so it doesn't cost anything. (Log is not part of metadata in this sense.) When you lose a leg (and use dmeventd), the metadata on the remaining PVs is updated to say that. The leg is also yanked from the mirror. You can add it as a fresh image (with full resync) if it ever comes back.
If you yank disks out of mirrors often (and plug them back afterwards), you should maybe use replication instead of mirroring.
Yours,
Petr.
Quoted 2 lines · Petr Rockai
Yes, (write) performance is a concern. Otherwise, what you describe is perfectly valid and achievable with LVM.
It is? What happens when the log device fails?
Quoted 11 lines
Quoted 4 lines
I don't know how LVM handles issues of PVs that are not permanently dead and come back on line again later. The md driver uses a generation count in the super block to determine which is newer. (How is that updated without loss of efficiency?)
LVM has a generation counter as well. It's only updated on metadata writes though, so it doesn't cost anything. (Log is not part of metadata in this sense.) When you lose a leg (and use dmeventd), the metadata on the remaining PVs is updated to say that. The leg is also yanked from the mirror. You can add it as a fresh image (with full resync) if it ever comes back.
Why a full resync? With mdadm, it just keeps flagging the dirty chunks so it only has to copy those when the other disk returns.
Quoted 9 lines · Phillip Susi
Quoted 6 lines
LVM has a generation counter as well. It's only updated on metadata writes though, so it doesn't cost anything. (Log is not part of metadata in this sense.) When you lose a leg (and use dmeventd), the metadata on the remaining PVs is updated to say that. The leg is also yanked from the mirror. You can add it as a fresh image (with full resync) if it ever comes back.
Why a full resync? With mdadm, it just keeps flagging the dirty chunks so it only has to copy those when the other disk returns.
Because it's really hard to guarantee anything when you experience a failure. The safe route is to discard any data on the failed drive. Certainly better than winding up with a subtly corrupt mirror.
As I said before, if you experience that a lot, you should be using replication not mirroring.
Yours,
Petr.