vgimport/vgexport commands when logically moving disks between systems
Hello,
I believe I might be misunderstanding whether vgimport and vgexport are needed in my particular situation. It would be great to get some feedback for clarification.
THE SETUP:
LVM1 (1.0.8-14) on two RedHat AS3 systems (kernel: 2.4.21-47.ELsmp).
I think the same concept applies for LVM2 as well though.
Machine A is primary and Machine B is backup in a two-node Linux heartbeat cluster. Both machines are connected to the same SAN via fiber, and see the same disks, where the volume groups reside.
THE STRATEGY:
The idea is for A to have the LVM file systems mounted, and when a failure is detected, have the LVM file systems "moved" to (or seen by) system B. The way this is currently accomplished is for A to do the following upon detection of a failure:
+ unmount file systems
+ deactivate (vgchange -an $vg)
+ export (vgexport $vg)
then, on system B:
+ import & activate (vgimport $vg $disks)
+ mount file systems
THE ISSUE:
The export works as expected on A, but upon import on B, a return code of 4 is returned meaning "volume group already exists". The mounting works properly, but all the disks are shown like this:
"inactive PV /dev/sdx is in EXPORTED VG $vg"
when inspected with pvscan.
Does a vgimport and/or vgexport mark the disks themselves, or simply update the system on which the commands are run??? I suppose that is essentially the heart of this issue.
I'm starting to believe that for our strategy the vgexport and vgimport commands are not necessary, and are actually causing the problem. (The HOWTO mentions these commands are used to move disks between systems, but perhaps that is meant to refer to disks that are only physically moved?)
Instead, the following strategy might be correct in case system A fails (Note: no vgimport or vgexport commands):
+ unmount fs
+ vgchange -an $vg
then, on system B:
+ vgchange -ay $vg
+ mount fs
IS THIS CORRECT???
Thanks in advance for any helpful feedback.
Cheers,
Dave
____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com
Quoted 34 lines · Dave
Hello,
I believe I might be misunderstanding whether vgimport and vgexport are needed in my particular situation. It would be great to get some feedback for clarification.
THE SETUP:
LVM1 (1.0.8-14) on two RedHat AS3 systems (kernel: 2.4.21-47.ELsmp).
I think the same concept applies for LVM2 as well though.
Machine A is primary and Machine B is backup in a two-node Linux heartbeat cluster. Both machines are connected to the same SAN via fiber, and see the same disks, where the volume groups reside.THE STRATEGY:
The idea is for A to have the LVM file systems mounted, and when a failure is detected, have the LVM file systems "moved" to (or seen by) system B. The way this is currently accomplished is for A to do the following upon detection of a failure:
+ unmount file systems
+ deactivate (vgchange -an $vg)
+ export (vgexport $vg)then, on system B:
+ import & activate (vgimport $vg $disks)
+ mount file systemsTHE ISSUE:
The export works as expected on A, but upon import on B, a return code of 4 is returned meaning "volume group already exists". The mounting works properly, but all the disks are shown like this:
Quoted 8 lines
"inactive PV
/dev/sdxis in EXPORTED VG $vg"when inspected with pvscan.
Does a vgimport and/or vgexport mark the disks themselves, or simply update the system on which the commands are run??? I suppose that is essentially the heart of this issue.
Yes, vgimport/export marks the disks in the volume group. It's really for moving disks between systems where the target system might have a volume group with the same name as the one to be imported.
Quoted 11 lines
I'm starting to believe that for our strategy the vgexport and vgimport commands are not necessary, and are actually causing the problem. (The HOWTO mentions these commands are used to move disks between systems, but perhaps that is meant to refer to disks that are only physically moved?)
Instead, the following strategy might be correct in case system A fails (Note: no vgimport or vgexport commands):
+ unmount fs
+ vgchange -an $vg
Quoted 8 lines
then, on system B:
+ vgchange -ay $vg
+ mount fsIS THIS CORRECT???
Yes. IF YOU'RE VERY CAREFUL!
vgimport/vgexport are not the tools you want for this job.
Signature · 1 line
patrick
Hi Patrick, et al.
Thanks for your comments!
I've been doing some testing on this topic, and encountering a bit of strange behavior which seems to confirm the murkiness with switching VGs between hosts. Please bear with my lengthy description, as I'm trying to be as clear as possible. I really need to work this out.
THE TEST:
Hosts are P10 (primary node) and P11 (backup node)
VG is activated on P10 and fs is mounted. To test a switch to P11, I deactivate the VG on P10, but DO NOT run vgexport (following your suggestion). I then, run vgimport on P11 (but P11 reports it already knows about the VG - that's fine), and then run an activate. However, when I try to run an e2fsck on the fs, I get the following error:
/sbin/e2fsck: No such device or address while trying to open /dev/tux-ao/app Possibly non-existent or swap device?
However, the device does exist, and looks identical to the one on P10:
[P11]$ ls -l /dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 14:46 /dev/tux-ao/app
I was able to fix the problem by putting vgexport back into the mix. In this case, I export the VG from P10 and then after an import on P11 I was able to run e2fsck and mount successfully.
Also, this (unknown VG) message is somewhat common in pvscan, if a vgexport is not performed:
Quoted 1 line
pvscan -- inactive PV "
/dev/sdk" is associated to unknown VG "tux-ao" (run
vgscan)
Here is another clear example of some unexpected behavior (at least to me)...
1. status when $vg on P11 successful - notice the status of ACTIVE on P11 and EXPORTED on P10
[P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- ACTIVE PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free]
[P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is in EXPORTED VG "tux-ao" [27.09 GB / 9.09 GB free]
Then...
2. status after $vg is deactivated on P11 and activated on P10 (no vgexport run on P11 before activation on P10) - Notice the "unknown VG" message on P10!!!
[P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free]
[P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is associated to unknown VG "tux-ao" (run vgscan)
Then...
3. Then, when I tried an activation on P10 I get no joy at all (ie. cannot perform operations on the logical volume), despite the fact that it exists:
[P10]$ ls -l /dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 15:04 /dev/tux-ao/app
/sbin/e2fsck: No such device or address while trying to open /dev/tux-ao/app Possibly non-existent or swap device?
Is this simply flakey behavior with LVM 1.0.8 ?
We have several VGs on the machine, and sometimes we need to move one or two at a time. LVM (v1) seems to have trouble here, at least without vgexport. Anyone know what might be happening behind the scenes to cause this behavior? It's looking to me like I really do need vgexport to make things work the way we want.
Thanks again for helping me to clarify this situation. Dave
Quoted 76 lines · --- Patrick Caulfield
Quoted 3 lines · Dave
Hello,
I believe I might be misunderstanding whether vgimport and vgexport are needed
in my
Quoted 7 lines
particular situation. It would be great to get some feedback for clarification.
THE SETUP:
LVM1 (1.0.8-14) on two RedHat AS3 systems (kernel: 2.4.21-47.ELsmp). I think the same concept applies for LVM2 as well though. Machine A is primary and Machine B is backup in a two-node Linux heartbeat
cluster.
Quoted 1 line
Both machines are connected to the same SAN via fiber, and see the same disks,
where
Quoted 6 lines
the volume groups reside.
THE STRATEGY:
The idea is for A to have the LVM file systems mounted, and when a failure is detected, have the LVM file systems "moved" to (or seen by) system B. The way
this
Quoted 1 line
is currently accomplished is for A to do the following upon detection of a
failure:
Quoted 7 lines
+ unmount file systems
+ deactivate (vgchange -an $vg)
+ export (vgexport $vg)then, on system B:
Quoted 7 lines
+ import & activate (vgimport $vg $disks)
+ mount file systemsTHE ISSUE:
The export works as expected on A, but upon import on B, a return code of 4 is returned meaning "volume group already exists". The mounting works properly,
but
Quoted 9 lines
all the disks are shown like this:
"inactive PV
/dev/sdxis in EXPORTED VG $vg"when inspected with pvscan.
Does a vgimport and/or vgexport mark the disks themselves, or simply update the system on which the commands are run??? I suppose that is essentially the heart
of
Quoted 1 line
this issue.
Yes, vgimport/export marks the disks in the volume group. It's really for moving disks between systems where the target system might
have a volume group with the same name as the one to be imported.Quoted 1 line
I'm starting to believe that for our strategy the vgexport and vgimport commands
are
Quoted 2 lines
not necessary, and are actually causing the problem. (The HOWTO mentions these commands are used to move disks between systems, but perhaps that is meant to
refer
Quoted 3 lines
to disks that are only physically moved?)
Instead, the following strategy might be correct in case system A fails (Note:
no
Quoted 11 lines
vgimport or vgexport commands):
+ unmount fs
+ vgchange -an $vgthen, on system B:
+ vgchange -ay $vg
+ mount fs
Quoted 17 lines
Quoted 2 lines
IS THIS CORRECT???
Yes. IF YOU'RE VERY CAREFUL!
vgimport/vgexport are not the tools you want for this job.Signature · 7 lines
patrick _______________________________________________ linux-lvm mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
____________________________________________________________________________________
Cheap talk?
Check out Yahoo! Messenger's low PC-to-Phone call rates.
http://voice.yahoo.com
Quoted 73 lines · Dave
Hi Patrick, et al.
Thanks for your comments!
I've been doing some testing on this topic, and encountering a bit of strange behavior which seems to confirm the murkiness with switching VGs between hosts. Please bear with my lengthy description, as I'm trying to be as clear as possible. I really need to work this out.THE TEST:
Hosts are P10 (primary node) and P11 (backup node)
VG is activated on P10 and fs is mounted. To test a switch to P11, I deactivate the VG on P10, but DO NOT run vgexport (following your suggestion). I then, run vgimport on P11 (but P11 reports it already knows about the VG - that's fine), and then run an activate. However, when I try to run an e2fsck on the fs, I get the following error:
/sbin/e2fsck: No such device or address while trying to open/dev/tux-ao/appPossibly non-existent or swap device?However, the device does exist, and looks identical to the one on P10:
[P11]$ ls -l/dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 14:46/dev/tux-ao/appI was able to fix the problem by putting vgexport back into the mix. In this case, I export the VG from P10 and then after an import on P11 I was able to run e2fsck and mount successfully.
Also, this (unknown VG) message is somewhat common in pvscan, if a vgexport is not performed:
Quoted 1 line
pvscan -- inactive PV "
/dev/sdk" is associated to unknown VG "tux-ao" (runvgscan)
Here is another clear example of some unexpected behavior (at least to me)...
1. status when $vg on P11 successful - notice the status of ACTIVE on P11 and EXPORTED on P10[P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- ACTIVE PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free][P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is in EXPORTED VG "tux-ao" [27.09 GB / 9.09 GB free]Then...
2. status after $vg is deactivated on P11 and activated on P10 (no vgexport run on P11 before activation on P10) - Notice the "unknown VG" message on P10!!![P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free][P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is associated to unknown VG "tux-ao" (run vgscan)Then...
3. Then, when I tried an activation on P10 I get no joy at all (ie. cannot perform operations on the logical volume), despite the fact that it exists:
[P10]$ ls -l/dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 15:04/dev/tux-ao/app
/sbin/e2fsck: No such device or address while trying to open/dev/tux-ao/appPossibly non-existent or swap device?
Quoted 8 lines
Is this simply flakey behavior with LVM 1.0.8 ?
We have several VGs on the machine, and sometimes we need to move one or two at a time. LVM (v1) seems to have trouble here, at least without vgexport. Anyone know what might be happening behind the scenes to cause this behavior? It's looking to me like I really do need vgexport to make things work the way we want.
Thanks again for helping me to clarify this situation.
It's rather worrying that the two nodes seem to be reading different data from the same disks. I can't remember off-hand whether lvm1 does direct-io when it updates metadata, possibly not. In which case you might have to upgrade to lvm2 (which does).
lvm1 is /not/ a clustering tool ;-)
Signature · 1 line
patrick
Hi Patrick,
Thanks for clarifying the status of LVM1 (ie. *not* a clustering tool). That's somewhat surprising since it does include an LVM script in the resource.d directory.
In any case, we're not in a huge rush to upgrade to LVM2 unless it really is
necessary, as these are RH AS3 boxes running the 2.4.21-47.ELsmp kernel, and the docs I've read suggest LVM2 really needs/wants to run under a 2.6.x kernel.
If direct-io to metadata could be set as an option under LVM1, assuming that isn't the default, that would be helpful. But, it sounds like no one really wants to support LVM1 anymore?!
Assuming we want to go forward with an upgrade to LVM2 on the current systems, how involved is that procedure? Do we risk losing all our data?
Thanks,
Dave
Quoted 95 lines · --- Patrick Caulfield
Quoted 6 lines · Dave
Hi Patrick, et al.
Thanks for your comments!
I've been doing some testing on this topic, and encountering a bit of strange behavior which seems to confirm the murkiness with switching VGs between hosts. Please bear with my lengthy description, as I'm trying to be as clear aspossible.
Quoted 6 lines
I really need to work this out.
THE TEST:
Hosts are P10 (primary node) and P11 (backup node)
VG is activated on P10 and fs is mounted. To test a switch to P11, I deactivatethe
Quoted 2 lines
VG on P10, but DO NOT run vgexport (following your suggestion). I then, run vgimport on P11 (but P11 reports it already knows about the VG - that's fine),
and
Quoted 11 lines
then run an activate. However, when I try to run an e2fsck on the fs, I get the following error:
/sbin/e2fsck: No such device or address while trying to open/dev/tux-ao/appPossibly non-existent or swap device?However, the device does exist, and looks identical to the one on P10:
[P11]$ ls -l/dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 14:46/dev/tux-ao/appI was able to fix the problem by putting vgexport back into the mix. In this
case,
Quoted 1 line
I export the VG from P10 and then after an import on P11 I was able to run
e2fsck
Quoted 3 lines
and mount successfully.
Also, this (unknown VG) message is somewhat common in pvscan, if a vgexport is
not
Quoted 15 lines
performed:
Quoted 1 line
pvscan -- inactive PV "
/dev/sdk" is associated to unknown VG "tux-ao" (runvgscan)
Here is another clear example of some unexpected behavior (at least to me)...
1. status when $vg on P11 successful - notice the status of ACTIVE on P11 and EXPORTED on P10[P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- ACTIVE PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free]Quoted 8 lines
[P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is in EXPORTED VG "tux-ao" [27.09 GB / 9.09 GB free]Then...
2. status after $vg is deactivated on P11 and activated on P10 (no vgexport runon
Quoted 11 lines
P11 before activation on P10) - Notice the "unknown VG" message on P10!!!
[P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free][P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is associated to unknown VG "tux-ao" (runvgscan)
Quoted 3 lines
Then...
3. Then, when I tried an activation on P10 I get no joy at all (ie. cannotperform
Quoted 10 lines
operations on the logical volume), despite the fact that it exists:
[P10]$ ls -l/dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 15:04/dev/tux-ao/app
/sbin/e2fsck: No such device or address while trying to open/dev/tux-ao/appPossibly non-existent or swap device?Is this simply flakey behavior with LVM 1.0.8 ?
We have several VGs on the machine, and sometimes we need to move one or two at
a
Quoted 1 line
time. LVM (v1) seems to have trouble here, at least without vgexport. Anyone
know
Quoted 1 line
what might be happening behind the scenes to cause this behavior? It's looking
to
Quoted 3 lines
me like I really do need vgexport to make things work the way we want.
Thanks again for helping me to clarify this situation.
Quoted 18 lines
It's rather worrying that the two nodes seem to be reading different data from the same disks.
I can't remember off-hand whether lvm1 does direct-io when it updates metadata, possibly not.
In which case you might have to upgrade to lvm2 (which does).lvm1 is /not/ a clustering tool ;-)
Signature · 7 lines
patrick _______________________________________________ linux-lvm mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
____________________________________________________________________________________
Yahoo! Music Unlimited
Access over 1 million songs.
http://music.yahoo.com/unlimited
Quoted 7 lines · Dave
Hi Patrick,
Thanks for clarifying the status of LVM1 (ie. *not* a clustering tool). That's somewhat surprising since it does include an LVM script in the resource.d directory.
In any case, we're not in a huge rush to upgrade to LVM2 unless it really is
necessary, as these are RH AS3 boxes running the 2.4.21-47.ELsmp kernel, and the docs I've read suggest LVM2 really needs/wants to run under a 2.6.x kernel.
Yes. there are device-mapper patches for 2.4 but it's really recommended to run 2.6.
Quoted 3 lines
If direct-io to metadata could be set as an option under LVM1, assuming that isn't the default, that would be helpful. But, it sounds like no one really wants to support LVM1 anymore?!
That's true. lvm1 is in maintenance mode for RHEL3.
Quoted 2 lines
Assuming we want to go forward with an upgrade to LVM2 on the current systems, how involved is that procedure? Do we risk losing all our data?
No. lvm2 can read and write lvm1 metadata. you can upgrade it to lvm2 format metadata with a single command (vgconvert)
Signature · 1 line
patrick
Quoted 1 line · Patrick Caulfield
Yes. there are device-mapper patches for 2.4 but it's really recommended to run 2.6.
The 2.4 patches have not been updated for a long time so they include bugs that got fixed in 2.6 and they are missing features.
Anyone fancy updating them?
Alasdair
Signature · 1 line
[email protected]