Skip to contentSkip to conversation

vgimport/vgexport commands when logically moving disks between systems

linux-lvm Dave 3 participants 7 messages Dec 5, 2006 – Dec 14, 2006

lore
Share & tools
lore

Dave

author
#0 lore raw

Hello,

I believe I might be misunderstanding whether vgimport and vgexport are needed in my particular situation. It would be great to get some feedback for clarification.

THE SETUP:

LVM1 (1.0.8-14) on two RedHat AS3 systems (kernel: 2.4.21-47.ELsmp).
I think the same concept applies for LVM2 as well though.
Machine A is primary and Machine B is backup in a two-node Linux heartbeat cluster. Both machines are connected to the same SAN via fiber, and see the same disks, where the volume groups reside.

THE STRATEGY:

The idea is for A to have the LVM file systems mounted, and when a failure is detected, have the LVM file systems "moved" to (or seen by) system B. The way this is currently accomplished is for A to do the following upon detection of a failure:

+ unmount file systems
+ deactivate (vgchange -an $vg)
+ export (vgexport $vg)

then, on system B:

+ import & activate (vgimport $vg $disks)
+ mount file systems

THE ISSUE:

The export works as expected on A, but upon import on B, a return code of 4 is returned meaning "volume group already exists". The mounting works properly, but all the disks are shown like this:

"inactive PV /dev/sdx is in EXPORTED VG $vg"

when inspected with pvscan.

Does a vgimport and/or vgexport mark the disks themselves, or simply update the system on which the commands are run??? I suppose that is essentially the heart of this issue.

I'm starting to believe that for our strategy the vgexport and vgimport commands are not necessary, and are actually causing the problem. (The HOWTO mentions these commands are used to move disks between systems, but perhaps that is meant to refer to disks that are only physically moved?)

Instead, the following strategy might be correct in case system A fails (Note: no vgimport or vgexport commands):

+ unmount fs
+ vgchange -an $vg

then, on system B:

+ vgchange -ay $vg
+ mount fs

IS THIS CORRECT???

Thanks in advance for any helpful feedback.
Cheers,
Dave

____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

#1 lore raw
Quoted 34 lines · Dave

Hello,

I believe I might be misunderstanding whether vgimport and vgexport are needed in my particular situation. It would be great to get some feedback for clarification.

THE SETUP:

LVM1 (1.0.8-14) on two RedHat AS3 systems (kernel: 2.4.21-47.ELsmp).
I think the same concept applies for LVM2 as well though.
Machine A is primary and Machine B is backup in a two-node Linux heartbeat cluster. Both machines are connected to the same SAN via fiber, and see the same disks, where the volume groups reside.

THE STRATEGY:

The idea is for A to have the LVM file systems mounted, and when a failure is detected, have the LVM file systems "moved" to (or seen by) system B. The way this is currently accomplished is for A to do the following upon detection of a failure:

+ unmount file systems
+ deactivate (vgchange -an $vg)
+ export (vgexport $vg)

then, on system B:

+ import & activate (vgimport $vg $disks)
+ mount file systems

THE ISSUE:

The export works as expected on A, but upon import on B, a return code of 4 is returned meaning "volume group already exists". The mounting works properly, but all the disks are shown like this:

Quoted 8 lines

"inactive PV /dev/sdx is in EXPORTED VG $vg"

when inspected with pvscan.

Does a vgimport and/or vgexport mark the disks themselves, or simply update the system on which the commands are run??? I suppose that is essentially the heart of this issue.

Yes, vgimport/export marks the disks in the volume group. It's really for moving disks between systems where the target system might have a volume group with the same name as the one to be imported.

Quoted 11 lines

I'm starting to believe that for our strategy the vgexport and vgimport commands are not necessary, and are actually causing the problem. (The HOWTO mentions these commands are used to move disks between systems, but perhaps that is meant to refer to disks that are only physically moved?)

Instead, the following strategy might be correct in case system A fails (Note: no vgimport or vgexport commands):

+ unmount fs
+ vgchange -an $vg

Quoted 8 lines

then, on system B:

+ vgchange -ay $vg
+ mount fs

IS THIS CORRECT???

Yes. IF YOU'RE VERY CAREFUL!
vgimport/vgexport are not the tools you want for this job.

Signature · 1 line
patrick
author
#2 lore raw

Hi Patrick, et al.

Thanks for your comments!
I've been doing some testing on this topic, and encountering a bit of strange behavior which seems to confirm the murkiness with switching VGs between hosts. Please bear with my lengthy description, as I'm trying to be as clear as possible. I really need to work this out.

THE TEST:

Hosts are P10 (primary node) and P11 (backup node)
VG is activated on P10 and fs is mounted. To test a switch to P11, I deactivate the VG on P10, but DO NOT run vgexport (following your suggestion). I then, run vgimport on P11 (but P11 reports it already knows about the VG - that's fine), and then run an activate. However, when I try to run an e2fsck on the fs, I get the following error:

/sbin/e2fsck: No such device or address while trying to open /dev/tux-ao/app Possibly non-existent or swap device?

However, the device does exist, and looks identical to the one on P10:
[P11]$ ls -l /dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 14:46 /dev/tux-ao/app

I was able to fix the problem by putting vgexport back into the mix. In this case, I export the VG from P10 and then after an import on P11 I was able to run e2fsck and mount successfully.

Also, this (unknown VG) message is somewhat common in pvscan, if a vgexport is not performed:

Quoted 1 line

pvscan -- inactive PV "/dev/sdk" is associated to unknown VG "tux-ao" (run

vgscan)

Here is another clear example of some unexpected behavior (at least to me)...
1. status when $vg on P11 successful - notice the status of ACTIVE on P11 and EXPORTED on P10

[P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- ACTIVE PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free]

[P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is in EXPORTED VG "tux-ao" [27.09 GB / 9.09 GB free]

Then...
2. status after $vg is deactivated on P11 and activated on P10 (no vgexport run on P11 before activation on P10) - Notice the "unknown VG" message on P10!!!

[P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free]

[P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is associated to unknown VG "tux-ao" (run vgscan)

Then...
3. Then, when I tried an activation on P10 I get no joy at all (ie. cannot perform operations on the logical volume), despite the fact that it exists:
[P10]$ ls -l /dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 15:04 /dev/tux-ao/app

/sbin/e2fsck: No such device or address while trying to open /dev/tux-ao/app Possibly non-existent or swap device?

Is this simply flakey behavior with LVM 1.0.8 ?

We have several VGs on the machine, and sometimes we need to move one or two at a time. LVM (v1) seems to have trouble here, at least without vgexport. Anyone know what might be happening behind the scenes to cause this behavior? It's looking to me like I really do need vgexport to make things work the way we want.

Thanks again for helping me to clarify this situation. Dave

Quoted 76 lines · --- Patrick Caulfield
Quoted 3 lines · Dave

Hello,

I believe I might be misunderstanding whether vgimport and vgexport are needed

in my

Quoted 7 lines

particular situation. It would be great to get some feedback for clarification.

THE SETUP:

LVM1 (1.0.8-14) on two RedHat AS3 systems (kernel: 2.4.21-47.ELsmp). I think the same concept applies for LVM2 as well though. Machine A is primary and Machine B is backup in a two-node Linux heartbeat

cluster.

Quoted 1 line

Both machines are connected to the same SAN via fiber, and see the same disks,

where

Quoted 6 lines

the volume groups reside.

THE STRATEGY:

The idea is for A to have the LVM file systems mounted, and when a failure is detected, have the LVM file systems "moved" to (or seen by) system B. The way

this

Quoted 1 line

is currently accomplished is for A to do the following upon detection of a

failure:

Quoted 7 lines

+ unmount file systems
+ deactivate (vgchange -an $vg)
+ export (vgexport $vg)

then, on system B:

Quoted 7 lines

+ import & activate (vgimport $vg $disks)
+ mount file systems

THE ISSUE:

The export works as expected on A, but upon import on B, a return code of 4 is returned meaning "volume group already exists". The mounting works properly,

but

Quoted 9 lines

all the disks are shown like this:

"inactive PV /dev/sdx is in EXPORTED VG $vg"

when inspected with pvscan.

Does a vgimport and/or vgexport mark the disks themselves, or simply update the system on which the commands are run??? I suppose that is essentially the heart

of

Quoted 1 line

this issue.

Yes, vgimport/export marks the disks in the volume group. It's really for moving disks between systems where the target system might
have a volume group with the same name as the one to be imported.

Quoted 1 line

I'm starting to believe that for our strategy the vgexport and vgimport commands

are

Quoted 2 lines

not necessary, and are actually causing the problem. (The HOWTO mentions these commands are used to move disks between systems, but perhaps that is meant to

refer

Quoted 3 lines

to disks that are only physically moved?)

Instead, the following strategy might be correct in case system A fails (Note:

no

Quoted 11 lines

vgimport or vgexport commands):

+ unmount fs
+ vgchange -an $vg

then, on system B:

+ vgchange -ay $vg
+ mount fs

Quoted 17 lines
Quoted 2 lines

IS THIS CORRECT???

Yes. IF YOU'RE VERY CAREFUL!
vgimport/vgexport are not the tools you want for this job.

Signature · 7 lines
patrick

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

____________________________________________________________________________________
Cheap talk?
Check out Yahoo! Messenger's low PC-to-Phone call rates.
http://voice.yahoo.com

#3 lore raw
Quoted 73 lines · Dave

Hi Patrick, et al.

Thanks for your comments!
I've been doing some testing on this topic, and encountering a bit of strange behavior which seems to confirm the murkiness with switching VGs between hosts. Please bear with my lengthy description, as I'm trying to be as clear as possible. I really need to work this out.

THE TEST:

Hosts are P10 (primary node) and P11 (backup node)
VG is activated on P10 and fs is mounted. To test a switch to P11, I deactivate the VG on P10, but DO NOT run vgexport (following your suggestion). I then, run vgimport on P11 (but P11 reports it already knows about the VG - that's fine), and then run an activate. However, when I try to run an e2fsck on the fs, I get the following error:

/sbin/e2fsck: No such device or address while trying to open /dev/tux-ao/app Possibly non-existent or swap device?

However, the device does exist, and looks identical to the one on P10:
[P11]$ ls -l /dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 14:46 /dev/tux-ao/app

I was able to fix the problem by putting vgexport back into the mix. In this case, I export the VG from P10 and then after an import on P11 I was able to run e2fsck and mount successfully.

Also, this (unknown VG) message is somewhat common in pvscan, if a vgexport is not performed:

Quoted 1 line

pvscan -- inactive PV "/dev/sdk" is associated to unknown VG "tux-ao" (run

vgscan)

Here is another clear example of some unexpected behavior (at least to me)...
1. status when $vg on P11 successful - notice the status of ACTIVE on P11 and EXPORTED on P10

[P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- ACTIVE PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free]

[P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is in EXPORTED VG "tux-ao" [27.09 GB / 9.09 GB free]

Then...
2. status after $vg is deactivated on P11 and activated on P10 (no vgexport run on P11 before activation on P10) - Notice the "unknown VG" message on P10!!!

[P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free]

[P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is associated to unknown VG "tux-ao" (run vgscan)

Then...
3. Then, when I tried an activation on P10 I get no joy at all (ie. cannot perform operations on the logical volume), despite the fact that it exists:
[P10]$ ls -l /dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 15:04 /dev/tux-ao/app

/sbin/e2fsck: No such device or address while trying to open /dev/tux-ao/app Possibly non-existent or swap device?

Quoted 8 lines

Is this simply flakey behavior with LVM 1.0.8 ?

We have several VGs on the machine, and sometimes we need to move one or two at a time. LVM (v1) seems to have trouble here, at least without vgexport. Anyone know what might be happening behind the scenes to cause this behavior? It's looking to me like I really do need vgexport to make things work the way we want.

Thanks again for helping me to clarify this situation.

It's rather worrying that the two nodes seem to be reading different data from the same disks. I can't remember off-hand whether lvm1 does direct-io when it updates metadata, possibly not. In which case you might have to upgrade to lvm2 (which does).

lvm1 is /not/ a clustering tool ;-)

Signature · 1 line
patrick
author
#4 lore raw

Hi Patrick,

Thanks for clarifying the status of LVM1 (ie. *not* a clustering tool). That's somewhat surprising since it does include an LVM script in the resource.d directory.
In any case, we're not in a huge rush to upgrade to LVM2 unless it really is
necessary, as these are RH AS3 boxes running the 2.4.21-47.ELsmp kernel, and the docs I've read suggest LVM2 really needs/wants to run under a 2.6.x kernel.

If direct-io to metadata could be set as an option under LVM1, assuming that isn't the default, that would be helpful. But, it sounds like no one really wants to support LVM1 anymore?!

Assuming we want to go forward with an upgrade to LVM2 on the current systems, how involved is that procedure? Do we risk losing all our data?

Thanks,
Dave

Quoted 95 lines · --- Patrick Caulfield
Quoted 6 lines · Dave

Hi Patrick, et al.

Thanks for your comments!
I've been doing some testing on this topic, and encountering a bit of strange behavior which seems to confirm the murkiness with switching VGs between hosts. Please bear with my lengthy description, as I'm trying to be as clear as

possible.

Quoted 6 lines

I really need to work this out.

THE TEST:

Hosts are P10 (primary node) and P11 (backup node)
VG is activated on P10 and fs is mounted. To test a switch to P11, I deactivate

the

Quoted 2 lines

VG on P10, but DO NOT run vgexport (following your suggestion). I then, run vgimport on P11 (but P11 reports it already knows about the VG - that's fine),

and

Quoted 11 lines

then run an activate. However, when I try to run an e2fsck on the fs, I get the following error:

/sbin/e2fsck: No such device or address while trying to open /dev/tux-ao/app Possibly non-existent or swap device?

However, the device does exist, and looks identical to the one on P10:
[P11]$ ls -l /dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 14:46 /dev/tux-ao/app

I was able to fix the problem by putting vgexport back into the mix. In this

case,

Quoted 1 line

I export the VG from P10 and then after an import on P11 I was able to run

e2fsck

Quoted 3 lines

and mount successfully.

Also, this (unknown VG) message is somewhat common in pvscan, if a vgexport is

not

Quoted 15 lines

performed:

Quoted 1 line

pvscan -- inactive PV "/dev/sdk" is associated to unknown VG "tux-ao" (run

vgscan)

Here is another clear example of some unexpected behavior (at least to me)...
1. status when $vg on P11 successful - notice the status of ACTIVE on P11 and EXPORTED on P10

[P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- ACTIVE PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free]

Quoted 8 lines

[P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is in EXPORTED VG "tux-ao" [27.09 GB / 9.09 GB free]

Then...
2. status after $vg is deactivated on P11 and activated on P10 (no vgexport run

on

Quoted 11 lines

P11 before activation on P10) - Notice the "unknown VG" message on P10!!!

[P11]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" of VG "tux-ao" [27.09 GB / 9.09 GB free]

[P10]$ sudo pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- inactive PV "/dev/sdk" is associated to unknown VG "tux-ao" (run

vgscan)

Quoted 3 lines

Then...
3. Then, when I tried an activation on P10 I get no joy at all (ie. cannot

perform

Quoted 10 lines

operations on the logical volume), despite the fact that it exists:
[P10]$ ls -l /dev/tux-ao/app
brw-rw---- 1 root disk 58, 14 Dec 6 15:04 /dev/tux-ao/app

/sbin/e2fsck: No such device or address while trying to open /dev/tux-ao/app Possibly non-existent or swap device?

Is this simply flakey behavior with LVM 1.0.8 ?

We have several VGs on the machine, and sometimes we need to move one or two at

a

Quoted 1 line

time. LVM (v1) seems to have trouble here, at least without vgexport. Anyone

know

Quoted 1 line

what might be happening behind the scenes to cause this behavior? It's looking

to

Quoted 3 lines

me like I really do need vgexport to make things work the way we want.

Thanks again for helping me to clarify this situation.

Quoted 18 lines

It's rather worrying that the two nodes seem to be reading different data from the same disks.
I can't remember off-hand whether lvm1 does direct-io when it updates metadata, possibly not.
In which case you might have to upgrade to lvm2 (which does).

lvm1 is /not/ a clustering tool ;-)

Signature · 7 lines
patrick

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

____________________________________________________________________________________
Yahoo! Music Unlimited
Access over 1 million songs.
http://music.yahoo.com/unlimited

#5 lore raw
Quoted 7 lines · Dave

Hi Patrick,

Thanks for clarifying the status of LVM1 (ie. *not* a clustering tool). That's somewhat surprising since it does include an LVM script in the resource.d directory.
In any case, we're not in a huge rush to upgrade to LVM2 unless it really is
necessary, as these are RH AS3 boxes running the 2.4.21-47.ELsmp kernel, and the docs I've read suggest LVM2 really needs/wants to run under a 2.6.x kernel.

Yes. there are device-mapper patches for 2.4 but it's really recommended to run 2.6.

Quoted 3 lines

If direct-io to metadata could be set as an option under LVM1, assuming that isn't the default, that would be helpful. But, it sounds like no one really wants to support LVM1 anymore?!

That's true. lvm1 is in maintenance mode for RHEL3.

Quoted 2 lines

Assuming we want to go forward with an upgrade to LVM2 on the current systems, how involved is that procedure? Do we risk losing all our data?

No. lvm2 can read and write lvm1 metadata. you can upgrade it to lvm2 format metadata with a single command (vgconvert)

Signature · 1 line
patrick
#6 lore raw
Quoted 1 line · Patrick Caulfield

Yes. there are device-mapper patches for 2.4 but it's really recommended to run 2.6.

The 2.4 patches have not been updated for a long time so they include bugs that got fixed in 2.6 and they are missing features.

Anyone fancy updating them?

Alasdair

Signature · 1 line
[email protected]