Skip to contentSkip to conversation

Q: active/inactive/imported/exported group ?

linux-lvm Gerhard Fuernkranz 4 participants 8 messages Nov 22, 1999 – Nov 23, 1999

lore
Share & tools
lore
#0 lore raw

Recently I've made my first steps with LVM and I think I really like it. I've just a few questions:

I'm wondering, what's the actual difference betweent an inactive, imported and an exported group?

My impression was, that exported groups are ignored by "vgscan" and therefore cannot be activated later. Is this merely the only difference?

Furthermore I tried to move an inactive but *imported* group to another host:

on host A:
vgchange -an <group>

move disks to host B
on host B:
vgscan
vgchange -ay <group>

... and it seemed to work.

Is there anything wrong, if I transport disks
in this way? Can there be any bad side effects, if I do *not* export on host A and then import on host B, but only deactivate on host A and then scan and activate the group on host B?

Is there a reason why I have to specify phys. disks for the "vgimport" command? Would'nt it also be possible to scan all disks for the group that is to be imported?

And one last question:
When does LVM assign minor device numbers to a volume? Will the minor number of a volume remain constant as long as the volume exists in the group or does LVM assign new minor numbers each time a group or volume gets activated?

Thank you very much in advance,
Gerhard

#1 lore raw
Quoted 7 lines

Recently I've made my first steps with LVM and I think I really like it. I've just a few questions:

I'm wondering, what's the actual difference betweent an inactive, imported and an exported group?

Inactive - LVM driver doesn't have table information about a VG
("vgchange -ay" activates VG(s))
Imported - Metadata on PV(s) is set up for activation
("vgscan" can find these VG(s) and can build
/etc/lvmtab + /etc/lvmtab.d/*)
Exported - Metadata on PV(s) is set up to move PV(s) to a different system

Quoted 4 lines

My impression was, that exported groups are ignored by "vgscan" and therefore cannot be activated later. Is this merely the only difference?

Yes.

Quoted 7 lines

Furthermore I tried to move an inactive but *imported* group to another host:

on host A:
vgchange -an <group>

Quoted 6 lines

move disks to host B
on host B:
vgscan
vgchange -ay <group>

... and it seemed to work.

Yes it does.
But only in case that there's no VG with the same name on this system before. Then you will not be able to scan and activate any of these VGs because the VG name is not unique.

Quoted 3 lines

Is there anything wrong, if I transport disks
in this way?

If you don't have a VG with the same name on the other system, it's o.k. for now.
One of my TODO items for the future still is to have UUIDs (Uniform Unique Identifiers) for VGs and for system ownership of a VG.

Until this will be implemented, you don't run into trouble. So far it's a good possiblity to multiply installations 8*)

Quoted 4 lines

Can there be any bad side effects,
if I do *not* export on host A and then import on host B, but only deactivate on host A and then scan and activate the group on host B?

See above (VG name collision).

Quoted 4 lines

Is there a reason why I have to specify phys. disks for the "vgimport" command? Would'nt it also be possible to scan all disks for the group that is to be imported?

Yes, that's possible.
But this is an enhancement which has to be implemented.

A simple sh+grep+sed workaround for today is to do a pvscan, figure out all PVs belonging to the exported VG, parse the PV name out of this information and set up a corresponding vgimport command line.

Quoted 3 lines

And one last question:
When does LVM assign minor device numbers to a volume?

At lvcreate time _and_ at vgscan time.

Quoted 3 lines

Will the minor number of a volume remain constant as long as the volume exists in the group or does LVM assign new minor numbers each time a group or volume gets activated?

See my previous statement.

Quoted 5 lines

Thank you very much in advance,
Gerhard

You're welcome.

Regards,
Heinz

--

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Systemmanagement CS-TS T-Nova
Entwicklungszentrum Darmstadt
Heinz Mauelshagen Otto-Roehm-Strasse 71c
Senior Systems Engineer Postfach 10 05 41

                                                 64205 Darmstadt
[email protected]                      Germany
                                                 +49 6151 886-425
                                                          FAX-386

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw

Heinz,

thanks for the fast answer -
just a few more questions:

Heinz Mauelshagen wrote:
...

Quoted 4 lines
Quoted 2 lines

And one last question:
When does LVM assign minor device numbers to a volume?

At lvcreate time _and_ at vgscan time.

Assume I have a group g1 with volume v1 and a group g2 with volume v2. Now I turn off the machine, remove g1's physical disks and power on the machine again. Does this mean, that vgscan may assign a *different* minor number to v2 after the reboot?

Concernig my other question (imported/exported/active/inactive) probably I should explain, what I want to do:

I'd like to use LVM in a cluster with multihosted disks. Each host *can* access all disks, but I'll guarantee (howsoever), that only one host *does* access a disk/volume/volume-group at a time.

My idea was, that all hosts see all groups on the shared disks, but only *one* host activates a volume group at a time, while all other hosts don't activate the same group (but probably different ones). If I want to (logically) switch a volume group to a different host, then I deactivate it on the source host, re-"vgscan" on the target host and then activate the group on the target host.

Do you think, this strategy is ok?
I assume, that LVM does not write anything to a physical disk belonging to a deactivated group, correct?
Can I just re-"vgscan" a (deactivated) group at any time in order to see all changes a different host has made to the group while it was deactivated on the local host?

I did not want to export/import for two reasons:

- import does not search the disks for the group
- if a host with an activated group crashes, then it has no chance to export the group anyway. Nevertheless in this case I'd like to access the group from different host in a cluster.

For operation in a cluster I'd also desire, that
only one host can activate a group at time (-> i.e.
the host claims ownership and e.g. writes its host name to all disks of the group) in order that no other host can accidently activate the same group at the same time. But there should also be a "forced" activation method to override this security mechanism (-> in case that the owner crashes). IMHO something like this is NYI?

Thanks,
Gerhard

#3 lore raw
Quoted 11 lines

Heinz,

thanks for the fast answer -
just a few more questions:

Heinz Mauelshagen wrote:
...

Quoted 3 lines
Quoted 2 lines

And one last question:
When does LVM assign minor device numbers to a volume?

Quoted 7 lines
Quoted 1 line

At lvcreate time _and_ at vgscan time.

Assume I have a group g1 with volume v1 and a group g2 with volume v2. Now I turn off the machine, remove g1's physical disks and power on the machine again. Does this mean, that vgscan may assign a *different* minor number to v2 after the reboot?

Yes.
But that's only a local resource for sure.

Quoted 18 lines

Concernig my other question (imported/exported/active/inactive) probably I should explain, what I want to do:

I'd like to use LVM in a cluster with multihosted disks. Each host *can* access all disks, but I'll guarantee (howsoever), that only one host *does* access a disk/volume/volume-group at a time.

My idea was, that all hosts see all groups on the shared disks, but only *one* host activates a volume group at a time, while all other hosts don't activate the same group (but probably different ones). If I want to (logically) switch a volume group to a different host, then I deactivate it on the source host, re-"vgscan" on the target host and then activate the group on the target host.

Quoted 1 line

Do you think, this strategy is ok?

Yes.

Quoted 2 lines

I assume, that LVM does not write anything to a physical disk belonging to a deactivated group, correct?

Not till now.

Quoted 3 lines

Can I just re-"vgscan" a (deactivated) group at any time in order to see all changes a different host has made to the group while it was deactivated on the local host?

Yes.

The read/write routines in the LVM library flush the buffer cache contents of a physical volume to be sure to read actual data from disk(s) and not to rely on cached data.
If like in your example one host changes a VG, the other one should read actual VG data afterwards while doing a vgscan.

Quoted 4 lines

I did not want to export/import for two reasons:

- import does not search the disks for the group

Correct.
See my workaround proposal in my previous mail.

If you only have one exported VG you could import it for eg. with:

pvscan|grep EXPORTED|sed 's/^.*PV "//;s/".*$//'|xargs vgimport VGNAME

Quoted 11 lines

- if a host with an activated group crashes, then it has no chance to export the group anyway. Nevertheless in this case I'd like to access the group from different host in a cluster.

For operation in a cluster I'd also desire, that only one host can activate a group at time (-> i.e. the host claims ownership and e.g. writes its host name to all disks of the group)
in order that no other host
can accidently activate the same group at the same time.

I want to achive this by keeping a UUID of the VG owning sytem in the VGDA.

IMO we would need a cluster manager to take care of mastership of shared resources.

Quoted 2 lines

But there should also be a "forced" activation method to override this security mechanism (-> in case that the owner crashes).

But in this case a crashing host is not able to release the VG anyway. So there must be a --force option to gain ownership of such a VG _and_ the sure knowlegde that the crashed system is down _and_ stays down. If not so there could be tricky races where the original 'owner' system of our VG would try to regain ownership.

Quoted 1 line

IMHO something like this is NYI?

Not today 8*(

Quoted 5 lines

Thanks,
Gerhard

Cheers,
Heinz

Signature · 11 lines
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Systemmanagement CS-TS                           T-Nova
                                                 Entwicklungszentrum Darmstadt
Heinz Mauelshagen                                Otto-Roehm-Strasse 71c
Senior Systems Engineer                          Postfach 10 05 41
                                                 64205 Darmstadt
[email protected]                      Germany
                                                 +49 6151 886-425
                                                          FAX-386
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
#4 lore raw
Quoted 8 lines · [email protected] wrote:
Quoted 5 lines

Assume I have a group g1 with volume v1 and a group g2 with volume v2. Now I turn off the machine, remove g1's physical disks and power on the machine again. Does this mean, that vgscan may assign a *different* minor number to v2 after the reboot?

Yes.
But that's only a local resource for sure.

That's what I was afraid of.
The problem is the following:
The NFS server identifies files by a file handle, which also contains the major/minor number of the disk, where the exported filesystem resides. Assume there is a NFS server, which exports a filesystem residing on a LVM volume. There also exist NFS clients, which have hard-mounted this filesystem.
If the NFS server crashes and reboots, then the clients should hang on NFS calls while the server is down, but should resume normal operation once the server is back (w/o umounting and re-mounting the FS!). But this only works, if the volumes get the same minor device number after the reboot in order that the old filehandle still references the same file on the same device.

So I think there's also the reqirement for a "sticky" minor device number for volumes (or at least for a specific subset of volumes). VxVM also has recognized this and provides a method to assign minor device numbers to volumes.

Do you have an idea, how I could integrate such a
feature into LVM? My 1st idea was the following:

- I have a table e.g. /etc/lvm_minors, which contains {volume_name, minor_number} pairs for all sticky volumes

- Change vgscan to do the following:

     ...
     for (all volumes of the group) {
        ...
        if (entry for volume exists in /etc/lvm_minors) {
            if (minor number from lvm_minors already in use
                for a different volume)
            {
                fail, print an error message and skip this volume
            }
            else {
               assign the minor number from /etc/lvm_minors
               to the volume
            }
        }
        else {
            assign a free minor number to this volume
            similar as it currently does
        }
        ...
     }

Any comments?

Quoted 1 line

I want to achive this by keeping a UUID of the VG owning sytem in the VGDA.

The question is: What is the "owning" system? Is it the system which has imported the group? Or the system which has activated the group at last?

Quoted 2 lines

IMO we would need a cluster manager to take care of mastership of shared resources.

I currently don't want to use "real" shared resources, which are configured at more than one host at the same time.

I'll guarantee, that only one host at a time will activate a specific group. If a host crashes, I'll guarantee, that the crashed host is actually down and stays down (of course it may reboot, but it won't re-activate the volume after the reboot automatically).

...

Quoted 5 lines

But in this case a crashing host is not able to release the VG anyway. So there must be a --force option to gain ownership of such a VG _and_ the sure knowlegde that the crashed system is down _and_ stays down. If not so there could be tricky races where the original 'owner' system of our VG would try to regain ownership.

Currently I won't care very much about such race conditions as I'll only bring the group online on one host at a time automatically. I'd rather see the ownership only as additional security feature in order that the sysadmin cannot accidently activate the same group at the same time on a different host.

Gerhard

#5 lore raw
Quoted 4 lines · [email protected] wrote:

So I think there's also the reqirement for a "sticky" minor device number for volumes (or at least for a specific subset of volumes). VxVM also has recognized this and provides a method to assign minor device numbers to volumes.

Your aproach with "sticky minors" might be a good idea until we finally get additional options due to bigger major/minors.

Quoted 9 lines

Do you have an idea, how I could integrate such a feature into LVM? My 1st idea was the following:

- I have a table e.g. /etc/lvm_minors, which contains {volume_name, minor_number} pairs for all sticky volumes

- Change vgscan to do the following:

     ...

This should be rather easy as it's almost all done in userspace. Most functionality is in the liblvm which probably can be extended without interfering with the kernel proper.

Quoted 5 lines
Quoted 1 line

I want to achive this by keeping a UUID of the VG owning sytem in the VGDA.

The question is: What is the "owning" system? Is it the system which has imported the group? Or the system which has activated the group at last?

IMO this is close. If you imported the VG or activated it recently, another system trying to access the VG better get's in contact with you. The UUID could serve for two purposes: Avoid the possible issues which could be caused by different SCSI configurations and it could also serve as a lock (more of a hint).

The VG could be marked as shared (accessible from different systems) and LVM might call some external utility to let it resolve the situation. If a HA framework exists, LVM could make use of it or the admin could implement a simple rule in a shell script.

Michael

--

Michael Marxmeier           Marxmeier Software AG
E-Mail: [email protected]      Besenbruchstrasse 9
Phone : +49 202 2431440     42285 Wuppertal, Germany
Fax   : +49 202 2431420     http://www.msede.com/
#6 lore raw
In reply to Gerhard Fuernkranz
Quoted 27 lines
Quoted 8 lines · [email protected] wrote:
Quoted 5 lines

Assume I have a group g1 with volume v1 and a group g2 with volume v2. Now I turn off the machine, remove g1's physical disks and power on the machine again. Does this mean, that vgscan may assign a *different* minor number to v2 after the reboot?

Yes.
But that's only a local resource for sure.

That's what I was afraid of.
The problem is the following:
The NFS server identifies files by a file handle, which also contains the major/minor number of the disk, where the exported filesystem resides. Assume there is a NFS server, which exports a filesystem residing on a LVM volume. There also exist NFS clients, which have hard-mounted this filesystem.
If the NFS server crashes and reboots, then the clients should hang on NFS calls while the server is down, but should resume normal operation once the server is back (w/o umounting and re-mounting the FS!). But this only works, if the volumes get the same minor device number after the reboot in order that the old filehandle still references the same file on the same device.

Quoted 4 lines

So I think there's also the reqirement for a "sticky" minor device number for volumes (or at least for a specific subset of volumes). VxVM also has recognized this and provides a method to assign minor device numbers to volumes.

If you don't change your IO configuration vgscan will not produce different LV minors anyway.

Quoted 30 lines

Do you have an idea, how I could integrate such a
feature into LVM? My 1st idea was the following:

- I have a table e.g. /etc/lvm_minors, which contains {volume_name, minor_number} pairs for all sticky volumes

- Change vgscan to do the following:

     ...
     for (all volumes of the group) {
        ...
        if (entry for volume exists in /etc/lvm_minors) {
            if (minor number from lvm_minors already in use
                for a different volume)
            {
                fail, print an error message and skip this volume
            }
            else {
               assign the minor number from /etc/lvm_minors
               to the volume
            }
        }
        else {
            assign a free minor number to this volume
            similar as it currently does
        }
        ...
     }
Quoted 1 line

Any comments?

This is an option.

I'ld rather prefer to let vgscan deal with 'sticky' minors based on existing lvmtab entries and let it only use free minors for new VGs.

Quoted 11 lines
Quoted 3 lines

I want to achive this by keeping a UUID of the VG owning sytem in the VGDA.

The question is: What is the "owning" system? Is it the system which has imported the group? Or the system which has activated the group at last?

Quoted 2 lines

IMO we would need a cluster manager to take care of mastership of shared resources.

Quoted 9 lines

I currently don't want to use "real" shared resources, which are configured at more than one host at the same time.

I'll guarantee, that only one host at a time will activate a specific group. If a host crashes, I'll guarantee, that the crashed host is actually down and stays down (of course it may reboot, but it won't re-activate the volume after the reboot automatically).

If you garanty that you must already have a cluster manager... 8*)

Quoted 7 lines

...

Quoted 5 lines

But in this case a crashing host is not able to release the VG anyway. So there must be a --force option to gain ownership of such a VG _and_ the sure knowlegde that the crashed system is down _and_ stays down. If not so there could be tricky races where the original 'owner' system of our VG would try to regain ownership.

Quoted 6 lines

Currently I won't care very much about such race conditions as I'll only bring the group online on one host at a time automatically. I'd rather see the ownership only as additional security feature in order that the sysadmin cannot accidently activate the same group at the same time on a different host.

O.k., UUIDs are future anway.

Heinz

Signature · 11 lines
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Systemmanagement CS-TS                           T-Nova
                                                 Entwicklungszentrum Darmstadt
Heinz Mauelshagen                                Otto-Roehm-Strasse 71c
Senior Systems Engineer                          Postfach 10 05 41
                                                 64205 Darmstadt
[email protected]                      Germany
                                                 +49 6151 886-425
                                                          FAX-386
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
#7 lore raw
Quoted 2 lines · [email protected] wrote:

If you don't change your IO configuration vgscan will not produce different LV minors anyway.

see below.

Quoted 6 lines
Quoted 1 line

Any comments?

This is an option.

I'ld rather prefer to let vgscan deal with 'sticky' minors based on existing lvmtab entries and let it only use free minors for new VGs.

The problem is again the cluster. Additionally to shared volumes I also may have local volumes on each host. Each host will usually see the same set of shared volumes, but the sets of each host's local volumes may differ. But I also want, that a shared volume gets the same minor number on every host in the cluster in order that NFS server failover from one host to another one works.

So if I add new (shared) volumes on host A, then it is not guaranteed, that the new minor number host A assigns to the volume is not already in use on host B. With my sticky volume table I'd use the following procedure to create a new shared volume:

1. find a free minor number (manually)
2. enter it to the table on every host in the cluster (manually)
3. then craete the volume: vgcreate on 1st host / vgscan on all other hosts - vgcreate/vgscan will assign the number from the sticky table.

Of course my sticky table can also be integrated in lvmtab.

A different approach could be:

- I've seen in the pvdata output, that the volume descriptor already contains a major/minor number.

- So the desired minor number could reside in the volume descriptor on the disk together with a sticky flag for the volume.

- such a sticky voulume could e.g. be created with "lvcreate ... --sticky=<minor> ...",
(which only succeeds if the minor number is not already in use - either by a currently active volume or another sticky volume).

- vgscan will
1. go through the currently active volumes
2. through the sticky volumes and try to assign exactly the sticky minor number
3. all other volumes and assign a free minor number

Quoted 1 line

If you garanty that you must already have a cluster manager... 8*)

Yes.

Thanks,
Gerhard