vgchange doesn't enable found VGs
Strange:
+ /usr/sbin/vgscan --ignorelockingfailure Reading all physical volumes. This may take a while... Found volume group "foobar" using metadata type lvm2
+ /usr/sbin/vgchange -a y --ignorelockingfailure No volume groups found
It seems I have to tell vgchange to enable the foobar VG explicitly.
Why is this?
b.
Quoted 11 lines · Brian J. Murrell
Strange:
+
/usr/sbin/vgscan--ignorelockingfailure Reading all physical volumes. This may take a while... Found volume group "foobar" using metadata type lvm2 +/usr/sbin/vgchange-a y --ignorelockingfailure No volume groups foundIt seems I have to tell vgchange to enable the foobar VG explicitly.
Why is this?
It's difficult to say from the output above. Try running the tools with "-vvv" and looking at the detailed output. They'll produce a lot of output when run this way - you may want to use something like pastebin & send a link to the list rather than posting the entire output.
Why do you need the --ignorelockingfailure?
Bryn.
Quoted 4 lines · Bryn M. Reeves
It's difficult to say from the output above. Try running the tools with "-vvv" and looking at the detailed output. They'll produce a lot of output when run this way
Not so much in fact.
Quoted 1 line
Why do you need the --ignorelockingfailure?
Because this is being done during system boot.
Here's "vgchange -v -v -v -a n" output:
Processing: vgchange -v -v -v -a n
O_DIRECT will be used
Setting global/locking_type to 1
File-based locking selected.
Setting global/locking_dir to /var/lock/lvm
Finding all volume groups
/dev/ramdisk: Not using O_DIRECT
Opened /dev/ramdisk RO
/dev/ramdisk: block size is 4096 bytes
/dev/ramdisk: No label detected
Closed /dev/ramdisk
/dev/ram: Not using O_DIRECT
Opened /dev/ram RO
/dev/ram: block size is 4096 bytes
/dev/ram: No label detected
Closed /dev/ram
/dev/ram2: Not using O_DIRECT
Opened /dev/ram2 RO
/dev/ram2: block size is 4096 bytes
/dev/ram2: No label detected
Closed /dev/ram2
...
/dev/ram14: Not using O_DIRECT
Opened /dev/ram14 RO
/dev/ram14: block size is 4096 bytes
/dev/ram14: No label detected
Closed /dev/ram14
/dev/ram15: Not using O_DIRECT
Opened /dev/ram15 RO
/dev/ram15: block size is 4096 bytes
/dev/ram15: No label detected
Closed /dev/ram15
No volume groups found
The eclipses represents ram3, ram4, ram5, to ram 13.
When I specify the volume group, I see this:
Processing: vgchange -v -v -v -a n lustre
O_DIRECT will be used
Setting global/locking_type to 1 File-based locking selected.
Setting global/locking_dir to /var/lock/lvm
Using volume group(s) on command line
Locking /var/lock/lvm/V_lustre RB
Finding volume group "lustre"
/dev/ramdisk: Not using O_DIRECT Opened /dev/ramdisk RO /dev/ramdisk: block size is 4096 bytes
/dev/ramdisk: No label detected
Closed /dev/ramdisk
/dev/ram: Not using O_DIRECT Opened /dev/ram RO
/dev/ram: block size is 4096 bytes
/dev/ram: No label detected
Closed /dev/ram
/dev/ram2: Not using O_DIRECT Opened /dev/ram2 RO
/dev/ram2: block size is 4096 bytes
/dev/ram2: No label detected
Closed /dev/ram2
...
Wiping cache of LVM-capable devices
/dev/console: Not a block device /dev/core: Not a block device /dev/evtchn: Not a block device /dev/fd: Symbolic link to directory
...
and it considers a whole boatload more devices, including /dev/sda, where my volume groups is. Now the one thing that just occurred to me as relevant is that my root filesystem (/) is read-only.
b.
Quoted 3 lines · Brian J. Murrell
/dev/ram14: Not usingO_DIRECTOpened/dev/ram14RO/dev/ram14:
block size is 4096 bytes/dev/ram14: No label detected Closed/dev/ram14/dev/ram15: Not usingO_DIRECTOpened/dev/ram15RO/dev/ram15: block size is 4096 bytes/dev/ram15: No label detected Closed/dev/ram15No volume groups found
So it's not scanning /dev/sda at all (only ramdisks?). Do you have any filter settings in lvm.conf and are you sure that the SD device nodes will have been created at this point in the boot process?
You don't mention the distribution you're using but most 2.6 based setups will use udev to manage driver loading and device node creation during boot. It's possible for these to get out of sync in some cases.
Is there any way you can grab a listing of /dev at the moment the vgchange runs during boot?
Quoted 6 lines
When I specify the volume group, I see this:
Processing: vgchange -v -v -v -a n lustre
O_DIRECTwill be used Setting global/locking_type to 1 File-based locking selected. Setting global/locking_dir to/var/lock/lvmUsing volume group(s) on command line Locking/var/lock/lvm/V_lustreRB Finding volume
group "lustre" ...
Did it scan /dev/sda this time?
Quoted 7 lines
Wiping cache of LVM-capable devices
/dev/console: Not a block device/dev/core: Not a block device/dev/evtchn: Not a block device/dev/fd: Symbolic link to directory ...and it considers a whole boatload more devices, including
/dev/sda,
where my volume groups is. Now the one thing that just occurred
to me as relevant is that my root filesystem (/) is read-only.
Shouldn't matter.
Just to be clear, are you only seeing this problem during boot or are you able to reproduce once the system has completed booting (i.e. vgchange -ay fails but vgchange -ay <vg name> succeeds)?
Your earlier messages didn't make that clear.
Regards,
Bryn.
Quoted 2 lines · Bryn M. Reeves
So it's not scanning
/dev/sdaat all (only ramdisks?).
So it seems, yes.
Quoted 2 lines
Do you have any
filter settings in lvm.conf
devices {
dir = "/dev"
scan = [ "/dev" ]
preferred_names = [ ]
filter = [ "a/.*/" ]
cache_dir = "/etc/lvm/cache"
cache_file_prefix = ""
write_cache_state = 1
sysfs_scan = 1
md_component_detection = 1
ignore_suspended_devices = 0
}
Quoted 2 lines
and are you sure that the SD device nodes
will have been created at this point in the boot process?
Well, the data from the previous message was done on a fully booted machine, complete with /dev/sda being present for many, many minutes if not hours.
Quoted 1 line
You don't mention the distribution you're using
RHEL5.2.
Quoted 3 lines
but most 2.6 based
setups will use udev to manage driver loading and device node creation during boot. It's possible for these to get out of sync in some cases.
Yes, but again, the data I am giving you is being taken from a completely up and running system.
Quoted 2 lines
Is there any way you can grab a listing of /dev at the moment the vgchange runs during boot?
Not relevant per above.
Quoted 1 line
Did it scan
/dev/sdathis time?
Yes.
Quoted 3 lines
Just to be clear, are you only seeing this problem during boot or are you able to reproduce once the system has completed booting (i.e. vgchange -ay fails but vgchange -ay <vg name> succeeds)?
As per the above, I can reproduce on a completed booted system.
b.
Quoted 1 line · Brian J. Murrell
cache_dir= "/etc/lvm/cache"
The cache_dir must be *writeable*.
If you're attempting to use read-only root, then move it somewhere else (is /var writeable? or make a ramfs) or delete that directory *and* change
Quoted 1 line
write_cache_state= 1
to 0.
(This will slow commands down.)
Or if all devices on your system are the same on every boot, place the correct version of that file into your read only image. This won't work if devices can move about unless you include all the possible locations in there.
Quoted 3 lines
Quoted 2 lines
Is there any way you can grab a listing of /dev at the moment the vgchange runs during boot?
Not relevant per above.
-vvvv output provides this information already.
Alasdair
Signature · 1 line
[email protected]
Quoted 4 lines · Alasdair G Kergon
Quoted 1 line · Brian J. Murrell
cache_dir= "/etc/lvm/cache"The
cache_dirmust be *writeable*.
And persistent or just writable?
Interestingly enough, RHEL5.2's /etc/rwtab (the files and directories that need to be made "magically" writable -- through bind mounting to either persistent storage or tmpfs) has:
files /etc/lvm/.cache
That's starting to look like a type methinks.
Quoted 3 lines
If you're attempting to use read-only root, then move it somewhere else (is /var writeable? or make a ramfs) or delete that directory *and* change
Ahhh. "a ramfs". So persistence is not a requirement it seems.
b.
Quoted 8 lines · Brian J. Murrell
Quoted 4 lines · Alasdair G Kergon
Quoted 1 line · Brian J. Murrell
cache_dir= "/etc/lvm/cache"The
cache_dirmust be *writeable*.And persistent or just writable?
Quoted 7 lines
Interestingly enough, RHEL5.2's
/etc/rwtab(the files and directories that need to be made "magically" writable -- through bind mounting to either persistent storage or tmpfs) has:files
/etc/lvm/.cacheThat's starting to look like a type methinks.
But this is moot as the /etc/rwtab bind mounts are not configured until after lvm in the rc.sysinit and crappity crap crap: there is nothing mounted at that point in the boot that is writable.
But changing (only) "write_cache_state = 0" in the lvm.conf made no difference in vgchange's inability to find the volume group.
b.
You wrote:
my root filesystem (/) is read-only.
That's what I was reacting to.
With a read-only root, if /etc/lvm/cache/ is being used and is wrong it will be unable to correct itself.
You must either make the cache file writeable or delete it.
vgchange -ay
and
vgchange -ay XXX
behave differently.
The first activates all VGs it sees, respecting the cache file if it exists.
But with the second, if after doing that (and unless trustcache is set) it can't find XXX, it considers the cache file might be wrong and tries again without it, updating it afterwards iff write_cache_file is set.
Whether or not to preserve that file across boot depends on your system, but typically vgscan will be run each time to regenerate it.
Alasdair
Signature · 1 line
[email protected]