F7 will not boot after running backup w/snapshot

I ran a backup today with a snapshot. Sometime after the backup completed but before I could unmount and lvremove the snapshot the system became hung. No choice but to cold boot. When the system came up it showed a kernel panic on the screen. Looking back up the screen I could see that it could not find a device with UUID blahblah for VolGroup00 (which is where the / drive is located) and then VolGroup00 not found. So ok, no problem, I just figure that the VolGroup00 is inconsistent and I can deal with this in rescue mode. Wrong. When I boot the F7 rescue from the dvd it says "You don't have any Linux partitions..." What!? I go into the shell but there is nothing under /mnt/sysimage. So how do I recover from this? How could a snapshot cause all this?

Gerry

In rescue, I tried running:
===============================
lvm vgchange -P -ay
Partial mode. Incomplete volume groups will be activated read-only. Couldn't find device with uuid 'blahblah'
Couldn't find device with uuid 'blahblah'
Couldn't find device with uuid 'blahblah'
Couldn't find device with uuid 'blahblah'
Device /dev/ioerror not found.
device-mapper reload ioctl failed: invalid argument device-mapper reload ioctl failed: invalid argument Couldn't find device with uuid 'blahblah'
Couldn't find device with uuid 'blahblah'
2 logical volume(s) in volume group "VolGroup00" now active.

lvm vgdisplay -v VolGroup00
Segmentation fault
===============================

In fact every 'lvm' command you try to run after that now gives a Segmentation fault.

What I want to do is to try:
lvm vgreduce VolGroup00 --removemissing

which I think might make the volume consistent again but I cannot even run the command without segfault. Any ideas on how to get rid of the segfaulting?

Gerry

So I got the vgreduce to work:

reboot into rescue and issue it as the very first command:
lvm vgreduce VolGroup00 --removemissing
lvm vgdisplay -v VolGroup00
<no error this time>
but I do see that my LogVol00 is missing (root disk) (this is what got corrupted) lvm lvchange -ay /dev/VolGroup00/LogVol01
<no errors, good>
vgdisplay now shows LogVol01 as available, yea!
so now what about the missing LogVol00?
what I want to try is replacing it at exactly the same extents with a new LogVol00 lvm lvcreate --extents 284 --name=/dev/VolGroup00/LogVol00 VolGroup00
<it succeeds>
vgdisplay now shows both log vols with exactly the same size as originally before the problem. vgchange -ay
<succeeds and activates VG and both LV)
mount -t ext3 /dev/mapper/VolGroup00-LogVol00 /mnt/sysimage
<fails: invalid argument>
so now I think that something needs to be done with uuid but I'm not familiar with how to do this. even if I have no access to data on the logvol, if I can mount it I can restore it. Anyone know how to make the new LogVol00 mount?

Gerry

I ran e2fsck against the new logvol and it found the filesystem and the journal but the filesystem was corrupted with hundreds of errors. At this point I just reformatted the device and now I'm loading the backup onto it.

How do I handle the LVM metadata? The restored metadata will not be correct. Or does LVM have a way to figure this out?

Gerry

Gerry, for the benefit of the rest of us who are depending on snapshot -

Was there much writing and/or reading going on between the time your backup was made, and the time of the hang?

I know that snapshot works by putting an extra load on every write to the origin volume. Is there any reason, known to knowledgeable LVM designers on this list, why this should lead to "hundreds of errors" when a system hang takes place? I assume stuff got caught in a half-committed state, but _hundreds_... ???

Thanks,

Larry

Larry,
There was only maybe ten minutes between the time the backup finished
and the hang. The system was very quiet at the time. The backups run overnight when there is almost no system activity. I had only recently begun using snapshots and I was surprised to see a machine get into this state while running a backup w/snapshot. I use a 256M ramdisk as the snapshot vol. Prior to introducing snapshots this machine had run without any problems during backups. I could not tell what exactly caused the hang. There was no indication that the snapshot had run out of space and I doubt that it did. The issue for me is that no matter what happened to cause a hang that LVM should never get into a state where it cannot start the system. The backup had completed backing up the root filesystem (which was last) when the hang occurred. What that did was to leave the volume that contained the root filesystem in an inconsistent state at reboot since the snapshot logvol was not removed. LVM should have recognized that the inconsistency was due to a snapshot logvol and corrected the problem at boot and brought up the VolGroup sans snapshot device since it's worthless at that point anyway. My concern is that there is a bug somewhere in the snapshot mechanism. I've seen other people report what they said were corruption problems resulting from snapshots.

Regards,
Gerry

Just to finish documenting the recovery from the corruption:

I was able to fully restore the root logvol and then I rebooted the system. The system came up fine without having to make any special adjustments to LVM metadata. It figured it out. The only thing I noticed was a blizzard of SELinux avc denials. So I ran a relabel and that seemed to cure those. I did a 'cat /etc/lvm/backup/VolGroup00' and took a look at what was in there. It was in the state that included the snapshot0 and my snapshot logvol. I then did a vgdisplay -v and LVM rewrote the meta backup and when I checked it again the snapshot0 and my snapshot were gone. So the system appears to be recovered and working again.

Gerry

This is quite scary to someone like me who is new to LVM and using snapshots for backups (whether automated or not)...

I'd really like to hear from some of the devs on this, and what they think may have caused it...

Incidentally... this is the first time I've heard of someone using a Ramdisk as the snapshot volume... any chance that could have been involved somehow?

Best regards,

Charles

Charles,
Here's a snapshot ramdisk reference:

http://linuxsoftware.co.nz/blog/2008/03/11/lvm-snapshot-with-no-free-diskspace

The ramdisk has been working without problem. I tested it quite a bit
before putting it into production. But I don't know LVM internals so I cannot say whether it could have had anything to do with the hang that occurred.

Regards,
Gerry

Please, do *not* use this configuration, it is very dangerous!

Note that you extend Volume Group on you physical disk with another PV in volatile memory.

Now imagine that your system crash (oh, wait, it happened already!) After reboot you have no ramdisk, and the Volume Group is incomplete (because you didn't removed PV on ramdisk).

If there is only snapshot, you can recover it, if some operation placed here part of another volume, you will lose data.

If there is root volume, system will not boot. Period.

Milan
--
[email protected]

And this is no different than if snapshot was on any other device such as esata-hdd, usb-hdd or usb stick. LVM should handle this. If the snapshot device goes away, then just vgreduce it on the reboot.
Trying to retrofit snapshot into existing systems is far easier if you can use ramdisk, esata-hdd, usb-hdd, usb-stick because most systems have allocated all space and rather than struggling through trying to compact and reduce VG, LV, PV, partition, filesystem to gain space it is much easier to use other devices. And again this is something that LVM should be able to handle.

part of another volume placed where? on the snapshot? I don't understand this part.

Regards,
Gerry

LVM is not going to distinguish between your /dev/ram1 and other PV's. So, if you happen to do any 'lvextend/pvmove/lvcreate/anyother' operations, LVM may place such data on your ram disk. The only operation that you do after placing /dev/ram1 under LVM is snapshot creation, I don't see big problem though. You should be very careful to delete the ram disk PV before you do any operation that involves allocating space.

--Malahal.

If you mount the origin device with missing snapshot, you destroy the snapshot (even if you don't touch it). The snapshot can no longer be repaired.

So it is safer to not activate device in this case then destroy data.

Imagine, for example, you have origin and snapshot, you reconfigure disks in some weird way that the snapshot disk is inaccessible, you boot, and the system automatically starts without the snapshot. And you lose any data that you stored on that snapshot.

You can with dmsetup (but it has deadlocks). Maybe someone could write non-deadlocky snapshot-managing tool that wouldn't depend on lvm vgs, pvs and lvs.

Mikulas

Why? What value is the old snapshot at this point? You just had a system reboot in the middle of a snapshotted backup so all you need to do is get the system up, redo another snapshot and retake your backup. I'm not interested in the old snapshot.

What is on the old snapshot at this point is probably indeterminate anyway.

Have not used dmsetup. If it has deadlocks, I don't think I want to use it.

Gerry

? It holds all of the changes to your original (snapshotted) volume, so I imagine it would be pretty important to most people?

I'm not really understanding all the inner-workings of LVM, but now I can see why using a ram disk for an LVM snapshot might not be a very good idea for this one reason - it won't survive a reboot...

I would be... but thats just me...

Best regards,

Charles

(a copy of)

It's like yesterday's newspaper. Why do you need to keep it?

Gerry

A wild guess... because the changes haven't been synced back with the original yet (which happens when the snapshot is released)?

Best regards,

Charles

My understanding is that the snapshot is a copy-on-write (COW) to the original volume. If that is not the case, then I guess things must have changed and I have some old information.

Gerry

You could be right... like I said, I'm not totally understanding how it all works yet...

It would be nice if someone who knows could clear this up...

Best regards,

Charles

At least in the case where the snapshot is read-only (LVM1 default, LVM2 by config?), if the snapshot is lost, invalid, not removed from VG prior to reboot, when LVM comes back up it should see this and immediately know that it can just vgreduce VG --removemissing for the old snapshot. In the case of rw (no LVM1, LVM2 default), it should be a user choice and LVM should prompt the user at boot as to whether to remove this old snapshot so it can attempt to activate the VG. Unless the user knows that there were non-backup related lvm mods written during the snapshot (eg: pvmove) then the user will just answer yes and the system should boot. This is how LVM should operate in this scenario.

Gerry

If you want your system to do that, update your initrd/initscripts accordingly to run the appropriate lvm2 commands to do that!

Alasdair

[email protected]

I can certainly do that. But I think this applies in the general case and should be included as standard behavior in LVM. This is a much more robust means of dealing with this scenario that having LVM just refuse to mount the volume when the only issue is a bad snapshot.

Gerry

Question...

In Gerry's scenario here, if the snapshot volume had NOT been on a ram disk, would he have had the problem he had or not?

Best regards,

Charles

Charles,
Without knowing the exact cause of the hang there is no way to know if
the ramdisk was at issue. I doubt it myself, I ran a whole series of backups last night using the ramdisk snapshot and everything went fine.

Gerry

Actually, I'm not talking about the hang as much as I am the problem you had recovering from it...

My questions is, had the snapshot volume been on a regular hard drive, would you have had the same problems recovering from the situation.

Best regards,

Charles

I don't think anyone could possibly say.

There was corruption involved and that could have occurred for lots of reasons. My thought is that even if the ramdisk were not present on the next boot there should be no reason for that to cause corruption. There were no lvm mods made to the rw snapshot which means in essence it was read-only. The VG did not mount and nothing wrote to any filesystem on the VG which means the corruption is probably totally unrelated or is a bug. LVM should be robust enough to fully recover from this without any corruption. It just should have discarded the old snapshot. The original volume should not be affected. But as I said, I suspect there may be a bug. Probably some scenario is not being handled in the code.

Gerry

Once again, lvm2 provides the distribution or sysadmin with the facilities to handle these scenarios. It's up to your initrd and/or initscripts whether or not and how to use those facilities.

Alasdair

[email protected]

Alasdair,
Is it possible to define an init script that would run early enough
(before LVM tries to activate the volumes) so that I could vgreduce the old snapshot?
Or does this require a new initrd?

Gerry

I started looking at this and then found that Fedora appears to have the device mapper and snapshot support as modules:

/lib/modules/2.6.20-1.2952.fc6xen/kernel/drivers/md/dm-snapshot.ko /lib/modules/2.6.20-1.2952.fc6xen/kernel/drivers/md/dm-mod.ko

So what is going to happen if you have a snapshot of root partition active and then a reboot happens (sound familiar?). The kernel on restart can no longer read device mapper or snapshot module or root partition and it will panic and who knows what else will happen. Corruption maybe???

Gerry

The modules are copied to initrd by mkinitrd, and loaded from there. They are treated the same as device drivers for the disk containing
the root filesystem. Mkinitrd can also copy md drivers to initrd
so that root can be on a software raid partition.

	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Here is the snip from Fedora/RHEL for activating the VG containing the root filesystem (note: kernel modules and lvm command are contained in initrd filesystem, and do not depend on /lib/modules/...):

...
echo "Loading dm-mod.ko module"
insmod /lib/dm-mod.ko
echo "Loading dm-mirror.ko module"
insmod /lib/dm-mirror.ko
echo "Loading dm-zero.ko module"
insmod /lib/dm-zero.ko
echo "Loading dm-snapshot.ko module"
insmod /lib/dm-snapshot.ko
echo Making device-mapper control node
mkdmnod
mkblkdevs
echo Scanning logical volumes
lvm vgscan --ignorelockingfailure
echo Activating logical volumes
lvm vgchange -ay --ignorelockingfailure rootvg
...

This will fail if any of the physical volumes are missing. More logic in initrd is required to boot with missing PVs.

So yes, a ram disk is guaranteed to be missing on reboot, and thus to fail in the RH/Fedora distros. There is no problem taking snapshots of the root fs otherwise. If there is a robust way to reduce missing PVs automagically in the above script from initrd, Fedora could use the contribution.

	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

So right ^^^here^^^ is where we need to test if old snapshot was read-only or read-write (how to do that? grep vgdisplay -v for some value?) So maybe something like this?:

set removemissing flag false
if old snapshot is the only PV missing:
--if old snapshot was read-write:
------stop boot sequence
------show user some info about old snapshot
------prompt whether to remove the old snapshot from the VG so it can be activated
------if response yes:
----------set removemissing flag true
------else
----------alert user that manual action is required to correct problem related to old snapshot before VG can be activated ----------abort boot
--elsif old snapshot was read-only:
------set removemissing flag true
if removemissing flag true:
----vgreduce --removemissing VG PV(old snapshot)

Gerry

Hi Gerry,

I don't think that automatically making changes to the VG is a good idea. Imagine that you have a snapshot on disk mapped from SAN storage array and that becomes temporary unavailable (SAN switch failure or whatever). It would be a bad idea to remove this PV from VG just because it is TEMPORARY unavailable. And how do you distinguish between ram disk and normal disk and if it is totally gone or it is only temp issue? Also if real HDD fails, you don't want to vgreduce it, you want to replace it, vgcfgrestore it and sync it (in case you are using mirroring). Also imagine that you can have also real data on the same PV, not only snapshot.

What would help is the ability to activate the VG also with missing PVs, I'm not sure if that is possible in linux LVM.

  bYE, Marki

This was part of my point about the ramdisk, in that other types of devices can also disappear not just ramdisk. Once the device with the snapshot disappears, I'm not sure I would trust that snapshot even if the device comes back online. The backup process would try to keep running and it would probably get all kinds of errors at this point so to me it makes sense to consider that snapshot invalid from that standpoint of the backup. Now if it was a read-write snapshot and you've used some lvm commands to pvmove something onto the snapshot then that is why I suggest that the user be able to make decision at reboot about whether to remove the PV. If they answer 'no' to the removal then the boot will stop and they will have to perform some recovery through the rescue mode to see if they can recover some data from the device before allowing the PV to be removed from the VG so the system can boot.

You don't vgreduce the entire hdd. Only for the PV that contained the snapshot. If you used an entire hdd to be the snapshot then you can restore it. Recover data in the case of read-write and then vgreduce it from the VG so system can boot.

You cannot activate a VG with missing PV's because at some point then LVM would attempt to write on one of the missing PV and then you get panic.

Regards,
Gerry

Sure - but a ramdisk is *much* more volatile than a hard drive... and data on a hard drive will almost *always* survive a reboot, where data on a ramdisk will *never* survive a reboot.

Personally, for me, the dangers of using a ramdisk as a snapshot volume *far* outweigh any questionable benefits...

Best regards,

Charles

The COW operation implies you can trust that snapshot if, and only if, nothing was written to the origin volume between when the snapshot disappeared and when it came back on line. Perhaps a "Has been written" flag could be maintained for the origin volume starting at reboot.

Larry Dickson

Hello,

As I told - I'm not sure if it is possible on Linux, but for example on HP-UX LVM it is normally possible to activate VG even with some missing PVs (there is also special Low Quorum mode - which is not enabled by default - to activate VG when you have available <=50% of mirrored PVs).

Of course trying to read/write from missing PV should (and on HP-UX does) produce an I/O error to application, but should NOT panic. From my point there is almost no difference between activating VG with missing PV and loosing that PV while VG is activated (SAN or HDD failure).

Having snaphot on failed/missing PV is a little bit different, because there is question what to do with original lvol. But that can/should prevent activating the lvol and not entire VG (which can have more lvols which don't have any data/snapshots on the missing PV).

  bYE, Marki

When activating a VG, missing PVs should be marked "missing", and show as such with pvs. Missing PVs should behave exactly as if they were not missing, but have I/O errors on every operation. Every attempted I/O on a missing PV should cause a synthetic "missing PV" error. This should make mirrors, etc, do the right thing. LVs partially on the missing PV could be mountable, getting errors for spots on the missing PV.

The could be an option on vgchange for --ignoremissingpv to activate with missing PVs. You might not want to activate with a root filesystem partially on a missing PV (filesystem corruption).

It could be helpful if LVs partially on missing PVs were made read-only. If the PV was missing only temporarily, this prevents filesystem corruption. On the other hand, if the PV is really gone, you want to be able to read it to recover as much data as you can. This determination is straightforward for regular and snapshot LVs (snapshots should be read-only if they or their origin is partially missing). It would be more complicated for mirrored LVs.

Finally, there is the issue of metadata changes made while some PVs are missing. Suppose there are 2 PVs, one is temporarily missing, and you make changes to LVM (add/remove LVs and snapshots). When you reboot and the missing PV comes back, which version of metadata do you use? "Most recent version" has some failure scenarios. AIX solves this problem with "quorum". A quorum of PVs must have the same metadata version for the VG to be automatically activated. Otherwise, an admin must pick which version to use. A quorum is a simple majority by default.

In any case, it should be obvious that supporting activation of VGs with missing PVs is not trivial, and using a ram disk for a PV is insane until missing PVs are fully supported.

	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am I missing something here? vgchange --partial seems like it will do what you want (man 8 lvm):

- -P | --partial
When set, the tools will do their best to provide access to volume groups that are only partially available. Where part of a logical volume is missing, /dev/ioerror will be substituted, and you could use dmsetup (8) to set this up to return I/O errors when accessed, or create it as a large block device of nulls. Metadata may not be changed with this option. To insert a replacement physical volume of the same or large size use pvcreate -u to set the uuid to match the original followed by vgcfgrestore (8).

You can also choose the device used for mapping segments on any incomplete logical volumes by setting the missing_stripe_filler parameter in the activation section of lvm.conf.

Regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFIGyme6YSQoMYUY94RAuqLAJ0SNvliyJFj+oPca084RnCrjhtLPQCeMc42 mPcsR4yUr4WSJq1K+0eyZQY=
=Fu2C
-----END PGP SIGNATURE-----

Hello,

It is not the same. I was doing this about 2 years ago when one HDD died and I wanted to recover as much data as possible (I mean data changes since last backup), but it was not possible to activate the VG with missing PV (or activate only read-only, which prevented me from mounting the filesystem, because it required fsck, which cannot be done on readonly lvol). I had to google quite a lot on how to create this "virtual" PV with dmsetup which returned all zeros (or I/O errors). It had to be the same size as original (but fortunately you can create 100 GB sparse file in RAM and use loop to turn it into PV)... But this is only for recovery - you are making "virtual" identical PV instead of the original missing... So it is definitely not the same as activating VG/LV with missing PV.

  bYE, Marki

It does part of what I want. It would at least allow a system to boot from a partial VG containing the root fs. It does not allow any metadata changes while VG is partial - not even reducing missing PV. If a PV is temporarily missing, and you using the pvcreate -u option, then the original PV comes back as well, you have fireworks. Also, it doesn't ensure partially missing LVs are readonly (including where a snapshot is on a missing PV).

In any case, the --partial might be good for initrd. Although a newbie admin would be terribly confused by the "no metadatachanges" restriction.

	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Well, what good is this if you cannot fix anything? You are stuck in 'broken' mode.

Regards,
Gerry

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

You recover what you can and then either remove the stray PVs from the volume group via vgreduce --removemissing or replace them with new devices that you have prepared using pvcreate --uuid --restorefile=.

Regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFIGzTy6YSQoMYUY94RAszRAJ9d0xHvZJbktyps+iF1dhTRSM4oKgCfeWz2 WnV7jmyJLWIknLsddgDNpuM=
=vnhu
-----END PGP SIGNATURE-----

If you use snapshots for something other than backups (for example version-control using snapshots --- to enable admin to revert changes if he messes something), then the snapshot is valuable and should survive reboot.

Anyway, placing a ramdisk to volume group is bad idea and it must not be done on any production system --- note that any lvcreate, lvconvert, etc command can allocate anything on that ramdisk --- without the administrator knowing it. --- so I don't see any reason why we should do extra hacks to lvm for people who placed ramdisk into vg.

To use temporary storage for snapshot, a special command for lvm would be more appropriate --- a command that would setup snapshot and write nothing about it to metadata, so that the snapshot would be forgotten on next reboot --- then, you can setup the snapshot on any device outside the volume group. You can already do this with dmsetup.

Mikulas

This is an excellent idea - and not just for ramdisks. The hardest part of using snapshots for backups is making sure they are deleted afterward. Not having to check at reboot would be one less 'i' to dot.

Now supposing this feature is added to lvm - what if an enterprising admin decides to create *two* snapshots using the same block device? This could be a caveat emptor - or it could check that the block device was not in use by anything else, like another snapshot.

Couldn't a LV be used for a temporary snapshot also? It would just look like the COW instead of the snapshot image. I could allocate a backup_cow LV for backup use only.

The only problem I see is newbies getting very confused wondering why their "snapshot" won't mount. That's why I suggest *not* calling the examples (of the temporary snapshot feature) in the docs anything with "snap" in the name. I suggest "cow" as above.

	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

I love this idea because it nicely covers one of the main purposes of snapshotting, namely, making backups at a well-defined time point. Would there ever be any use of a temporary read-write snapshot? If not, perhaps making it read-only would simplify the design.

Larry Dickson
Cutting Edge Networked Storage

Ditto...

I feel much better now about using snapshots after realizing the OP's problem was due to the use of a RAMdisk for a snapshot volume (combined with an unclean shutdown), but anything that makes these safer and easier gets my vite...

Best regards,

Charles