Can't mount after crash

Hi all!

My lvm-system crashed a day ago and I was really lost in how to do. Buy I found the metadatafile and could with
pvcreate --uuid ... --restorefile ... .... /hdg5,
pvcreate --uuid ... --restorefile ... .... /hdi1,
restore those two drives. It didnt work with just /hdg/hdi as the metafile decribes. hde couldn't be fixed with the restoredfile command, it said that it was to small or equiv, so I just did a pvcreate --uuid ... /hde1

my lvm version is 2.00.29.

After this I did the vgcfgrestore from the metadatafile, my_volume_group, which worked fine, and finally I hade to do a vgmknodes my_volume_group which also worked fine. But then when I tried to the lv I get this

roth:/dev# mount /dev/my_volume_group/mylv /disk
/dev/my_volume_group/mylv: Invalid argument
mount: you must specify the filesystem type
roth:/dev# mount -t ext2 /dev/my_volume_group/mylv /disk
mount: wrong fs type, bad option, bad superblock on /dev/my_volume_group/mylv,
      missing codepage, or too many mounted file systems
      (could this be the IDE device where you in fact use
      ide-scsi so that sr0 or sda or so is needed?)

so I tried to fsck it but nothing worked. What have I done wrong? How can I get back my data? I'll paste some shelldumps as help.

Regards

Daniel

----

roth:/dev# lvscan
 ACTIVE            '/dev/my_volume_group/mylv' [282.05 GB] inherit
roth:/dev# lvmdiskscan
 /dev/hda1  [        2.05 GB]
 /dev/hde1  [       76.69 GB] LVM physical volume
 /dev/hdg1  [        1.00 KB]
 /dev/hdi1  [      149.05 GB] LVM physical volume
 /dev/hda2  [        1.00 KB]
 /dev/hda5  [      346.47 MB]
 /dev/hdg5  [       57.25 GB] LVM physical volume
 0 disks
 4 partitions
 0 LVM physical volume whole disks
 3 LVM physical volumes
roth:/dev# lvdisplay
 --- Logical volume ---
 LV Name                /dev/my_volume_group/mylv
 VG Name                my_volume_group
 LV UUID                jUl5rP-jVIH-5Ne1-p0y9-VTeL-P8su-UF2o2F
 LV Write Access        read/write
 LV Status              available
 # open                 0
 LV Size                282.05 GB
 Current LE             72204
 Segments               3
 Allocation             inherit
 Read ahead sectors     0
 Block device           254:0

roth:/dev# pvdisplay
 --- Physical volume ---
 PV Name               /dev/hdi1
 VG Name               my_volume_group
 PV Size               149.05 GB / not usable 0
 Allocatable           yes (but full)
 PE Size (KByte)       4096
 Total PE              38156
 Free PE               0
 Allocated PE          38156
 PV UUID               3Pv4bt-9E47-aDii-mJbB-6OYc-pCFS-7dOBv9

--- Physical volume ---

 PV Name               /dev/hde1
 VG Name               my_volume_group
 PV Size               76.69 GB / not usable 0
 Allocatable           yes (but full)
 PE Size (KByte)       4096
 Total PE              19633
 Free PE               0
 Allocated PE          19633
 PV UUID               jfqDXu-rwu3-9SyS-U39k-O6TL-M7Kj-0QDpwB

--- Physical volume ---

 PV Name               /dev/hdg5
 VG Name               my_volume_group
 PV Size               57.25 GB / not usable 0
 Allocatable           yes
 PE Size (KByte)       4096
 Total PE              14656
 Free PE               241
 Allocated PE          14415
 PV UUID               6erEsN-BDBw-KFnO-voBg-FtJu-vUVD-D0SH0p

roth:/dev# cat /etc/lvm/backup/my_volume_group
# Generated by LVM2: Fri Dec 10 22:31:29 2004

contents = "Text Format Volume Group"
version = 1

description = "Created *after* executing '/sbin/vgcfgbackup'"

creation_host = "roth" # Linux roth 2.4.22 #8 SMP Mon Apr 5 05:19:01 CEST 2004 i686
creation_time = 1102714289 # Fri Dec 10 22:31:29 2004

my_volume_group {
       id = "73sskf-3Yve-YD3u-iHIt-pX5X-Va4f-iXC1i7"
       seqno = 4
       status = ["RESIZEABLE", "READ", "WRITE"]
       extent_size = 8192              # 4 Megabytes
       max_lv = 255
       max_pv = 255

       physical_volumes {

               pv0 {
                       id = "3Pv4bt-9E47-aDii-mJbB-6OYc-pCFS-7dOBv9"
                       device = "/dev/hdi"     # Hint only

                       status = ["ALLOCATABLE"]
                       pe_start = 384
                       pe_count = 38156        # 149.047 Gigabytes
               }

               pv1 {
                       id = "jfqDXu-rwu3-9SyS-U39k-O6TL-M7Kj-0QDpwB"
                       device = "/dev/hde"     # Hint only

                       status = ["ALLOCATABLE"]
                       pe_start = 384
                       pe_count = 19633        # 76.6914 Gigabytes
               }

               pv2 {
                       id = "6erEsN-BDBw-KFnO-voBg-FtJu-vUVD-D0SH0p"
                       device = "/dev/hdg"     # Hint only

                       status = ["ALLOCATABLE"]
                       pe_start = 384
                       pe_count = 14656        # 57.25 Gigabytes
               }
       }

       logical_volumes {

               mylv {
                       id = "jUl5rP-jVIH-5Ne1-p0y9-VTeL-P8su-UF2o2F"
                       status = ["READ", "WRITE", "VISIBLE"]
                       segment_count = 3

                       segment1 {
                               start_extent = 0
                               extent_count = 38156    # 149.047 Gigabytes

                               type = "striped"
                               stripe_count = 1        # linear

                               stripes = [
                                       "pv0", 0
                               ]
                       }
                       segment2 {
                               start_extent = 38156
                               extent_count = 19633    # 76.6914 Gigabytes

                               type = "striped"
                               stripe_count = 1        # linear

                               stripes = [
                                       "pv1", 0
                               ]
                       }
                       segment3 {
                               start_extent = 57789
                               extent_count = 14415    # 56.3086 Gigabytes

                               type = "striped"
                               stripe_count = 1        # linear

                               stripes = [
                                       "pv2", 0
                               ]
                       }
               }
       }
}
roth:/dev# pvscan
....
 PV /dev/hdi1   VG my_volume_group   lvm2 [149.05 GB / 0    free]
 PV /dev/hde1   VG my_volume_group   lvm2 [76.69 GB / 0    free]
 PV /dev/hdg5   VG my_volume_group   lvm2 [57.25 GB / 964.00 MB free]
 Total: 3 [282.99 GB] / in use: 3 [282.99 GB] / in no VG: 0 [0   ]

roth:/dev# vgscan
 Found volume group "my_volume_group" using metadata type lvm2

Hi!

With pvscan --partial -vvv I found some extra info.

/dev/mapper/my_volume_group-mylv: Added to device cache /dev/my_volume_group/mylv: Aliased to
/dev/mapper/my_volume_group-mylv in device cache (preferred name)
/dev/my_volume_group/mylv: Skipping: Unrecognised LVM device
type 254 <---
...
Finding volume group "my_volume_group"
Opened /dev/hde1
Opened /dev/hdi1
/dev/hdi1: lvm2 label detected
/dev/hde1: lvm2 label detected
Opened /dev/hdg5
/dev/hdg5: lvm2 label detected
Read my_volume_group metadata (5) from /dev/hde1 at 4096 size 1341
/dev/hdi1: lvm2 label detected
/dev/hde1: lvm2 label detected
/dev/hdg5: lvm2 label detected
Read my_volume_group metadata (5) from /dev/hdi1 at 4096 size 1341
/dev/hdi1: lvm2 label detected
/dev/hde1: lvm2 label detected
/dev/hdg5: lvm2 label detected
Read my_volume_group metadata (5) from /dev/hdg5@4096 size 1341
Found volume group "my_volume_group" using metadata type lvm2

Is this perhaps why I cant mount it? How can I restore? I still am a bit confused with Metadata file saying:

              pv0 {
                      id = "3Pv4bt-9E47-aDii-mJbB-6OYc-pCFS-7dOBv9"
                      device = "/dev/hdi"     # Hint only
              }

when I pv(re)created hdi1

Regards,
Daniel

Hi all,

Not anyone out there having a clue on what to do? It seems that about the same thing has happened to the submitter of https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=142737 "Fedora Core 3 system with lvm2 won't boot". Is this acceptable? Because of its impact., this is the kind of bug my company would consider a "show-stopper", and relabel the project "beta" again.

So if there is no way to fix the error, is there any way to recover any of my ~300GB of data? Fourtunately I have backuped all the crucial stuff, but there are a lot of things I would want to recover.

Best regards,

Daniel

FWIW:

1) I suspect that if you install FC3 without LVM2, you may not suffer from this kind of problem

2) I've posted on this mailing list (lvm), the redhat buzilla bug mentioned below, the fedora users list, the e2salvage list, the ext3 users list, and don't seem to be getting satisfaction.

3) I've tried fsck, fsck with a huge variety of different superblocks, e2salvage, e2extract, and nothing is working.

4) I've finally sent a message to gmane.linux.kernel (moderated), inquiring as to the availability of a better data recovery program, or alternatively, the on-disk datastructures used by ext3 and lvm2, with the hope of finding time to write a program to recover my ext3 data.

You don't say how you actually first discovered something had gone wrong, or give any clues about what might have led up to the problem causing it.

Your message doesn't show any LVM problems.

Did something scribble over your disks?
Have you tried mounting with alternative superblocks?

You should only use those if you've lost a disk or accidentally overwritten the start of one and vgcfgrestore isn't working.

Alasdair

[email protected]

But you tried to mount it as ext2?

Is it meant to be ext2 or ext3?

Probably because you're not giving people all the information they need to help you; and maybe even some of the things you've tried have made the problem worse - I don't understand yet why you did the pvcreates, for example, and are you certain the lvm2 backup file you used was completely up-to-date?

Have you a kept a complete log (i.e. shell history) of every command you ran on the system while trying to recover things?

Alasdair

[email protected]

It's ext3, but ext2 tools are supposed to often work on ext3 - especially if you're applying them to a -copy- of your data. :)

I have my data back now. As a reminder, what I did was:

On FC3's rescue disk...

1) Do startup network interfaces
2) Don't try to automatically mount the filesystems - not even readonly
3) lvm vgchange --ignorelockingfailure -P -a y
4) fdisk -l, and guess which partition is which based on size: the small one was /boot, and the large one was /
5) mkdir /mnt/boot
6) mount /dev/hda1 /mnt/boot
7) Look up the device node for the root filesystem
in /mnt/boot/grub/grub.conf
8) A first tentative step, to see if things are working: fsck - n /dev/VolGroup00/LogVol00
9) Dive in: fsck -f -y /dev/VolGroup00/LogVol00
10) Wait a while... Be patient. Don't interrupt it
11) Reboot

Nope, but all seems fine now.

Thanks!

Ok. I did an upgrade of my Debian system.apt-get update && apt-get upgrade *. lvm version got updated as well. Afterwards I did a reboot to put in some extra hw (memory). After reboot I couldn't mount my volume at all. No VG was found et.c. I tried to vgcfgrestore without any halp whatsoever It only said that the pv could not be found or equiv so I restored them with the pvcreate line below, which worked fine on the volumes and a vg and lvm was set up appropriately. Still unmountable, Still unable to fsck.ext2 it with some diffrent superblocks.

I'd be happy to give any specific information you might need to find out any way to restore, since I have run completely out of ideas. If the file system only is "wrapped" into the lvm I'd like to find a tool which can scan disks and recognize superblocks, since I only have tried the 3-4 first I could count up to.

"or accidentally overwritten the start of one" is maybe just what has happened.

Daniel

Thanks Dan! This also restored all my data. Thanks!

Daniel