LVM1 and a dying disk

One of the disks in my /dev/files_vg vg have been making weird noises for a while. This vg is a JBOD with the following disks in it:

majestix root # pvscan
pvscan -- reading all physical volumes (this may take a while...)
pvscan -- ACTIVE PV "/dev/ide/host2/bus1/target0/lun0/disc" of VG "f2"

     [149 GB / 0 free]

pvscan -- inactive PV "/dev/ide/host2/bus0/target0/lun0/disc" is in EXPORTED VG
"files_vg" [149.03 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host2/bus0/target1/lun0/disc" is in EXPORTED VG
"files_vg" [111.77 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host0/bus1/target1/lun0/disc" is in EXPORTED VG
"files_vg" [93.14 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host0/bus0/target0/lun0/disc" is in EXPORTED VG
"files_vg" [111.77 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host0/bus0/target1/lun0/disc" is in EXPORTED VG
"files_vg" [93.14 GB / 0 free]
pvscan -- total: 6 [708 GB] / in use: 6 [708 GB] / in no VG: 0 [0]

Exported, you say? Yep. The machine that this vg was in suffered a fatal root filesystem corruption last friday - not related to this disk (I name it Black Friday, that same day my VW Passat dropped the rear muffler and my upstream ISP had some major network issues that still isn't resolved). The f2 vg is the new vg I'm building to move data over to.

majestix root # vgscan
vgscan -- reading all physical volumes (this may take a while...) vgscan -- found active volume group "f2"
vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of volume group "files_vgPV_EXP" from physical volume(s)
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created vgscan -- WARNING: This program does not do a VGDA backup of your volume groups

And devfs? Gentoo. Devfs is a major and total pain when trying to do a vgimport, listing 6 of those long paths on the command line is no fun. And, it doesn't work:

vgimport -- ERROR "pv_read(): PV identifier invalid" reading physical volume "/d ev/ide/host0/bus1/target0/lun0/disc"

I did manage to export the vg before the machine died permanently. The backups I had of the root drive are older then the last major change to the vg (addition of the Seagate 160GB that shows up as 149.03 GB above). The troublesome disk is not showing in the pvscan, it's currently on /dev/ide/host0/bus1/target0/lun0/disc and a pvdata for it reveals:

majestix root # pvdata /dev/ide/host0/bus1/target0/lun0/disc --- Physical volume ---

PV Name               /dev/ide/host0/bus1/target0/lun0/disc
VG Name               ÇD?
PV Size               111.79 GB [234441648 secs] / NOT usable 16.19 MB [LVM: 155 KB]
PV#                   0
PV Status             NOT available
Allocatable           yes (but full)
Cur LV                9306113
PE Size (KByte)       16384
Total PE              7153
Free PE               0
Allocated PE          7153
PV UUID               rDBb94-Kb8v-GMDa-HxIi-???

Segmentation fault

As you can see, not all fun and games.

Now, is there any way to re-initialize the pv data on the failing disk long enough to get it back into the vg and extract the data? The data left in the vg is not critically important (I started backing it up as soon as I noticed the noises), but it would be really nice to get it back.

I have considered pvcreating it, but I'm not sure how I'd get it into the vg afterwards - would a vgscan/vgchange suffice? Would a reiserfsck --rebuild_tree then prune the fs of anything that was on this disk?

/ Rickard Olsson,IT-Konsult/
/ Telefon: +46 70 635 01 42/
/ http://www.webhackande.se/

Rickard,

you say that you don't have a recent LVM metadata backup reflecting your VG with /dev/ide/host0/bus1/target0/lun0/disc in it ?

If you had, normal procedure would be to

# pvcreate -ff /dev/ide/host0/bus1/target0/lun0/disc

and to

# vgcfgrestore -n YourVGName /dev/ide/host0/bus1/target0/lun0/disc
# vgscan
# vgchange -ay

(Because you exported, you need to pvcreate+vgcfgrestore to all PVs)

If /dev/ide/host0/bus1/target0/lun0/disc took a serious overwrite (and it looks like it eyeing the pvdata output below), there's almost no chance to cook up the correct metadata by hand I'm afraid.

People, please take LVM metadata backups!

Regards,
Heinz -- The LVM Guy --

*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf Germany
[email protected] +49 2626 141200

                                                       FAX 924446

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Well, possibly. Many of the files could be rescued off the damaged root file system, even if the directory structure was shot. What file(s) would I be looking for? There's 18487 numbered directories in lost+found with a large number of subdirectories - but if I have a name to go on, I may be able to find something useful in there.

/ Rickard Olsson,IT-Konsult/
/ Telefon: +46 70 635 01 42/
/ http://www.webhackande.se/

I found a files_vg.conf and put it in /etc/lvmconf, did pvcreate and successful vgcfgrestores on all of them. I had to move the disks around a bit to make them appear in their old places since they're now in a new machine (vgcfgrestore complained otherwise). The bad disk is now at /dev/ide/host2/bus1/target1/lun0/disc

majestix root # pvscan
pvscan -- reading all physical volumes (this may take a while...)
pvscan -- ACTIVE PV "/dev/ide/host4/bus1/target0/lun0/disc" of VG "f2"

      [

149 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host2/bus1/target0/lun0/disc" is associated to unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host2/bus1/target1/lun0/disc" is associated to unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host2/bus0/target0/lun0/disc" is associated to unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host2/bus0/target1/lun0/disc" is associated to unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host0/bus1/target0/lun0/disc" is associated to unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host0/bus1/target1/lun0/disc" is associated to unknown VG "files_vg" (run vgscan)
pvscan -- total: 7 [819.79 GB] / in use: 7 [819.79 GB] / in no VG: 0 [0]

majestix root # vgscan
vgscan -- reading all physical volumes (this may take a while...) vgscan -- found active volume group "f2"
vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of volume group "files_vg" from physical volume(s)
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created vgscan -- WARNING: This program does not do a VGDA backup of your volume groups

majestix root # pvdata /dev/ide/host2/bus1/target1/lun0/disc --- Physical volume ---

PV Name               /dev/ide/host2/bus1/target1/lun0/disc
VG Name               files_vg
PV Size               93.16 GB [195371568 secs] / NOT usable 16.19 MB [LVM: 151
KB]
PV#                   4
PV Status             NOT available
Allocatable           yes (but full)
Cur LV                1
PE Size (KByte)       16384
Total PE              5961
Free PE               0
Allocated PE          5961
PV UUID               KsET9a-qoKI-a2gr-z1GB-jwM1-vFw7-qHgxeZ

--- Volume group ---
VG Name

VG Access             read/write
VG Status             NOT available/resizable
VG #                  0
MAX LV                256
Cur LV                1
Open LV               0
MAX LV Size           1023.97 GB
Max PV                256
Cur PV                6
Act PV                6
VG Size               670.61 GB
PE Size               16 MB
Total PE              42919
Alloc PE / Size       42919 / 670.61 GB
Free  PE / Size       0 / 0
VG UUID               Z9MJ9w-UjoI-BO9X-rAb0-DPtF-pXqO-Haiji7

--- List of logical volumes ---

pvdata -- logical volume "/dev/files_vg/files_lv" at offset 0
pvdata -- logical volume struct at offset 1 is empty
[...]

What did I do wrong (this time)?

/ Rickard Olsson,IT-Konsult/
/ Telefon: +46 70 635 01 42/
/ http://www.webhackande.se/

"vgcfgrestore -o ..." does the trick without moving disks around (which might be completely impossible with certain path changes anyway). "-o" takes the old path, which has been stored in the metadata backup.

"vgcfgrestore -ll ..." long lists the VG metadata stored in the backup.

Hmmm, if you pvcreated+vgcfgrestored every PV fine, nothing.

Could you send the output of "pvdata -UPP" for all your PVs of "files_vg" and files_vg.conf to me in private email, please.

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
[email protected]                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

# vgchange -ay
vgchange -- volume group "files_vg" successfully activated

I had a duplicate PV uuid for some reason.

The LVM2 vgcfgrestore didn't like the old LVM1 metadata format and LVM2's vgconvert said files_vg doesn't exist. Catch 22. Combining the -u switch in LVM2's pvcreate with a regular pvcreate and vgcfgrestore in LVM1 finally solved that.

Very, very scary. Now to see if reiserfsck's --rebuild_tree works. :-)

/ Rickard Olsson,IT-Konsult/
/ Telefon: +46 70 635 01 42/
/ http://www.webhackande.se/