Serious problem and I don't know where to turn.

I screwed up, big time. If it was just one or two mistakes than I think I could of solved it, but now I am not sure how to fix this.

Some back ground: I recently picked up a DSN-3200 cheap off ebay. Its basically an Iscsi SAN system that holds 15 drives. VERY simplistic but hey, does RAID5. I added 5 drives and set up a simple LVM2 group off the first partition of the array. (/dev/sdb1) Followed the directions and made a logical drive, formatted to ext3 and everything was great.

A few months latter Fry's was having a 500gb drive sale, so I picked up 5 more drives and expanded the array (The DSN only tapes on extra drives to the existing one, rather than expanding the array a drive at a time. Think JOBD with RAID5 arrays) I now had an extra, or exact 1.7tb of space. I create another LVM2 partition (/dev/sdb2), expand both the VG and LV.

This would have been all fine, except I ran into an issue with e2fsck. To expand the ext3, it wanted me to run e2fsck -f on the volume. The problem was that eveytime I run the command, it would seem to lock up the iscsi driver. I tried looking for any kind of hints on why this was with no avail. After hours of working on it, I threw up my hands. I figured if I can never e2fsck the volume, it might be bad for me to extend it in the first place. At the time, I didn't have a way to back up the array, so I did something stupid.

I deleted the /dev/sdb2 partition. Mind you, this was after a few days and the line of thinking was that I would delete it, create an ext3 partition, and copy the data over. Of course it gave a lot of errors in LVM2 afterwards. I could still mount the volume so I thought I was ok.

A day or two passes and I wake up to find the server has locked up. I do a hard reboot to find the little 160gig IDE drive in it has failed. Doesn't spin up on power. There goes my backup metadata.

Sigh. I would have been better off with an ext3 partition to begin with, but I liked the flexibility that LVM2 offered. To be honest I don't remember a lot of the commands I have used either. Any help would be appreciated. It even looks like I lost the logical drive. But I know the data is still there, is there any kind of utility I can just "get" the ext3 partition out of there?

[root@server archive]# vgscan

Reading all physical volumes. This may take a while...

Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1

Found volume group "VolGroup00" using metadata type lvm2

Found volume group "ANIME" using metadata type lvm2

[root@server archive]# ls

VolGroup00_00000.vg

[root@server archive]# vgchange -a y

Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1

2 logical volume(s) in volume group "VolGroup00" now active

0 logical volume(s) in volume group "ANIME" now active

[root@server archive]# pvscan

Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1

PV /dev/hda2 VG VolGroup00 lvm2 [148.94 GB / 0 free]

PV /dev/sdb1 VG ANIME lvm2 [1.82 TB / 1.82 TB free]

PV /dev/sda2 lvm2 [1.82 TB]

Total: 3 [3.78 TB] / in use: 2 [1.96 TB] / in no VG: 1 [1.82 TB]

[root@server archive]# lvscan

Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1

ACTIVE '/dev/VolGroup00/LogVol00' [147.00 GB] inherit

ACTIVE '/dev/VolGroup00/LogVol01' [1.94 GB] inherit

[root@server archive]#

How did this happen? Why is the PV magic number on /dev/sdb1 exactly the same as on /dev/sda1?

I looks like your lvs are up. Can't you just mount them?

Sincerely,

Richard van den Berg

AH, well for the first issue it was because I had the iScsi portal's incorrect. So it was doing two separate connections under my two Ethernet cards instead of a single bonded connection, that issue is fixed.

The two ACTIVE lvm's are the boot drive. This is a default Centros install so it sets the drive up like that. Here is a better out

[root@server ~]# lvscan -a

ACTIVE '/dev/VolGroup00/LogVol00' [147.00 GB] inherit

ACTIVE '/dev/VolGroup00/LogVol01' [1.94 GB] inherit

[root@server ~]# vgscan

Reading all physical volumes. This may take a while...

Found volume group "VolGroup00" using metadata type lvm2

Found volume group "ANIME" using metadata type lvm2

Basicly, I have a volume called "ANIME-logical" I am trying to restore, but I don't have the metadata. Is there a way I can get to it?

From: [email protected] [mailto:[email protected]] On Behalf Of Richard van den Berg
Sent: Thursday, June 28, 2007 4:52 PM
To: LVM general discussion and development
Subject: Re: [linux-lvm] Serious problem and I don't know where to turn.

WarlockD wrote:

[root@server archive]# vgscan

Reading all physical volumes. This may take a while...

Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1

How did this happen? Why is the PV magic number on /dev/sdb1 exactly the same as on /dev/sda1?

[root@server archive]# lvscan

Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1

ACTIVE '/dev/VolGroup00/LogVol00' [147.00 GB] inherit

ACTIVE '/dev/VolGroup00/LogVol01' [1.94 GB] inherit

[root@server archive]#

I looks like your lvs are up. Can't you just mount them?

Sincerely,

Richard van den Berg

So the vg "ANIME" is found, but is not activated? That kind of makes sense because it is missing the pv on the deleted /dev/sdb2 partition. According to the vgreduce man pages you can:

vgreduce --removemissing ANIME

to remove all missing physical volumes from the volume group and make the volume group consistent again. It's a good idea to run this option with --test first to find out what it would remove before running it for real. There is also "--partial" option you can use to activate vgs and lvs even if they are missing data. Read the man pages and be very careful before you proceed.

Without the metadata that's going to be tough. If the filesystem is all continuous on your RAID5 partition, technically you should be able to mount it if you can create a device pointing at the right start, and with the right size. Good luck figuring that out.

Another approach could be to recreate the /dev/sdb2 partition in exactly the same way you did before. The metadata might still be there, and the vg might come up.

Sincerely,

Richard van den Berg

The good news is the logical drive came up. I just recreated the logical drive exactly as it was and its even detected, with a missing superblock, as an ext3 partition. The bad news is that e2fsck still froze on me. I can’t tell if it’s the iSCSI driver that is doing it (The logical drive comes up as “busy” when I try to access it after killing e2fsck) or something with lvm2 as doubtful as that is.

I picked up some hard drives and put it in my old raid card for a RAID0, and doing a dd if=/dev/ANIME/testlv of=/dev/sdc1. If this works, and e2fsck works, then I got my data back, yea!

Now what to do with it:P

From: [email protected] [mailto:[email protected]] On Behalf Of Richard van den Berg
Sent: Friday, June 29, 2007 1:23 AM
To: LVM general discussion and development
Subject: Re: [linux-lvm] Serious problem and I don't know where to turn.

WarlockD wrote:

[root@server ~]# lvscan -a

ACTIVE '/dev/VolGroup00/LogVol00' [147.00 GB] inherit

ACTIVE '/dev/VolGroup00/LogVol01' [1.94 GB] inherit

[root@server ~]# vgscan

Reading all physical volumes. This may take a while...

Found volume group "VolGroup00" using metadata type lvm2

Found volume group "ANIME" using metadata type lvm2

So the vg "ANIME" is found, but is not activated? That kind of makes sense because it is missing the pv on the deleted /dev/sdb2 partition. According to the vgreduce man pages you can:

vgreduce --removemissing ANIME

to remove all missing physical volumes from the volume group and make the volume group consistent again. It's a good idea to run this option with --test first to find out what it would remove before running it for real. There is also "--partial" option you can use to activate vgs and lvs even if they are missing data. Read the man pages and be very careful before you proceed.

Basicly, I have a volume called “ANIME-logical” I am trying to restore, but I don’t have the metadata. Is there a way I can get to it?

Without the metadata that's going to be tough. If the filesystem is all continuous on your RAID5 partition, technically you should be able to mount it if you can create a device pointing at the right start, and with the right size. Good luck figuring that out.

Another approach could be to recreate the /dev/sdb2 partition in exactly the same way you did before. The metadata might still be there, and the vg might come up.

Sincerely,

Richard van den Berg