Skip to contentSkip to conversation

Recovering "broken" disk ( 17th )

linux-lvm Brian McCullough 3 participants 8 messages Oct 18, 2021 – Oct 21, 2021

lore
Share & tools
lore
#0 lore raw

Folks,

I have had a disk go bad on me, causing me to lose one PV.

I seem to have retrieved the partition using ddrescue, but it also seems to be missing some label information, because pvscan doesn't see it.

Using hexdump, I see the string " LVM2 " at 0x1004, but nothing before that. The whole phrase is:

0x01000 16 d6 8e db 20 4c 56 4d 32 20 78 5b 35 41 25 72

I find what appears to be an LVM2 configuration section at 0x1200, and so I was able to read the UUID that this PV should have.

On another machine, I dumped a PV partition, and find "LABLEONE" at 0x200, with the same " LVM2 " at 0x01000.

I was concerned that my dump was offset, but the comparison to the "good" one suggests that that isn't the problem, but just the missing "LABLEONE" and related information at 0x0200.

If I do a "pvcreate --uuid xxxx" would this fix that recovered partition so that pvscan and friends can work properly, and I can finally boot that machine?

Thank you,
Brian

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#1 lore raw

You will need a lvm backup file for the pvcreate --uuid I believe (there may be some option to get around needing the backup file).

That will put the header back on if you either have an lvm backup and/or archive file, you might also need a vgcfgrestore afterwards depending on if anything else is missing.

I have never done it, but it looks possible to make a lvm backup file by reading it directly off the disk with dd, so that you will have a file that pvcreate is ok with, that is if there is no way to force it without a backup file.

But, this should get the pv back showing up with whatever sectors that you successfully recovered.

Quoted 35 lines · Brian McCullough

Folks,

I have had a disk go bad on me, causing me to lose one PV.

I seem to have retrieved the partition using ddrescue, but it also seems to be missing some label information, because pvscan doesn't see it.

Using hexdump, I see the string " LVM2 " at 0x1004, but nothing before that. The whole phrase is:

0x01000 16 d6 8e db 20 4c 56 4d 32 20 78 5b 35 41 25 72

I find what appears to be an LVM2 configuration section at 0x1200, and so I was able to read the UUID that this PV should have.

On another machine, I dumped a PV partition, and find "LABLEONE" at 0x200, with the same " LVM2 " at 0x01000.

I was concerned that my dump was offset, but the comparison to the "good" one suggests that that isn't the problem, but just the missing "LABLEONE" and related information at 0x0200.

If I do a "pvcreate --uuid xxxx" would this fix that recovered partition so that pvscan and friends can work properly, and I can finally boot that machine?

Thank you,
Brian

Quoted 7 lines

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
Quoted 7 lines · Roger Heflin

You will need a lvm backup file for the pvcreate --uuid I believe (there may be some option to get around needing the backup file).

That will put the header back on if you either have an lvm backup and/or archive file, you might also need a vgcfgrestore afterwards depending on if anything else is missing.

Quoted 7 lines

I have never done it, but it looks possible to make a lvm backup file by reading it directly off the disk with dd, so that you will have a file that pvcreate is ok with, that is if there is no way to force it without a backup file.

But, this should get the pv back showing up with whatever sectors that you successfully recovered.

Thank you, Roger.

I might be able to do that on the original machine, which won't boot at the moment because of this failure, but I have been using a working machine so far to use ddrestore etc.

Yes, as you said, I was able to use dd to copy off the vgconfig data into a file. So you think that I might be able to use that as the reference in pvcreate? That makes some sense.

I started this out asking ( but not clearly ) whether the data on the disk appeared to be in the correct place and whether what was there was correct, but I think that I answered that for myself with the comparison to the working PV partition.

Thanks again,
Brian

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#3 lore raw

I would edit the vgconfig you dd'ed with an editor and make sure it looks reasonable for what you think you had.

When you do the pvcreate --uuid it won't use anything except the uuid info so the rest may not need to be exactly right, if you have to do a vgcfgrestore to get it to read the rest of the info will be used.

I have seen some weird disk controller failures that appeared to zero out the first bit of the disk (enough to get the partition table, grub, and the pv header depending on where the first partition starts).

You will need to reinstall grub if this was the bootable disk, since there were 384 bytes of grub in the sector with the partition table that you know are missing.

Quoted 35 lines · Brian McCullough
Quoted 5 lines · Roger Heflin

You will need a lvm backup file for the pvcreate --uuid I believe (there may be some option to get around needing the backup file).

That will put the header back on if you either have an lvm backup and/or archive file, you might also need a vgcfgrestore afterwards depending on

if

Quoted 4 lines

anything else is missing.

I have never done it, but it looks possible to make a lvm backup file by reading it directly off the disk with dd, so that you will have a file

that

Quoted 4 lines

pvcreate is ok with, that is if there is no way to force it without a backup file.

But, this should get the pv back showing up with whatever sectors that

you

Quoted 1 line

successfully recovered.

Thank you, Roger.

I might be able to do that on the original machine, which won't boot at the moment because of this failure, but I have been using a working machine so far to use ddrestore etc.

Yes, as you said, I was able to use dd to copy off the vgconfig data into a file. So you think that I might be able to use that as the reference in pvcreate? That makes some sense.

I started this out asking ( but not clearly ) whether the data on the disk appeared to be in the correct place and whether what was there was correct, but I think that I answered that for myself with the comparison to the working PV partition.

Quoted 10 lines

Thanks again,
Brian

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#4 lore raw
Quoted 2 lines · Roger Heflin

I would edit the vgconfig you dd'ed with an editor and make sure it looks reasonable for what you think you had.

It turns out, comparing the information that I pulled off of the drive with what I find in /etc/lvm/backup, that the first part of the vgconfig information is missing. As I said in one of my messages, the information that I retrieved from the disk starts at 0x1200. I don't know whether that is correct or not. It does not appear to be a proper "backup" file, which I think it should be.

I rebooted ( partially ) the machine and copied the vgconfig backup file from that, but am somewhat concerned, because I don't seem to be able to match the UUIDs. The one that I seem to see in the vgconfig data that I pulled off of the drive vs what I got out of /etc/lvm/backup. Maybe I am just mis-reading it. I will continue my research for a bit.

Quoted 3 lines

When you do the pvcreate --uuid it won't use anything except the uuid info so the rest may not need to be exactly right, if you have to do a vgcfgrestore to get it to read the rest of the info will be used.

Oh, thank you. I did see that things got somewhat different on the
target drive when I did "pvcreate --uuid --restorefile." I got paranoid when I saw that, and re-copied the ddrestore file back to the target
drive before I did anything else. Should I do "pvcreate --uuid
--norestorefile," instead? Then, once it is back in the machine, do the pvscan and vgcfgrestore, and expect good things?

Quoted 3 lines

I have seen some weird disk controller failures that appeared to zero out the first bit of the disk (enough to get the partition table, grub, and the pv header depending on where the first partition starts).

I APPEAR to have a partition table, containing an NTFS partition, an LVM partiton ( the one that I am concentrating on ) and a Linux partion. I would have thought that it was all LVM, but my memory could easily be wrong.

Quoted 3 lines

You will need to reinstall grub if this was the bootable disk, since there were 384 bytes of grub in the sector with the partition table that you know are missing.

Fortunately, this is all data, nothing to do with the boot sequence, except that the machine will not boot with the missing PV.

Thank you,
Brian

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#5 lore raw

is the pv in the root device vg? if not changing fstab to not mount the
missing fs(es) should get it bootable. I have a practice of putting
",nofail" on all non-root filesystems (ie defaults,nofail) since priority #1 is getting the machine up and on the network after a reboot such that it can be ssh'ed to and fixed as needed.

If it is not the root device then on the root device there should be several prior archive copy in /etc/lvm/archive/<vgname>*, and maybe some copies in /etc/lvm/backup.

In the vg backup file there will be a bunch of uuids, you want the specific pv uuid and not the vg/lv uuids. Each pv has a uuid and each lv has a uuid and the vg has a uuid.

Quoted 57 lines · Brian McCullough
Quoted 2 lines · Roger Heflin

I would edit the vgconfig you dd'ed with an editor and make sure it looks reasonable for what you think you had.

It turns out, comparing the information that I pulled off of the drive with what I find in /etc/lvm/backup, that the first part of the vgconfig information is missing. As I said in one of my messages, the information that I retrieved from the disk starts at 0x1200. I don't know whether that is correct or not. It does not appear to be a proper "backup" file, which I think it should be.

I rebooted ( partially ) the machine and copied the vgconfig backup file from that, but am somewhat concerned, because I don't seem to be able to match the UUIDs. The one that I seem to see in the vgconfig data that I pulled off of the drive vs what I got out of /etc/lvm/backup. Maybe I am just mis-reading it. I will continue my research for a bit.

Quoted 1 line

When you do the pvcreate --uuid it won't use anything except the uuid

info

Quoted 2 lines

so the rest may not need to be exactly right, if you have to do a vgcfgrestore to get it to read the rest of the info will be used.

Oh, thank you. I did see that things got somewhat different on the
target drive when I did "pvcreate --uuid --restorefile." I got paranoid when I saw that, and re-copied the ddrestore file back to the target
drive before I did anything else. Should I do "pvcreate --uuid
--norestorefile," instead? Then, once it is back in the machine, do the pvscan and vgcfgrestore, and expect good things?

Quoted 2 lines

I have seen some weird disk controller failures that appeared to zero out the first bit of the disk (enough to get the partition table, grub, and

the

Quoted 1 line

pv header depending on where the first partition starts).

I APPEAR to have a partition table, containing an NTFS partition, an LVM partiton ( the one that I am concentrating on ) and a Linux partion. I would have thought that it was all LVM, but my memory could easily be wrong.

Quoted 1 line

You will need to reinstall grub if this was the bootable disk, since

there

Quoted 1 line

were 384 bytes of grub in the sector with the partition table that you

know

Quoted 1 line

are missing.

Fortunately, this is all data, nothing to do with the boot sequence, except that the machine will not boot with the missing PV.

Quoted 10 lines

Thank you,
Brian

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#6 lore raw

replying to my last email.

do the pvcreate -uuid and then do a pvs/lvs/vgs and see if the vg/lv's look
like that are there. if so do a vgchange -ay <vgname> and then test
mounting the fs.

And if with the fs either commented out and/or ,nofail the normal os boots up work from there as you should have backup files.

Quoted 10 lines · Roger Heflin

is the pv in the root device vg? if not changing fstab to not mount the
missing fs(es) should get it bootable. I have a practice of putting
",nofail" on all non-root filesystems (ie defaults,nofail) since priority #1 is getting the machine up and on the network after a reboot such that it can be ssh'ed to and fixed as needed.

If it is not the root device then on the root device there should be several prior archive copy in /etc/lvm/archive/<vgname>*, and maybe some copies in /etc/lvm/backup.

Quoted 76 lines

In the vg backup file there will be a bunch of uuids, you want the specific pv uuid and not the vg/lv uuids. Each pv has a uuid and each lv has a uuid and the vg has a uuid.

Quoted 69 lines · Brian McCullough
Quoted 1 line · Roger Heflin

I would edit the vgconfig you dd'ed with an editor and make sure it

looks

Quoted 1 line

reasonable for what you think you had.

It turns out, comparing the information that I pulled off of the drive with what I find in /etc/lvm/backup, that the first part of the vgconfig information is missing. As I said in one of my messages, the information that I retrieved from the disk starts at 0x1200. I don't know whether that is correct or not. It does not appear to be a proper "backup" file, which I think it should be.

I rebooted ( partially ) the machine and copied the vgconfig backup file from that, but am somewhat concerned, because I don't seem to be able to match the UUIDs. The one that I seem to see in the vgconfig data that I pulled off of the drive vs what I got out of /etc/lvm/backup. Maybe I am just mis-reading it. I will continue my research for a bit.

Quoted 1 line

When you do the pvcreate --uuid it won't use anything except the uuid

info

Quoted 2 lines

so the rest may not need to be exactly right, if you have to do a vgcfgrestore to get it to read the rest of the info will be used.

Oh, thank you. I did see that things got somewhat different on the
target drive when I did "pvcreate --uuid --restorefile." I got paranoid when I saw that, and re-copied the ddrestore file back to the target
drive before I did anything else. Should I do "pvcreate --uuid
--norestorefile," instead? Then, once it is back in the machine, do the pvscan and vgcfgrestore, and expect good things?

Quoted 1 line

I have seen some weird disk controller failures that appeared to zero

out

Quoted 1 line

the first bit of the disk (enough to get the partition table, grub, and

the

Quoted 1 line

pv header depending on where the first partition starts).

I APPEAR to have a partition table, containing an NTFS partition, an LVM partiton ( the one that I am concentrating on ) and a Linux partion. I would have thought that it was all LVM, but my memory could easily be wrong.

Quoted 1 line

You will need to reinstall grub if this was the bootable disk, since

there

Quoted 1 line

were 384 bytes of grub in the sector with the partition table that you

know

Quoted 1 line

are missing.

Fortunately, this is all data, nothing to do with the boot sequence, except that the machine will not boot with the missing PV.

Thank you,
Brian

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#7 lore raw
Quoted 1 line · Roger Heflin

replying to my last email.

Thank you for your help, Roger. I will answer with more detail when the sun comes around to my side of the planet, but will say that this is a "secondary" VG, so your suggestion of changing fstab is very attractive.

B

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/