Skip to contentSkip to conversation

LVM label lost on MD RAID5 reshape?

linux-lvm Bob Bell 4 participants 6 messages Sep 23, 2008 – Sep 24, 2008

lore
Share & tools
lore

Bob Bell

author
#0 lore raw

I'm up a creek, and hoping someone out there can help rescue my data, or at least tell me what went wrong so that I don't have a repeat event. I'm starting with this question on linux-lvm, though let me know if you think the discussion needs to be on linux-raid as well.

I'm setting up a new server running Ubuntu's Hardy Heron release. lvm version reports:
LVM version: 2.02.26 (2007-06-15)
Library version: 1.02.20 (2007-06-15)
Driver version: 4.12.0
uname -a reports:
Linux sherwood 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00 UTC 2008 i686 GNU/Linux

I initially created an md RAID5 device with only two components (matching 320 GB SATA HDDs). I created a single Physical Volume using the entirety of that md device (320 GB), and then created several Logical Volumes for different filesystems (all ext3). This was done using the Ubuntu installer. After installing I used lvresize to increase the size of a few of the Logical Volumes, as I was conservative regarding the size during installation. These filesystems hold data that is not a critical part of the system (mail, music, video, etc.).

(I also have a similar setup with a couple of IDE drives that I use for the system (/home, /var, /boot, /), but I haven't touched those since the install and they continue to work fine.)

I then copied data from a third matching 320 GB SATA HDD to one of the Logical Volume filesystems on the md device. After freeing up that drive (by relocating its contents to the LVM/MD setup), I added the drive to the md device, which brought the total to 2 active devices and 1 spare device. I then grew the number of devices to 3 and waited for
the reshape to finish (increasing the capacity to 640 GB). I bumped
the values in /proc/sys/dev/raid/ so that I wouldn't have to wait as long.

Now that the reshaping is completed, LVM can't find the physical volume on that device anymore. I tried rebooting the system, but the problem remained. Checking /proc/mdstat shows that the md device is up and healthy. The pvdisplay command only shows my other Physical Volume (for the IDE drives). I found the pvck command and ran that on the md device, and it states that there is no LVM label on the device.

It is my understanding that the steps I outlined should have worked. I planned to follow them with pvresize, then lvresize, then umount, resize2fs, and mount again. I've seen this procedure outlined a few different places, including at http://gentoo-wiki.com/Resize_LVM2_on_RAID5.

Did I do something wrong? Is there anyway to rescue my data? I saved the contents of /etc/lvm/backup/ when I noticed the problem -- perhaps that might help? If there's no way of saving the data, I'd at least like to figure out what happened in the first place.

Thank you. Your thoughtfulness and help is appreciated.

Signature · 1 line
Bob Bell
#1 lore raw

On Tue, 23 Sep 2008, Bob Bell wrote:

Greetings, Bob,

Quoted 6 lines

I'm up a creek, and hoping someone out there can help rescue my data, or at least tell me what went wrong so that I don't have a repeat event. I'm starting with this question on linux-lvm, though let me know if you think the discussion needs to be on linux-raid as well.

I'm setting up a new server running Ubuntu's Hardy Heron release. lvm version reports:

LVM version:     2.02.26 (2007-06-15)
Library version: 1.02.20 (2007-06-15)
Driver version:  4.12.0
Quoted 8 lines

uname -a reports:
Linux sherwood 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00 UTC 2008 i686 GNU/Linux

I initially created an md RAID5 device with only two components (matching 320 GB SATA HDDs). I created a single Physical Volume using the entirety of that md device (320 GB), and then created several Logical Volumes for different filesystems (all ext3). This was done using the Ubuntu installer. After installing I used lvresize to increase the size of a few of the Logical Volumes, as I was conservative regarding the size during installation. These filesystems hold data that is not a critical part of the system (mail, music, video, etc.).

(I also have a similar setup with a couple of IDE drives that I use for the system (/home, /var, /boot, /), but I haven't touched those since the install and they continue to work fine.)

I then copied data from a third matching 320 GB SATA HDD to one of the Logical Volume filesystems on the md device. After freeing up that drive (by relocating its contents to the LVM/MD setup), I added the drive to the md device, which brought the total to 2 active devices and 1 spare device. I then grew the number of devices to 3 and waited for the reshape to finish (increasing the capacity to 640 GB). I bumped the values in /proc/sys/dev/raid/ so that I wouldn't have to wait as long.

My understanding of RAID5 is that once you configure the set of disks to
the array, you can't just add an additional disk to it. Attempting to do
so does more than just "reshape" the array, it changes the logical layout
of the sectors completely. It's no wonder that LVM can't find anything,
since it's data (if it survived the reorg) is no longer where it should
be. And, once you've done this, there's really no going back to recover. This is assuming my understanding of RAID5 is correct.

Quoted 7 lines

Now that the reshaping is completed, LVM can't find the physical volume on that device anymore. I tried rebooting the system, but the problem remained. Checking /proc/mdstat shows that the md device is up and healthy. The pvdisplay command only shows my other Physical Volume (for the IDE drives). I found the pvck command and ran that on the md device, and it states that there is no LVM label on the device.

It is my understanding that the steps I outlined should have worked. I planned to follow them with pvresize, then lvresize, then umount, resize2fs, and mount again. I've seen this procedure outlined a few different places, including at http://gentoo-wiki.com/Resize_LVM2_on_RAID5.

Did I do something wrong? Is there anyway to rescue my data? I saved the contents of /etc/lvm/backup/ when I noticed the problem -- perhaps that might help? If there's no way of saving the data, I'd at least like to figure out what happened in the first place.

Thank you. Your thoughtfulness and help is appreciated.

Signature · 2 lines
Peter A. Castro <[email protected]> or <[email protected]>
	"Cats are just autistic Dogs" -- Dr. Tony Attwood
#2 lore raw

Hi Bob,

Notes interpolated.

Quoted 23 lines · Peter A. Castro

On Tue, 23 Sep 2008, Bob Bell wrote:

Greetings, Bob,

I'm up a creek, and hoping someone out there can help rescue my data, or at

Quoted 15 lines

least tell me what went wrong so that I don't have a repeat event. I'm starting with this question on linux-lvm, though let me know if you think the discussion needs to be on linux-raid as well.

I'm setting up a new server running Ubuntu's Hardy Heron release. lvm version reports:

LVM version:     2.02.26 (2007-06-15)
Library version: 1.02.20 (2007-06-15)
Driver version:  4.12.0

uname -a reports:
Linux sherwood 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00 UTC 2008 i686 GNU/Linux

I initially created an md RAID5 device with only two components (matching 320 GB SATA HDDs).

This is puzzling, since by the usual definition, there is no such thing as a 2-disk RAID5. Your description of its virtual size indicates it is a RAID1, or equivalent.

I created a single Physical Volume using the entirety of that md device

Quoted 7 lines
Quoted 5 lines

(320 GB), and then created several Logical Volumes for different filesystems (all ext3). This was done using the Ubuntu installer. After installing I used lvresize to increase the size of a few of the Logical Volumes, as I was conservative regarding the size during installation. These filesystems hold data that is not a critical part of the system (mail, music, video, etc.).

The LVs live atop the virtual md device so this should not have done anything essentially disturbing.

(I also have a similar setup with a couple of IDE drives that I use for the

Quoted 11 lines
Quoted 9 lines

system (/home, /var, /boot, /), but I haven't touched those since the install and they continue to work fine.)

I then copied data from a third matching 320 GB SATA HDD to one of the Logical Volume filesystems on the md device. After freeing up that drive (by relocating its contents to the LVM/MD setup), I added the drive to the md device, which brought the total to 2 active devices and 1 spare device.
I then grew the number of devices to 3 and waited for the reshape to
finish (increasing the capacity to 640 GB).

If this is a reshape in md, then LVM should have seen nothing (since the 320 GB it knew about should still be there).

I bumped the values in /proc/sys/dev/raid/ so that I wouldn't have to

Quoted 10 lines
Quoted 2 lines

wait as long.

My understanding of RAID5 is that once you configure the set of disks to the array, you can't just add an additional disk to it. Attempting to do so does more than just "reshape" the array, it changes the logical layout of the sectors completely. It's no wonder that LVM can't find anything, since it's data (if it survived the reorg) is no longer where it should be. And, once you've done this, there's really no going back to recover. This is assuming my understanding of RAID5 is correct.

I disagree, because assuming the md reshape operates intuitively, the old 320 GB of data should still be visible as the first 320 GB of data on the new VIRTUAL 640GB device. And LVM should know nothing about the underlying RAIDing, so it should be happy to look there. However, I have no idea what "bumped the values in /proc/sys/dev/raid/" means, or if it causes LVM to look in the wrong places for things.

Larry Dickson
Cutting Edge Networked Storage

Now that the reshaping is completed, LVM can't find the physical volume on

Quoted 29 lines
Quoted 11 lines

that device anymore. I tried rebooting the system, but the problem remained. Checking /proc/mdstat shows that the md device is up and healthy.
The pvdisplay command only shows my other Physical Volume (for the IDE
drives). I found the pvck command and ran that on the md device, and it states that there is no LVM label on the device.

It is my understanding that the steps I outlined should have worked. I planned to follow them with pvresize, then lvresize, then umount, resize2fs, and mount again. I've seen this procedure outlined a few different places, including at http://gentoo-wiki.com/Resize_LVM2_on_RAID5.

Quoted 7 lines

Did I do something wrong? Is there anyway to rescue my data? I saved the contents of /etc/lvm/backup/ when I noticed the problem -- perhaps that might help? If there's no way of saving the data, I'd at least like to figure out what happened in the first place.

Thank you. Your thoughtfulness and help is appreciated.

--
Peter A. Castro <[email protected]> or <[email protected]>
"Cats are just autistic Dogs" -- Dr. Tony Attwood

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#3 lore raw
Quoted 24 lines · Larry Dickson

Hi Bob,

Notes interpolated.

Quoted 19 lines · Peter A. Castro

On Tue, 23 Sep 2008, Bob Bell wrote:

Greetings, Bob,

I'm up a creek, and hoping someone out there can help rescue my data, or at

Quoted 13 lines

least tell me what went wrong so that I don't have a repeat event. I'm starting with this question on linux-lvm, though let me know if you think the discussion needs to be on linux-raid as well.

I'm setting up a new server running Ubuntu's Hardy Heron release. lvm version reports:

LVM version:     2.02.26 (2007-06-15)
Library version: 1.02.20 (2007-06-15)
Driver version:  4.12.0

uname -a reports:
Linux sherwood 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00 UTC 2008 i686 GNU/Linux

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I initially created an md RAID5 device with only two components (matching 320 GB SATA HDDs).

This is puzzling, since by the usual definition, there is no such thing as a 2-disk RAID5. Your description of its virtual size indicates it is a RAID1, or equivalent.

While it's unusual to have only 2 disks in a "RAID5" (usally you want 3 drives min), it is possible, if unreliable (can you say "half-of-a-mirror" ?)

Quoted 10 lines

I created a single Physical Volume using the entirety of that md device

Quoted 7 lines
Quoted 5 lines

(320 GB), and then created several Logical Volumes for different filesystems (all ext3). This was done using the Ubuntu installer. After installing I used lvresize to increase the size of a few of the Logical Volumes, as I was conservative regarding the size during installation. These filesystems hold data that is not a critical part of the system (mail, music, video, etc.).

The LVs live atop the virtual md device so this should not have done anything essentially disturbing.

Yep, I have several VG's with LV's on top a several MDs without any trouble.

Quoted 28 lines

(I also have a similar setup with a couple of IDE drives that I use for the

Quoted 11 lines
Quoted 9 lines

system (/home, /var, /boot, /), but I haven't touched those since the install and they continue to work fine.)

I then copied data from a third matching 320 GB SATA HDD to one of the Logical Volume filesystems on the md device. After freeing up that drive (by relocating its contents to the LVM/MD setup), I added the drive to the md device, which brought the total to 2 active devices and 1 spare device.
I then grew the number of devices to 3 and waited for the reshape to
finish (increasing the capacity to 640 GB).

If this is a reshape in md, then LVM should have seen nothing (since the 320 GB it knew about should still be there).

I bumped the values in /proc/sys/dev/raid/ so that I wouldn't have to

Quoted 10 lines
Quoted 2 lines

wait as long.

My understanding of RAID5 is that once you configure the set of disks to the array, you can't just add an additional disk to it. Attempting to do so does more than just "reshape" the array, it changes the logical layout of the sectors completely. It's no wonder that LVM can't find anything, since it's data (if it survived the reorg) is no longer where it should be. And, once you've done this, there's really no going back to recover. This is assuming my understanding of RAID5 is correct.

Quoted 6 lines

I disagree, because assuming the md reshape operates intuitively, the old 320 GB of data should still be visible as the first 320 GB of data on the new VIRTUAL 640GB device. And LVM should know nothing about the underlying RAIDing, so it should be happy to look there. However, I have no idea what "bumped the values in /proc/sys/dev/raid/" means, or if it causes LVM to look in the wrong places for things.

I don't believe the MD/RAID5 operates in this way. He started with a 2-disk RAID5 (effectively having the data logically interleaved between the two drives), then he added an additional disk. The way RAID works, it doesn't just add the disk space to the end of the array (that would not create a recoverable scenario should it or one of the others fail), it re-integrates and effectively re-interleaves the logical sectors to re-spread the data across all three drives. The trouble is, I don't think you can actually do this with current MD tech (I could be wrong).

Perhaps a better question would be to ask how, exactly, Bob create his array initially (what commands) and what steps he took to add the new drive (again, what commands did he issue). That would give us a better idea of how the array looked before and after.

All of this is orthogonal to LVM, however. I suspect a RAID/MD email list would be a better forum to discuss this.

Quoted 34 lines

Larry Dickson
Cutting Edge Networked Storage

Now that the reshaping is completed, LVM can't find the physical volume on

Quoted 29 lines
Quoted 11 lines

that device anymore. I tried rebooting the system, but the problem remained. Checking /proc/mdstat shows that the md device is up and healthy.
The pvdisplay command only shows my other Physical Volume (for the IDE
drives). I found the pvck command and ran that on the md device, and it states that there is no LVM label on the device.

It is my understanding that the steps I outlined should have worked. I planned to follow them with pvresize, then lvresize, then umount, resize2fs, and mount again. I've seen this procedure outlined a few different places, including at http://gentoo-wiki.com/Resize_LVM2_on_RAID5.

Quoted 7 lines

Did I do something wrong? Is there anyway to rescue my data? I saved the contents of /etc/lvm/backup/ when I noticed the problem -- perhaps that might help? If there's no way of saving the data, I'd at least like to figure out what happened in the first place.

Thank you. Your thoughtfulness and help is appreciated.

--
Peter A. Castro <[email protected]> or <[email protected]>
"Cats are just autistic Dogs" -- Dr. Tony Attwood

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Signature · 2 lines
Peter A. Castro <[email protected]> or <[email protected]>
	"Cats are just autistic Dogs" -- Dr. Tony Attwood

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#4 lore raw
Quoted 44 lines · Peter A. Castro

On Tue, 23 Sep 2008, Larry Dickson wrote:

Hi Bob,

Quoted 33 lines

Notes interpolated.

Quoted 28 lines · Peter A. Castro

On Tue, 23 Sep 2008, Bob Bell wrote:

Greetings, Bob,

I'm up a creek, and hoping someone out there can help rescue my data, or at

Quoted 18 lines

least tell me what went wrong so that I don't have a repeat event. I'm starting with this question on linux-lvm, though let me know if you think
the discussion needs to be on linux-raid as well.

I'm setting up a new server running Ubuntu's Hardy Heron release. lvm version reports:

LVM version:     2.02.26 (2007-06-15)
Library version: 1.02.20 (2007-06-15)
Driver version:  4.12.0

uname -a reports:
Linux sherwood 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00 UTC 2008 i686 GNU/Linux

I initially created an md RAID5 device with only two components (matching
320 GB SATA HDDs).

Quoted 6 lines
Quoted 1 line

This is puzzling, since by the usual definition, there is no such thing

as a
2-disk RAID5. Your description of its virtual size indicates it is a RAID1,
or equivalent.

Quoted 3 lines

While it's unusual to have only 2 disks in a "RAID5" (usally you want 3 drives min), it is possible, if unreliable (can you say "half-of-a-mirror" ?)

If you are meaning a degraded 3-disk RAID5, then he should have had a virtual size of 640GB. He reported 320GB which would be the value expected of a RAID1. You can apply the RAID5 algorithm to 2 disks, but then you just get the equivalent of a RAID1, since a parity sector made so the XOR sum of it and ++ one ++ data sector is zero must equal that data sector.

Larry

I created a single Physical Volume using the entirety of that md device

Quoted 138 lines
Quoted 14 lines
Quoted 14 lines

(320 GB), and then created several Logical Volumes for different

Quoted 10 lines

filesystems
(all ext3). This was done using the Ubuntu installer. After installing I
used lvresize to increase the size of a few of the Logical Volumes, as I was
conservative regarding the size during installation. These filesystems hold
data that is not a critical part of the system (mail, music, video, etc.).

Quoted 3 lines
Quoted 1 line

The LVs live atop the virtual md device so this should not have done

anything essentially disturbing.

Yep, I have several VG's with LV's on top a several MDs without any trouble.

Quoted 36 lines

(I also have a similar setup with a couple of IDE drives that I use for the

Quoted 15 lines

system (/home, /var, /boot, /), but I haven't touched those since the

Quoted 12 lines

install and they continue to work fine.)

I then copied data from a third matching 320 GB SATA HDD to one of the Logical Volume filesystems on the md device. After freeing up that drive
(by relocating its contents to the LVM/MD setup), I added the drive to the
md device, which brought the total to 2 active devices and 1 spare device.
I then grew the number of devices to 3 and waited for the reshape to
finish (increasing the capacity to 640 GB).

Quoted 1 line

If this is a reshape in md, then LVM should have seen nothing (since the

320
GB it knew about should still be there).

I bumped the values in /proc/sys/dev/raid/ so that I wouldn't have to

Quoted 11 lines

wait as long.

Quoted 2 lines

My understanding of RAID5 is that once you configure the set of disks to the array, you can't just add an additional disk to it. Attempting to do so does more than just "reshape" the array, it changes the logical layout of the sectors completely. It's no wonder that LVM can't find anything, since it's data (if it survived the reorg) is no longer where it should be. And, once you've done this, there's really no going back to recover. This is assuming my understanding of RAID5 is correct.

Quoted 7 lines

I disagree, because assuming the md reshape operates intuitively, the old 320 GB of data should still be visible as the first 320 GB of data on the new VIRTUAL 640GB device. And LVM should know nothing about the underlying RAIDing, so it should be happy to look there. However, I have no idea what "bumped the values in /proc/sys/dev/raid/" means, or if it causes LVM to look in the wrong places for things.

I don't believe the MD/RAID5 operates in this way. He started with a 2-disk RAID5 (effectively having the data logically interleaved between the two drives), then he added an additional disk. The way RAID works, it doesn't just add the disk space to the end of the array (that would not create a recoverable scenario should it or one of the others fail), it re-integrates and effectively re-interleaves the logical sectors to re-spread the data across all three drives. The trouble is, I don't think you can actually do this with current MD tech (I could be wrong).

Perhaps a better question would be to ask how, exactly, Bob create his array initially (what commands) and what steps he took to add the new drive (again, what commands did he issue). That would give us a better idea of how the array looked before and after.

All of this is orthogonal to LVM, however. I suspect a RAID/MD email list would be a better forum to discuss this.

Larry Dickson

Quoted 39 lines

Cutting Edge Networked Storage

Now that the reshaping is completed, LVM can't find the physical volume on

Quoted 34 lines

that device anymore. I tried rebooting the system, but the problem

Quoted 13 lines

remained. Checking /proc/mdstat shows that the md device is up and healthy.
The pvdisplay command only shows my other Physical Volume (for the IDE
drives). I found the pvck command and ran that on the md device, and it states that there is no LVM label on the device.

It is my understanding that the steps I outlined should have worked. I planned to follow them with pvresize, then lvresize, then umount, resize2fs,
and mount again. I've seen this procedure outlined a few different places,
including at http://gentoo-wiki.com/Resize_LVM2_on_RAID5.

Quoted 9 lines

Did I do something wrong? Is there anyway to rescue my data? I saved the
contents of /etc/lvm/backup/ when I noticed the problem -- perhaps that might help? If there's no way of saving the data, I'd at least like to figure out what happened in the first place.

Thank you. Your thoughtfulness and help is appreciated.

--
Peter A. Castro <[email protected]> or <[email protected]>
"Cats are just autistic Dogs" -- Dr. Tony Attwood

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

--
Peter A. Castro <[email protected]> or <[email protected]>
"Cats are just autistic Dogs" -- Dr. Tony Attwood

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#5 lore raw
In reply to Peter A. Castro
Quoted 24 lines · Peter A. Castro
Quoted 8 lines
Quoted 7 lines

My understanding of RAID5 is that once you configure the set of disks to the array, you can't just add an additional disk to it. Attempting to do so does more than just "reshape" the array, it changes the logical layout of the sectors completely. It's no wonder that LVM can't find anything, since it's data (if it survived the reorg) is no longer where it should be. And, once you've done this, there's really no going back to recover. This is assuming my understanding of RAID5 is correct.

Quoted 6 lines

I disagree, because assuming the md reshape operates intuitively, the old 320 GB of data should still be visible as the first 320 GB of data on the new VIRTUAL 640GB device. And LVM should know nothing about the underlying RAIDing, so it should be happy to look there. However, I have no idea what "bumped the values in /proc/sys/dev/raid/" means, or if it causes LVM to look in the wrong places for things.

I don't believe the MD/RAID5 operates in this way. He started with a 2-disk RAID5 (effectively having the data logically interleaved between the two drives), then he added an additional disk. The way RAID works, it doesn't just add the disk space to the end of the array (that would not create a recoverable scenario should it or one of the others fail), it re-integrates and effectively re-interleaves the logical sectors to re-spread the data across all three drives. The trouble is, I don't think you can actually do this with current MD tech (I could be wrong).

Quoted 8 lines

Perhaps a better question would be to ask how, exactly, Bob create his array initially (what commands) and what steps he took to add the new drive (again, what commands did he issue). That would give us a better idea of how the array looked before and after.

All of this is orthogonal to LVM, however. I suspect a RAID/MD email list would be a better forum to discuss this.

Current linux MD-raid can reshape raid-5 arrays, the operation just takes really long time and there is a critical section at the start, which means if your computer crashes during the first second or two at the start all data is pretty much lost.

The raid-5 reshape is quite new feature in md-raid and I don't know how stable it is.