removing a bad HD
Hi.
Recently a harddrive in my LVM started giving me read-errors, so naturally I wanted to replace the drive with a new one. Now I've got the replacement disk, and I've moved nearly all the PE's with pvmove.
But, there is one PE that it can't move
--8<--
# pvmove -v /dev/hdk:646 [...] pvmove -- /dev/hdk [PE 646 [lv0 [LE 15369]] -> /dev/hdg [PE 17557] [1/1] /dev/vg0/group::/dev/vg0/lv0: 3900 10600832, 2200 287670784 pvmove -- ERROR "Input/output error" copying extent from "/dev/hdk"
pvmove -- ERROR "Input/output error" moving physical extents --8<--
I've also tried with -i, but it had noe effect.
During the move-attempt, the syslog gets a few of these messages:
--8<--
Oct 4 21:13:03 2h0qoj kernel: hdk: read_intr: status=0x59 { DriveReady SeekComplete DataRequest Error }
Oct 4 21:13:03 2h0qoj kernel: hdk: read_intr: error=0x40 { UncorrectableError }, LBAsect=10609463, sector=10609456
Oct 4 21:13:03 2h0qoj kernel: end_request: I/O error, dev 39:00 (hdk), sector 10609456--8<--
Anyway, because of this, vgreduce won't let me remove the drive from the LVM
--8<--
# vgreduce -v vg0 /dev/hdk vgreduce -- locking logical volume manager vgreduce -- checking volume group name "vg0" vgreduce -- checking volume group "vg0" existence vgreduce -- checking volume group "vg0" activity vgreduce -- reading data of volume group "vg0" from disk(s) vgreduce -- reducing VGDA structures of volume group "vg0" vgreduce -- ERROR: can't reduce volume group "vg0" by used physical volume "/dev/hdk" --8<--
So, uhm. Now what? What will happen if I just turn off the coputer and remove the drive? Will I be able to use the rest of the LVM without losing anything other than the 8MB in the still active PE from the faulty drive?
Oh, and how do I determine which files will be affected? Is there any way to find out which inodes that reside on the faulty LE?
I suspect that this is sort of a FAQ, and I apologise if I missed an obvious entry in the HOWTO or the FAQ, but my searches have been unsuccessful. The closest I've come is instructions on how to use the device-mapper, which I understand is a LVM2 feature. I'm running LVM 1.0.3
more info on the drive and the volume group:
--8<--
# pvdisplay /dev/hdk --- Physical volume --- PV Name /dev/hdk VG Name vg0 PV Size 74.53 GB [156301488 secs] / NOT usable 8.19 MB [LVM: 165 KB] PV# 2 PV Status available Allocatable yes Cur LV 1 PE Size (KByte) 8192 Total PE 9538 Free PE 9537 Allocated PE 1 PV UUID T3DIq4-sVkG-mM9j-gxJp-pjsa-U00w-X12d3w
# vgdisplay vg0 --- Volume group --- VG Name vg0 VG Access read/write VG Status available/resizable VG # 0 MAX LV 256 Cur LV 2 Open LV 0 MAX LV Size 511.98 GB Max PV 256 Cur PV 5 Act PV 5 VG Size 568.46 GB PE Size 8 MB Total PE 72763 Alloc PE / Size 61243 / 478.46 GB Free PE / Size 11520 / 90 GB VG UUID VIwKSd-wcyb-huBo-fUPR-bmp0-XRR5-DH2Y7q --8<--
Signature · 1 line
Trond Michelsen
Quoted 2 lines · Trond Michelsen
Recently a harddrive in my LVM started giving me read-errors, so naturally I wanted to replace the drive with a new one.
I recently went through this (with the added complication of some bad metadata) and you have a few options.
The best one may be to use dd (with NOERROR) to raw copy the contents from the old disk to a new one of the same size and then use the new one when doing the pvmove.
Another path is to upgrade to LVM2 (you may have to do this anyway, I needed some functionality in the LVM2 pvcreate to pull my stunt off) and use the "partial" feature. I haven't tried this myself.
There are more details of my tribulations just a few weeks back in the list archives, but feel free to ask if there's something missing (I had some private conversations with Heinz on the topic as well).
/ Rickard Olsson,IT-Konsult/
/ Telefon: +46 70 635 01 42/
/ http://www.webhackande.se/
Quoted 12 lines · Rickard Olsson
Quoted 2 lines · Trond Michelsen
Recently a harddrive in my LVM started giving me read-errors, so naturally I wanted to replace the drive with a new one.
I recently went through this (with the added complication of some bad metadata) and you have a few options.
The best one may be to use dd (with NOERROR) to raw copy the contents from the old disk to a new one of the same size and then use the new one when doing the pvmove.
Quoted 7 lines
Another path is to upgrade to LVM2 (you may have to do this anyway, I needed some functionality in the LVM2 pvcreate to pull my stunt off) and use the "partial" feature. I haven't tried this myself.
There are more details of my tribulations just a few weeks back in the list archives, but feel free to ask if there's something missing (I had some private conversations with Heinz on the topic as well).
Thanks, but is it really necessary to copy the PE at all? It's broken, and I don't care about that 8MB of data. Isn't it possible to just reassign the LE that's mapped to the broken PE to a completely empty PE at some other disk?
At the moment I'm a bit tempted to just set up a new VG, and migrate data from the old VG file by file, disk by disk.
Signature · 2 lines
// Trond Michelsen \X/ [email protected]
Quoted 4 lines · Trond Michelsen
Thanks, but is it really necessary to copy the PE at all? It's broken, and I don't care about that 8MB of data. Isn't it possible to just reassign the LE that's mapped to the broken PE to a completely empty PE at some other disk?
It's possible that Heinz or one of the other gurus know of a way, but I don't. If pvmove -i can't get past the read errors... Theoretically, you could be able to hexedit the LVM metadata but I really don't want to read about that in the news afterwards, if you know what I mean.
Quoted 2 lines
At the moment I'm a bit tempted to just set up a new VG, and migrate data from the old VG file by file, disk by disk.
If you have a spare disk to start the new VG with, that's probably a good idea for several reasons (fresh filesystem, fresh metadata, the ability to check disks offline before adding them to the VG and so on).
I did this too since my old extents were rapidly becoming too small. New LVs default to a max size of 2TB, when I originally created mine I believe they were 250GB unless specified otherwise.
/ Rickard Olsson,IT-Konsult/
/ Telefon: +46 70 635 01 42/
/ http://www.webhackande.se/
Quoted 9 lines · Rickard Olsson
Quoted 2 lines
At the moment I'm a bit tempted to just set up a new VG, and migrate data from the old VG file by file, disk by disk.
If you have a spare disk to start the new VG with, that's probably a good idea for several reasons (fresh filesystem, fresh metadata, the ability to check disks offline before adding them to the VG and so on).
I did this too since my old extents were rapidly becoming too small. New LVs default to a max size of 2TB, when I originally created mine I believe they were 250GB unless specified otherwise.
Yeah, it occured to me that this was probably the best opportunity I had to increase the PE-size. So I set it to 32MB, giving me a maximum LV-size of 2TB.
BTW: What are the effects of choosing a large PE-size? Apart from getting larger LV's, of course? Does it effect performance or memory usage at all?
Signature · 1 line
Trond Michelsen
Quoted 3 lines · Trond Michelsen
Yeah, it occured to me that this was probably the best opportunity I had to increase the PE-size. So I set it to 32MB, giving me a maximum LV-size of 2TB.
"That should be enough for anyone." ;-)
Quoted 3 lines
BTW: What are the effects of choosing a large PE-size? Apart from getting larger LV's, of course? Does it effect performance or memory usage at all?
None that I have noticed. Then again, my main fileserver gets something like 1% load average and 10% mem usage (unless I'm shuffling large files locally, the gigabit adapter and Samba's buffers eat both up pretty quickly). Pulling notions out of my ass, I'd guess that larger PE-sizes are better now than they were before since most disks have larger buffers, our boxes have more RAM and Moore's law is seldom broken.
Heinz, is this the reasoning behind upping the default PE size or were there other considerations?
/ Rickard Olsson,IT-Konsult/
/ Telefon: +46 70 635 01 42/
/ http://www.webhackande.se/