Strange pvmove problem
Hi!
I'm trying to move PEs away from two PVs (because I want to remove the disks from the system), but pvmove always fails:
# pvmove /dev/hdd1 /dev/sda3
Wiping cache of LVM-capable devices
Finding volume group "LVM"
Archiving volume group "LVM" metadata (seqno 51).
Creating logical volume pvmove0
Moving 3468 extents of logical volume LVM/data
Found volume group "LVM"
Updating volume group metadata
Creating volume group backup "/etc/lvm/backup/LVM" (seqno 52).
Found volume group "LVM"
Found volume group "LVM"
Suspending LVM-data (253:1)
Found volume group "LVM"
Creating LVM-pvmove0
Loading LVM-pvmove0 table
device-mapper: reload ioctl failed: Invalid argument
ABORTING: Temporary mirror activation failed. Run pvmove --abort.
Found volume group "LVM"
Loading LVM-pvmove0 table
device-mapper: reload ioctl failed: Invalid argument
Loading LVM-data table
device-mapper: reload ioctl failed: Invalid argumentAfter that the system remains in an unstable state. All [vg|pv|lv]* commands hang and the LV becomes unaccessible too.
After rebooting and excecuting "pvmove" without any arguments the VG becomes accessible again and I'm back where I started (without any PE moved). Trying to move data from another PV makes no difference.
The system's running with Debian Testing and Kernel 2.6.16.16.
Can anyone please help me finding out whats causing this?
ciao
Dariush
# pvdisplay File descriptor 3 left open File descriptor 4 left open File descriptor 6 left open File descriptor 8 left open --- Physical volume --- PV Name /dev/hda3 VG Name LVM PV Size 129.48 GB / not usable 0 Allocatable yes PE Size (KByte) 4096 Total PE 33147 Free PE 33147 Allocated PE 0 PV UUID uc4kn4-xMQ7-M4vw-1thl-OcCF-LIV9-aeRTn0
--- Physical volume ---
PV Name /dev/hdd1 VG Name LVM PV Size 37.27 GB / not usable 0 Allocatable yes PE Size (KByte) 4096 Total PE 9541 Free PE 6073 Allocated PE 3468 PV UUID tNzm5d-ClTt-0vJ6-ZQA9-0eYQ-IwKy-tbxV6o
--- Physical volume ---
PV Name /dev/sda3 VG Name LVM PV Size 214.19 GB / not usable 0 Allocatable yes PE Size (KByte) 4096 Total PE 54833 Free PE 54734 Allocated PE 99 PV UUID CfA2kz-5Gt8-Zshc-3Eou-fscj-1Lfn-xOEQSY
--- Physical volume ---
PV Name /dev/sdb1 VG Name LVM PV Size 232.88 GB / not usable 0 Allocatable yes (but full) PE Size (KByte) 4096 Total PE 59618 Free PE 0 Allocated PE 59618 PV UUID L6S26w-zr8S-t0RN-si42-zWEd-rJA5-Ms0JIb
# vgdisplay File descriptor 3 left open File descriptor 4 left open File descriptor 6 left open File descriptor 8 left open --- Volume group --- VG Name LVM System ID Format lvm2 Metadata Areas 4 Metadata Sequence No 55 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 4 Act PV 4 VG Size 613.82 GB PE Size 4.00 MB Total PE 157139 Alloc PE / Size 63185 / 246.82 GB Free PE / Size 93954 / 367.01 GB VG UUID GoVzA8-HPSg-FNhr-6fM3-Fo94-k2IG-462F9E
# lvdisplay File descriptor 3 left open File descriptor 4 left open File descriptor 6 left open File descriptor 8 left open --- Logical volume --- LV Name /dev/LVM/data VG Name LVM LV UUID zscrIa-BwvW-rktv-3L5P-QP67-BkfG-ri3NZL LV Write Access read/write LV Status available # open 1 LV Size 246.82 GB Current LE 63185 Segments 3 Allocation inherit Read ahead sectors 0 Block device 253:1
Quoted 2 lines · Dariush Forouher
I'm trying to move PEs away from two PVs (because I want to remove the disks from the system), but pvmove always fails:
I had some problems with pvmove some time ago, if the PV to empty contained parts from several different LVs. But this was due to some old bug with a 2.6.12 or 13 kernel. May be it's interesting to have a look into /etc/lvm/backup/LVM and the associated backups, to understand whats going on.
To overcome the problem of hanging LVM I shut down my server and performed a "pvmove --abort" in single user mode while the LVM was not activated. This restored it to the state before.
Dieter.
Quoted 9 lines · Dieter St�ken
Quoted 2 lines · Dariush Forouher
I'm trying to move PEs away from two PVs (because I want to remove the disks from the system), but pvmove always fails:
I had some problems with pvmove some time ago, if the PV to empty contained parts from several different LVs. But this was due to some old bug with a 2.6.12 or 13 kernel. May be it's interesting to have a look into
/etc/lvm/backup/LVMand the associated backups, to understand whats going on.
Uhm, this is the file created right before I started pvmove:
<SNIP>
# Generated by LVM2: Fri Jul 28 10:15:24 2006
contents = "Text Format Volume Group"
version = 1
description = "Created *before* executing 'pvmove -v /dev/hdd1 /dev/hda3'"
creation_host = "palomar" # Linux palomar 2.6.16.16 #2 PREEMPT Sun
May 14 13:56:04 CEST 2006 i686
creation_time = 1154074524 # Fri Jul 28 10:15:24 2006
LVM {
id = "GoVzA8-HPSg-FNhr-6fM3-Fo94-k2IG-462F9E"
seqno = 39
status = ["RESIZEABLE", "READ", "WRITE"]
extent_size = 8192 # 4 Megabytes
max_lv = 0
max_pv = 0
physical_volumes {
pv0 {
id = "uc4kn4-xMQ7-M4vw-1thl-OcCF-LIV9-aeRTn0"
device = "/dev/hda3" # Hint only
status = ["ALLOCATABLE"]
pe_start = 384
pe_count = 33147 # 129.48 Gigabytes
}
pv1 {
id = "tNzm5d-ClTt-0vJ6-ZQA9-0eYQ-IwKy-tbxV6o"
device = "/dev/hdd1" # Hint only
status = ["ALLOCATABLE"]
pe_start = 384
pe_count = 9541 # 37.2695 Gigabytes
}
pv2 {
id = "CfA2kz-5Gt8-Zshc-3Eou-fscj-1Lfn-xOEQSY"
device = "/dev/sda3" # Hint only
status = ["ALLOCATABLE"]
pe_start = 384
pe_count = 54833 # 214.191 Gigabytes
}
pv3 {
id = "L6S26w-zr8S-t0RN-si42-zWEd-rJA5-Ms0JIb"
device = "/dev/sdb1" # Hint only
status = ["ALLOCATABLE"]
pe_start = 384
pe_count = 59618 # 232.883 Gigabytes
}
}
logical_volumes {
data {
id = "zscrIa-BwvW-rktv-3L5P-QP67-BkfG-ri3NZL"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 3
segment1 {
start_extent = 0
extent_count = 59618 # 232.883 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv3", 0
]
}
segment2 {
start_extent = 59618
extent_count = 3468 # 13.5469 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv1", 6073
]
}
segment3 {
start_extent = 63086
extent_count = 99 # 396 Megabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv2", 54734
]
}
}
}
}</SNIP>
I diff'ed it againt more recent backups and against /etc/lvm/backup/LVM. They are pretty much the same (except the "seqno" attribute).
I'd be already happy if I could free /dev/sda3 (which is _nearly_ unused). Then I could create a new VG and move the files around by hand (like in the old days, when we had to use partitions). But even thats not possible since all disks big enough to hold the data are bound by this (damaged?) VG.
ciao
Dariush
Quoted 1 line · Dariush Forouher
Uhm, this is the file created right before I started pvmove:
....
seems the pvmove was not even started because of a problems with the underlaying device-mapper:
Quoted 1 line
device-mapper: reload ioctl failed: Invalid argument
sorry, no idea :-(
any messages from syslog / dmesg?
does it work while LV not active (i.E. boot from rescue CD)?
Dieter.
Quoted 1 line · Dieter St�ken
any messages from syslog / dmesg?
Jupp, these:
device-mapper: unrecognised sync argument to mirror log: 2
device-mapper: dm-mirror: Error creating mirror dirty log
device-mapper: error adding target to table
device-mapper: unrecognised sync argument to mirror log: 2
device-mapper: dm-mirror: Error creating mirror dirty log
device-mapper: error adding target to table
device-mapper: device 253:1 too small for target
device-mapper: error adding target to table
Quoted 1 line
does it work while LV not active (i.E. boot from rescue CD)?
Nope. :(
ciao
Dariush
I was able to workaround the problem by creating a new LV (which I placed exactly on those PVs which I intended to keep) and then moving all files to the the new LV by hand. This way I didn't have to use pvmove.
Anyway, thanks for your help!
ciao
Dariush
Quoted 21 lines · Dieter St�ken
Quoted 1 line · Dariush Forouher
Uhm, this is the file created right before I started pvmove:
....
seems the pvmove was not even started because of a problems with the underlaying device-mapper:
Quoted 1 line
device-mapper: reload ioctl failed: Invalid argument
sorry, no idea :-(
any messages from syslog / dmesg?
does it work while LV not active (i.E. boot from rescue CD)?Dieter.
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
You've got a kernel/userspace version mismatch on device-mapper and lvm2.
Check your config an upgrade.
Heinz
Quoted 19 lines · Dariush Forouher
Quoted 1 line · Dieter St�ken
any messages from syslog / dmesg?
Jupp, these:
device-mapper: unrecognised sync argument to mirror log: 2
device-mapper: dm-mirror: Error creating mirror dirty log
device-mapper: error adding target to table
device-mapper: unrecognised sync argument to mirror log: 2
device-mapper: dm-mirror: Error creating mirror dirty log
device-mapper: error adding target to table
device-mapper: device 253:1 too small for target
device-mapper: error adding target to tableQuoted 1 line
does it work while LV not active (i.E. boot from rescue CD)?
Nope. :(
Quoted 8 lines
ciao
Dariush_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Red Hat GmbH
Consulting Development Engineer Am Sonnenhang 11
Storage Development 56242 Marienrachdorf
Germany
[email protected] PHONE +49 171 7803392
FAX +49 2626 924446=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Quoted 4 lines · Heinz Mauelshagen
You've got a kernel/userspace version mismatch on device-mapper and lvm2.
Check your config an upgrade.
Ah, I thought with using Debian Testing I would have been using bleeding edge versions allready. Obviously I was wrong.
Thanks for the hint!
ciao
Dariush