activation mode=degraded doesn't work with thin pool
Hi!
I just tested lv activation with a degraded raid1 thin pool. Unfortunately it looks like activation mode=degraded only works for plain raid1 lvs. If you add a thin pool, lvm won't activate it in degraded mode. (Unless you specify --activationmode partial, which is IMHO rather dangerous.)
Is this the intended behavior?
-------------------- 8< --------------------
# lvcreate --type raid1 -m 1 -n thinmeta -L100m vg1 /dev/sda2 /dev/sdb2 Logical volume "thinmeta" created. # lvcreate --type raid1 -m 1 -n Thin -L2g vg1 /dev/sda2 /dev/sdb2 Logical volume "Thin" created. # lvconvert -y --type thin-pool --poolmetadata vg1/thinmeta vg1/Thin Thin pool volume with chunk size 64.00 KiB can address@most 15.81 TiB of data. WARNING: Converting logical volume vg1/Thin and vg1/thinmeta to thin pool's data and metadata volumes with metadata wiping. THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.) Converted vg1/Thin_tdata to thin pool.
# vgchange -a n vg1 0 logical volume(s) in volume group "vg1" now active
### filter one leg of the raid: global_filter = [ "r|^/dev/sdb2|" ]
# systemctl restart lvm2-lvmetad.service # pvscan --cache
# vgchange -a y WARNING: Device for PV WzDZS1-Hj9B-RZCa-uhOc-pY9L-1rLN-GB5Zs1 not found or rejected by a filter. Refusing activation of partial LV vg1/Thin. Use '--activationmode partial' to override. 0 logical volume(s) in volume group "vg1" now active -------------------- 8< --------------------
(--activationmode partial works)
I am using lvm2-2.02.171-8.el7.x86_64.
Cheers,
--leo
Signature · 3 lines
e-mail ::: Leo.Bergolth (at) wu.ac.at fax ::: +43-1-31336-906050 location ::: IT-Services | Vienna University of Economics | Austria
[linux-lvm] unable to recover from degraded raid1 with thin pool (was: activation mode=degraded doesn't work with thin pool)
Quoted 5 lines · Alexander 'Leo' Bergolth
I just tested lv activation with a degraded raid1 thin pool. Unfortunately it looks like activation mode=degraded only works for plain raid1 lvs. If you add a thin pool, lvm won't activate it in degraded mode. (Unless you specify --activationmode partial, which is IMHO rather dangerous.)
Unfortunately I cannot even replace a faulty PV if a thin pool is present.
lvm thinks that the thin pool is a partial (not just degraded) LV, so removing the missing PV would also remove the thin pool! (see below)
Is there any way to work around this problem?
Cheers,
--leo
P.S.: I filed a bugreport for this:
https://bugzilla.redhat.com/show_bug.cgi?id=1504044
-------------------- 8< --------------------
# pvcreate /dev/vdb Physical volume "/dev/vdb" successfully created. # pvcreate /dev/vdc Physical volume "/dev/vdc" successfully created. # vgcreate vg_test /dev/vdb /dev/vdc Volume group "vg_test" successfully created # lvcreate --type raid1 -m 1 -n thinmeta -L100m vg_test /dev/vdb /dev/vdc Logical volume "thinmeta" created. # lvcreate --type raid1 -m 1 -n Thin -L2g vg_test /dev/vdb /dev/vdc Logical volume "Thin" created. # lvconvert -y --type thin-pool --poolmetadata vg_test/thinmeta vg_test/Thin Thin pool volume with chunk size 64.00 KiB can address@most 15.81 TiB of data. WARNING: Converting logical volume vg_test/Thin and vg_test/thinmeta to thin pool's data and metadata volumes with metadata wiping. THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.) Converted vg_test/Thin_tdata to thin pool.
# vgchange -a n vg_test 0 logical volume(s) in volume group "vg_test" now active
### add global_filter = [ "r|^/dev/vdc$|" ] to lvm.conf
# systemctl restart lvm2-lvmetad.service # pvscan --cache
# pvs WARNING: Device for PV na0HS1-ZcQt-bnt0-tFfB-Zl20-irIg-FffGxH not found or rejected by a filter. PV VG Fmt Attr PSize PFree /dev/vda2 vg_sys lvm2 a-- <12.00g <2.00g /dev/vdb vg_test lvm2 a-- <12.00g 9.79g [unknown] vg_test lvm2 a-m <12.00g 9.89g
# vgchange -a y vg_test WARNING: Device for PV na0HS1-ZcQt-bnt0-tFfB-Zl20-irIg-FffGxH not found or rejected by a filter. Refusing activation of partial LV vg_test/Thin. Use '--activationmode partial' to override. 0 logical volume(s) in volume group "vg_test" now active
# vgchange -a y --activationmode=partial vg_test PARTIAL MODE. Incomplete logical volumes will be processed. WARNING: Device for PV na0HS1-ZcQt-bnt0-tFfB-Zl20-irIg-FffGxH not found or rejected by a filter. 1 logical volume(s) in volume group "vg_test" now active
### removing the missing PV would also remove my thin pool!
# vgreduce --removemissing vg_test WARNING: Device for PV na0HS1-ZcQt-bnt0-tFfB-Zl20-irIg-FffGxH not found or rejected by a filter. WARNING: Partial LV Thin needs to be repaired or removed. WARNING: Partial LV Thin_tmeta needs to be repaired or removed. WARNING: Partial LV Thin_tdata needs to be repaired or removed. WARNING: Partial LV Thin_tmeta_rimage_1 needs to be repaired or removed. WARNING: Partial LV Thin_tmeta_rmeta_1 needs to be repaired or removed. WARNING: Partial LV Thin_tdata_rimage_1 needs to be repaired or removed. WARNING: Partial LV Thin_tdata_rmeta_1 needs to be repaired or removed. There are still partial LVs in VG vg_test. To remove them unconditionally use: vgreduce --removemissing --force. WARNING: Proceeding to remove empty missing PVs.
-------------------- 8< --------------------
Quoted 34 lines
Is this the intended behavior?
-------------------- 8< --------------------
# lvcreate --type raid1 -m 1 -n thinmeta -L100m vg1 /dev/sda2 /dev/sdb2 Logical volume "thinmeta" created. # lvcreate --type raid1 -m 1 -n Thin -L2g vg1 /dev/sda2 /dev/sdb2 Logical volume "Thin" created. # lvconvert -y --type thin-pool --poolmetadata vg1/thinmeta vg1/Thin Thin pool volume with chunk size 64.00 KiB can address at most 15.81 TiB of data. WARNING: Converting logical volume vg1/Thin and vg1/thinmeta to thin pool's data and metadata volumes with metadata wiping. THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.) Converted vg1/Thin_tdata to thin pool.# vgchange -a n vg1 0 logical volume(s) in volume group "vg1" now active### filter one leg of the raid:
global_filter= [ "r|^/dev/sdb2|" ]# systemctl restart lvm2-lvmetad.service # pvscan --cache# vgchange -a y WARNING: Device for PV WzDZS1-Hj9B-RZCa-uhOc-pY9L-1rLN-GB5Zs1 not found or rejected by a filter. Refusing activation of partial LV vg1/Thin. Use '--activationmode partial' to override. 0 logical volume(s) in volume group "vg1" now active -------------------- 8< --------------------(--activationmode partial works)
I am using lvm2-2.02.171-8.el7.x86_64.
Cheers,
--leo
Signature · 3 lines
e-mail ::: Leo.Bergolth (at) wu.ac.at fax ::: +43-1-31336-906050 location ::: IT-Services | Vienna University of Economics | Austria
Re: [linux-lvm] unable to recover from degraded raid1 with thin pool (was: activation mode=degraded doesn't work with thin pool)
Il 19-10-2017 13:45 Alexander 'Leo' Bergolth ha scritto:
Quoted 8 lines · Alexander 'Leo' Bergolth
Quoted 5 lines
I just tested lv activation with a degraded raid1 thin pool. Unfortunately it looks like activation mode=degraded only works for plain raid1 lvs. If you add a thin pool, lvm won't activate it in degraded mode. (Unless you specify --activationmode partial, which is IMHO rather dangerous.)
Unfortunately I cannot even replace a faulty PV if a thin pool is present.
From you command history, I can see you are trying to remove the failed device (to re-add a good device later). Instead, can you try with lvconvert --repair or lvconvert --replace? Do they works?
Thanks.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Re: [linux-lvm] unable to recover from degraded raid1 with thin pool
Quoted 8 lines · Gionatan Danti
Il 19-10-2017 13:45 Alexander 'Leo' Bergolth ha scritto:
Quoted 7 lines · Alexander 'Leo' Bergolth
Quoted 5 lines
I just tested lv activation with a degraded raid1 thin pool. Unfortunately it looks like activation mode=degraded only works for plain raid1 lvs. If you add a thin pool, lvm won't activate it in degraded mode. (Unless you specify --activationmode partial, which is IMHO rather dangerous.)
Quoted 7 lines
Quoted 2 lines
Unfortunately I cannot even replace a faulty PV if a thin pool is present.
From you command history, I can see you are trying to remove the failed device (to re-add a good device later).
Instead, can you try with lvconvert --repair or lvconvert --replace? Do they works?
lvconvert --repair doesn't work because there is also a linear LV ("boot") on this VG, which already has the error target assigned instead of the missing PV:
-------------------- 8< --------------------
# lvconvert --repair /dev/vg_test/boot /dev/vdd WARNING: Disabling lvmetad cache for repair command. WARNING: Not using lvmetad because of repair. Couldn't find device with uuid 5ORr1v-kn3T-ROJW-quEg-0Lp5-3kdX-M3jr8S. Attempt to replace failed RAID images (requires full device resync)? [y/n]: y vg_test/boot has components with error targets that must be removed first: vg_test/boot_rimage_1. Try removing the PV list and rerun. the command. Failed to remove the specified images from vg_test/boot. Failed to replace faulty devices in vg_test/boot. -------------------- 8< --------------------
(But removing the missing PV doesn't work because of the Thin volume which is considered as partial as described previously.)
lvconvert --replace also refuses to process bcause of the missing PV:
-------------------- 8< --------------------
# lvconvert --replace 5ORr1v-kn3T-ROJW-quEg-0Lp5-3kdX-M3jr8S vg_test/Thin_tdata /dev/vdd Couldn't find device with uuid 5ORr1v-kn3T-ROJW-quEg-0Lp5-3kdX-M3jr8S. Cannot change VG vg_test while PVs are missing. Consider vgreduce --removemissing. Cannot process volume group vg_test -------------------- 8< --------------------
--leo
Signature · 3 lines
e-mail ::: Leo.Bergolth (at) wu.ac.at fax ::: +43-1-31336-906050 location ::: IT-Services | Vienna University of Economics | Austria
Re: [linux-lvm] unable to recover from degraded raid1 with thin pool
In reply to Alexander 'Leo' BergolthQuoted 8 lines · Alexander 'Leo' Bergolth
Quoted 5 lines · Alexander 'Leo' Bergolth
I just tested lv activation with a degraded raid1 thin pool. Unfortunately it looks like activation mode=degraded only works for plain raid1 lvs. If you add a thin pool, lvm won't activate it in degraded mode. (Unless you specify --activationmode partial, which is IMHO rather dangerous.)
Unfortunately I cannot even replace a faulty PV if a thin pool is present.
FYI: I managed to remove the missing PV using a rather dirty workaround:
I replaced the missing devices with the error target using dmsetup and edited and restored the VG config:
-------------------- 8< --------------------
### starting point:
### one PV is missing (one leg of the boot LV and of the Thin-subvolumes)
# lvs -a -olv_name,lv_attr,segtype,devices vg_test Couldn't find device with uuid cUHx7d-Gl8O-TX2l-mi5B-QCx6-so00-cS0M4z. LV Attr Type Devices Thin twi-aotzp- thin-pool Thin_tdata(0) [Thin_tdata] rwi-aor-p- raid1 Thin_tdata_rimage_0(0),Thin_tdata_rimage_1(0) [Thin_tdata_rimage_0] iwi-aor--- linear /dev/vdb(53) [Thin_tdata_rimage_1] Iwi-aor-p- linear [unknown](53) [Thin_tdata_rmeta_0] ewi-aor--- linear /dev/vdb(52) [Thin_tdata_rmeta_1] ewi-aor-p- linear [unknown](52) [Thin_tmeta] ewi-aor-p- raid1 Thin_tmeta_rimage_0(0),Thin_tmeta_rimage_1(0) [Thin_tmeta_rimage_0] iwi-aor--- linear /dev/vdb(27) [Thin_tmeta_rimage_1] Iwi-aor-p- linear [unknown](27) [Thin_tmeta_rmeta_0] ewi-aor--- linear /dev/vdb(26) [Thin_tmeta_rmeta_1] ewi-aor-p- linear [unknown](26) boot rwi-a-r-p- raid1 boot_rimage_0(0),boot_rimage_1(0) [boot_rimage_0] iwi-aor--- linear /dev/vdb(1) [boot_rimage_1] Iwi-aor-p- linear [unknown](1) [boot_rmeta_0] ewi-aor--- linear /dev/vdb(0) [boot_rmeta_1] ewi-aor-p- linear [unknown](0) [lvol0_pmspare] ewi-----p- linear [unknown](2613) thintest Vwi-aotzp- thin
# vgreduce --removemissing vg_test --force Couldn't find device with uuid cUHx7d-Gl8O-TX2l-mi5B-QCx6-so00-cS0M4z. WARNING: Removing partial LV vg_test/Thin. Logical volume vg_test/thintest in use.
### the Thin subvolumes still have the missing PV attached:
# lvs -a -olv_name,lv_attr,segtype,devices vg_test Couldn't find device with uuid cUHx7d-Gl8O-TX2l-mi5B-QCx6-so00-cS0M4z. LV Attr Type Devices Thin twi-aotzp- thin-pool Thin_tdata(0) [Thin_tdata] rwi-aor-p- raid1 Thin_tdata_rimage_0(0),Thin_tdata_rimage_1(0) [Thin_tdata_rimage_0] iwi-aor--- linear /dev/vdb(53) [Thin_tdata_rimage_1] Iwi-aor-p- linear [unknown](53) [Thin_tdata_rmeta_0] ewi-aor--- linear /dev/vdb(52) [Thin_tdata_rmeta_1] ewi-aor-p- linear [unknown](52) [Thin_tmeta] ewi-aor-p- raid1 Thin_tmeta_rimage_0(0),Thin_tmeta_rimage_1(0) [Thin_tmeta_rimage_0] iwi-aor--- linear /dev/vdb(27) [Thin_tmeta_rimage_1] Iwi-aor-p- linear [unknown](27) [Thin_tmeta_rmeta_0] ewi-aor--- linear /dev/vdb(26) [Thin_tmeta_rmeta_1] ewi-aor-p- linear [unknown](26) boot rwi-a-r-r- raid1 boot_rimage_0(0),boot_rimage_1(0) [boot_rimage_0] iwi-aor--- linear /dev/vdb(1) [boot_rimage_1] vwi-aor-r- error [boot_rmeta_0] ewi-aor--- linear /dev/vdb(0) [boot_rmeta_1] ewi-aor-r- error [lvol0_pmspare] ewi-----p- linear [unknown](2613) thintest Vwi-aotzp- thin
### create and execute the dmsetup commands used to replace it:
vg="vg_test"
devs="Thin_tdata_rimage_1 Thin_tdata_rmeta_1 Thin_tmeta_rimage_1 Thin_tmeta_rmeta_1" for d in $devs; do
d2="$vg-$d"
t1="$(dmsetup table "$d2")"
t2="${t1/ linear */ error}"
echo dmsetup reload "$d2" --table \""$t2"\"
echo dmsetup suspend --noflush "$d2"
echo dmsetup resume "$d2"
done
dmsetup reload vg_test-Thin_tdata_rimage_1 --table "0 20971520 error" dmsetup suspend --noflush vg_test-Thin_tdata_rimage_1 dmsetup resume vg_test-Thin_tdata_rimage_1
dmsetup reload vg_test-Thin_tdata_rmeta_1 --table "0 8192 error" dmsetup suspend --noflush vg_test-Thin_tdata_rmeta_1 dmsetup resume vg_test-Thin_tdata_rmeta_1
dmsetup reload vg_test-Thin_tmeta_rimage_1 --table "0 204800 error" dmsetup suspend --noflush vg_test-Thin_tmeta_rimage_1 dmsetup resume vg_test-Thin_tmeta_rimage_1
dmsetup reload vg_test-Thin_tmeta_rmeta_1 --table "0 8192 error" dmsetup suspend --noflush vg_test-Thin_tmeta_rmeta_1 dmsetup resume vg_test-Thin_tmeta_rmeta_1
\cp /etc/lvm/backup/vg_test /etc/lvm/vg_test-before-error-target.cfg \cp /etc/lvm/backup/vg_test /etc/lvm/vg_test-error-target.cfg ### edit vg_test-error-target.cfg:
### remove the missing PV ### edit Thin_tdata_rimage_1 Thin_tdata_rmeta_1 Thin_tmeta_rimage_1 Thin_tmeta_rmeta_1 ### you may have to remove the lvol0_pmspare volume if it resided on the missing PV
# diff -u /etc/lvm/vg_test-before-error-target.cfg /etc/lvm/vg_test-error-target.cfg --- /etc/lvm/vg_test-before-error-target.cfg 2017-10-27 11:54:31.546944978 +0200 +++ /etc/lvm/vg_test-error-target.cfg 2017-10-27 12:01:15.156352344 +0200 @@ -21,17 +21,6 @@
physical_volumes {- pv0 {
- id = "cUHx7d-Gl8O-TX2l-mi5B-QCx6-so00-cS0M4z"
- device = "[unknown]" # Hint only
-
- status = ["ALLOCATABLE"]
- flags = ["MISSING"]
- dev_size = 25165824 # 12 Gigabytes
- pe_start = 2048
- pe_count = 3071 # 11.9961 Gigabytes
- }
-
pv1 {
id = "FlCr4G-d80M-g9zW-5A2r-qvim-Ayi1-Ce1Rsa"
device = "/dev/vdb" # Hint only@@ -295,12 +284,7 @@
start_extent = 0
extent_count = 25 # 100 Megabytes
- type = "striped"
- stripe_count = 1 # linear
-
- stripes = [
- "pv0", 27
- ]
+ type = "error"
}
}@@ -316,12 +300,7 @@
start_extent = 0
extent_count = 1 # 4 Megabytes
- type = "striped"
- stripe_count = 1 # linear
-
- stripes = [
- "pv0", 26
- ]
+ type = "error"
}
}@@ -379,12 +358,7 @@
start_extent = 0
extent_count = 2560 # 10 Gigabytes
- type = "striped"
- stripe_count = 1 # linear
-
- stripes = [
- "pv0", 53
- ]
+ type = "error"
}
}@@ -400,33 +374,7 @@
start_extent = 0
extent_count = 1 # 4 Megabytes
- type = "striped"
- stripe_count = 1 # linear
-
- stripes = [
- "pv0", 52
- ]
- }
- }
-
- lvol0_pmspare {
- id = "1tNgGe-mRW5-fKXs-1eVG-t91E-40JP-YdGqZi" - status = ["READ", "WRITE"] - flags = [] - creation_time = 1509097937 # 2017-10-27 11:52:17 +0200 - creation_host = "bastel-c7-lvm.wu.ac.at" - segment_count = 1
-
- segment1 {
- start_extent = 0
- extent_count = 25 # 100 Megabytes
-
- type = "striped"
- stripe_count = 1 # linear
-
- stripes = [
- "pv0", 2613
- ]
+ type = "error"
}
}
}### write this config to all configured PVs
# vgcfgrestore --force -f /etc/lvm/vg_test-error-target.cfg vg_test WARNING: Forced restore of Volume Group vg_test with thin volumes. Restored volume group vg_test
### now the missing PV is completely removed from the VG:
# lvs -a -olv_name,lv_attr,segtype,devices vg_test LV Attr Type Devices Thin twi-aotz-- thin-pool Thin_tdata(0) [Thin_tdata] rwi-aor-r- raid1 Thin_tdata_rimage_0(0),Thin_tdata_rimage_1(0) [Thin_tdata_rimage_0] iwi-aor--- linear /dev/vdb(53) [Thin_tdata_rimage_1] vwi-aor-r- error [Thin_tdata_rmeta_0] ewi-aor--- linear /dev/vdb(52) [Thin_tdata_rmeta_1] ewi-aor-r- error [Thin_tmeta] ewi-aor-r- raid1 Thin_tmeta_rimage_0(0),Thin_tmeta_rimage_1(0) [Thin_tmeta_rimage_0] iwi-aor--- linear /dev/vdb(27) [Thin_tmeta_rimage_1] vwi-aor-r- error [Thin_tmeta_rmeta_0] ewi-aor--- linear /dev/vdb(26) [Thin_tmeta_rmeta_1] ewi-aor-r- error boot rwi-a-r-r- raid1 boot_rimage_0(0),boot_rimage_1(0) [boot_rimage_0] iwi-aor--- linear /dev/vdb(1) [boot_rimage_1] vwi-aor-r- error [boot_rmeta_0] ewi-aor--- linear /dev/vdb(0) [boot_rmeta_1] ewi-aor-r- error thintest Vwi-aotz-- thin
### lvconvert --repair will work now:
# lvconvert -y --repair /dev/vg_test/boot WARNING: Disabling lvmetad cache for repair command. WARNING: Not using lvmetad because of repair. Faulty devices in vg_test/boot successfully replaced. # lvconvert -y --repair /dev/vg_test/Thin_tmeta WARNING: Disabling lvmetad cache for repair command. WARNING: Not using lvmetad because of repair. Faulty devices in vg_test/Thin_tmeta successfully replaced. # lvconvert -y --repair /dev/vg_test/Thin_tdata WARNING: Disabling lvmetad cache for repair command. WARNING: Not using lvmetad because of repair. Faulty devices in vg_test/Thin_tdata successfully replaced.
# lvs -a -olv_name,lv_attr,sync_percent,segtype,devices vg_test LV Attr Cpy%Sync Type Devices Thin twi-aotz-- thin-pool Thin_tdata(0) [Thin_tdata] rwi-aor--- 100.00 raid1 Thin_tdata_rimage_0(0),Thin_tdata_rimage_1(0) [Thin_tdata_rimage_0] iwi-aor--- linear /dev/vdb(53) [Thin_tdata_rimage_1] iwi-aor--- linear /dev/vdd(53) [Thin_tdata_rmeta_0] ewi-aor--- linear /dev/vdb(52) [Thin_tdata_rmeta_1] ewi-aor--- linear /dev/vdd(52) [Thin_tmeta] ewi-aor--- 100.00 raid1 Thin_tmeta_rimage_0(0),Thin_tmeta_rimage_1(0) [Thin_tmeta_rimage_0] iwi-aor--- linear /dev/vdb(27) [Thin_tmeta_rimage_1] iwi-aor--- linear /dev/vdd(27) [Thin_tmeta_rmeta_0] ewi-aor--- linear /dev/vdb(26) [Thin_tmeta_rmeta_1] ewi-aor--- linear /dev/vdd(26) boot rwi-a-r--- 100.00 raid1 boot_rimage_0(0),boot_rimage_1(0) [boot_rimage_0] iwi-aor--- linear /dev/vdb(1) [boot_rimage_1] iwi-aor--- linear /dev/vdd(1) [boot_rmeta_0] ewi-aor--- linear /dev/vdb(0) [boot_rmeta_1] ewi-aor--- linear /dev/vdd(0) thintest Vwi-aotz-- thin -------------------- 8< --------------------
For the wishlist: :-)
An lvm command to (maybe forcibly) manually set a raid sub-lv to faulty (i.e. replace the PV with the error target) would be nice.
Cheers,
--leo
Signature · 2 lines
e-mail ::: Leo.Bergolth (at) wu.ac.at fax ::: +43-1-31336-906050 location ::: IT-Services | Vienna University of Economics | Austria