Skip to contentSkip to conversation

lvm raid1 metadata on different pv

linux-lvm Alexander 'Leo' Bergolth 3 participants 4 messages Sep 25, 2017 – Sep 26, 2017

lore
Share & tools
lore

Alexander 'Leo' Bergolth

author
#0 lore raw

Hi!

I tried to move the raid1 metadata subvolumes to different PVs (SSD devices for performance).

Moving with pvmove works fine but activation fails when both legs of the metadata had been moved to external devices. (See below.)

Interestingly moving just one metadata LV to another device works fine. (Raid LV can be activated afterwards.)

I guess raid1 metadata on different PVs is not supported (yet)?

I am using Centos 7.4 and kernel 3.10.0-693.el7.x86_64.

Cheers,
--leo

-------------------- 8< --------------------
modprobe zram num_devices=2
echo 300M > /sys/block/zram0/disksize
echo 300M > /sys/block/zram1/disksize

pvcreate /dev/sda2
pvcreate /dev/sdb2
pvcreate /dev/zram0
pvcreate /dev/zram1

vgcreate vg_sys /dev/sda2 /dev/sdb2 /dev/zram0 /dev/zram1
lvcreate --type raid1 -m 1 --regionsize 64M -L 500m -n lv_boot vg_sys /dev/sda2 /dev/sdb2

pvmove -n 'lv_boot_rmeta_0' /dev/sda2 /dev/zram0

# and maybe
# pvmove -n 'lv_boot_rmeta_1' /dev/sdb2 /dev/zram1

-------------------- 8< --------------------

    Creating vg_sys-lv_boot
        dm create vg_sys-lv_boot LVM-l6Eg7Uvcm2KieevnXDjLLje3wqmSVGa1e56whxycwUR2RvGvcQNLy1GdfpzlZuQk [ noopencount flush ]   [16384] (*1)
    Loading vg_sys-lv_boot table (253:7)
      Getting target version for raid
        dm versions   [ opencount flush ]   [16384] (*1)
      Found raid target v1.12.0.
        Adding target to (253:7): 0 1024000 raid raid1 3 0 region_size 8192 2 253:3 253:4 253:5 253:6
        dm table   (253:7) [ opencount flush ]   [16384] (*1)
        dm reload   (253:7) [ noopencount flush ]   [16384] (*1)
  device-mapper: reload ioctl on  (253:7) failed: Input/output error
-------------------- 8< --------------------
[ 8130.110467] md/raid1:mdX: active with 2 out of 2 mirrors
[ 8130.111361] mdX: failed to create bitmap (-5)
[ 8130.112254] device-mapper: table: 253:7: raid: Failed to run raid array
[ 8130.113154] device-mapper: ioctl: error adding target to table
-------------------- 8< --------------------
# lvs -a -o+devices
  LV                 VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices                                
  lv_boot            vg_sys rwi---r--- 500.00m                                                     lv_boot_rimage_0(0),lv_boot_rimage_1(0)
  [lv_boot_rimage_0] vg_sys Iwi-a-r-r- 500.00m                                                     /dev/sda2(1)                           
  [lv_boot_rimage_1] vg_sys Iwi-a-r-r- 500.00m                                                     /dev/sdb2(1)                           
  [lv_boot_rmeta_0]  vg_sys ewi-a-r-r-   4.00m                                                     /dev/zram0(0)                          
  [lv_boot_rmeta_1]  vg_sys ewi-a-r-r-   4.00m                                                     /dev/zram1(0)                          
-------------------- 8< --------------------

Full vgchange output can be found at:
http://leo.kloburg.at/tmp/lvm-raid1-ext-meta/

Signature · 3 lines
e-mail   ::: Leo.Bergolth (at) wu.ac.at   
fax      ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria
#1 lore raw

I did as follow and it works, without storing the metadata on zram:

truncate -s500M disk1.dd
truncate -s500M disk2.dd
losetup /dev/loop0 /root/disk1.dd
losetup /dev/loop1 /root/disk2.dd

pvcreate /dev/loop0
pvcreate /dev/loop1
vgcreate vg00/dev/loop0 /dev/loop1
vgcreate vg00 /dev/loop0 /dev/loop1

lvcreate --type raid1 --name raid1vol -L 400M vg00

[root@puppetserver:~# lvs -ao +devices
  LV                  VG   Attr     LSize   Pool Origin Data%  Move Log
Copy%  Convert Devices
  raid1vol            vg00 rwi-a-m- 400.00m
100.00         raid1vol_rimage_0(0),raid1vol_rimage_1(0)
  [raid1vol_rimage_0] vg00 iwi-aor-
400.00m
/dev/loop0(1)
  [raid1vol_rimage_1] vg00 iwi-aor-
400.00m
/dev/loop1(1)
  [raid1vol_rmeta_0]  vg00 ewi-aor-
4.00m
/dev/loop0(0)
  [raid1vol_rmeta_1]  vg00 ewi-aor-
4.00m
/dev/loop1(0)

Now I will the metadata to other pvs:

truncate -s 100M meta1.dd
truncate -s 100M meta2.dd

losetup /dev/loop2 meta1.dd
losetup /dev/loop3 meta2.dd

vgextend vg00 /dev/loop2
vgextend vg00 /dev/loop3

pvmove -n 'raid1vol_rmeta_0' /dev/loop0 /dev/loop2 pvmove -n 'raid1vol_rmeta_1' /dev/loop1 /dev/loop3

vgchange -an vg00
0 logical volume(s) in volume group "vg00" now active

vgchange -ay vg00
1 logical volume(s) in volume group "vg00" now active

2017-09-25 11:30 GMT+02:00 Alexander 'Leo' Bergolth <[email protected]>:

Quoted 85 lines

Hi!

I tried to move the raid1 metadata subvolumes to different PVs (SSD devices for performance).

Moving with pvmove works fine but activation fails when both legs of the metadata had been moved to external devices. (See below.)

Interestingly moving just one metadata LV to another device works fine. (Raid LV can be activated afterwards.)

I guess raid1 metadata on different PVs is not supported (yet)?

I am using Centos 7.4 and kernel 3.10.0-693.el7.x86_64.

Cheers,
--leo

-------------------- 8< --------------------
modprobe zram num_devices=2
echo 300M > /sys/block/zram0/disksize
echo 300M > /sys/block/zram1/disksize

pvcreate /dev/sda2
pvcreate /dev/sdb2
pvcreate /dev/zram0
pvcreate /dev/zram1

vgcreate vg_sys /dev/sda2 /dev/sdb2 /dev/zram0 /dev/zram1 lvcreate --type raid1 -m 1 --regionsize 64M -L 500m -n lv_boot vg_sys /dev/sda2 /dev/sdb2

pvmove -n 'lv_boot_rmeta_0' /dev/sda2 /dev/zram0

# and maybe
# pvmove -n 'lv_boot_rmeta_1' /dev/sdb2 /dev/zram1

-------------------- 8< --------------------
Creating vg_sys-lv_boot
dm create vg_sys-lv_boot LVM-l6Eg7Uvcm2KieevnXDjLLje3wqmSVG
a1e56whxycwUR2RvGvcQNLy1GdfpzlZuQk [ noopencount flush ] [16384] (*1)

    Loading vg_sys-lv_boot table (253:7)
      Getting target version for raid
        dm versions   [ opencount flush ]   [16384] (*1)
      Found raid target v1.12.0.
        Adding target to (253:7): 0 1024000 raid raid1 3 0 region_size
8192 2 253:3 253:4 253:5 253:6
        dm table   (253:7) [ opencount flush ]   [16384] (*1)
        dm reload   (253:7) [ noopencount flush ]   [16384] (*1)
  device-mapper: reload ioctl on  (253:7) failed: Input/output error
-------------------- 8< --------------------
[ 8130.110467] md/raid1:mdX: active with 2 out of 2 mirrors
[ 8130.111361] mdX: failed to create bitmap (-5)
[ 8130.112254] device-mapper: table: 253:7: raid: Failed to run raid array
[ 8130.113154] device-mapper: ioctl: error adding target to table
-------------------- 8< --------------------
# lvs -a -o+devices
  LV                 VG     Attr       LSize   Pool Origin Data%  Meta%
Move Log Cpy%Sync Convert Devices
  lv_boot            vg_sys rwi---r--- 500.00m
                         lv_boot_rimage_0(0),lv_boot_rimage_1(0)
  [lv_boot_rimage_0] vg_sys Iwi-a-r-r- 500.00m
                         /dev/sda2(1)
  [lv_boot_rimage_1] vg_sys Iwi-a-r-r- 500.00m
                         /dev/sdb2(1)
  [lv_boot_rmeta_0]  vg_sys ewi-a-r-r-   4.00m
                         /dev/zram0(0)
  [lv_boot_rmeta_1]  vg_sys ewi-a-r-r-   4.00m
                         /dev/zram1(0)
-------------------- 8< --------------------

Full vgchange output can be found at:
http://leo.kloburg.at/tmp/lvm-raid1-ext-meta/

--
e-mail ::: Leo.Bergolth (at) wu.ac.at
fax ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Signature · 5 lines
  .~.
  /V\
 //  \\
/(   )\
^`~'^
#2 lore raw
In reply to Alexander 'Leo' Bergolth

I’m not entirely sure how I would advocate moving the metadata area of a RAID LV, but trying it in a way similar to yours seemed to work for me:

[root@bp-01 ~]# devices vg
LV Attr Cpy%Sync Devices

  lv            rwi-a-r--- 100.00   lv_rimage_0(0),lv_rimage_1(0)
  [lv_rimage_0] iwi-aor---          /dev/sdb1(1)
  [lv_rimage_1] iwi-aor---          /dev/sdc1(1)
  [lv_rmeta_0]  ewi-aor---          /dev/sdb1(0)
  [lv_rmeta_1]  ewi-aor---          /dev/sdc1(0)

[root@bp-01 ~]# pvmove /dev/sdb1:0 /dev/sdd1
/dev/sdb1: Moved: 0.00%
/dev/sdb1: Moved: 100.00%
[root@bp-01 ~]# devices vg
LV Attr Cpy%Sync Devices

  lv            rwi-a-r--- 100.00   lv_rimage_0(0),lv_rimage_1(0)
  [lv_rimage_0] iwi-aor---          /dev/sdb1(1)
  [lv_rimage_1] iwi-aor---          /dev/sdc1(1)
  [lv_rmeta_0]  ewi-aor---          /dev/sdd1(0)
  [lv_rmeta_1]  ewi-aor---          /dev/sdc1(0)

[root@bp-01 ~]# pvmove /dev/sdc1:0 /dev/sde1
/dev/sdc1: Moved: 0.00%
/dev/sdc1: Moved: 100.00%
[root@bp-01 ~]# devices vg
LV Attr Cpy%Sync Devices

  lv            rwi-a-r--- 100.00   lv_rimage_0(0),lv_rimage_1(0)
  [lv_rimage_0] iwi-aor---          /dev/sdb1(1)
  [lv_rimage_1] iwi-aor---          /dev/sdc1(1)
  [lv_rmeta_0]  ewi-aor---          /dev/sdd1(0)
  [lv_rmeta_1]  ewi-aor---          /dev/sde1(0)

[root@bp-01 ~]# vgchange -an vg
0 logical volume(s) in volume group "vg" now active
[root@bp-01 ~]# vgchange -ay vg
1 logical volume(s) in volume group "vg" now active
[root@bp-01 ~]# devices vg
LV Attr Cpy%Sync Devices

  lv            rwi-a-r--- 100.00   lv_rimage_0(0),lv_rimage_1(0)
  [lv_rimage_0] iwi-aor---          /dev/sdb1(1)
  [lv_rimage_1] iwi-aor---          /dev/sdc1(1)
  [lv_rmeta_0]  ewi-aor---          /dev/sdd1(0)
  [lv_rmeta_1]  ewi-aor---          /dev/sde1(0)

If this does turn out to be a bug, we should probably fix it. However, I probably wouldn’t do what you are attempting to do for the following reasons:
1) performance is probably not impacted /that/ much by having the metadata area on the same device.
Although I am sure that you can find some benchmark that proves me wrong, there have been a number of things done to mitigate the effect of having a write intent bitmap. I don’t believe the overhead is much in normal use.)
2) You are reducing your redundancy.
Before, 1 device made up a side. Now two devices make up a side. If you loose any component of a side, it will effectively mean that side has failed. The likelihood of a failure from 2 drives is more than the likelihood of failure from just one of them.

brassow

Quoted 74 lines · Alexander 'Leo' Bergolth

Hi!

I tried to move the raid1 metadata subvolumes to different PVs (SSD devices for performance).

Moving with pvmove works fine but activation fails when both legs of the metadata had been moved to external devices. (See below.)

Interestingly moving just one metadata LV to another device works fine. (Raid LV can be activated afterwards.)

I guess raid1 metadata on different PVs is not supported (yet)?

I am using Centos 7.4 and kernel 3.10.0-693.el7.x86_64.

Cheers,
--leo

-------------------- 8< --------------------
modprobe zram num_devices=2
echo 300M > /sys/block/zram0/disksize
echo 300M > /sys/block/zram1/disksize

pvcreate /dev/sda2
pvcreate /dev/sdb2
pvcreate /dev/zram0
pvcreate /dev/zram1

vgcreate vg_sys /dev/sda2 /dev/sdb2 /dev/zram0 /dev/zram1
lvcreate --type raid1 -m 1 --regionsize 64M -L 500m -n lv_boot vg_sys /dev/sda2 /dev/sdb2

pvmove -n 'lv_boot_rmeta_0' /dev/sda2 /dev/zram0

# and maybe
# pvmove -n 'lv_boot_rmeta_1' /dev/sdb2 /dev/zram1

-------------------- 8< --------------------
Creating vg_sys-lv_boot
dm create vg_sys-lv_boot LVM-l6Eg7Uvcm2KieevnXDjLLje3wqmSVGa1e56whxycwUR2RvGvcQNLy1GdfpzlZuQk [ noopencount flush ] [16384] (*1)
Loading vg_sys-lv_boot table (253:7)

     Getting target version for raid
       dm versions   [ opencount flush ]   [16384] (*1)
     Found raid target v1.12.0.
       Adding target to (253:7): 0 1024000 raid raid1 3 0 region_size 8192 2 253:3 253:4 253:5 253:6
       dm table   (253:7) [ opencount flush ]   [16384] (*1)
       dm reload   (253:7) [ noopencount flush ]   [16384] (*1)
 device-mapper: reload ioctl on  (253:7) failed: Input/output error
-------------------- 8< --------------------
[ 8130.110467] md/raid1:mdX: active with 2 out of 2 mirrors
[ 8130.111361] mdX: failed to create bitmap (-5)
[ 8130.112254] device-mapper: table: 253:7: raid: Failed to run raid array
[ 8130.113154] device-mapper: ioctl: error adding target to table
-------------------- 8< --------------------
# lvs -a -o+devices
 LV                 VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices                                
 lv_boot            vg_sys rwi---r--- 500.00m                                                     lv_boot_rimage_0(0),lv_boot_rimage_1(0)
 [lv_boot_rimage_0] vg_sys Iwi-a-r-r- 500.00m                                                     /dev/sda2(1)                           
 [lv_boot_rimage_1] vg_sys Iwi-a-r-r- 500.00m                                                     /dev/sdb2(1)                           
 [lv_boot_rmeta_0]  vg_sys ewi-a-r-r-   4.00m                                                     /dev/zram0(0)                          
 [lv_boot_rmeta_1]  vg_sys ewi-a-r-r-   4.00m                                                     /dev/zram1(0)                          
-------------------- 8< --------------------

Full vgchange output can be found at:
http://leo.kloburg.at/tmp/lvm-raid1-ext-meta/

Signature · 9 lines
e-mail   ::: Leo.Bergolth (at) wu.ac.at   
fax      ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
author
#3 lore raw
Quoted 3 lines · Brassow Jonathan

I’m not entirely sure how I would advocate moving the metadata area of a RAID LV, but trying it in a way similar to yours seemed to work for me.

Strange..

Quoted 1 line

If this does turn out to be a bug, we should probably fix it.

Hmm. Looks like it was a false-alarm.. :-)
The issue seems to be connected to my test-target-devices of type zram. If I use brd devices (/dev/ram0 and /dev/ram1) for my test, activation works as expected:

modprobe brd rd_size=307200 max_part=0 rd_nr=2

# pvcreate /dev/ram0
  Physical volume "/dev/ram0" successfully created.
# pvcreate /dev/ram1
  Physical volume "/dev/ram1" successfully created.
# vgextend vg1 /dev/ram0 /dev/ram1
  Volume group "vg1" successfully extended
# pvmove /dev/sdc:0 /dev/ram0
  /dev/sdc: Moved: 0.00%
  /dev/sdc: Moved: 100.00%
# pvmove /dev/sdd:0 /dev/ram1
  /dev/sdd: Moved: 0.00%
# vgchange -a n
  0 logical volume(s) in volume group "vg1" now active
# vgchange -a y
  1 logical volume(s) in volume group "vg1" now active

Maybe the different block size characteristics of the zram device are causing the error?
(See below for a diff.)

Quoted 2 lines

1) performance is probably not impacted
2) You are reducing your redundancy.

I am aware of that. Nevertheless I think it might be reasonable for my usage scenario:

I'd like to build an SSD-cached raid1 using two harddisks for data and and two SSDs for cache and metadata.
The use case is big volumes using cheap sata disks that should be pimped in terms of access time by the SSD cache.

My (raid1-backed) cache-pool in writeback mode already decreases my fault-tolerance, so putting raid metadata on the corresponding SSDs won't do further harm.

Cheers,
--leo

-------------------- 8< --------------------

# cd /sys/block
# diff -aryW10 --suppress-common-lines ram0/queue zram0/queue
diff -aryW10 --suppress-common-lines ram0/queue/hw_sector_size zram0/queue/hw_sector_size
51  |   40
diff -aryW10 --suppress-common-lines ram0/queue/logical_block_size zram0/queue/logical_block_size
51  |   40
diff -aryW10 --suppress-common-lines ram0/queue/max_hw_sectors_kb zram0/queue/max_hw_sectors_kb
51  |   12
diff -aryW10 --suppress-common-lines ram0/queue/max_sectors_kb zram0/queue/max_sectors_kb
51  |   12
diff -aryW10 --suppress-common-lines ram0/queue/minimum_io_size zram0/queue/minimum_io_size
51  |   40
diff -aryW10 --suppress-common-lines ram0/queue/optimal_io_size zram0/queue/optimal_io_size
0   |   40
diff -aryW10 --suppress-common-lines ram0/queue/physical_block_size zram0/queue/physical_block_size
51  |   40
diff -aryW10 --suppress-common-lines ram0/queue/read_ahead_kb zram0/queue/read_ahead_kb
40  |   12
diff -aryW10 --suppress-common-lines ram0/queue/rotational zram0/queue/rotational
1   |   0
-------------------- 8< --------------------
Quoted 82 lines
Quoted 60 lines · Alexander 'Leo' Bergolth

Hi!

I tried to move the raid1 metadata subvolumes to different PVs (SSD devices for performance).

Moving with pvmove works fine but activation fails when both legs of the metadata had been moved to external devices. (See below.)

Interestingly moving just one metadata LV to another device works fine. (Raid LV can be activated afterwards.)

I guess raid1 metadata on different PVs is not supported (yet)?

I am using Centos 7.4 and kernel 3.10.0-693.el7.x86_64.

Cheers,
--leo

-------------------- 8< --------------------
modprobe zram num_devices=2
echo 300M > /sys/block/zram0/disksize
echo 300M > /sys/block/zram1/disksize

pvcreate /dev/sda2
pvcreate /dev/sdb2
pvcreate /dev/zram0
pvcreate /dev/zram1

vgcreate vg_sys /dev/sda2 /dev/sdb2 /dev/zram0 /dev/zram1
lvcreate --type raid1 -m 1 --regionsize 64M -L 500m -n lv_boot vg_sys /dev/sda2 /dev/sdb2

pvmove -n 'lv_boot_rmeta_0' /dev/sda2 /dev/zram0

# and maybe
# pvmove -n 'lv_boot_rmeta_1' /dev/sdb2 /dev/zram1

-------------------- 8< --------------------
Creating vg_sys-lv_boot
dm create vg_sys-lv_boot LVM-l6Eg7Uvcm2KieevnXDjLLje3wqmSVGa1e56whxycwUR2RvGvcQNLy1GdfpzlZuQk [ noopencount flush ] [16384] (*1)
Loading vg_sys-lv_boot table (253:7)

     Getting target version for raid
       dm versions   [ opencount flush ]   [16384] (*1)
     Found raid target v1.12.0.
       Adding target to (253:7): 0 1024000 raid raid1 3 0 region_size 8192 2 253:3 253:4 253:5 253:6
       dm table   (253:7) [ opencount flush ]   [16384] (*1)
       dm reload   (253:7) [ noopencount flush ]   [16384] (*1)
 device-mapper: reload ioctl on  (253:7) failed: Input/output error
-------------------- 8< --------------------
[ 8130.110467] md/raid1:mdX: active with 2 out of 2 mirrors
[ 8130.111361] mdX: failed to create bitmap (-5)
[ 8130.112254] device-mapper: table: 253:7: raid: Failed to run raid array
[ 8130.113154] device-mapper: ioctl: error adding target to table
-------------------- 8< --------------------
# lvs -a -o+devices
 LV                 VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices                                
 lv_boot            vg_sys rwi---r--- 500.00m                                                     lv_boot_rimage_0(0),lv_boot_rimage_1(0)
 [lv_boot_rimage_0] vg_sys Iwi-a-r-r- 500.00m                                                     /dev/sda2(1)                           
 [lv_boot_rimage_1] vg_sys Iwi-a-r-r- 500.00m                                                     /dev/sdb2(1)                           
 [lv_boot_rmeta_0]  vg_sys ewi-a-r-r-   4.00m                                                     /dev/zram0(0)                          
 [lv_boot_rmeta_1]  vg_sys ewi-a-r-r-   4.00m                                                     /dev/zram1(0)                          
-------------------- 8< --------------------
Quoted 14 lines

Full vgchange output can be found at:
http://leo.kloburg.at/tmp/lvm-raid1-ext-meta/

Signature · 9 lines
e-mail   ::: Leo.Bergolth (at) wu.ac.at   
fax      ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Signature · 3 lines
e-mail   ::: Leo.Bergolth (at) wu.ac.at
fax      ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria