Skip to contentSkip to conversation

Problems creating/converting large logical volume to RAID1

linux-lvm Tony Arnold 3 participants 7 messages Nov 24, 2015 – Nov 27, 2015

lore
Share & tools
lore
#0 lore raw

I'm trying to convert one of my logical volumes to be RAID1 mirrored using LVM on my Ubuntu 15.10 system. When I do, the command lvs shows the mirror set is 100% synchronized immediately. syslog show a failure saying one of the devices in the mirror set has failed and it reverts to a single device mirror set.

Errors include saying attempt to access beyond end of device. There is also an error showing code=-5 for 'super_written'. A full extract of syslog is below.

The volume group is configured so:

root@the-circus:~# pvs
  PV         VG        Fmt  Attr PSize PFree
  /dev/sda3  vg0       lvm2 a--  2.73t 1.09t
  /dev/sdb3  vg0       lvm2 a--  2.73t 2.68t

So it's trying to create the mirror devices on /dev/sdb3.

The disks are 3TB Seagate Barracuda disks. They both brand new.

Experimentation shows I only get these errors for logical volumes bigger than 31GB, anything smaller behaves correctly.

My version of Ubuntu:

# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 15.10
Release:	15.10
Codename:	wily

Kernel is:

# uname -a
Linux the-circus 4.2.0-18-generic #22-Ubuntu SMP Fri Nov 6 18:25:50 UTC
2015 x86_64 x86_64 x86_64 GNU/Linux

Anyone any ideas? I've googled this until the cows come home but cannot find anything helpful.

Regards,
Tony.

Nov 21 17:29:52 the-circus kernel: [  231.538111] device-mapper: raid:
Loading target version 1.7.0
Nov 21 17:29:52 the-circus kernel: [  231.541976] device-mapper: raid:
Superblocks created for new array
Nov 21 17:29:52 the-circus kernel: [  231.542894] md/raid1:mdX: not
clean -- starting background reconstruction
Nov 21 17:29:52 the-circus kernel: [  231.542897] md/raid1:mdX: active
with 2 out of 2 mirrors
Nov 21 17:29:52 the-circus kernel: [  231.542899] Choosing daemon_sleep
default (5 sec)
Nov 21 17:29:52 the-circus kernel: [  231.542903] created bitmap (50
pages) for device mdX
Nov 21 17:29:52 the-circus kernel: [  231.542906] attempt to access
beyond end of device
Nov 21 17:29:52 the-circus kernel: [  231.542908] dm-7: rw=13329,
want=18446744073709551608, limit=8192
Nov 21 17:29:52 the-circus kernel: [  231.542909] md: super_written gets
error=-5, uptodate=0
Nov 21 17:29:52 the-circus kernel: [  231.542920] md/raid1:mdX: Disk
failure on dm-8, disabling device.
Nov 21 17:29:52 the-circus kernel: [  231.542920] md/raid1:mdX:
Operation continuing on 1 devices.
Nov 21 17:29:52 the-circus kernel: [  231.542933] attempt to access
beyond end of device
Nov 21 17:29:52 the-circus kernel: [  231.542936] dm-5: rw=13329,
want=18446744073709551608, limit=8192
Nov 21 17:29:52 the-circus kernel: [  231.542939] md: super_written gets
error=-5, uptodate=0
Nov 21 17:29:52 the-circus kernel: [  231.569851] attempt to access
beyond end of device
Nov 21 17:29:52 the-circus kernel: [  231.569859] dm-5: rw=13329,
want=18446744073709551608, limit=8192
Nov 21 17:29:52 the-circus kernel: [  231.569863] md: super_written gets
error=-5, uptodate=0
Nov 21 17:29:52 the-circus kernel: [  231.596802] mdX: bitmap file is
out of date, doing full recovery
Nov 21 17:29:52 the-circus kernel: [  231.712193] mdX: bitmap
initialized from disk: read 4 pages, set 102400 of 102400 bits
Nov 21 17:29:52 the-circus systemd[1]: Started Device-mapper event daemon.
Nov 21 17:29:52 the-circus dmeventd[3132]: dmeventd ready for processing.
Nov 21 17:29:52 the-circus kernel: [  231.786858] RAID1 conf printout:
Nov 21 17:29:52 the-circus kernel: [  231.786863]  --- wd:1 rd:2
Nov 21 17:29:52 the-circus kernel: [  231.786866]  disk 0, wo:0, o:1,
dev:dm-6
Nov 21 17:29:52 the-circus kernel: [  231.786868]  disk 1, wo:1, o:0,
dev:dm-8
Nov 21 17:29:52 the-circus kernel: [  231.793942] RAID1 conf printout:
Nov 21 17:29:52 the-circus kernel: [  231.793947]  --- wd:1 rd:2
Nov 21 17:29:52 the-circus kernel: [  231.793949]  disk 0, wo:0, o:1,
dev:dm-6
Nov 21 17:29:52 the-circus kernel: [  231.794033] md: resync of RAID
array mdX
Nov 21 17:29:52 the-circus kernel: [  231.794037] md: minimum
_guaranteed_  speed: 1000 KB/sec/disk.
Nov 21 17:29:52 the-circus kernel: [  231.794039] md: using maximum
available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
Nov 21 17:29:52 the-circus kernel: [  231.794042] md: using 128k window,
over a total of 52428800k.
Nov 21 17:29:52 the-circus kernel: [  231.794196] md: mdX: resync done.
Nov 21 17:29:53 the-circus kernel: [  232.037242] RAID1 conf printout:
Nov 21 17:29:53 the-circus kernel: [  232.037248]  --- wd:1 rd:2
Nov 21 17:29:53 the-circus kernel: [  232.037251]  disk 0, wo:0, o:1,
dev:dm-6
Nov 21 17:29:53 the-circus lvm[3132]: Monitoring RAID device vg0-testlv
for events.
Nov 21 17:29:53 the-circus lvm[3132]: Device #1 of raid1 array,
vg0-testlv, has failed.
Nov 21 17:29:53 the-circus lvm[3132]: Use 'lvconvert --repair

vg0/testlv' to replace failed device.

Signature · 9 lines
Tony Arnold, IT Security Analyst, University of Manchester.
T: +44 (0) 161 275 6093, F: +44 (0) 705 344 3082,
M: +44 (0) 773 330 0039, E: [email protected]

-- 
ubuntu-users mailing list
[email protected]
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
#1 lore raw

Tony,

Can I make a suggestion to you? Instead of trying to do the mirroring in LVM, build yourself an MD mirror, then layer LVM ontop of that instead. You get nice seperation.

You can also take that /dev/sdb3, make it into an MD RAID1 array with just one device, add it into the VG, then move the LVs onto the new device. Once that's done, you remove the sda3 from the VG, then add it into the MD device. Something like this...

Quoted 8 lines

mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb3 missing cat /proc/mdstat
pvcreate -v /dev/md1
vgextend -v vg0 /dev/md1
pvmove -v sda3
vgreduce -v vg0 sda3
mdadm --add /dev/md0 /dev/sda3
cat /proc/mdstat

And you will now be mirrored, etc.

Tony> I'm trying to convert one of my logical volumes to be RAID1 mirrored Tony> using LVM on my Ubuntu 15.10 system. When I do, the command lvs shows Tony> the mirror set is 100% synchronized immediately. syslog show a failure Tony> saying one of the devices in the mirror set has failed and it reverts to Tony> a single device mirror set.

Tony> Errors include saying attempt to access beyond end of device. There is Tony> also an error showing code=-5 for 'super_written'. A full extract of Tony> syslog is below.

Tony> The volume group is configured so:

Tony> root@the-circus:~# pvs

Tony>   PV         VG        Fmt  Attr PSize PFree
Tony>   /dev/sda3  vg0       lvm2 a--  2.73t 1.09t
Tony>   /dev/sdb3  vg0       lvm2 a--  2.73t 2.68t

Tony> So it's trying to create the mirror devices on /dev/sdb3.

Tony> The disks are 3TB Seagate Barracuda disks. They both brand new.

Tony> Experimentation shows I only get these errors for logical volumes bigger Tony> than 31GB, anything smaller behaves correctly.

Tony> My version of Ubuntu:

Tony> # lsb_release -a
Tony> No LSB modules are available.
Tony> Distributor ID: Ubuntu
Tony> Description: Ubuntu 15.10
Tony> Release: 15.10
Tony> Codename: wily

Tony> Kernel is:

Tony> # uname -a
Tony> Linux the-circus 4.2.0-18-generic #22-Ubuntu SMP Fri Nov 6 18:25:50 UTC Tony> 2015 x86_64 x86_64 x86_64 GNU/Linux

Tony> Anyone any ideas? I've googled this until the cows come home but cannot Tony> find anything helpful.

Tony> Regards,
Tony> Tony.

Tony> Nov 21 17:29:52 the-circus kernel: [ 231.538111] device-mapper: raid:
Tony> Loading target version 1.7.0
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.541976] device-mapper: raid:
Tony> Superblocks created for new array
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542894] md/raid1:mdX: not Tony> clean -- starting background reconstruction
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542897] md/raid1:mdX: active Tony> with 2 out of 2 mirrors
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542899] Choosing daemon_sleep Tony> default (5 sec)
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542903] created bitmap (50 Tony> pages) for device mdX
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542906] attempt to access Tony> beyond end of device
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542908] dm-7: rw=13329, Tony> want=18446744073709551608, limit=8192
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542909] md: super_written gets Tony> error=-5, uptodate=0
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542920] md/raid1:mdX: Disk Tony> failure on dm-8, disabling device.
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542920] md/raid1:mdX:
Tony> Operation continuing on 1 devices.
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542933] attempt to access Tony> beyond end of device
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542936] dm-5: rw=13329, Tony> want=18446744073709551608, limit=8192
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542939] md: super_written gets Tony> error=-5, uptodate=0
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.569851] attempt to access Tony> beyond end of device
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.569859] dm-5: rw=13329, Tony> want=18446744073709551608, limit=8192
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.569863] md: super_written gets Tony> error=-5, uptodate=0
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.596802] mdX: bitmap file is Tony> out of date, doing full recovery
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.712193] mdX: bitmap Tony> initialized from disk: read 4 pages, set 102400 of 102400 bits Tony> Nov 21 17:29:52 the-circus systemd[1]: Started Device-mapper event daemon. Tony> Nov 21 17:29:52 the-circus dmeventd[3132]: dmeventd ready for processing.

Tony> Nov 21 17:29:52 the-circus kernel: [  231.786858] RAID1 conf printout:
Tony> Nov 21 17:29:52 the-circus kernel: [  231.786863]  --- wd:1 rd:2
Tony> Nov 21 17:29:52 the-circus kernel: [  231.786866]  disk 0, wo:0, o:1,
Tony> dev:dm-6
Tony> Nov 21 17:29:52 the-circus kernel: [  231.786868]  disk 1, wo:1, o:0,
Tony> dev:dm-8
Tony> Nov 21 17:29:52 the-circus kernel: [  231.793942] RAID1 conf printout:
Tony> Nov 21 17:29:52 the-circus kernel: [  231.793947]  --- wd:1 rd:2
Tony> Nov 21 17:29:52 the-circus kernel: [  231.793949]  disk 0, wo:0, o:1,

Tony> dev:dm-6
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.794033] md: resync of RAID Tony> array mdX
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.794037] md: minimum Tony> _guaranteed_ speed: 1000 KB/sec/disk.
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.794039] md: using maximum Tony> available idle IO bandwidth (but not more than 200000 KB/sec) for resync. Tony> Nov 21 17:29:52 the-circus kernel: [ 231.794042] md: using 128k window, Tony> over a total of 52428800k.

Tony> Nov 21 17:29:52 the-circus kernel: [  231.794196] md: mdX: resync done.
Tony> Nov 21 17:29:53 the-circus kernel: [  232.037242] RAID1 conf printout:
Tony> Nov 21 17:29:53 the-circus kernel: [  232.037248]  --- wd:1 rd:2
Tony> Nov 21 17:29:53 the-circus kernel: [  232.037251]  disk 0, wo:0, o:1,

Tony> dev:dm-6
Tony> Nov 21 17:29:53 the-circus lvm[3132]: Monitoring RAID device vg0-testlv Tony> for events.
Tony> Nov 21 17:29:53 the-circus lvm[3132]: Device #1 of raid1 array, Tony> vg0-testlv, has failed.
Tony> Nov 21 17:29:53 the-circus lvm[3132]: Use 'lvconvert --repair Tony> vg0/testlv' to replace failed device.

Tony> --
Tony> Tony Arnold, IT Security Analyst, University of Manchester. Tony> T: +44 (0) 161 275 6093, F: +44 (0) 705 344 3082, Tony> M: +44 (0) 773 330 0039, E: [email protected]

Tony> --
Tony> ubuntu-users mailing list
Tony> [email protected]
Tony> Modify settings or unsubscribe at:
Tony> https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

Tony> _______________________________________________ Tony> linux-lvm mailing list
Tony> [email protected]
Tony> https://www.redhat.com/mailman/listinfo/linux-lvm
Tony> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw

John,

Quoted 18 lines · John Stoffel

Can I make a suggestion to you? Instead of trying to do the mirroring in LVM, build yourself an MD mirror, then layer LVM ontop of that instead. You get nice seperation.

You can also take that /dev/sdb3, make it into an MD RAID1 array with just one device, add it into the VG, then move the LVs onto the new device. Once that's done, you remove the sda3 from the VG, then add it into the MD device. Something like this...

Quoted 8 lines

mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb3 missing cat /proc/mdstat
pvcreate -v /dev/md1
vgextend -v vg0 /dev/md1
pvmove -v sda3
vgreduce -v vg0 sda3
mdadm --add /dev/md0 /dev/sda3
cat /proc/mdstat

Quoted 1 line

And you will now be mirrored, etc.

Thanks for your suggestion which I have now implemented. The pvmove took a few hours (1.5TB to move!) and then also some more hours while the mirror set synchronised. But it's all done and I now have my volumes mirrored.

I'm still curious though why the LVM based RAID1 didn't work. I'll just have to stay curious!

Thanks once again.

Regards,
Tony.

Signature · 3 lines
Tony Arnold, IT Security Analyst, University of Manchester.
T: +44 (0) 161 275 6093, F: +44 (0) 705 344 3082,
M: +44 (0) 773 330 0039, E: [email protected]
#3 lore raw

Dne 26.11.2015 v 11:29 Tony Arnold napsal(a):

Quoted 24 lines

John,

Quoted 18 lines · John Stoffel

Can I make a suggestion to you? Instead of trying to do the mirroring in LVM, build yourself an MD mirror, then layer LVM ontop of that instead. You get nice seperation.

You can also take that /dev/sdb3, make it into an MD RAID1 array with just one device, add it into the VG, then move the LVs onto the new device. Once that's done, you remove the sda3 from the VG, then add it into the MD device. Something like this...

Quoted 8 lines

mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb3 missing cat /proc/mdstat
pvcreate -v /dev/md1
vgextend -v vg0 /dev/md1
pvmove -v sda3
vgreduce -v vg0 sda3
mdadm --add /dev/md0 /dev/sda3
cat /proc/mdstat

Quoted 1 line

And you will now be mirrored, etc.

Quoted 7 lines

Thanks for your suggestion which I have now implemented. The pvmove took a few hours (1.5TB to move!) and then also some more hours while the mirror set synchronised. But it's all done and I now have my volumes mirrored.

I'm still curious though why the LVM based RAID1 didn't work. I'll just have to stay curious!

4.1 & 4.2 kernel has broken RAID

DO NOT USE (at least with lvm2 - but since the flaw there is md driver I assume mdadm is affected as well)

Switch to 4.3 please.

Regards

Zdenek

#4 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Zdenek" == Zdenek Kabelac <[email protected]> writes:

Zdenek> Dne 26.11.2015 v 11:29 Tony Arnold napsal(a):

Quoted 24 lines
Quoted 24 lines

John,

Quoted 19 lines · John Stoffel

Can I make a suggestion to you? Instead of trying to do the mirroring in LVM, build yourself an MD mirror, then layer LVM ontop of that instead. You get nice seperation.

You can also take that /dev/sdb3, make it into an MD RAID1 array with just one device, add it into the VG, then move the LVs onto the new device. Once that's done, you remove the sda3 from the VG, then add it into the MD device. Something like this...

Quoted 8 lines

mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb3 missing cat /proc/mdstat
pvcreate -v /dev/md1
vgextend -v vg0 /dev/md1
pvmove -v sda3
vgreduce -v vg0 sda3
mdadm --add /dev/md0 /dev/sda3
cat /proc/mdstat

And you will now be mirrored, etc.

Quoted 7 lines
Quoted 7 lines

Thanks for your suggestion which I have now implemented. The pvmove took a few hours (1.5TB to move!) and then also some more hours while the mirror set synchronised. But it's all done and I now have my volumes mirrored.

I'm still curious though why the LVM based RAID1 didn't work. I'll just have to stay curious!

Zdenek> 4.1 & 4.2 kernel has broken RAID

Zdenek> DO NOT USE (at least with lvm2 - but since the flaw there is md driver Zdenek> I assume mdadm is affected as well)

Zdenek> Switch to 4.3 please.

Was this fixed in 4.2.6 by any chance? That's what I'm currently running...

#5 lore raw
In reply to Tony Arnold
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Tony" == Tony Arnold <[email protected]> writes:

Tony> John,

Quoted 18 lines · John Stoffel
Quoted 18 lines

Can I make a suggestion to you? Instead of trying to do the mirroring in LVM, build yourself an MD mirror, then layer LVM ontop of that instead. You get nice seperation.

You can also take that /dev/sdb3, make it into an MD RAID1 array with just one device, add it into the VG, then move the LVs onto the new device. Once that's done, you remove the sda3 from the VG, then add it into the MD device. Something like this...

Quoted 8 lines

mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb3 missing cat /proc/mdstat
pvcreate -v /dev/md1
vgextend -v vg0 /dev/md1
pvmove -v sda3
vgreduce -v vg0 sda3
mdadm --add /dev/md0 /dev/sda3
cat /proc/mdstat

Quoted 1 line
Quoted 1 line

And you will now be mirrored, etc.

Tony> Thanks for your suggestion which I have now implemented. The pvmove took Tony> a few hours (1.5TB to move!) and then also some more hours while the Tony> mirror set synchronised. But it's all done and I now have my volumes Tony> mirrored.

Tony> I'm still curious though why the LVM based RAID1 didn't work. I'll just Tony> have to stay curious!

I suspect it could have been made to work, but in my experience, there's a very active MD developer (Neil, who would love more help!) who does a great job of the pure RAID features that MD provides.

The DM guys are more based on logical block management in my experience, and so don't do nearly as much in the RAID side of things.

But in any case, the above setup works really well, and gives you a good seperation of the layers and is easy to manage and control.

I'm planning on migrating all my setup to a lvm-cache setup, with a pair of 4TB disk mirrored (and possibly a third mirror of two striped 2Tb drives... belt and suspenders!) with a pair of MX200 SSDS, 500Gb for boot and cache duties. Should be a nice fast setup, and take less power as well.

John

#6 lore raw

John,

Quoted 9 lines · John Stoffel

I suspect it could have been made to work, but in my experience, there's a very active MD developer (Neil, who would love more help!) who does a great job of the pure RAID features that MD provides.

The DM guys are more based on logical block management in my experience, and so don't do nearly as much in the RAID side of things.

Quoted 8 lines

But in any case, the above setup works really well, and gives you a good seperation of the layers and is easy to manage and control.

I'm planning on migrating all my setup to a lvm-cache setup, with a pair of 4TB disk mirrored (and possibly a third mirror of two striped 2Tb drives... belt and suspenders!) with a pair of MX200 SSDS, 500Gb for boot and cache duties. Should be a nice fast setup, and take less power as well.

That sounds interesting and also fun!

My next step is to get the rest of the partitions on my two disks (/boot etc) mirrored so if I lose one of the disks the system will still boot up and run. I've seen some guidance on how to do this.

Regards,
Tony.

Signature · 3 lines
Tony Arnold, IT Security Analyst, University of Manchester.
T: +44 (0) 161 275 6093, F: +44 (0) 705 344 3082,
M: +44 (0) 773 330 0039, E: [email protected]