Problems creating/converting large logical volume to RAID1
I'm trying to convert one of my logical volumes to be RAID1 mirrored using LVM on my Ubuntu 15.10 system. When I do, the command lvs shows the mirror set is 100% synchronized immediately. syslog show a failure saying one of the devices in the mirror set has failed and it reverts to a single device mirror set.
Errors include saying attempt to access beyond end of device. There is also an error showing code=-5 for 'super_written'. A full extract of syslog is below.
The volume group is configured so:
root@the-circus:~# pvs PV VG Fmt Attr PSize PFree /dev/sda3 vg0 lvm2 a-- 2.73t 1.09t /dev/sdb3 vg0 lvm2 a-- 2.73t 2.68t
So it's trying to create the mirror devices on /dev/sdb3.
The disks are 3TB Seagate Barracuda disks. They both brand new.
Experimentation shows I only get these errors for logical volumes bigger than 31GB, anything smaller behaves correctly.
My version of Ubuntu:
# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 15.10 Release: 15.10 Codename: wily
Kernel is:
# uname -a Linux the-circus 4.2.0-18-generic #22-Ubuntu SMP Fri Nov 6 18:25:50 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Anyone any ideas? I've googled this until the cows come home but cannot find anything helpful.
Regards,
Tony.
Nov 21 17:29:52 the-circus kernel: [ 231.538111] device-mapper: raid: Loading target version 1.7.0 Nov 21 17:29:52 the-circus kernel: [ 231.541976] device-mapper: raid: Superblocks created for new array Nov 21 17:29:52 the-circus kernel: [ 231.542894] md/raid1:mdX: not clean -- starting background reconstruction Nov 21 17:29:52 the-circus kernel: [ 231.542897] md/raid1:mdX: active with 2 out of 2 mirrors Nov 21 17:29:52 the-circus kernel: [ 231.542899] Choosing daemon_sleep default (5 sec) Nov 21 17:29:52 the-circus kernel: [ 231.542903] created bitmap (50 pages) for device mdX Nov 21 17:29:52 the-circus kernel: [ 231.542906] attempt to access beyond end of device Nov 21 17:29:52 the-circus kernel: [ 231.542908] dm-7: rw=13329, want=18446744073709551608, limit=8192 Nov 21 17:29:52 the-circus kernel: [ 231.542909] md: super_written gets error=-5, uptodate=0 Nov 21 17:29:52 the-circus kernel: [ 231.542920] md/raid1:mdX: Disk failure on dm-8, disabling device. Nov 21 17:29:52 the-circus kernel: [ 231.542920] md/raid1:mdX: Operation continuing on 1 devices. Nov 21 17:29:52 the-circus kernel: [ 231.542933] attempt to access beyond end of device Nov 21 17:29:52 the-circus kernel: [ 231.542936] dm-5: rw=13329, want=18446744073709551608, limit=8192 Nov 21 17:29:52 the-circus kernel: [ 231.542939] md: super_written gets error=-5, uptodate=0 Nov 21 17:29:52 the-circus kernel: [ 231.569851] attempt to access beyond end of device Nov 21 17:29:52 the-circus kernel: [ 231.569859] dm-5: rw=13329, want=18446744073709551608, limit=8192 Nov 21 17:29:52 the-circus kernel: [ 231.569863] md: super_written gets error=-5, uptodate=0 Nov 21 17:29:52 the-circus kernel: [ 231.596802] mdX: bitmap file is out of date, doing full recovery Nov 21 17:29:52 the-circus kernel: [ 231.712193] mdX: bitmap initialized from disk: read 4 pages, set 102400 of 102400 bits Nov 21 17:29:52 the-circus systemd[1]: Started Device-mapper event daemon. Nov 21 17:29:52 the-circus dmeventd[3132]: dmeventd ready for processing. Nov 21 17:29:52 the-circus kernel: [ 231.786858] RAID1 conf printout: Nov 21 17:29:52 the-circus kernel: [ 231.786863] --- wd:1 rd:2 Nov 21 17:29:52 the-circus kernel: [ 231.786866] disk 0, wo:0, o:1, dev:dm-6 Nov 21 17:29:52 the-circus kernel: [ 231.786868] disk 1, wo:1, o:0, dev:dm-8 Nov 21 17:29:52 the-circus kernel: [ 231.793942] RAID1 conf printout: Nov 21 17:29:52 the-circus kernel: [ 231.793947] --- wd:1 rd:2 Nov 21 17:29:52 the-circus kernel: [ 231.793949] disk 0, wo:0, o:1, dev:dm-6 Nov 21 17:29:52 the-circus kernel: [ 231.794033] md: resync of RAID array mdX Nov 21 17:29:52 the-circus kernel: [ 231.794037] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. Nov 21 17:29:52 the-circus kernel: [ 231.794039] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync. Nov 21 17:29:52 the-circus kernel: [ 231.794042] md: using 128k window, over a total of 52428800k. Nov 21 17:29:52 the-circus kernel: [ 231.794196] md: mdX: resync done. Nov 21 17:29:53 the-circus kernel: [ 232.037242] RAID1 conf printout: Nov 21 17:29:53 the-circus kernel: [ 232.037248] --- wd:1 rd:2 Nov 21 17:29:53 the-circus kernel: [ 232.037251] disk 0, wo:0, o:1, dev:dm-6 Nov 21 17:29:53 the-circus lvm[3132]: Monitoring RAID device vg0-testlv for events. Nov 21 17:29:53 the-circus lvm[3132]: Device #1 of raid1 array, vg0-testlv, has failed. Nov 21 17:29:53 the-circus lvm[3132]: Use 'lvconvert --repair
vg0/testlv' to replace failed device.
Signature · 9 lines
Tony Arnold, IT Security Analyst, University of Manchester. T: +44 (0) 161 275 6093, F: +44 (0) 705 344 3082, M: +44 (0) 773 330 0039, E: [email protected] -- ubuntu-users mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Tony,
Can I make a suggestion to you? Instead of trying to do the mirroring in LVM, build yourself an MD mirror, then layer LVM ontop of that instead. You get nice seperation.
You can also take that /dev/sdb3, make it into an MD RAID1 array with just one device, add it into the VG, then move the LVs onto the new device. Once that's done, you remove the sda3 from the VG, then add it into the MD device. Something like this...
Quoted 8 lines
mdadm --create
/dev/md1--level=1 --raid-devices=2/dev/sdb3missing cat/proc/mdstat
pvcreate -v/dev/md1
vgextend -v vg0/dev/md1
pvmove -v sda3
vgreduce -v vg0 sda3
mdadm --add/dev/md0/dev/sda3
cat/proc/mdstat
And you will now be mirrored, etc.
Tony> I'm trying to convert one of my logical volumes to be RAID1 mirrored Tony> using LVM on my Ubuntu 15.10 system. When I do, the command lvs shows Tony> the mirror set is 100% synchronized immediately. syslog show a failure Tony> saying one of the devices in the mirror set has failed and it reverts to Tony> a single device mirror set.
Tony> Errors include saying attempt to access beyond end of device. There is Tony> also an error showing code=-5 for 'super_written'. A full extract of Tony> syslog is below.
Tony> The volume group is configured so:
Tony> root@the-circus:~# pvs
Tony> PV VG Fmt Attr PSize PFree Tony> /dev/sda3 vg0 lvm2 a-- 2.73t 1.09t Tony> /dev/sdb3 vg0 lvm2 a-- 2.73t 2.68t
Tony> So it's trying to create the mirror devices on /dev/sdb3.
Tony> The disks are 3TB Seagate Barracuda disks. They both brand new.
Tony> Experimentation shows I only get these errors for logical volumes bigger Tony> than 31GB, anything smaller behaves correctly.
Tony> My version of Ubuntu:
Tony> # lsb_release -a
Tony> No LSB modules are available.
Tony> Distributor ID: Ubuntu
Tony> Description: Ubuntu 15.10
Tony> Release: 15.10
Tony> Codename: wily
Tony> Kernel is:
Tony> # uname -a
Tony> Linux the-circus 4.2.0-18-generic #22-Ubuntu SMP Fri Nov 6 18:25:50 UTC Tony> 2015 x86_64 x86_64 x86_64 GNU/Linux
Tony> Anyone any ideas? I've googled this until the cows come home but cannot Tony> find anything helpful.
Tony> Regards,
Tony> Tony.
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.538111] device-mapper: raid:
Tony> Loading target version 1.7.0
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.541976] device-mapper: raid:
Tony> Superblocks created for new array
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542894] md/raid1:mdX: not Tony> clean -- starting background reconstruction
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542897] md/raid1:mdX: active Tony> with 2 out of 2 mirrors
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542899] Choosing daemon_sleep Tony> default (5 sec)
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542903] created bitmap (50 Tony> pages) for device mdX
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542906] attempt to access Tony> beyond end of device
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542908] dm-7: rw=13329, Tony> want=18446744073709551608, limit=8192
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542909] md: super_written gets Tony> error=-5, uptodate=0
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542920] md/raid1:mdX: Disk Tony> failure on dm-8, disabling device.
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542920] md/raid1:mdX:
Tony> Operation continuing on 1 devices.
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542933] attempt to access Tony> beyond end of device
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542936] dm-5: rw=13329, Tony> want=18446744073709551608, limit=8192
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.542939] md: super_written gets Tony> error=-5, uptodate=0
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.569851] attempt to access Tony> beyond end of device
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.569859] dm-5: rw=13329, Tony> want=18446744073709551608, limit=8192
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.569863] md: super_written gets Tony> error=-5, uptodate=0
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.596802] mdX: bitmap file is Tony> out of date, doing full recovery
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.712193] mdX: bitmap Tony> initialized from disk: read 4 pages, set 102400 of 102400 bits Tony> Nov 21 17:29:52 the-circus systemd[1]: Started Device-mapper event daemon. Tony> Nov 21 17:29:52 the-circus dmeventd[3132]: dmeventd ready for processing.
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.786858] RAID1 conf printout: Tony> Nov 21 17:29:52 the-circus kernel: [ 231.786863] --- wd:1 rd:2 Tony> Nov 21 17:29:52 the-circus kernel: [ 231.786866] disk 0, wo:0, o:1, Tony> dev:dm-6 Tony> Nov 21 17:29:52 the-circus kernel: [ 231.786868] disk 1, wo:1, o:0, Tony> dev:dm-8 Tony> Nov 21 17:29:52 the-circus kernel: [ 231.793942] RAID1 conf printout: Tony> Nov 21 17:29:52 the-circus kernel: [ 231.793947] --- wd:1 rd:2 Tony> Nov 21 17:29:52 the-circus kernel: [ 231.793949] disk 0, wo:0, o:1,
Tony> dev:dm-6
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.794033] md: resync of RAID Tony> array mdX
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.794037] md: minimum Tony> _guaranteed_ speed: 1000 KB/sec/disk.
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.794039] md: using maximum Tony> available idle IO bandwidth (but not more than 200000 KB/sec) for resync. Tony> Nov 21 17:29:52 the-circus kernel: [ 231.794042] md: using 128k window, Tony> over a total of 52428800k.
Tony> Nov 21 17:29:52 the-circus kernel: [ 231.794196] md: mdX: resync done. Tony> Nov 21 17:29:53 the-circus kernel: [ 232.037242] RAID1 conf printout: Tony> Nov 21 17:29:53 the-circus kernel: [ 232.037248] --- wd:1 rd:2 Tony> Nov 21 17:29:53 the-circus kernel: [ 232.037251] disk 0, wo:0, o:1,
Tony> dev:dm-6
Tony> Nov 21 17:29:53 the-circus lvm[3132]: Monitoring RAID device vg0-testlv Tony> for events.
Tony> Nov 21 17:29:53 the-circus lvm[3132]: Device #1 of raid1 array, Tony> vg0-testlv, has failed.
Tony> Nov 21 17:29:53 the-circus lvm[3132]: Use 'lvconvert --repair Tony> vg0/testlv' to replace failed device.
Tony> --
Tony> Tony Arnold, IT Security Analyst, University of Manchester. Tony> T: +44 (0) 161 275 6093, F: +44 (0) 705 344 3082, Tony> M: +44 (0) 773 330 0039, E: [email protected]
Tony> --
Tony> ubuntu-users mailing list
Tony> [email protected]
Tony> Modify settings or unsubscribe at:
Tony> https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Tony> _______________________________________________ Tony> linux-lvm mailing list
Tony> [email protected]
Tony> https://www.redhat.com/mailman/listinfo/linux-lvm
Tony> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
John,
Quoted 18 lines · John Stoffel
Can I make a suggestion to you? Instead of trying to do the mirroring in LVM, build yourself an MD mirror, then layer LVM ontop of that instead. You get nice seperation.
You can also take that
/dev/sdb3, make it into an MD RAID1 array with just one device, add it into the VG, then move the LVs onto the new device. Once that's done, you remove the sda3 from the VG, then add it into the MD device. Something like this...Quoted 8 lines
mdadm --create
/dev/md1--level=1 --raid-devices=2/dev/sdb3missing cat/proc/mdstat
pvcreate -v/dev/md1
vgextend -v vg0/dev/md1
pvmove -v sda3
vgreduce -v vg0 sda3
mdadm --add/dev/md0/dev/sda3
cat/proc/mdstat
Quoted 1 line
And you will now be mirrored, etc.
Thanks for your suggestion which I have now implemented. The pvmove took a few hours (1.5TB to move!) and then also some more hours while the mirror set synchronised. But it's all done and I now have my volumes mirrored.
I'm still curious though why the LVM based RAID1 didn't work. I'll just have to stay curious!
Thanks once again.
Regards,
Tony.
Signature · 3 lines
Tony Arnold, IT Security Analyst, University of Manchester. T: +44 (0) 161 275 6093, F: +44 (0) 705 344 3082, M: +44 (0) 773 330 0039, E: [email protected]
Dne 26.11.2015 v 11:29 Tony Arnold napsal(a):
Quoted 24 lines
John,
Quoted 18 lines · John Stoffel
Can I make a suggestion to you? Instead of trying to do the mirroring in LVM, build yourself an MD mirror, then layer LVM ontop of that instead. You get nice seperation.
You can also take that
/dev/sdb3, make it into an MD RAID1 array with just one device, add it into the VG, then move the LVs onto the new device. Once that's done, you remove the sda3 from the VG, then add it into the MD device. Something like this...Quoted 8 lines
mdadm --create
/dev/md1--level=1 --raid-devices=2/dev/sdb3missing cat/proc/mdstat
pvcreate -v/dev/md1
vgextend -v vg0/dev/md1
pvmove -v sda3
vgreduce -v vg0 sda3
mdadm --add/dev/md0/dev/sda3
cat/proc/mdstatQuoted 1 line
And you will now be mirrored, etc.
Quoted 7 lines
Thanks for your suggestion which I have now implemented. The pvmove took a few hours (1.5TB to move!) and then also some more hours while the mirror set synchronised. But it's all done and I now have my volumes mirrored.
I'm still curious though why the LVM based RAID1 didn't work. I'll just have to stay curious!
4.1 & 4.2 kernel has broken RAID
DO NOT USE (at least with lvm2 - but since the flaw there is md driver I assume mdadm is affected as well)
Switch to 4.3 please.
Regards
Zdenek
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Zdenek" == Zdenek Kabelac <[email protected]> writes:
Zdenek> Dne 26.11.2015 v 11:29 Tony Arnold napsal(a):
Quoted 24 lines
Quoted 24 lines
John,
Quoted 19 lines · John Stoffel
Can I make a suggestion to you? Instead of trying to do the mirroring in LVM, build yourself an MD mirror, then layer LVM ontop of that instead. You get nice seperation.
You can also take that
/dev/sdb3, make it into an MD RAID1 array with just one device, add it into the VG, then move the LVs onto the new device. Once that's done, you remove the sda3 from the VG, then add it into the MD device. Something like this...Quoted 8 lines
mdadm --create
/dev/md1--level=1 --raid-devices=2/dev/sdb3missing cat/proc/mdstat
pvcreate -v/dev/md1
vgextend -v vg0/dev/md1
pvmove -v sda3
vgreduce -v vg0 sda3
mdadm --add/dev/md0/dev/sda3
cat/proc/mdstatAnd you will now be mirrored, etc.
Quoted 7 lines
Quoted 7 lines
Thanks for your suggestion which I have now implemented. The pvmove took a few hours (1.5TB to move!) and then also some more hours while the mirror set synchronised. But it's all done and I now have my volumes mirrored.
I'm still curious though why the LVM based RAID1 didn't work. I'll just have to stay curious!
Zdenek> 4.1 & 4.2 kernel has broken RAID
Zdenek> DO NOT USE (at least with lvm2 - but since the flaw there is md driver Zdenek> I assume mdadm is affected as well)
Zdenek> Switch to 4.3 please.
Was this fixed in 4.2.6 by any chance? That's what I'm currently running...
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Tony" == Tony Arnold <[email protected]> writes:
Tony> John,
Quoted 18 lines · John Stoffel
Quoted 18 lines
Can I make a suggestion to you? Instead of trying to do the mirroring in LVM, build yourself an MD mirror, then layer LVM ontop of that instead. You get nice seperation.
You can also take that
/dev/sdb3, make it into an MD RAID1 array with just one device, add it into the VG, then move the LVs onto the new device. Once that's done, you remove the sda3 from the VG, then add it into the MD device. Something like this...Quoted 8 lines
mdadm --create
/dev/md1--level=1 --raid-devices=2/dev/sdb3missing cat/proc/mdstat
pvcreate -v/dev/md1
vgextend -v vg0/dev/md1
pvmove -v sda3
vgreduce -v vg0 sda3
mdadm --add/dev/md0/dev/sda3
cat/proc/mdstat
Quoted 1 line
Quoted 1 line
And you will now be mirrored, etc.
Tony> Thanks for your suggestion which I have now implemented. The pvmove took Tony> a few hours (1.5TB to move!) and then also some more hours while the Tony> mirror set synchronised. But it's all done and I now have my volumes Tony> mirrored.
Tony> I'm still curious though why the LVM based RAID1 didn't work. I'll just Tony> have to stay curious!
I suspect it could have been made to work, but in my experience, there's a very active MD developer (Neil, who would love more help!) who does a great job of the pure RAID features that MD provides.
The DM guys are more based on logical block management in my experience, and so don't do nearly as much in the RAID side of things.
But in any case, the above setup works really well, and gives you a good seperation of the layers and is easy to manage and control.
I'm planning on migrating all my setup to a lvm-cache setup, with a pair of 4TB disk mirrored (and possibly a third mirror of two striped 2Tb drives... belt and suspenders!) with a pair of MX200 SSDS, 500Gb for boot and cache duties. Should be a nice fast setup, and take less power as well.
John
John,
Quoted 9 lines · John Stoffel
I suspect it could have been made to work, but in my experience, there's a very active MD developer (Neil, who would love more help!) who does a great job of the pure RAID features that MD provides.
The DM guys are more based on logical block management in my experience, and so don't do nearly as much in the RAID side of things.
Quoted 8 lines
But in any case, the above setup works really well, and gives you a good seperation of the layers and is easy to manage and control.
I'm planning on migrating all my setup to a lvm-cache setup, with a pair of 4TB disk mirrored (and possibly a third mirror of two striped 2Tb drives... belt and suspenders!) with a pair of MX200 SSDS, 500Gb for boot and cache duties. Should be a nice fast setup, and take less power as well.
That sounds interesting and also fun!
My next step is to get the rest of the partitions on my two disks (/boot etc) mirrored so if I lose one of the disks the system will still boot up and run. I've seen some guidance on how to do this.
Regards,
Tony.
Signature · 3 lines
Tony Arnold, IT Security Analyst, University of Manchester. T: +44 (0) 161 275 6093, F: +44 (0) 705 344 3082, M: +44 (0) 773 330 0039, E: [email protected]