Skip to contentSkip to conversation

Is MD array (containing a PV) reducible?

linux-lvm Brian J. Murrell 3 participants 11 messages Oct 2, 2023 – Oct 5, 2023

lore
Share & tools
lore
#0 lore raw

Hi.

Just hoping to get a sanity check here. I have a PV on an mdadm raid array. the PV's size is:

  PV Size               3.64 TiB / not usable 2.84 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              953829
  Free PE               35813
  Allocated PE          918016

The mdadm array's size is:

        Array Size : 3,906,887,512 (3.64 TiB 4.00 TB)
     Used Dev Size : 3,906,887,512 (3.64 TiB 4.00 TB)

I can shrink this array down to 3,906,886,464 without impacting the PV (and it's contents) correct?

From my calculations there is 4,022,272 bytes of unused space after the PV to the end of the array so shrinking it by 1,073,152 shouldn't have any impact, correct?

Cheers,
b.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#1 lore raw
Quoted 2 lines · Brian J. Murrell

I can shrink this array down to 3,906,886,464 without impacting the PV (and it's contents) correct?

Looks that way. That would be how much space is left in the array that is not quite enough for another 4 MiB PE.

Quoted 3 lines

From my calculations there is 4,022,272 bytes of unused space after the PV to the end of the array so shrinking it by 1,073,152 shouldn't have any impact, correct?

It shouldn't, but why bother?

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
Quoted 4 lines · Phillip Susi

Looks that way.  That would be how much space is left in the array that
is not quite enough for another 4 MiB PE.

Good. Thanks for the sanity check.

Quoted 1 line

It shouldn't, but why bother?

Because I need to add another member to the array that is just slightly smaller than the existing array. It's the same sized disk as what's in the array currently but is itself an MD array with luks, which is reserving it's own metadata region from the total disk making it slightly smaller.

So yes, ultimately I am going to have both an encrypted and unencrypted member of this raid-1 array. At least until the encrypted member is added and then I will remove the unencrypted member and add it to the luks encrypted array.

The general idea is to (safely -- as in be able to achieve in specific sepearate steps that have a back-out plan) convert a non-encrypted raid-1 array into an encrypted raid-1 array.

So what I have right now is:

sdc                                             8:32   0   3.7T  0 disk  
└─md0                                           9:0    0   3.7T  0 raid1 
  ├─backups-backups                           253:13   0   2.5T  0 lvm   /.snapshots
  ├─backups-borg                              253:14   0   300G  0 lvm   
  └─backups-vdo--test                         253:15   0   700G  0 lvm   
    └─vdo-backups                             253:73   0   1.8T  0 vdo   
sdd                                             8:48   0   3.7T  0 disk  
└─md1                                           9:1    0   3.7T  0 raid1 
  └─luks-backup                               253:40   0   3.7T  0 crypt 

I am going to reduce the size of md0 so that it's the same size as md1, then add md1 to md0 (so yes, I will have a raid-1 array with a raid-1 member). Then I am going to remove md0 from the md1 array and then add sdc to md1 so that md0 no longer exists and md1 is the primary raid-1 array. Ideally, rename md1 to md0 once done but that's aesthetic.

Ultimately I should end up (prior to any attempt to rename) md1 with sdd and sdc as an encrypted raid-1 array. The goal is to be able to remove a disk from the raid array and safely discard/return/recycle/etc. it knowing that it's content is relatively secure.

The most problematic scenario of course is a disk dies and you want to return it under warranty but you cannot access the disk to "scrub" it and you either send it back with all kinds of sensitive data on it, or you eat it.

Cheers,
b.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#3 lore raw
Quoted 3 lines

The general idea is to (safely -- as in be able to achieve in specific sepearate steps that have a back-out plan) convert a non-encrypted raid-1 array into an encrypted raid-1 array.

i have a very similar setup
can you document the steps as you do the conversion

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#4 lore raw
In reply to Brian J. Murrell
Quoted 5 lines · Brian J. Murrell

Because I need to add another member to the array that is just slightly smaller than the existing array. It's the same sized disk as what's in the array currently but is itself an MD array with luks, which is reserving it's own metadata region from the total disk making it slightly smaller.

You want to put an MD array inside of another MD array? Why?

Quoted 10 lines

So yes, ultimately I am going to have both an encrypted and unencrypted member of this raid-1 array. At least until the encrypted member is added and then I will remove the unencrypted member and add it to the luks encrypted array.

The general idea is to (safely -- as in be able to achieve in specific sepearate steps that have a back-out plan) convert a non-encrypted raid-1 array into an encrypted raid-1 array.

So what I have right now is:

Quoted 9 lines
sdc                                             8:32   0   3.7T  0 disk  
└─md0                                           9:0    0   3.7T  0 raid1 
  ├─backups-backups                           253:13   0   2.5T  0 lvm   /.snapshots
  ├─backups-borg                              253:14   0   300G  0 lvm   
  └─backups-vdo--test                         253:15   0   700G  0 lvm   
    └─vdo-backups                             253:73   0   1.8T  0 vdo   
sdd                                             8:48   0   3.7T  0 disk  
└─md1                                           9:1    0   3.7T  0 raid1 
  └─luks-backup                               253:40   0   3.7T  0 crypt 

I would suggest that rather than

Quoted 5 lines

I am going to reduce the size of md0 so that it's the same size as md1, then add md1 to md0 (so yes, I will have a raid-1 array with a raid-1 member). Then I am going to remove md0 from the md1 array and then add sdc to md1 so that md0 no longer exists and md1 is the primary raid-1 array. Ideally, rename md1 to md0 once done but that's aesthetic.

I would suggest instead that you add the luks volume as a new PV to your VG, then use pvmove to migrate all of your logical volumes from md0 to md1, then vgreduce md0 from the VG, stop md0, mdadm --zero-superblocks /dev/sdc, then add sdc to md1.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#5 lore raw
Quoted 2 lines · Phillip Susi

You want to put an MD array inside of another MD array?  Why?

Just to take advantage of MD's online/background sync to get the data from md0 array to the md1 array.

Quoted 7 lines

I would suggest instead that you add the luks volume as a new PV to your
VG, then use pvmove to migrate all of your logical volumes from md0 to
md1, then vgreduce md0 from the VG, stop md0, mdadm --zero- superblocks
/dev/sdc, then add sdc to md1.

That's an interesting alternative. Thanks for the suggestion.

Cheers,
b.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#7 lore raw
Quoted 10 lines · Phillip Susi
Quoted 3 lines · Brian J. Murrell

Just to take advantage of MD's online/background sync to get the data
from md0 array to the md1 array.

If you weren't using LVM, then that would be a good idea, but LVM can do
that too.

So just to confirm, given:

sdc                                             8:32   0   3.7T  0 disk  
└─md0                                           9:0    0   3.7T  0 raid1 
  ├─backups-backups                           253:13   0   2.5T  0 lvm   /.snapshots
  ├─backups-borg                              253:14   0   300G  0 lvm   
  └─backups-vdo--test                         253:15   0   700G  0 lvm   
    └─vdo-backups                             253:73   0   1.8T  0 vdo   
sdd                                             8:48   0   3.7T  0 disk  
└─md1                                           9:1    0   3.7T  0 raid1 
  └─luks-backup                               253:40   0   3.7T  0 crypt 

I want to do:

# pvcreate /dev/mapper/luks-backup
# vgextend backups /dev/mapper/luks-backup
# pvmove /dev/md0 /dev/luks-backup
# vgreduce backups /dev/md0
# mdadm --stop /dev/md0
# mdadm --zero-superblocks /dev/sdc
# mdadm /dev/md1 --add /dev/sdc

Correct?

Cheers,
b.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#8 lore raw
Quoted 22 lines · Brian J. Murrell

So just to confirm, given:

sdc                                             8:32   0   3.7T  0 disk  
└─md0                                           9:0    0   3.7T  0 raid1 
  ├─backups-backups                           253:13   0   2.5T  0 lvm   /.snapshots
  ├─backups-borg                              253:14   0   300G  0 lvm   
  └─backups-vdo--test                         253:15   0   700G  0 lvm   
    └─vdo-backups                             253:73   0   1.8T  0 vdo   
sdd                                             8:48   0   3.7T  0 disk  
└─md1                                           9:1    0   3.7T  0 raid1 
  └─luks-backup                               253:40   0   3.7T  0 crypt 

I want to do:

# pvcreate /dev/mapper/luks-backup
# vgextend backups /dev/mapper/luks-backup
# pvmove /dev/md0 /dev/luks-backup
# vgreduce backups /dev/md0
# mdadm --stop /dev/md0
# mdadm --zero-superblocks /dev/sdc
# mdadm /dev/md1 --add /dev/sdc
Quoted 1 line

Correct?

Yep.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#9 lore raw
Quoted 2 lines · Phillip Susi

Yep.

Much thanks.

But I did just think of the other benefit of using MD to sync the new device and that's having a recovery path.

That is, if I add luks-backup to /dev/md0 and then wait for it to finish syncing, I can then shut down the machine, remove /dev/sdc from the machine and start it back up and ensure that all of the needed initialization bits are in place to make the luks component come up and be readable. If it is, I just remove it from md0 and use it stand- alone (as it is itself a RAID-1 recall). Reboot again to make sure it's all still working and good.

If any of the above goes sideways, I still have the sdc disk as an md0 member and can put it back in the machine to get back to my starting position and try again, trying to figure out where I went wrong.

With pvmove, once the move is complete there is no going back to my starting position if something is not right and I cannot access my new luks device.

So given that, and given that I have stared my pvmove with --atomic and --interval, do I just SIGINT the pvmove? Or should I do pvmove --abort on another terminal? How can I know when the the abort is complete and that I can vgreduce luks-backup out of the backups volume group?

Once I have vgreduced, is there anything I should do to /dev/mapper/luks-backup before I mdadm --manage /dev/md0 --add /dev/mapper/luks-backup just to wipe any remnants of it being a PV previously?

I really do appreciate all of the help, advise and patience you have given/shown me.

Cheers,
b.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#10 lore raw
Quoted 11 lines · Brian J. Murrell

But I did just think of the other benefit of using MD to sync the new device and that's having a recovery path.

That is, if I add luks-backup to /dev/md0 and then wait for it to finish syncing, I can then shut down the machine, remove /dev/sdc from the machine and start it back up and ensure that all of the needed initialization bits are in place to make the luks component come up and be readable. If it is, I just remove it from md0 and use it stand- alone (as it is itself a RAID-1 recall). Reboot again to make sure it's all still working and good.

Quoted 7 lines

If any of the above goes sideways, I still have the sdc disk as an md0 member and can put it back in the machine to get back to my starting position and try again, trying to figure out where I went wrong.

With pvmove, once the move is complete there is no going back to my starting position if something is not right and I cannot access my new luks device.

Instead of pvmove, you can use lvconvert to have LVM mirror across the drives, then break the mirror, like you would with mdadm.

Quoted 4 lines

So given that, and given that I have stared my pvmove with --atomic and --interval, do I just SIGINT the pvmove? Or should I do pvmove --abort on another terminal? How can I know when the the abort is complete and that I can vgreduce luks-backup out of the backups volume group?

lvs or lvdisplay will show if the move is still in progress.

Quoted 4 lines

Once I have vgreduced, is there anything I should do to /dev/mapper/luks-backup before I mdadm --manage /dev/md0 --add /dev/mapper/luks-backup just to wipe any remnants of it being a PV previously?

pvremove will wipe the pv label.

Quoted 2 lines

I really do appreciate all of the help, advise and patience you have given/shown me.

You are welcome.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/