Skip to contentSkip to conversation

balance conversion from RAID-1 to RAID-10 leaves some metadata in RAID-1?

DISCUSSION Tomasz Chmielewski 3 participants 5 messages Mar 1, 2020 – Mar 2, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Just finished a conversion from RAID-1 to RAID-10, on Linux 5.5.6:

# time btrfs balance start -dconvert=raid10 -mconvert=raid10 /data/lxd
Done, had to relocate 2064 out of 2064 chunks
real    1811m51.427s
user    0m0.000s
sys     1259m34.448s
# echo $?
0
# dmesg
(...)
[211862.205521] BTRFS info (device sda2): found 40805 extents
[211910.773888] BTRFS info (device sda2): found 40805 extents
[211965.321150] BTRFS info (device sda2): relocating block group 3492979671040 flags data|raid1
[211972.743476] BTRFS info (device sda2): found 37221 extents
[212020.193342] BTRFS info (device sda2): found 37221 extents
[212084.580739] BTRFS info (device sda2): balance: ended with status: 0

To my surprise, some metadata is still RAID-1 - is it expected?

# btrfs fi usage /data/lxd
Overall:
    Device size:                   6.91TiB
    Device allocated:              6.70TiB
    Device unallocated:          210.40GiB
    Device missing:                  0.00B
    Used:                          3.96TiB
    Free (estimated):              1.46TiB      (min: 1.46TiB)
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,RAID10: Size:3.32TiB, Used:1.97TiB (59.20%)

   /dev/sda2       1.66TiB
   /dev/sdd2       1.66TiB
   /dev/sdc2       1.66TiB
   /dev/sdb2       1.66TiB

Metadata,RAID1: Size:2.00GiB, Used:85.02MiB (4.15%)
/dev/sdc2 2.00GiB
/dev/sdb2 2.00GiB

Metadata,RAID10: Size:29.00GiB, Used:12.55GiB (43.27%)

   /dev/sda2      14.50GiB
   /dev/sdd2      14.50GiB
   /dev/sdc2      14.50GiB
   /dev/sdb2      14.50GiB

System,RAID10: Size:64.00MiB, Used:400.00KiB (0.61%)

   /dev/sda2      32.00MiB
   /dev/sdd2      32.00MiB
   /dev/sdc2      32.00MiB
   /dev/sdb2      32.00MiB

Unallocated:

   /dev/sda2      53.60GiB
   /dev/sdd2      53.60GiB
   /dev/sdc2      51.60GiB
   /dev/sdb2      51.60GiB
# btrfs fi df /data/lxd
Data, RAID10: total=3.32TiB, used=1.97TiB
System, RAID10: total=64.00MiB, used=400.00KiB
Metadata, RAID10: total=29.00GiB, used=12.55GiB
Metadata, RAID1: total=2.00GiB, used=85.02MiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi show /data/lxd
Label: 'lxd5'  uuid: 2b77b498-a644-430b-9dd9-2ad3d381448a
        Total devices 4 FS bytes used 1.98TiB
        devid    1 size 1.73TiB used 1.67TiB path /dev/sda2
        devid    3 size 1.73TiB used 1.67TiB path /dev/sdd2
        devid    4 size 1.73TiB used 1.68TiB path /dev/sdc2
        devid    5 size 1.73TiB used 1.68TiB path /dev/sdb2

Tomasz Chmielewski
https://lxadm.com

#1 lore raw
Quoted 2 lines · Tomasz Chmielewski

To my surprise, some metadata is still RAID-1 - is it expected?

I'd say it's not expected but also balance is pretty complicated. Try

'btrfs balance start -dconvert=raid10,soft /mountpoint/'

What does dmesg report for that command? And are those raid1 bg's converted to raid10? I don't think it should matter in this case, but what's the btrfs-progs version?

Signature · 1 line
Chris Murphy
#2 lore raw
Quoted 11 lines · Chris Murphy
Quoted 2 lines · Tomasz Chmielewski

To my surprise, some metadata is still RAID-1 - is it expected?

I'd say it's not expected but also balance is pretty complicated. Try

'btrfs balance start -dconvert=raid10,soft /mountpoint/'

What does dmesg report for that command? And are those raid1 bg's converted to raid10? I don't think it should matter in this case, but what's the btrfs-progs version?

I did one more balance, for metadata only:

btrfs balance start -dconvert=raid10 /mountpoint/

And it fully converted the remaining RAID-1 metadata to RAID-10.

Still, a bit "weird".

btrfs-progs is 5.4.1.

Tomasz Chmielewski
https://lxadm.com

#3 lore raw
Quoted 11 lines · Tomasz Chmielewski
Quoted 9 lines · Chris Murphy
Quoted 2 lines · Tomasz Chmielewski

To my surprise, some metadata is still RAID-1 - is it expected?

I'd say it's not expected but also balance is pretty complicated. Try

'btrfs balance start -dconvert=raid10,soft /mountpoint/'

Quoted 7 lines
Quoted 3 lines

What does dmesg report for that command? And are those raid1 bg's converted to raid10? I don't think it should matter in this case, but what's the btrfs-progs version?

I did one more balance, for metadata only:

btrfs balance start -dconvert=raid10 /mountpoint/

Sorry mine had a typo too; should be '-mconvert=raid10,soft /mountpoint/'

Quoted 4 lines

And it fully converted the remaining RAID-1 metadata to RAID-10.

Still, a bit "weird".

Yep, I think Hugo's guess is a likely explanation.

Signature · 1 line
Chris Murphy
#4 lore raw
In reply to Tomasz Chmielewski
Quoted 24 lines · Tomasz Chmielewski

Just finished a conversion from RAID-1 to RAID-10, on Linux 5.5.6:

# time btrfs balance start -dconvert=raid10 -mconvert=raid10 /data/lxd
Done, had to relocate 2064 out of 2064 chunks
real    1811m51.427s
user    0m0.000s
sys     1259m34.448s
# echo $?
0
# dmesg
(...)
[211862.205521] BTRFS info (device sda2): found 40805 extents
[211910.773888] BTRFS info (device sda2): found 40805 extents
[211965.321150] BTRFS info (device sda2): relocating block group
3492979671040 flags data|raid1
[211972.743476] BTRFS info (device sda2): found 37221 extents
[212020.193342] BTRFS info (device sda2): found 37221 extents
[212084.580739] BTRFS info (device sda2): balance: ended with status: 0
Quoted 1 line

To my surprise, some metadata is still RAID-1 - is it expected?

It's definitely something that can happen. I think what happens is
that new chunks are created while the balance process is going, so they never get picked up by the converter. I'd guess it's most likely to happen on an FS with active writes on it while the balance is running.

If it happens again, you can finish off just the unconverted chunks
by adding the "soft" option to the conversion filter:

# btrfs balance start -dconvert=raid10,sort -mconvert=raid10,soft /data/lxd

which will convert only those chunks that aren't already in the target RAID level.

Hugo.

Quoted 62 lines
# btrfs fi usage /data/lxd
Overall:
    Device size:                   6.91TiB
    Device allocated:              6.70TiB
    Device unallocated:          210.40GiB
    Device missing:                  0.00B
    Used:                          3.96TiB
    Free (estimated):              1.46TiB      (min: 1.46TiB)
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,RAID10: Size:3.32TiB, Used:1.97TiB (59.20%)

   /dev/sda2       1.66TiB
   /dev/sdd2       1.66TiB
   /dev/sdc2       1.66TiB
   /dev/sdb2       1.66TiB

Metadata,RAID1: Size:2.00GiB, Used:85.02MiB (4.15%)
/dev/sdc2 2.00GiB
/dev/sdb2 2.00GiB

Metadata,RAID10: Size:29.00GiB, Used:12.55GiB (43.27%)

   /dev/sda2      14.50GiB
   /dev/sdd2      14.50GiB
   /dev/sdc2      14.50GiB
   /dev/sdb2      14.50GiB

System,RAID10: Size:64.00MiB, Used:400.00KiB (0.61%)

   /dev/sda2      32.00MiB
   /dev/sdd2      32.00MiB
   /dev/sdc2      32.00MiB
   /dev/sdb2      32.00MiB

Unallocated:

   /dev/sda2      53.60GiB
   /dev/sdd2      53.60GiB
   /dev/sdc2      51.60GiB
   /dev/sdb2      51.60GiB
# btrfs fi df /data/lxd
Data, RAID10: total=3.32TiB, used=1.97TiB
System, RAID10: total=64.00MiB, used=400.00KiB
Metadata, RAID10: total=29.00GiB, used=12.55GiB
Metadata, RAID1: total=2.00GiB, used=85.02MiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi show /data/lxd
Label: 'lxd5'  uuid: 2b77b498-a644-430b-9dd9-2ad3d381448a
        Total devices 4 FS bytes used 1.98TiB
        devid    1 size 1.73TiB used 1.67TiB path /dev/sda2
        devid    3 size 1.73TiB used 1.67TiB path /dev/sdd2
        devid    4 size 1.73TiB used 1.68TiB path /dev/sdc2
        devid    5 size 1.73TiB used 1.68TiB path /dev/sdb2

Tomasz Chmielewski
https://lxadm.com

Signature · 4 lines
Hugo Mills             | The problem with programmers and the law is not that
hugo@... carfax.org.uk | they treat laws as code, but that they don't
http://carfax.org.uk/  | understand the VM it runs on.
PGP: E2AB1DE4          |