RAID-1 refuses to balance large drive

I have a RAID 1, and was running a bit low, so replaced a 2TB drive with
a 6TB. The other drives are a 3TB and a 4TB. After switching the
drive, I did a balance and ... essentially nothing changed. It did not balance clusters over to the 6TB drive off of the other 2 drives. I found it odd, and wondered if it would do it as needed, but as time went on, the filesys got full for real.

Making inquiries on the IRC channel, it was suggested perhaps the drives were too full for a balance, but they had at least 50gb free I would
estimate, when I swapped. As a test, I added a 4th drive, a spare
20gb partition and did a balance. The balance did indeed balance the 3 small drives, so they now each have 6gb unallocated, but the big drive
remained unchanged. The balance reported it operated on almost all the
clusters, though.

Linux kernel 4.2.0 (Ubuntu Wiley)

Label: 'butter' uuid: a91755d4-87d8-4acd-ae08-c11e7f1f5438
Total devices 4 FS bytes used 3.88TiB
devid 1 size 3.62TiB used 3.62TiB path /dev/sdi2
devid 2 size 2.73TiB used 2.72TiB path /dev/sdh
devid 3 size 5.43TiB used 1.42TiB path /dev/sdg2
devid 4 size 20.00GiB used 14.00GiB path /dev/sda1

btrfs fi usage /local

Overall:
Device size: 11.81TiB
Device allocated: 7.77TiB
Device unallocated: 4.04TiB
Device missing: 0.00B
Used: 7.76TiB
Free (estimated): 2.02TiB (min: 2.02TiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)

Data,RAID1: Size:3.87TiB, Used:3.87TiB

    /dev/sda1      14.00GiB
    /dev/sdg2       1.41TiB
    /dev/sdh        2.72TiB
    /dev/sdi2       3.61TiB

Metadata,RAID1: Size:11.00GiB, Used:9.79GiB

    /dev/sdg2       5.00GiB
    /dev/sdh        7.00GiB
    /dev/sdi2      10.00GiB

System,RAID1: Size:32.00MiB, Used:572.00KiB

    /dev/sdg2      32.00MiB
    /dev/sdi2      32.00MiB

Unallocated:

    /dev/sda1       6.00GiB
    /dev/sdg2       4.02TiB
    /dev/sdh        5.52GiB
    /dev/sdi2       7.36GiB

----------------------
btrfs fi df /local
Data, RAID1: total=3.87TiB, used=3.87TiB System, RAID1: total=32.00MiB, used=572.00KiB Metadata, RAID1: total=11.00GiB, used=9.79GiB GlobalReserve, single: total=512.00MiB, used=0.00B

I would have presumed that a balance would take blocks found on both the 3TB and 4TB, and move one of them over to the 6TB until all had 1.3TB of unallocated space. But this does not happen. Any clues on how to make it happen?

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Brad Templeton wrote on 2016/03/22 17:47 -0700:

Quoted 6 lines

I have a RAID 1, and was running a bit low, so replaced a 2TB drive with
a 6TB. The other drives are a 3TB and a 4TB. After switching the
drive, I did a balance and ... essentially nothing changed. It did not balance clusters over to the 6TB drive off of the other 2 drives. I found it odd, and wondered if it would do it as needed, but as time went on, the filesys got full for real.

Did you resized the replaced deivces to max?
Without resize, btrfs still consider it can only use 2T of the 6T devices.

Thanks,
Qu

Quoted 71 lines

Making inquiries on the IRC channel, it was suggested perhaps the drives were too full for a balance, but they had at least 50gb free I would
estimate, when I swapped. As a test, I added a 4th drive, a spare
20gb partition and did a balance. The balance did indeed balance the 3 small drives, so they now each have 6gb unallocated, but the big drive
remained unchanged. The balance reported it operated on almost all the
clusters, though.

Linux kernel 4.2.0 (Ubuntu Wiley)

Label: 'butter' uuid: a91755d4-87d8-4acd-ae08-c11e7f1f5438
Total devices 4 FS bytes used 3.88TiB
devid 1 size 3.62TiB used 3.62TiB path /dev/sdi2
devid 2 size 2.73TiB used 2.72TiB path /dev/sdh
devid 3 size 5.43TiB used 1.42TiB path /dev/sdg2
devid 4 size 20.00GiB used 14.00GiB path /dev/sda1

btrfs fi usage /local

Overall:
Device size: 11.81TiB
Device allocated: 7.77TiB
Device unallocated: 4.04TiB
Device missing: 0.00B
Used: 7.76TiB
Free (estimated): 2.02TiB (min: 2.02TiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)

Data,RAID1: Size:3.87TiB, Used:3.87TiB

    /dev/sda1      14.00GiB
    /dev/sdg2       1.41TiB
    /dev/sdh        2.72TiB
    /dev/sdi2       3.61TiB

Metadata,RAID1: Size:11.00GiB, Used:9.79GiB

    /dev/sdg2       5.00GiB
    /dev/sdh        7.00GiB
    /dev/sdi2      10.00GiB

System,RAID1: Size:32.00MiB, Used:572.00KiB

    /dev/sdg2      32.00MiB
    /dev/sdi2      32.00MiB

Unallocated:

    /dev/sda1       6.00GiB
    /dev/sdg2       4.02TiB
    /dev/sdh        5.52GiB
    /dev/sdi2       7.36GiB

----------------------
btrfs fi df /local
Data, RAID1: total=3.87TiB, used=3.87TiB System, RAID1: total=32.00MiB, used=572.00KiB Metadata, RAID1: total=11.00GiB, used=9.79GiB GlobalReserve, single: total=512.00MiB, used=0.00B

I would have presumed that a balance would take blocks found on both the 3TB and 4TB, and move one of them over to the 6TB until all had 1.3TB of unallocated space. But this does not happen. Any clues on how to make it happen?

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

That's rather counter intuitive behaviour. In most FSs, resizes are needed when you do things like change the size of an underlying partition, or you weren't using all the partition. When you add one drive with device add, and you then remove another with device delete, why and how would the added device know to size itself to the device
that you are planning to delete? Ie. I don't see how it could know
(you add the new drive before even telling it you want to remove the old one) and I also can't see a reason it would not use all the drive you tell it to add.

In any event, I did a btrfs fi resize 3:max /local on the 6TB as you suggest, and have another balance running but it appears like all the others to be doing nothing, though of course it will take hours. Are you sure it works that way? Even before the resize, as you see below, it indicates the volume is 6TB with 4TB of unallocated space. It is only the df that says full (and the fact that there is no unallocated space on the 3TB and 4TB drives.)

Actually, the URL suggests that all the space will be used, which is what I had read about btrfs, that it handled this.

But again, how could it possibly know to restrict the new device to only using 2TB?

Stage one: Add the new 6TB device. The 2TB device is still present.

Stage two: Remove the 2TB device.

The system copies everything on it
to the device which has the most space, the empty 6TB device. But you are saying it decides to _shrink_ the 6TB device now that we know it is a 2TB device being removed?

We didn't know the 2TB would be removed
when we added the 6TB, so I just can't fathom why the code would do that. In addition, the stats I get back say it didn't do that.

More to the point, after the resize, the balance is still not changing any size numbers. It should be moving blocks to the most empty device,
should it not? There is almost no space on devids 1 and 2, so it
would not copy any chunks there.

I'm starting to think this is a bug, but I'll keep plugging.

Quoted 2 lines · Brad Templeton

Actually, the URL suggests that all the space will be used, which is what I had read about btrfs, that it handled this.

It will. But it does this by dominating writes to the devices that have the most free space, until all devices have the same free space.

Quoted 2 lines

But again, how could it possibly know to restrict the new device to only using 2TB?

In your case, before resizing it, it's just inheriting the size from the device being replaced.

Quoted 4 lines

Stage one: Add the new 6TB device. The 2TB device is still present.

Stage two: Remove the 2TB device.

OK this is confusing. In your first post you said replaced. That suggests you used 'btrfs replace start' rather than 'btrfs device add' followed by 'btrfs device remove'. So which did you do?

If you did the latter, then there's no resize necessary.

Quoted 4 lines

The system copies everything on it
to the device which has the most space, the empty 6TB device. But you are saying it decides to _shrink_ the 6TB device now that we know it is a 2TB device being removed?

No I'm not. The source of confusion appears to be that you're unfamiliar with 'btrfs replace' so you mean 'dev add' followed by 'dev remove' to mean replaced.

This line:
devid 3 size 5.43TiB used 1.42TiB path /dev/sdg2

suggests it's using the entire 6TB of the newly added drive, it's already at max size.

Quoted 3 lines

We didn't know the 2TB would be removed
when we added the 6TB, so I just can't fathom why the code would do that. In addition, the stats I get back say it didn't do that.

I don't understand the first part. Whether you asked for 'dev remove' or you used 'replace' both of those mean removing some device. You have to specify the device to be removed.

Now might be a good time to actually write out the exact commands you've used.

Quoted 7 lines

More to the point, after the resize, the balance is still not changing any size numbers. It should be moving blocks to the most empty device,
should it not? There is almost no space on devids 1 and 2, so it
would not copy any chunks there.

I'm starting to think this is a bug, but I'll keep plugging.

Could be a bug. Three drive raid1 of different sizes is somewhat uncommon so it's possible it's hit an edge case somehow. Qu will know more about how to find out why it's not allocating mostly to the larger drive. The eventual work around might end up being to convert data chunks to single, then convert back to raid1. But before doing that it'd be better to find out why it's not doing the right thing the normal way.

Thanks for assist. To reiterate what I said in private:

a) I am fairly sure I swapped drives by adding the 6TB drive and then removing the 2TB drive, which would not have made the 6TB think it was
only 2TB. The btrfs statistics commands have shown from the beginning
the size of the device as 6TB, and that after the remove, it haad 4TB unallocated.

So I am looking for other options, or if people have commands I might execute to diagnose this (as it seems to be a flaw in balance) let me know.

Some options remaining open to me:

a) I could re-add the 2TB device, which is still there. Then balance
again, which hopefully would move a lot of stuff. Then remove it again
and hopefully the new stuff would distribute mostly to the large drive.
Then I could try balance again.

b) It was suggested I could (with a good backup) convert the drive to non-RAID1 to free up tons of space and then re-convert. What's the precise procedure for that? Perhaps I can do it with a limit to see how
it works as an experiment? Any way to specifically target the blocks
that have their two copies on the 2 smaller drives for conversion?

c) Finally, I could take a full-full backup (my normal backups don't bother with cached stuff and certain other things that you can recover) and take the system down for a while to just wipe and restore the volumes. That doesn't find the bug, however.

Quoted 7 lines · Brad Templeton

Thanks for assist. To reiterate what I said in private:

a) I am fairly sure I swapped drives by adding the 6TB drive and then removing the 2TB drive, which would not have made the 6TB think it was
only 2TB. The btrfs statistics commands have shown from the beginning
the size of the device as 6TB, and that after the remove, it haad 4TB unallocated.

I agree this seems to be consistent with what's been reported.

Quoted 3 lines

So I am looking for other options, or if people have commands I might execute to diagnose this (as it seems to be a flaw in balance) let me know.

What version of btrfs-progs is this? I'm vaguely curious what 'btrfs check' reports (without --repair). Any version is OK but it's better to use something fairly recent since the check code continues to change a lot.

Another thing you could try is a newer kernel. Maybe there's a related bug in 4.2.0. I think it may be more likely this is just an edge case bug that's always been there, but it's valuable to know if recent kernels exhibit the problem.

And before proceeding with a change in layout (converting to another profile) I suggest taking an image of the metadata with btrfs-image, it might come in handy for a developer.

Quoted 7 lines

Some options remaining open to me:

a) I could re-add the 2TB device, which is still there. Then balance
again, which hopefully would move a lot of stuff. Then remove it again
and hopefully the new stuff would distribute mostly to the large drive.
Then I could try balance again.

Yeah, to do this will require -f to wipe the signature info from that drive when you add it. But I don't think this is a case of needing more free space, I think it might be due to the odd number of drives that are also fairly different in size.

But then what happens when you delete the 2TB drive after the balance? Do you end up right back in this same situation?

Quoted 6 lines

b) It was suggested I could (with a good backup) convert the drive to non-RAID1 to free up tons of space and then re-convert. What's the precise procedure for that? Perhaps I can do it with a limit to see how
it works as an experiment? Any way to specifically target the blocks
that have their two copies on the 2 smaller drives for conversion?

btrfs balance -dconvert=single -mconvert=single -f ## you have to
use -f to force reduction in redundancy
btrfs balance -dconvert=raid1 -mconvert=raid1

There is the devid= filter but I'm not sure of the consequences of limiting the conversion to two of three devices, that's kinda confusing and is sufficiently an edge case I wonder how many bugs you're looking to find today? :-)

Quoted 4 lines

c) Finally, I could take a full-full backup (my normal backups don't bother with cached stuff and certain other things that you can recover) and take the system down for a while to just wipe and restore the volumes. That doesn't find the bug, however.

I'd have the full backup no matter what choice you make. At any time for any reason any filesystem can face plant without warning.

But yes this should definitely work or else you've definitely found a bug. Finding the bug in your current scenario is harder because the history of this volume makes it really non-deterministic whereas if you start with a 3 disk volume at mkfs time, and then you reproduce this problem, for sure it's a bug. And fairly straightforward to reproduce.

I still recommend a newer kernel and progs though, just because there's no work being done on 4.2 anymore. I suggest 4.4.6 and 4.4.1 progs. And then if you reproduce it, it's not just a bug, it's a current bug.

It is Ubuntu wily, which is 4.2 and btrfs-progs 0.4. I will upgrade to Xenial in April but probably not before, I don't have days to spend on
this. Is there a fairly safe ppa to pull 4.4 or 4.5?

would patch and build my kernels from source but I just don't have time for all the long-term sysadmin burden that creates any more.

Also, I presume if this is a bug, it's in btrfsprogs, though the new one presumably needs a newer kernel too.

I am surprised to hear it said that having the mixed sizes is an odd case. That was actually one of the more compelling features of btrfs
that made me switch from mdadm, lvm and the rest. I presumed most
people were the same. You need more space, you go out and buy a new drive and of course the new drive is bigger than the old drives you bought because they always get bigger. Under mdadm the bigger drive still helped, because it replaced at smaller drive, the one that was holding the RAID back, but you didn't get to use all the big drive until a year later when you had upgraded them all. In the meantime you used the extra space in other RAIDs. (For example, a raid-5 plus a raid-1 on the 2 bigger drives) Or you used the extra space as non-RAID space, ie. space for static stuff that has offline backups. In fact, most of my storage is of that class (photo archives, reciprocal backups of other systems) where RAID is not needed.

So the long story is, I think most home users are likely to always have different sizes and want their FS to treat it well.

Since 6TB is a relatively new size, I wonder if that plays a role. More than 4TB of free space to balance into, could that confuse it?

Off to do a backup (good idea anyway.)

Quoted 83 lines · Chris Murphy

Quoted 7 lines · Brad Templeton

Thanks for assist. To reiterate what I said in private:

a) I am fairly sure I swapped drives by adding the 6TB drive and then removing the 2TB drive, which would not have made the 6TB think it was
only 2TB. The btrfs statistics commands have shown from the beginning
the size of the device as 6TB, and that after the remove, it haad 4TB unallocated.

I agree this seems to be consistent with what's been reported.

Quoted 3 lines

So I am looking for other options, or if people have commands I might execute to diagnose this (as it seems to be a flaw in balance) let me know.

What version of btrfs-progs is this? I'm vaguely curious what 'btrfs check' reports (without --repair). Any version is OK but it's better to use something fairly recent since the check code continues to change a lot.

Another thing you could try is a newer kernel. Maybe there's a related bug in 4.2.0. I think it may be more likely this is just an edge case bug that's always been there, but it's valuable to know if recent kernels exhibit the problem.

And before proceeding with a change in layout (converting to another profile) I suggest taking an image of the metadata with btrfs-image, it might come in handy for a developer.

Quoted 7 lines

Some options remaining open to me:

a) I could re-add the 2TB device, which is still there. Then balance
again, which hopefully would move a lot of stuff. Then remove it again
and hopefully the new stuff would distribute mostly to the large drive.
Then I could try balance again.

Yeah, to do this will require -f to wipe the signature info from that drive when you add it. But I don't think this is a case of needing more free space, I think it might be due to the odd number of drives that are also fairly different in size.

But then what happens when you delete the 2TB drive after the balance? Do you end up right back in this same situation?

Quoted 6 lines

b) It was suggested I could (with a good backup) convert the drive to non-RAID1 to free up tons of space and then re-convert. What's the precise procedure for that? Perhaps I can do it with a limit to see how
it works as an experiment? Any way to specifically target the blocks
that have their two copies on the 2 smaller drives for conversion?

btrfs balance -dconvert=single -mconvert=single -f ## you have to
use -f to force reduction in redundancy
btrfs balance -dconvert=raid1 -mconvert=raid1

There is the devid= filter but I'm not sure of the consequences of limiting the conversion to two of three devices, that's kinda confusing and is sufficiently an edge case I wonder how many bugs you're looking to find today? :-)

Quoted 4 lines

c) Finally, I could take a full-full backup (my normal backups don't bother with cached stuff and certain other things that you can recover) and take the system down for a while to just wipe and restore the volumes. That doesn't find the bug, however.

I'd have the full backup no matter what choice you make. At any time for any reason any filesystem can face plant without warning.

But yes this should definitely work or else you've definitely found a bug. Finding the bug in your current scenario is harder because the history of this volume makes it really non-deterministic whereas if you start with a 3 disk volume at mkfs time, and then you reproduce this problem, for sure it's a bug. And fairly straightforward to reproduce.

Quoted 7 lines

I still recommend a newer kernel and progs though, just because there's no work being done on 4.2 anymore. I suggest 4.4.6 and 4.4.1 progs. And then if you reproduce it, it's not just a bug, it's a current bug.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

It is Ubuntu wily, which is 4.2 and btrfs-progs 0.4. I will upgrade to Xenial in April but probably not before, I don't have days to spend on
this. Is there a fairly safe ppa to pull 4.4 or 4.5?

would patch and build my kernels from source but I just don't have time for all the long-term sysadmin burden that creates any more.

Also, I presume if this is a bug, it's in btrfsprogs, though the new one presumably needs a newer kernel too.

I am surprised to hear it said that having the mixed sizes is an odd case.

That was actually one of the more compelling features of btrfs
that made me switch from mdadm, lvm and the rest. I presumed most
people were the same. You need more space, you go out and buy a new drive and of course the new drive is bigger than the old drives you bought because they always get bigger.

Under mdadm the bigger drive
still helped, because it replaced at smaller drive, the one that was holding the RAID back, but you didn't get to use all the big drive until a year later when you had upgraded them all. In the meantime you used the extra space in other RAIDs. (For example, a raid-5 plus a raid-1 on the 2 bigger drives) Or you used the extra space as non-RAID space, ie. space for static stuff that has offline backups. In fact, most of my storage is of that class (photo archives, reciprocal backups of other systems) where RAID is not needed.

So the long story is, I think most home users are likely to always have different sizes and want their FS to treat it well.

Since 6TB is a relatively new size, I wonder if that plays a role. More than 4TB of free space to balance into, could that confuse it?

It is Ubuntu wily, which is 4.2 and btrfs-progs 0.4.

I will upgrade to
Xenial in April but probably not before, I don't have days to spend on
this. Is there a fairly safe ppa to pull 4.4 or 4.5? In olden days, I
would patch and build my kernels from source but I just don't have time for all the long-term sysadmin burden that creates any more.

Also, I presume if this is a bug, it's in btrfsprogs, though the new one presumably needs a newer kernel too.

Quoted 7 lines · Brad Templeton

Thanks for assist. To reiterate what I said in private:

a) I am fairly sure I swapped drives by adding the 6TB drive and then removing the 2TB drive, which would not have made the 6TB think it was
only 2TB. The btrfs statistics commands have shown from the
beginning the size of the device as 6TB, and that after the remove, it haad 4TB unallocated.

I agree this seems to be consistent with what's been reported.

Quoted 3 lines

It is Ubuntu wily, which is 4.2 and btrfs-progs 0.4. I will upgrade to Xenial in April but probably not before, I don't have days to spend on
this. Is there a fairly safe ppa to pull 4.4 or 4.5?

I'm not sure.

In olden days, I

Quoted 5 lines

would patch and build my kernels from source but I just don't have time for all the long-term sysadmin burden that creates any more.

Also, I presume if this is a bug, it's in btrfsprogs, though the new one presumably needs a newer kernel too.

No you can mix and match progs and kernel versions. You just don't get new features if you don't have a new kernel.

But the issue is the balance code is all in the kernel. It's activated by user space tools but it's all actually done by kernel code.

Quoted 2 lines

I am surprised to hear it said that having the mixed sizes is an odd case.

Not odd as in wrong, just uncommon compared to other arrangements being tested.

Quoted 5 lines

That was actually one of the more compelling features of btrfs
that made me switch from mdadm, lvm and the rest. I presumed most
people were the same. You need more space, you go out and buy a new drive and of course the new drive is bigger than the old drives you bought because they always get bigger.

Of course and I'm not saying it shouldn't work. The central problem here is we don't even know what the problem really is; we only know the manifestation of the problem isn't the desired or expected outcome. And how to find out the cause is different than how to fix it.

Quoted 13 lines

Under mdadm the bigger drive
still helped, because it replaced at smaller drive, the one that was holding the RAID back, but you didn't get to use all the big drive until a year later when you had upgraded them all. In the meantime you used the extra space in other RAIDs. (For example, a raid-5 plus a raid-1 on the 2 bigger drives) Or you used the extra space as non-RAID space, ie. space for static stuff that has offline backups. In fact, most of my storage is of that class (photo archives, reciprocal backups of other systems) where RAID is not needed.

Quoted 2 lines

So the long story is, I think most home users are likely to always have different sizes and want their FS to treat it well.

Yes of course. And at the expense of getting a frownie face....

"Btrfs is under heavy development, and is not suitable for any uses other than benchmarking and review."
https://www.kernel.org/doc/Documentation/filesystems/btrfs.txt

Despite that disclosure, what you're describing is not what I'd expect and not what I've previously experienced. But I haven't had three different sized drives, and they weren't particularly full, and I don't know if you started with three from the outset at mkfs time or if this is the result of two drives with a third added on later, etc. So the nature of file systems is actually really complicated and it's normal for there to be regressions - and maybe this is a regression, hard to say with available information.

Quoted 2 lines

Since 6TB is a relatively new size, I wonder if that plays a role. More than 4TB of free space to balance into, could that confuse it?

Seems unlikely.

About chunk allocation problem, I hope to get a clear view of the whole disk layout now.

What's the final disk layout?
Is that 4T + 3T + 6T + 20G layout?

If so, I'll say, in that case, only fully re-convert to single may help. As there is no enough space to allocate new raid1 chunks for balance them all.

Chris Murphy may have already mentioned, btrfs chunk allocation has some limitation, although it is already more flex than mdadm.

Btrfs chunk allocation will choose the device with most unallocated, and for raid1, it will ensure always pick 2 different devices to allocation.

This allocation does make btrfs raid1 allocation more space in a more flex method than mdadm raid1.
But that only works if you start from scratch.

I'll explain it that case first.

1) 6T and 4T devices only stage: Allocate 1T Raid1 chunk. As 6T and 4T devices have the most unallocated space, so the first 1T raid chunk will be allocated from them.
Remaining space: 3/3/5

2) 6T and 3/4 switch stage: Allocate 4T Raid1 chunk.
After stage 1), we have 3/3/5 remaining space, then btrfs will pick space from 5T remaining(6T devices), and switch between the other 3T remaining one.

Cause the remaining space to be 1/1/1.

3) Fake-even allocation stage: Allocate 1T raid chunk.
Now all devices have the same unallocated space, and there are 3 devices, we can't really balance all chunks across them. As we must and will only select 2 devices, in this stage, there will be 1T unallocated and never be used.

After all, you will get 1 +4 +1 = 6T, still smaller than (3 + 4 +6 ) /2 = 6.5T

Now let's talk about your 3 + 4 + 6 case.

For your initial state, 3 and 4 T devices is already filled up. Even your 6T device have about 4T available space, it's only 1 device, not 2 which raid1 needs.

So, no space for balance to allocate a new raid chunk. The extra 20G is so small that almost makes no sence.

The convert to single then back to raid1, will do its job partly. But according to other report from mail list.
The result won't be perfect even, even the reporter uses devices with all same size.

So to conclude:

1) Btrfs will use most of devices space for raid1.
2) 1) only happens if one fills btrfs from scratch
3) For already filled case, convert to single then convert back will work, but not perfectly.

Thanks,
Qu

Quoted 39 lines

Quoted 13 lines

Under mdadm the bigger drive
still helped, because it replaced at smaller drive, the one that was holding the RAID back, but you didn't get to use all the big drive until a year later when you had upgraded them all. In the meantime you used the extra space in other RAIDs. (For example, a raid-5 plus a raid-1 on the 2 bigger drives) Or you used the extra space as non-RAID space, ie. space for static stuff that has offline backups. In fact, most of my storage is of that class (photo archives, reciprocal backups of other systems) where RAID is not needed.

Quoted 2 lines

So the long story is, I think most home users are likely to always have different sizes and want their FS to treat it well.

Yes of course. And at the expense of getting a frownie face....

"Btrfs is under heavy development, and is not suitable for any uses other than benchmarking and review."
https://www.kernel.org/doc/Documentation/filesystems/btrfs.txt

Despite that disclosure, what you're describing is not what I'd expect and not what I've previously experienced. But I haven't had three different sized drives, and they weren't particularly full, and I don't know if you started with three from the outset at mkfs time or if this is the result of two drives with a third added on later, etc. So the nature of file systems is actually really complicated and it's normal for there to be regressions - and maybe this is a regression, hard to say with available information.

Quoted 2 lines

Since 6TB is a relatively new size, I wonder if that plays a role. More than 4TB of free space to balance into, could that confuse it?

Seems unlikely.

Quoted 26 lines

About chunk allocation problem, I hope to get a clear view of the whole disk layout now.

What's the final disk layout?
Is that 4T + 3T + 6T + 20G layout?

If so, I'll say, in that case, only fully re-convert to single may help. As there is no enough space to allocate new raid1 chunks for balance them all.

Chris Murphy may have already mentioned, btrfs chunk allocation has some limitation, although it is already more flex than mdadm.

Btrfs chunk allocation will choose the device with most unallocated, and for raid1, it will ensure always pick 2 different devices to allocation.

This allocation does make btrfs raid1 allocation more space in a more flex method than mdadm raid1.
But that only works if you start from scratch.

I'll explain it that case first.

Quoted 4 lines

1) 6T and 4T devices only stage: Allocate 1T Raid1 chunk. As 6T and 4T devices have the most unallocated space, so the first 1T raid chunk will be allocated from them.
Remaining space: 3/3/5

This stage never existed. We had a 4 + 3 + 2 stage, which was low-ish on space but not full. I mean it had hundreds of gb free.

Then we had 4 + 3 + 6 + 2, but did not add more files or balance.

Then we had a remove of the 2, which caused, as expected, all the chunks on the 2TB drive to be copied to the 6TB drive, as it was the most empty drive.

Then we had a balance. The balance (I would have expected) would have moved chunks found on both 3 and 4, taking one of them and moving it to
the 6. Generally alternating taking ones from the 3 and 4. I can see
no reason this should not work even if 3 and 4 are almost entirely full, but they were not.
But this did not happen.

Quoted 17 lines

2) 6T and 3/4 switch stage: Allocate 4T Raid1 chunk.
After stage 1), we have 3/3/5 remaining space, then btrfs will pick space from 5T remaining(6T devices), and switch between the other 3T remaining one.

Cause the remaining space to be 1/1/1.

3) Fake-even allocation stage: Allocate 1T raid chunk.
Now all devices have the same unallocated space, and there are 3 devices, we can't really balance all chunks across them. As we must and will only select 2 devices, in this stage, there will be 1T unallocated and never be used.

After all, you will get 1 +4 +1 = 6T, still smaller than (3 + 4 +6 ) /2 = 6.5T

Quoted 8 lines

Now let's talk about your 3 + 4 + 6 case.

For your initial state, 3 and 4 T devices is already filled up. Even your 6T device have about 4T available space, it's only 1 device, not 2 which raid1 needs.

So, no space for balance to allocate a new raid chunk. The extra 20G is so small that almost makes no sence.

Yes, it was added as an experiment on the suggestion of somebody on the IRC channel. I will be rid of it soon. Still, it seems to me that the lack of space even after I filled the disks should not interfere with the balance's ability to move chunks which are found on both 3 and 4 so that one remains and one goes to the 6. This action needs no spare
space. Now I presume the current algorithm perhaps does not work this way?

My next plan is to add the 2tb back. If I am right, balance will move chunks from 3 and 4 to the 2TB,

but it should not move any from the 6TB
because it has so much space.

LIkewise, when I re-remove the 2tb, all
its chunks should move to the 6tb, and I will be at least in a usable state.

Or is the single approach faster?

Quoted 59 lines

The convert to single then back to raid1, will do its job partly. But according to other report from mail list.
The result won't be perfect even, even the reporter uses devices with all same size.

So to conclude:

1) Btrfs will use most of devices space for raid1.
2) 1) only happens if one fills btrfs from scratch
3) For already filled case, convert to single then convert back will work, but not perfectly.

Thanks,
Qu

Quoted 39 lines

Quoted 13 lines

Under mdadm the bigger drive
still helped, because it replaced at smaller drive, the one that was holding the RAID back, but you didn't get to use all the big drive until a year later when you had upgraded them all. In the meantime you used the extra space in other RAIDs. (For example, a raid-5 plus a raid-1 on the 2 bigger drives) Or you used the extra space as non-RAID space, ie. space for static stuff that has offline backups. In fact, most of my storage is of that class (photo archives, reciprocal backups of other systems) where RAID is not needed.

Quoted 2 lines

So the long story is, I think most home users are likely to always have different sizes and want their FS to treat it well.

Yes of course. And at the expense of getting a frownie face....

"Btrfs is under heavy development, and is not suitable for any uses other than benchmarking and review."
https://www.kernel.org/doc/Documentation/filesystems/btrfs.txt

Despite that disclosure, what you're describing is not what I'd expect and not what I've previously experienced. But I haven't had three different sized drives, and they weren't particularly full, and I don't know if you started with three from the outset at mkfs time or if this is the result of two drives with a third added on later, etc. So the nature of file systems is actually really complicated and it's normal for there to be regressions - and maybe this is a regression, hard to say with available information.

Quoted 2 lines

Since 6TB is a relatively new size, I wonder if that plays a role. More than 4TB of free space to balance into, could that confuse it?

Seems unlikely.

The stage I talked about is only for you fill btrfs from scratch, with 3 4 6 devices.

Just as an example to explain how btrfs allocated space on un-even devices.

Quoted 40 lines

Then we had 4 + 3 + 6 + 2, but did not add more files or balance.

Then we had a remove of the 2, which caused, as expected, all the chunks on the 2TB drive to be copied to the 6TB drive, as it was the most empty drive.

Then we had a balance. The balance (I would have expected) would have moved chunks found on both 3 and 4, taking one of them and moving it to
the 6. Generally alternating taking ones from the 3 and 4. I can see
no reason this should not work even if 3 and 4 are almost entirely full, but they were not.
But this did not happen.

Quoted 17 lines

2) 6T and 3/4 switch stage: Allocate 4T Raid1 chunk.
After stage 1), we have 3/3/5 remaining space, then btrfs will pick space from 5T remaining(6T devices), and switch between the other 3T remaining one.

Cause the remaining space to be 1/1/1.

3) Fake-even allocation stage: Allocate 1T raid chunk.
Now all devices have the same unallocated space, and there are 3 devices, we can't really balance all chunks across them. As we must and will only select 2 devices, in this stage, there will be 1T unallocated and never be used.

After all, you will get 1 +4 +1 = 6T, still smaller than (3 + 4 +6 ) /2 = 6.5T

Quoted 8 lines

Now let's talk about your 3 + 4 + 6 case.

For your initial state, 3 and 4 T devices is already filled up. Even your 6T device have about 4T available space, it's only 1 device, not 2 which raid1 needs.

So, no space for balance to allocate a new raid chunk. The extra 20G is so small that almost makes no sence.

Quoted 7 lines

Yes, it was added as an experiment on the suggestion of somebody on the IRC channel. I will be rid of it soon. Still, it seems to me that the lack of space even after I filled the disks should not interfere with the balance's ability to move chunks which are found on both 3 and 4 so that one remains and one goes to the 6. This action needs no spare
space. Now I presume the current algorithm perhaps does not work
this way?

No, balance is not working like that.
Although most user consider balance is moving data, which is partly right. The fact is, balance is, copy-and-delete. And it needs spare space.

Means you must have enough space for the extents you are balancing, then btrfs will copy them, update reference, and then delete old data (with its block group).

So for balancing data in already filled device, btrfs needs to find space for them first.
Which will need 2 devices with unallocated space for RAID1.

And in you case, you only have 1 devices with unallocated space, so no space to balance.

Quoted 5 lines

My next plan is to add the 2tb back. If I am right, balance will move chunks from 3 and 4 to the 2TB,

Not only to 2TB, but to 2TB and 6TB. Never forgot that RAID1 needs 2 devices.
And if 2TB is filled and 3/4 and free space, it's also possible to 3/4 devices.

That will free 2TB in already filled up devices. But that's still not enough to get space even.

You may need to balance several times(maybe 10+) to make space a little even, as balance won't balance any chunk which is created by balance. (Or balance will loop infinitely).

Quoted 3 lines

but it should not move any from the 6TB
because it has so much space.

That's also wrong.
Whether balance will move data from 6TB devices, is only determined by if the src chunk has stripe on 6TB devices and there is enough space to copy them to.

Balance, unlike chunk allocation, is much simple and no complicated space calculation.

1) Check current chunk
If the chunk is out of chunk range (beyond last chunk, which means we are done and current chunk is newly created one) then we finish balance.

2) Check if we have enough space for current chunk. Including creating new chunks.

3) Copy all exntets in this chunk to new location

4) Update reference of all extents to point to new location And free old extents.

5) Goto next chunk.(bytenr order)

So, it's possible that some data in 6TB devices is moved to 6TB again, or to the empty 2TB devices.

It's chunk allocator which ensure the new chunk (destination chunk) is allocated from 6T and empty 2T devices.

Quoted 5 lines

LIkewise, when I re-remove the 2tb, all
its chunks should move to the 6tb, and I will be at least in a usable state.

Or is the single approach faster?

As mentioned, not that easy. The 2Tb devices is not the silver bullet at all.

Re-convert method is the preferred one, although it's not perfect.

Thanks,
Qu

Quoted 66 lines

Quoted 64 lines

The convert to single then back to raid1, will do its job partly. But according to other report from mail list.
The result won't be perfect even, even the reporter uses devices with all same size.

So to conclude:

1) Btrfs will use most of devices space for raid1.
2) 1) only happens if one fills btrfs from scratch
3) For already filled case, convert to single then convert back will work, but not perfectly.

Thanks,
Qu

Quoted 44 lines

Quoted 16 lines

Under mdadm the bigger drive
still helped, because it replaced at smaller drive, the one that was holding the RAID back, but you didn't get to use all the big drive until
a year later when you had upgraded them all. In the meantime you used the extra space in other RAIDs. (For example, a raid-5 plus a raid-1 on
the 2 bigger drives) Or you used the extra space as non-RAID space, ie.
space for static stuff that has offline backups. In fact, most of my storage is of that class (photo archives, reciprocal backups of other systems) where RAID is not needed.

Quoted 3 lines

So the long story is, I think most home users are likely to always have
different sizes and want their FS to treat it well.

Yes of course. And at the expense of getting a frownie face....

"Btrfs is under heavy development, and is not suitable for any uses other than benchmarking and review."
https://www.kernel.org/doc/Documentation/filesystems/btrfs.txt

Despite that disclosure, what you're describing is not what I'd expect and not what I've previously experienced. But I haven't had three different sized drives, and they weren't particularly full, and I don't know if you started with three from the outset at mkfs time or if this is the result of two drives with a third added on later, etc. So the nature of file systems is actually really complicated and it's normal for there to be regressions - and maybe this is a regression, hard to say with available information.

Quoted 3 lines

Since 6TB is a relatively new size, I wonder if that plays a role. More
than 4TB of free space to balance into, could that confuse it?

Seems unlikely.

Quoted 9 lines · Qu Wenruo

No, balance is not working like that.
Although most user consider balance is moving data, which is partly right. The fact is, balance is, copy-and-delete. And it needs spare space.

Means you must have enough space for the extents you are balancing, then btrfs will copy them, update reference, and then delete old data (with its block group).

Quoted 6 lines

So for balancing data in already filled device, btrfs needs to find space for them first.
Which will need 2 devices with unallocated space for RAID1.

And in you case, you only have 1 devices with unallocated space, so no space to balance.

Ah. I would class this as a bug, or at least a non-optimal design. If I understand, you say it tries to move both of the matching chunks to new homes. This makes no sense if there are 3 drives because it is
assured that one chunk is staying on the same drive. Even with 4 or
more drives, where this could make sense, in fact it would still be wise to attempt to move only one of the pair of chunks, and then move the other if that is also a good idea.

Quoted 13 lines

Quoted 7 lines

My next plan is to add the 2tb back. If I am right, balance will move chunks from 3 and 4 to the 2TB,

Not only to 2TB, but to 2TB and 6TB. Never forgot that RAID1 needs 2 devices.
And if 2TB is filled and 3/4 and free space, it's also possible to 3/4 devices.

Quoted 6 lines

That will free 2TB in already filled up devices. But that's still not enough to get space even.

You may need to balance several times(maybe 10+) to make space a little even, as balance won't balance any chunk which is created by balance. (Or balance will loop infinitely).

Now I understand -- I had not thought it would try to move 2 when that's so obviously wrong on a 3-drive, and so I was not thinking of the general case. So I can now calculate that if I add the 2TB, in an ideal situation, it will perhaps get 1TB of chunks and the 6TB will get 1TB of chunks and then the 4 drives will have 3 with 1TB free, and the 6TB will have 3TB free.

Quoted 63 lines

The stage I talked about is only for you fill btrfs from scratch, with 3 4 6 devices.

Just as an example to explain how btrfs allocated space on un-even devices.

Quoted 40 lines

Then we had 4 + 3 + 6 + 2, but did not add more files or balance.

Then we had a remove of the 2, which caused, as expected, all the chunks on the 2TB drive to be copied to the 6TB drive, as it was the most empty drive.

Then we had a balance. The balance (I would have expected) would have moved chunks found on both 3 and 4, taking one of them and moving it to
the 6. Generally alternating taking ones from the 3 and 4. I can see
no reason this should not work even if 3 and 4 are almost entirely full, but they were not.
But this did not happen.

Quoted 17 lines

2) 6T and 3/4 switch stage: Allocate 4T Raid1 chunk.
After stage 1), we have 3/3/5 remaining space, then btrfs will pick space from 5T remaining(6T devices), and switch between the other 3T remaining one.

Cause the remaining space to be 1/1/1.

3) Fake-even allocation stage: Allocate 1T raid chunk.
Now all devices have the same unallocated space, and there are 3 devices, we can't really balance all chunks across them. As we must and will only select 2 devices, in this stage, there will be 1T unallocated and never be used.

After all, you will get 1 +4 +1 = 6T, still smaller than (3 + 4 +6 ) /2 = 6.5T

Quoted 8 lines

Now let's talk about your 3 + 4 + 6 case.

For your initial state, 3 and 4 T devices is already filled up. Even your 6T device have about 4T available space, it's only 1 device, not 2 which raid1 needs.

So, no space for balance to allocate a new raid chunk. The extra 20G is so small that almost makes no sence.

Quoted 7 lines

Yes, it was added as an experiment on the suggestion of somebody on the IRC channel. I will be rid of it soon. Still, it seems to me that the lack of space even after I filled the disks should not interfere with the balance's ability to move chunks which are found on both 3 and 4 so that one remains and one goes to the 6. This action needs no spare
space. Now I presume the current algorithm perhaps does not work
this way?

No, balance is not working like that.
Although most user consider balance is moving data, which is partly right. The fact is, balance is, copy-and-delete. And it needs spare space.

Means you must have enough space for the extents you are balancing, then btrfs will copy them, update reference, and then delete old data (with its block group).

Quoted 6 lines

So for balancing data in already filled device, btrfs needs to find space for them first.
Which will need 2 devices with unallocated space for RAID1.

And in you case, you only have 1 devices with unallocated space, so no space to balance.

Ah. I would class this as a bug, or at least a non-optimal design. If I understand, you say it tries to move both of the matching chunks to new homes. This makes no sense if there are 3 drives because it is
assured that one chunk is staying on the same drive. Even with 4 or
more drives, where this could make sense, in fact it would still be wise to attempt to move only one of the pair of chunks, and then move the other if that is also a good idea.

Quoted 13 lines

Quoted 7 lines

My next plan is to add the 2tb back. If I am right, balance will move chunks from 3 and 4 to the 2TB,

Not only to 2TB, but to 2TB and 6TB. Never forgot that RAID1 needs 2 devices.
And if 2TB is filled and 3/4 and free space, it's also possible to 3/4 devices.

Quoted 6 lines

That will free 2TB in already filled up devices. But that's still not enough to get space even.

You may need to balance several times(maybe 10+) to make space a little even, as balance won't balance any chunk which is created by balance. (Or balance will loop infinitely).

Now I understand -- I had not thought it would try to move 2 when that's so obviously wrong on a 3-drive, and so I was not thinking of the general case. So I can now calculate that if I add the 2TB, in an ideal situation, it will perhaps get 1TB of chunks and the 6TB will get 1TB of chunks and then the 4 drives will have 3 with 1TB free, and the 6TB will
have 3TB free. Then when I remove the 2TB, the 6TB should get all its
chunks and will have 2TB free and the other two 1TB free and that's actually the right situation as all new blocks will appear on the 6TB and one of the other two drives.

I don't want to keep 4 drives because small drives consume power for little, better to move them to other purposes (offline backup etc.)

In the algorithm below, does "chunk" refer to both the redundant copies
of the data, or just to one of them?

I am guessing my misunderstanding
may come from it referring to both, and moving both?

The ability of it to move within the same device you describe is presumably there for combining things together to a chunk, but it appears it slows down the drive rebalancing plan.

Thanks for your explanations.

Quoted 113 lines

Quoted 3 lines

but it should not move any from the 6TB
because it has so much space.

That's also wrong.
Whether balance will move data from 6TB devices, is only determined by if the src chunk has stripe on 6TB devices and there is enough space to copy them to.

Balance, unlike chunk allocation, is much simple and no complicated space calculation.

1) Check current chunk
If the chunk is out of chunk range (beyond last chunk, which means we are done and current chunk is newly created one) then we finish balance.

2) Check if we have enough space for current chunk. Including creating new chunks.

3) Copy all exntets in this chunk to new location

4) Update reference of all extents to point to new location And free old extents.

5) Goto next chunk.(bytenr order)

So, it's possible that some data in 6TB devices is moved to 6TB again, or to the empty 2TB devices.

It's chunk allocator which ensure the new chunk (destination chunk) is allocated from 6T and empty 2T devices.

Quoted 5 lines

LIkewise, when I re-remove the 2tb, all
its chunks should move to the 6tb, and I will be at least in a usable state.

Or is the single approach faster?

As mentioned, not that easy. The 2Tb devices is not the silver bullet at all.

Re-convert method is the preferred one, although it's not perfect.

Thanks,
Qu

Quoted 66 lines

Quoted 64 lines

The convert to single then back to raid1, will do its job partly. But according to other report from mail list.
The result won't be perfect even, even the reporter uses devices with all same size.

So to conclude:

1) Btrfs will use most of devices space for raid1.
2) 1) only happens if one fills btrfs from scratch
3) For already filled case, convert to single then convert back will work, but not perfectly.

Thanks,
Qu

Quoted 44 lines

Quoted 16 lines

Under mdadm the bigger drive
still helped, because it replaced at smaller drive, the one that was holding the RAID back, but you didn't get to use all the big drive until
a year later when you had upgraded them all. In the meantime you used the extra space in other RAIDs. (For example, a raid-5 plus a raid-1 on
the 2 bigger drives) Or you used the extra space as non-RAID space, ie.
space for static stuff that has offline backups. In fact, most of my storage is of that class (photo archives, reciprocal backups of other systems) where RAID is not needed.

Quoted 3 lines

So the long story is, I think most home users are likely to always have
different sizes and want their FS to treat it well.

Yes of course. And at the expense of getting a frownie face....

"Btrfs is under heavy development, and is not suitable for any uses other than benchmarking and review."
https://www.kernel.org/doc/Documentation/filesystems/btrfs.txt

Despite that disclosure, what you're describing is not what I'd expect and not what I've previously experienced. But I haven't had three different sized drives, and they weren't particularly full, and I don't know if you started with three from the outset at mkfs time or if this is the result of two drives with a third added on later, etc. So the nature of file systems is actually really complicated and it's normal for there to be regressions - and maybe this is a regression, hard to say with available information.

Quoted 3 lines

Since 6TB is a relatively new size, I wonder if that plays a role. More
than 4TB of free space to balance into, could that confuse it?

Seems unlikely.

Quoted 17 lines

Quoted 7 lines

Still, it seems to me
that the lack of space even after I filled the disks should not interfere with the balance's ability to move chunks which are found on both 3 and 4 so that one remains and one goes to the 6. This action
needs no spare space. Now I presume the current algorithm perhaps
does not work this way?

No, balance is not working like that.
Although most user consider balance is moving data, which is partly right. The fact is, balance is, copy-and-delete. And it needs spare space.

Means you must have enough space for the extents you are balancing, then btrfs will copy them, update reference, and then delete old data (with its block group).

Quoted 6 lines

So for balancing data in already filled device, btrfs needs to find space for them first.
Which will need 2 devices with unallocated space for RAID1.

And in you case, you only have 1 devices with unallocated space, so no space to balance.

Ah. I would class this as a bug, or at least a non-optimal design. If I understand, you say it tries to move both of the matching chunks to new homes. This makes no sense if there are 3 drives because it is
assured that one chunk is staying on the same drive. Even with 4 or
more drives, where this could make sense, in fact it would still be wise to attempt to move only one of the pair of chunks, and then move the other if that is also a good idea.

Quoted 7 lines · Brad Templeton

Thanks for assist. To reiterate what I said in private:

a) I am fairly sure I swapped drives by adding the 6TB drive and then removing the 2TB drive, which would not have made the 6TB think it was
only 2TB. The btrfs statistics commands have shown from the beginning
the size of the device as 6TB, and that after the remove, it haad 4TB unallocated.

I agree this seems to be consistent with what's been reported.

Quoted 7 lines

Some options remaining open to me:

a) I could re-add the 2TB device, which is still there. Then balance
again, which hopefully would move a lot of stuff. Then remove it again
and hopefully the new stuff would distribute mostly to the large drive.
Then I could try balance again.

Yeah, to do this will require -f to wipe the signature info from that drive when you add it. But I don't think this is a case of needing more free space, I think it might be due to the odd number of drives that are also fairly different in size.

But then what happens when you delete the 2TB drive after the balance? Do you end up right back in this same situation?

Quoted 7 lines

b) It was suggested I could (with a good backup) convert the drive to non-RAID1 to free up tons of space and then re-convert. What's the precise procedure for that? Perhaps I can do it with a limit to see how
it works as an experiment? Any way to specifically target the blocks
that have their two copies on the 2 smaller drives for conversion?

btrfs balance -dconvert=single -mconvert=single -f ## you have to
use -f to force reduction in redundancy
btrfs balance -dconvert=raid1 -mconvert=raid1

Quoted 3 lines · Brad Templeton

I am surprised to hear it said that having the mixed sizes is an odd case.

Not odd as in wrong, just uncommon compared to other arrangements being tested.

Quoted 6 lines · Chris Murphy

Quoted 3 lines · Brad Templeton

I am surprised to hear it said that having the mixed sizes is an odd case.

Quoted 1 line

Not odd as in wrong, just uncommon compared to other arrangements being tested.

I think mixed drive sizes in raid1 is a killer feature for a home NAS, where you replace an old smaller drive with the latest and largest when you need more storage.

My raid1 currently consists of 6TB+3TB+3*2TB.

For the original OP situation, with chunks all filled op with extents and devices all filled up with chunks, 'integrating' a new 6TB drive in an 4TB+3TG+2TB raid1 array could probably be done in a bit unusual way in order to avoid immediate balancing needs:

- 'plug-in' the 6TB
- btrfs-replace 4TB by 6TB
- btrfs fi resize max 6TB_devID
- btrfs-replace 2TB by 4TB
- btrfs fi resize max 4TB_devID
- 'unplug' the 2TB

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was fairly
full. The problem was that after replacing (by add/delete) a small drive
with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if things were fixed now. I am running Xenial (kernel 4.4.0) and could consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Quoted 43 lines · Henk Slager

Quoted 8 lines · Patrik Lundquist

Quoted 7 lines · Chris Murphy

Quoted 3 lines · Brad Templeton

I am surprised to hear it said that having the mixed sizes is an odd case.

Quoted 8 lines

Quoted 2 lines

Not odd as in wrong, just uncommon compared to other arrangements being tested.

I think mixed drive sizes in raid1 is a killer feature for a home NAS, where you replace an old smaller drive with the latest and largest when you need more storage.

My raid1 currently consists of 6TB+3TB+3*2TB.

For the original OP situation, with chunks all filled op with extents and devices all filled up with chunks, 'integrating' a new 6TB drive in an 4TB+3TG+2TB raid1 array could probably be done in a bit unusual way in order to avoid immediate balancing needs:
- 'plug-in' the 6TB
- btrfs-replace 4TB by 6TB
- btrfs fi resize max 6TB_devID
- btrfs-replace 2TB by 4TB
- btrfs fi resize max 4TB_devID
- 'unplug' the 2TB

So then there would be 2 devices with roughly 2TB space available, so good for continued btrfs raid1 writes.

An offline variant with dd instead of btrfs-replace could also be done (I used to do that sometimes when btrfs-replace was not implemented). My experience is that btrfs-replace speed is roughly at max speed (so harddisk magnetic media transferspeed) during the whole replace process and it does in a more direct way what you actually want. So in total mostly way faster device replace/upgrade than with the add+delete method. And raid1 redundancy is active all the time. Of course it means first make sure the system runs up-to-date/latest kernel+tools.

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was
fairly full. The problem was that after replacing (by add/delete) a
small drive with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if
things were fixed now. I am running Xenial (kernel 4.4.0) and could
consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Quoted 27 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was fairly
full. The problem was that after replacing (by add/delete) a small drive
with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if things were fixed now. I am running Xenial (kernel 4.4.0) and could consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 50 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Quoted 43 lines · Henk Slager

Quoted 8 lines · Patrik Lundquist

Quoted 7 lines · Chris Murphy

Quoted 3 lines · Brad Templeton

I am surprised to hear it said that having the mixed sizes is an odd case.

Quoted 8 lines

Quoted 2 lines

Not odd as in wrong, just uncommon compared to other arrangements being tested.

I think mixed drive sizes in raid1 is a killer feature for a home NAS, where you replace an old smaller drive with the latest and largest when you need more storage.

My raid1 currently consists of 6TB+3TB+3*2TB.

For the original OP situation, with chunks all filled op with extents and devices all filled up with chunks, 'integrating' a new 6TB drive in an 4TB+3TG+2TB raid1 array could probably be done in a bit unusual way in order to avoid immediate balancing needs:
- 'plug-in' the 6TB
- btrfs-replace 4TB by 6TB
- btrfs fi resize max 6TB_devID
- btrfs-replace 2TB by 4TB
- btrfs fi resize max 4TB_devID
- 'unplug' the 2TB

So then there would be 2 devices with roughly 2TB space available, so good for continued btrfs raid1 writes.

An offline variant with dd instead of btrfs-replace could also be done (I used to do that sometimes when btrfs-replace was not implemented). My experience is that btrfs-replace speed is roughly at max speed (so harddisk magnetic media transferspeed) during the whole replace process and it does in a more direct way what you actually want. So in total mostly way faster device replace/upgrade than with the add+delete method. And raid1 redundancy is active all the time. Of course it means first make sure the system runs up-to-date/latest kernel+tools.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 11 lines · Brad Templeton

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was
fairly full. The problem was that after replacing (by add/delete) a
small drive with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

It's not that easy.
For balance, btrfs must first find a large enough space to locate both copy, then copy data.
Or if powerloss happens, it will cause data corruption.

So in your case, btrfs can only find enough space for one copy, thus unable to relocate any chunk.

Quoted 7 lines

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if
things were fixed now. I am running Xenial (kernel 4.4.0) and could
consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

Since there is nothing to fix, the behavior will not change at all.

Quoted 9 lines

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 3 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Btrfs chunk allocator is already trying to utilize all drivers for a long long time.
When allocate chunks, btrfs will choose the device with the most free space. However the nature of RAID1 needs btrfs to allocate extents from 2 different devices, which makes your replaced 4/4/6 a little complex. (If your 4/4/6 array is set up and then filled to current stage, btrfs should be able to utilize all the space)

Personally speaking, if you're confident enough, just add a new device, and then do balance.
If enough chunks get balanced, there should be enough space freed on existing disks.
Then remove the newly added device, then btrfs should handle the remaining space well.

Thanks,
Qu

Quoted 84 lines · Brad Templeton

Quoted 27 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was fairly
full. The problem was that after replacing (by add/delete) a small drive
with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if things were fixed now. I am running Xenial (kernel 4.4.0) and could consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 50 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Quoted 43 lines · Henk Slager

Quoted 8 lines · Patrik Lundquist

Quoted 7 lines · Chris Murphy

Quoted 3 lines · Brad Templeton

I am surprised to hear it said that having the mixed sizes is an odd case.

Quoted 8 lines

Quoted 2 lines

Not odd as in wrong, just uncommon compared to other arrangements being tested.

I think mixed drive sizes in raid1 is a killer feature for a home NAS, where you replace an old smaller drive with the latest and largest when you need more storage.

My raid1 currently consists of 6TB+3TB+3*2TB.

For the original OP situation, with chunks all filled op with extents and devices all filled up with chunks, 'integrating' a new 6TB drive in an 4TB+3TG+2TB raid1 array could probably be done in a bit unusual way in order to avoid immediate balancing needs:
- 'plug-in' the 6TB
- btrfs-replace 4TB by 6TB
- btrfs fi resize max 6TB_devID
- btrfs-replace 2TB by 4TB
- btrfs fi resize max 4TB_devID
- 'unplug' the 2TB

So then there would be 2 devices with roughly 2TB space available, so good for continued btrfs raid1 writes.

An offline variant with dd instead of btrfs-replace could also be done (I used to do that sometimes when btrfs-replace was not implemented). My experience is that btrfs-replace speed is roughly at max speed (so harddisk magnetic media transferspeed) during the whole replace process and it does in a more direct way what you actually want. So in total mostly way faster device replace/upgrade than with the add+delete method. And raid1 redundancy is active all the time. Of course it means first make sure the system runs up-to-date/latest kernel+tools.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

That is what did not work last time.

I say I think there can be a "fix" because I hope the goal of BTRFS
raid is to be superior to traditional RAID. That if one replaces a
drive, and asks to balance, it figures out what needs to be done to make that work. I understand that the current balance algorithm may
have trouble with that. In this situation, the ideal result would be
the system would take the 3 drives (4TB and 6TB full, 8TB with 4TB free) and move extents strictly from the 4TB and 6TB to the 8TB -- ie extents which are currently on both the 4TB and 6TB -- by moving only one copy.

It is not strictly a "bug" in that the code is operating as designed, but it is an undesired function.

The problem is the approach you describe did not work in the prior upgrade.

Quoted 150 lines · Brad Templeton

Quoted 11 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was
fairly full. The problem was that after replacing (by add/delete) a
small drive with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

It's not that easy.
For balance, btrfs must first find a large enough space to locate both copy, then copy data.
Or if powerloss happens, it will cause data corruption.

So in your case, btrfs can only find enough space for one copy, thus unable to relocate any chunk.

Quoted 7 lines

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if
things were fixed now. I am running Xenial (kernel 4.4.0) and could
consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

Since there is nothing to fix, the behavior will not change at all.

Quoted 9 lines

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 3 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Btrfs chunk allocator is already trying to utilize all drivers for a long long time.
When allocate chunks, btrfs will choose the device with the most free space. However the nature of RAID1 needs btrfs to allocate extents from 2 different devices, which makes your replaced 4/4/6 a little complex. (If your 4/4/6 array is set up and then filled to current stage, btrfs should be able to utilize all the space)

Personally speaking, if you're confident enough, just add a new device, and then do balance.
If enough chunks get balanced, there should be enough space freed on existing disks.
Then remove the newly added device, then btrfs should handle the remaining space well.

Thanks,
Qu

Quoted 84 lines · Brad Templeton

Quoted 27 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was fairly
full. The problem was that after replacing (by add/delete) a small drive
with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if things were fixed now. I am running Xenial (kernel 4.4.0) and could consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 50 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Quoted 43 lines · Henk Slager

Quoted 8 lines · Patrik Lundquist

Quoted 7 lines · Chris Murphy

Quoted 3 lines · Brad Templeton

I am surprised to hear it said that having the mixed sizes is an odd case.

Quoted 8 lines

Quoted 2 lines

Not odd as in wrong, just uncommon compared to other arrangements being tested.

I think mixed drive sizes in raid1 is a killer feature for a home NAS, where you replace an old smaller drive with the latest and largest when you need more storage.

My raid1 currently consists of 6TB+3TB+3*2TB.

For the original OP situation, with chunks all filled op with extents and devices all filled up with chunks, 'integrating' a new 6TB drive in an 4TB+3TG+2TB raid1 array could probably be done in a bit unusual way in order to avoid immediate balancing needs:
- 'plug-in' the 6TB
- btrfs-replace 4TB by 6TB
- btrfs fi resize max 6TB_devID
- btrfs-replace 2TB by 4TB
- btrfs fi resize max 4TB_devID
- 'unplug' the 2TB

So then there would be 2 devices with roughly 2TB space available, so good for continued btrfs raid1 writes.

An offline variant with dd instead of btrfs-replace could also be done (I used to do that sometimes when btrfs-replace was not implemented). My experience is that btrfs-replace speed is roughly at max speed (so harddisk magnetic media transferspeed) during the whole replace process and it does in a more direct way what you actually want. So in total mostly way faster device replace/upgrade than with the add+delete method. And raid1 redundancy is active all the time. Of course it means first make sure the system runs up-to-date/latest kernel+tools.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 11 lines · Brad Templeton

That is what did not work last time.

I say I think there can be a "fix" because I hope the goal of BTRFS
raid is to be superior to traditional RAID. That if one replaces a
drive, and asks to balance, it figures out what needs to be done to make that work. I understand that the current balance algorithm may
have trouble with that. In this situation, the ideal result would be
the system would take the 3 drives (4TB and 6TB full, 8TB with 4TB free) and move extents strictly from the 4TB and 6TB to the 8TB -- ie extents which are currently on both the 4TB and 6TB -- by moving only one copy.

Btrfs can only do balance in a chunk unit.
Thus btrfs can only do:
1) Create new chunk
2) Copy data
3) Remove old chunk.

So it can't do the way you mentioned.
But your purpose sounds pretty valid and maybe we could enhanace btrfs to do such thing.
(Currently only replace can behave like that)

Quoted 4 lines

It is not strictly a "bug" in that the code is operating as designed, but it is an undesired function.

The problem is the approach you describe did not work in the prior upgrade.

Would you please try 4/4/6 + 4 or 4/4/6 + 2 and then balance? And before/after balance, "btrfs fi usage" and "btrfs fi show" output could also help.

Thanks,
Qu

Quoted 152 lines · Qu Wenruo

Quoted 150 lines · Brad Templeton

Quoted 11 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was
fairly full. The problem was that after replacing (by add/delete) a
small drive with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

It's not that easy.
For balance, btrfs must first find a large enough space to locate both copy, then copy data.
Or if powerloss happens, it will cause data corruption.

So in your case, btrfs can only find enough space for one copy, thus unable to relocate any chunk.

Quoted 7 lines

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if
things were fixed now. I am running Xenial (kernel 4.4.0) and could
consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

Since there is nothing to fix, the behavior will not change at all.

Quoted 9 lines

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 3 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Btrfs chunk allocator is already trying to utilize all drivers for a long long time.
When allocate chunks, btrfs will choose the device with the most free space. However the nature of RAID1 needs btrfs to allocate extents from 2 different devices, which makes your replaced 4/4/6 a little complex. (If your 4/4/6 array is set up and then filled to current stage, btrfs should be able to utilize all the space)

Personally speaking, if you're confident enough, just add a new device, and then do balance.
If enough chunks get balanced, there should be enough space freed on existing disks.
Then remove the newly added device, then btrfs should handle the remaining space well.

Thanks,
Qu

Quoted 84 lines · Brad Templeton

Quoted 27 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was fairly
full. The problem was that after replacing (by add/delete) a small drive
with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if things were fixed now. I am running Xenial (kernel 4.4.0) and could consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 50 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Quoted 43 lines · Henk Slager

Quoted 8 lines · Patrik Lundquist

Quoted 7 lines · Chris Murphy

Quoted 3 lines · Brad Templeton

I am surprised to hear it said that having the mixed sizes is an odd case.

Quoted 8 lines

Quoted 2 lines

Not odd as in wrong, just uncommon compared to other arrangements being tested.

I think mixed drive sizes in raid1 is a killer feature for a home NAS, where you replace an old smaller drive with the latest and largest when you need more storage.

My raid1 currently consists of 6TB+3TB+3*2TB.

For the original OP situation, with chunks all filled op with extents and devices all filled up with chunks, 'integrating' a new 6TB drive in an 4TB+3TG+2TB raid1 array could probably be done in a bit unusual way in order to avoid immediate balancing needs:
- 'plug-in' the 6TB
- btrfs-replace 4TB by 6TB
- btrfs fi resize max 6TB_devID
- btrfs-replace 2TB by 4TB
- btrfs fi resize max 4TB_devID
- 'unplug' the 2TB

So then there would be 2 devices with roughly 2TB space available, so good for continued btrfs raid1 writes.

An offline variant with dd instead of btrfs-replace could also be done (I used to do that sometimes when btrfs-replace was not implemented). My experience is that btrfs-replace speed is roughly at max speed (so harddisk magnetic media transferspeed) during the whole replace process and it does in a more direct way what you actually want. So in total mostly way faster device replace/upgrade than with the add+delete method. And raid1 redundancy is active all the time. Of course it means first make sure the system runs up-to-date/latest kernel+tools.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Thanks. These are all things which take substantial fractions of a day to try, unfortunately.

Last time I ended up fixing it in a
fairly kluged way, which was to convert from raid-1 to single long enough to get enough single blocks that when I converted back to raid-1 they got distributed to the right drives.

But this is, aside
from being a kludge, a procedure with some minor risk. Of course I am taking a backup first, but still...

This strikes me as something that should be a fairly common event -- your raid is filling up, and so you expand it by replacing the oldest
and smallest drive with a new much bigger one. In the old days of
RAID, you could not do that, you had to grow all drives at the same time, and this is one of the ways that BTRFS is quite superior. When I had MD raid, I went through a strange process of always having a raid 5 that consisted of different sized drives. The raid-5 was based on the smallest of the 3 drives, and then the larger ones had extra space which could either be in raid-1, or more imply was in solo disk mode and used for less critical data (such as backups and old
archives.) Slowly, and in a messy way, each time I replaced the
smallest drive, I could then grow the raid 5. Yuck. BTRFS is so
much better, except for this issue.

So if somebody has a thought of a procedure that is fairly sure to work and doesn't involve too many copying passes -- copying 4tb is not a quick operation -- it is much appreciated and might be a good thing to add to a wiki page, which I would be happy to do.

Quoted 29 lines · Brad Templeton

Quoted 11 lines

That is what did not work last time.

I say I think there can be a "fix" because I hope the goal of BTRFS
raid is to be superior to traditional RAID. That if one replaces a
drive, and asks to balance, it figures out what needs to be done to make that work. I understand that the current balance algorithm may
have trouble with that. In this situation, the ideal result would be
the system would take the 3 drives (4TB and 6TB full, 8TB with 4TB free) and move extents strictly from the 4TB and 6TB to the 8TB -- ie extents which are currently on both the 4TB and 6TB -- by moving only one copy.

Btrfs can only do balance in a chunk unit.
Thus btrfs can only do:
1) Create new chunk
2) Copy data
3) Remove old chunk.

So it can't do the way you mentioned.
But your purpose sounds pretty valid and maybe we could enhanace btrfs to do such thing.
(Currently only replace can behave like that)

Quoted 3 lines

It is not strictly a "bug" in that the code is operating as designed, but it is an undesired function.

Quoted 162 lines

Quoted 1 line

The problem is the approach you describe did not work in the prior upgrade.

Would you please try 4/4/6 + 4 or 4/4/6 + 2 and then balance? And before/after balance, "btrfs fi usage" and "btrfs fi show" output could also help.

Thanks,
Qu

Quoted 152 lines · Qu Wenruo

Quoted 150 lines · Brad Templeton

Quoted 11 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was
fairly full. The problem was that after replacing (by add/delete) a
small drive with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

It's not that easy.
For balance, btrfs must first find a large enough space to locate both copy, then copy data.
Or if powerloss happens, it will cause data corruption.

So in your case, btrfs can only find enough space for one copy, thus unable to relocate any chunk.

Quoted 7 lines

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if
things were fixed now. I am running Xenial (kernel 4.4.0) and could
consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

Since there is nothing to fix, the behavior will not change at all.

Quoted 9 lines

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 3 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Btrfs chunk allocator is already trying to utilize all drivers for a long long time.
When allocate chunks, btrfs will choose the device with the most free space. However the nature of RAID1 needs btrfs to allocate extents from 2 different devices, which makes your replaced 4/4/6 a little complex. (If your 4/4/6 array is set up and then filled to current stage, btrfs should be able to utilize all the space)

Personally speaking, if you're confident enough, just add a new device, and then do balance.
If enough chunks get balanced, there should be enough space freed on existing disks.
Then remove the newly added device, then btrfs should handle the remaining space well.

Thanks,
Qu

Quoted 84 lines · Brad Templeton

Quoted 27 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was fairly
full. The problem was that after replacing (by add/delete) a small drive
with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if things were fixed now. I am running Xenial (kernel 4.4.0) and could consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 50 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Quoted 43 lines · Henk Slager

Quoted 8 lines · Patrik Lundquist

Quoted 7 lines · Chris Murphy

Quoted 3 lines · Brad Templeton

I am surprised to hear it said that having the mixed sizes is an odd case.

Quoted 8 lines

Quoted 2 lines

Not odd as in wrong, just uncommon compared to other arrangements being tested.

I think mixed drive sizes in raid1 is a killer feature for a home NAS, where you replace an old smaller drive with the latest and largest when you need more storage.

My raid1 currently consists of 6TB+3TB+3*2TB.

For the original OP situation, with chunks all filled op with extents and devices all filled up with chunks, 'integrating' a new 6TB drive in an 4TB+3TG+2TB raid1 array could probably be done in a bit unusual way in order to avoid immediate balancing needs:
- 'plug-in' the 6TB
- btrfs-replace 4TB by 6TB
- btrfs fi resize max 6TB_devID
- btrfs-replace 2TB by 4TB
- btrfs fi resize max 4TB_devID
- 'unplug' the 2TB

So then there would be 2 devices with roughly 2TB space available, so good for continued btrfs raid1 writes.

An offline variant with dd instead of btrfs-replace could also be done (I used to do that sometimes when btrfs-replace was not implemented). My experience is that btrfs-replace speed is roughly at max speed (so harddisk magnetic media transferspeed) during the whole replace process and it does in a more direct way what you actually want. So in total mostly way faster device replace/upgrade than with the add+delete method. And raid1 redundancy is active all the time. Of course it means first make sure the system runs up-to-date/latest kernel+tools.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 2 lines · Brad Templeton

Thanks. These are all things which take substantial fractions of a day to try, unfortunately.

Normally I would suggest just using VM and several small disks (~10G), along with fallocate (the fastest way to use space) to get a basic view of the procedure.

Quoted 4 lines

Last time I ended up fixing it in a
fairly kluged way, which was to convert from raid-1 to single long enough to get enough single blocks that when I converted back to raid-1 they got distributed to the right drives.

Yep, that's the ultimate one-fit-all solution.
Also, this reminds me about the fact we could do the RAID1->Single/DUP->Single downgrade in a much much faster way. I think it's worthy considering for later enhancement.

Quoted 18 lines

But this is, aside
from being a kludge, a procedure with some minor risk. Of course I am taking a backup first, but still...

This strikes me as something that should be a fairly common event -- your raid is filling up, and so you expand it by replacing the oldest
and smallest drive with a new much bigger one. In the old days of
RAID, you could not do that, you had to grow all drives at the same time, and this is one of the ways that BTRFS is quite superior. When I had MD raid, I went through a strange process of always having a raid 5 that consisted of different sized drives. The raid-5 was based on the smallest of the 3 drives, and then the larger ones had extra space which could either be in raid-1, or more imply was in solo disk mode and used for less critical data (such as backups and old
archives.) Slowly, and in a messy way, each time I replaced the
smallest drive, I could then grow the raid 5. Yuck. BTRFS is so
much better, except for this issue.

Quoted 4 lines

So if somebody has a thought of a procedure that is fairly sure to work and doesn't involve too many copying passes -- copying 4tb is not a quick operation -- it is much appreciated and might be a good thing to add to a wiki page, which I would be happy to do.

Anyway, "btrfs fi show" and "btrfs fi usage" would help before any further advice from community.

Thanks,
Qu

Quoted 198 lines · Qu Wenruo

Quoted 29 lines · Brad Templeton

Quoted 11 lines

That is what did not work last time.

I say I think there can be a "fix" because I hope the goal of BTRFS
raid is to be superior to traditional RAID. That if one replaces a
drive, and asks to balance, it figures out what needs to be done to make that work. I understand that the current balance algorithm may
have trouble with that. In this situation, the ideal result would be
the system would take the 3 drives (4TB and 6TB full, 8TB with 4TB free) and move extents strictly from the 4TB and 6TB to the 8TB -- ie extents which are currently on both the 4TB and 6TB -- by moving only one copy.

Btrfs can only do balance in a chunk unit.
Thus btrfs can only do:
1) Create new chunk
2) Copy data
3) Remove old chunk.

So it can't do the way you mentioned.
But your purpose sounds pretty valid and maybe we could enhanace btrfs to do such thing.
(Currently only replace can behave like that)

Quoted 3 lines

It is not strictly a "bug" in that the code is operating as designed, but it is an undesired function.

Quoted 162 lines

Quoted 1 line

The problem is the approach you describe did not work in the prior upgrade.

Would you please try 4/4/6 + 4 or 4/4/6 + 2 and then balance? And before/after balance, "btrfs fi usage" and "btrfs fi show" output could also help.

Thanks,
Qu

Quoted 152 lines · Qu Wenruo

Quoted 150 lines · Brad Templeton

Quoted 11 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was
fairly full. The problem was that after replacing (by add/delete) a
small drive with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

It's not that easy.
For balance, btrfs must first find a large enough space to locate both copy, then copy data.
Or if powerloss happens, it will cause data corruption.

So in your case, btrfs can only find enough space for one copy, thus unable to relocate any chunk.

Quoted 7 lines

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if
things were fixed now. I am running Xenial (kernel 4.4.0) and could
consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

Since there is nothing to fix, the behavior will not change at all.

Quoted 9 lines

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 3 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Btrfs chunk allocator is already trying to utilize all drivers for a long long time.
When allocate chunks, btrfs will choose the device with the most free space. However the nature of RAID1 needs btrfs to allocate extents from 2 different devices, which makes your replaced 4/4/6 a little complex. (If your 4/4/6 array is set up and then filled to current stage, btrfs should be able to utilize all the space)

Personally speaking, if you're confident enough, just add a new device, and then do balance.
If enough chunks get balanced, there should be enough space freed on existing disks.
Then remove the newly added device, then btrfs should handle the remaining space well.

Thanks,
Qu

Quoted 84 lines · Brad Templeton

Quoted 27 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was fairly
full. The problem was that after replacing (by add/delete) a small drive
with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if things were fixed now. I am running Xenial (kernel 4.4.0) and could consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 50 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Quoted 43 lines · Henk Slager

Quoted 8 lines · Patrik Lundquist

Quoted 7 lines · Chris Murphy

Quoted 3 lines · Brad Templeton

I am surprised to hear it said that having the mixed sizes is an odd case.

Quoted 8 lines

Quoted 2 lines

Not odd as in wrong, just uncommon compared to other arrangements being tested.

I think mixed drive sizes in raid1 is a killer feature for a home NAS, where you replace an old smaller drive with the latest and largest when you need more storage.

My raid1 currently consists of 6TB+3TB+3*2TB.

For the original OP situation, with chunks all filled op with extents and devices all filled up with chunks, 'integrating' a new 6TB drive in an 4TB+3TG+2TB raid1 array could probably be done in a bit unusual way in order to avoid immediate balancing needs:
- 'plug-in' the 6TB
- btrfs-replace 4TB by 6TB
- btrfs fi resize max 6TB_devID
- btrfs-replace 2TB by 4TB
- btrfs fi resize max 4TB_devID
- 'unplug' the 2TB

So then there would be 2 devices with roughly 2TB space available, so good for continued btrfs raid1 writes.

An offline variant with dd instead of btrfs-replace could also be done (I used to do that sometimes when btrfs-replace was not implemented). My experience is that btrfs-replace speed is roughly at max speed (so harddisk magnetic media transferspeed) during the whole replace process and it does in a more direct way what you actually want. So in total mostly way faster device replace/upgrade than with the add+delete method. And raid1 redundancy is active all the time. Of course it means first make sure the system runs up-to-date/latest kernel+tools.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Certainly. My apologies for not including them before.

As
described, the disks are reasonably balanced -- not as full as the last time. As such, it might be enough that balance would (slowly) free up enough chunks to get things going. And if I have to, I will
partially convert to single again. Certainly btrfs replace seems
like the most planned and simple path but it will result in a strange distribution of the chunks.

Label: 'butter' uuid: a91755d4-87d8-4acd-ae08-c11e7f1f5438
Total devices 3 FS bytes used 6.11TiB
devid 1 size 3.62TiB used 3.47TiB path /dev/sdj2Overall:
Device size: 12.70TiB
Device allocated: 12.25TiB
Device unallocated: 459.95GiB
Device missing: 0.00B
Used: 12.21TiB
Free (estimated): 246.35GiB (min: 246.35GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 1.32MiB)

Data,RAID1: Size:6.11TiB, Used:6.09TiB

  /dev/sda        3.48TiB
  /dev/sdi2       5.28TiB
  /dev/sdj2       3.46TiB

Metadata,RAID1: Size:14.00GiB, Used:12.38GiB
/dev/sda 8.00GiB
/dev/sdi2 7.00GiB
/dev/sdj2 13.00GiB

System,RAID1: Size:32.00MiB, Used:888.00KiB
/dev/sdi2 32.00MiB
/dev/sdj2 32.00MiB

Unallocated:

  /dev/sda      153.02GiB
  /dev/sdi2     154.56GiB
  /dev/sdj2     152.36GiB

devid 2 size 3.64TiB used 3.49TiB path /dev/sda
devid 3 size 5.43TiB used 5.28TiB path /dev/sdi2

Certainly. My apologies for not including them before. As
described, the disks are reasonably balanced -- not as full as the last time. As such, it might be enough that balance would (slowly) free up enough chunks to get things going. And if I have to, I will
partially convert to single again. Certainly btrfs replace seems
like the most planned and simple path but it will result in a strange distribution of the chunks.

Label: 'butter' uuid: a91755d4-87d8-4acd-ae08-c11e7f1f5438
Total devices 3 FS bytes used 6.11TiB
devid 1 size 3.62TiB used 3.47TiB path /dev/sdj2Overall:
Device size: 12.70TiB
Device allocated: 12.25TiB
Device unallocated: 459.95GiB
Device missing: 0.00B
Used: 12.21TiB
Free (estimated): 246.35GiB (min: 246.35GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 1.32MiB)

Data,RAID1: Size:6.11TiB, Used:6.09TiB

  /dev/sda        3.48TiB
  /dev/sdi2       5.28TiB
  /dev/sdj2       3.46TiB

Metadata,RAID1: Size:14.00GiB, Used:12.38GiB
/dev/sda 8.00GiB
/dev/sdi2 7.00GiB
/dev/sdj2 13.00GiB

System,RAID1: Size:32.00MiB, Used:888.00KiB
/dev/sdi2 32.00MiB
/dev/sdj2 32.00MiB

Unallocated:

  /dev/sda      153.02GiB
  /dev/sdi2     154.56GiB
  /dev/sdj2     152.36GiB

devid 2 size 3.64TiB used 3.49TiB path /dev/sda
devid 3 size 5.43TiB used 5.28TiB path /dev/sdi2

Quoted 248 lines · Qu Wenruo

Quoted 2 lines · Brad Templeton

Thanks. These are all things which take substantial fractions of a day to try, unfortunately.

Normally I would suggest just using VM and several small disks (~10G), along with fallocate (the fastest way to use space) to get a basic view of the procedure.

Quoted 4 lines

Last time I ended up fixing it in a
fairly kluged way, which was to convert from raid-1 to single long enough to get enough single blocks that when I converted back to raid-1 they got distributed to the right drives.

Yep, that's the ultimate one-fit-all solution.
Also, this reminds me about the fact we could do the RAID1->Single/DUP->Single downgrade in a much much faster way. I think it's worthy considering for later enhancement.

Quoted 18 lines

But this is, aside
from being a kludge, a procedure with some minor risk. Of course I am taking a backup first, but still...

This strikes me as something that should be a fairly common event -- your raid is filling up, and so you expand it by replacing the oldest
and smallest drive with a new much bigger one. In the old days of
RAID, you could not do that, you had to grow all drives at the same time, and this is one of the ways that BTRFS is quite superior. When I had MD raid, I went through a strange process of always having a raid 5 that consisted of different sized drives. The raid-5 was based on the smallest of the 3 drives, and then the larger ones had extra space which could either be in raid-1, or more imply was in solo disk mode and used for less critical data (such as backups and old
archives.) Slowly, and in a messy way, each time I replaced the
smallest drive, I could then grow the raid 5. Yuck. BTRFS is so
much better, except for this issue.

Quoted 4 lines

So if somebody has a thought of a procedure that is fairly sure to work and doesn't involve too many copying passes -- copying 4tb is not a quick operation -- it is much appreciated and might be a good thing to add to a wiki page, which I would be happy to do.

Anyway, "btrfs fi show" and "btrfs fi usage" would help before any further advice from community.

Thanks,
Qu

Quoted 198 lines · Qu Wenruo

Quoted 29 lines · Brad Templeton

Quoted 11 lines

That is what did not work last time.

I say I think there can be a "fix" because I hope the goal of BTRFS
raid is to be superior to traditional RAID. That if one replaces a
drive, and asks to balance, it figures out what needs to be done to make that work. I understand that the current balance algorithm may
have trouble with that. In this situation, the ideal result would be
the system would take the 3 drives (4TB and 6TB full, 8TB with 4TB free) and move extents strictly from the 4TB and 6TB to the 8TB -- ie extents which are currently on both the 4TB and 6TB -- by moving only one copy.

Btrfs can only do balance in a chunk unit.
Thus btrfs can only do:
1) Create new chunk
2) Copy data
3) Remove old chunk.

So it can't do the way you mentioned.
But your purpose sounds pretty valid and maybe we could enhanace btrfs to do such thing.
(Currently only replace can behave like that)

Quoted 3 lines

It is not strictly a "bug" in that the code is operating as designed, but it is an undesired function.

Quoted 162 lines

Quoted 1 line

The problem is the approach you describe did not work in the prior upgrade.

Would you please try 4/4/6 + 4 or 4/4/6 + 2 and then balance? And before/after balance, "btrfs fi usage" and "btrfs fi show" output could also help.

Thanks,
Qu

Quoted 152 lines · Qu Wenruo

Quoted 150 lines · Brad Templeton

Quoted 11 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was
fairly full. The problem was that after replacing (by add/delete) a
small drive with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

It's not that easy.
For balance, btrfs must first find a large enough space to locate both copy, then copy data.
Or if powerloss happens, it will cause data corruption.

So in your case, btrfs can only find enough space for one copy, thus unable to relocate any chunk.

Quoted 7 lines

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if
things were fixed now. I am running Xenial (kernel 4.4.0) and could
consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

Since there is nothing to fix, the behavior will not change at all.

Quoted 9 lines

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 3 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Btrfs chunk allocator is already trying to utilize all drivers for a long long time.
When allocate chunks, btrfs will choose the device with the most free space. However the nature of RAID1 needs btrfs to allocate extents from 2 different devices, which makes your replaced 4/4/6 a little complex. (If your 4/4/6 array is set up and then filled to current stage, btrfs should be able to utilize all the space)

Personally speaking, if you're confident enough, just add a new device, and then do balance.
If enough chunks get balanced, there should be enough space freed on existing disks.
Then remove the newly added device, then btrfs should handle the remaining space well.

Thanks,
Qu

Quoted 84 lines · Brad Templeton

Quoted 27 lines

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was fairly
full. The problem was that after replacing (by add/delete) a small drive
with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if things were fixed now. I am running Xenial (kernel 4.4.0) and could consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 50 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Quoted 43 lines · Henk Slager

Quoted 8 lines · Patrik Lundquist

Quoted 7 lines · Chris Murphy

Quoted 3 lines · Brad Templeton

I am surprised to hear it said that having the mixed sizes is an odd case.

Quoted 8 lines

Quoted 2 lines

Not odd as in wrong, just uncommon compared to other arrangements being tested.

I think mixed drive sizes in raid1 is a killer feature for a home NAS, where you replace an old smaller drive with the latest and largest when you need more storage.

My raid1 currently consists of 6TB+3TB+3*2TB.

For the original OP situation, with chunks all filled op with extents and devices all filled up with chunks, 'integrating' a new 6TB drive in an 4TB+3TG+2TB raid1 array could probably be done in a bit unusual way in order to avoid immediate balancing needs:
- 'plug-in' the 6TB
- btrfs-replace 4TB by 6TB
- btrfs fi resize max 6TB_devID
- btrfs-replace 2TB by 4TB
- btrfs fi resize max 4TB_devID
- 'unplug' the 2TB

So then there would be 2 devices with roughly 2TB space available, so good for continued btrfs raid1 writes.

An offline variant with dd instead of btrfs-replace could also be done (I used to do that sometimes when btrfs-replace was not implemented). My experience is that btrfs-replace speed is roughly at max speed (so harddisk magnetic media transferspeed) during the whole replace process and it does in a more direct way what you actually want. So in total mostly way faster device replace/upgrade than with the add+delete method. And raid1 redundancy is active all the time. Of course it means first make sure the system runs up-to-date/latest kernel+tools.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

BTW, I decided to follow the original double replace strategy suggested

replace 6TB with 8TB and replace 4TB with 6TB. That should be sure to leave the 2 large drives each with 2TB free once expanded, and thus able to fully use all space.

However, the first one has been going for 9 hours and is "189.7% done"
and still going. Some sort of bug in calculating the completion
status, obviously. With luck 200% will be enough?

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was
fairly full. The problem was that after replacing (by add/delete) a
small drive with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if
things were fixed now. I am running Xenial (kernel 4.4.0) and could
consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Quoted 27 lines · Brad Templeton

A few years ago, I encountered an issue (halfway between a bug and a problem) with attempting to grow a BTRFS 3 disk Raid 1 which was fairly
full. The problem was that after replacing (by add/delete) a small drive
with a larger one, there were now 2 full drives and one new half-full one, and balance was not able to correct this situation to produce the desired result, which is 3 drives, each with a roughly even amount of free space. It can't do it because the 2 smaller drives are full, and it doesn't realize it could just move one of the copies of a block off the smaller drive onto the larger drive to free space on the smaller drive, it wants to move them both, and there is nowhere to put them both.

I'm about to do it again, taking my nearly full array which is 4TB, 4TB, 6TB and replacing one of the 4TB with an 8TB. I don't want to repeat the very time consuming situation, so I wanted to find out if things were fixed now. I am running Xenial (kernel 4.4.0) and could consider the upgrade to bionic (4.15) though that adds a lot more to my plate before a long trip and I would prefer to avoid if I can.

So what is the best strategy:

a) Replace 4TB with 8TB, resize up and balance? (This is the "basic" strategy)
b) Add 8TB, balance, remove 4TB (automatic distribution of some blocks from 4TB but possibly not enough)
c) Replace 6TB with 8TB, resize/balance, then replace 4TB with recently vacated 6TB -- much longer procedure but possibly better

Quoted 50 lines

Or has this all been fixed and method A will work fine and get to the ideal goal -- 3 drives, with available space suitably distributed to allow full utilization over time?

Quoted 43 lines · Henk Slager

Quoted 8 lines · Patrik Lundquist

Quoted 7 lines · Chris Murphy

Quoted 3 lines · Brad Templeton

I am surprised to hear it said that having the mixed sizes is an odd case.

Quoted 8 lines

Quoted 2 lines

Not odd as in wrong, just uncommon compared to other arrangements being tested.

I think mixed drive sizes in raid1 is a killer feature for a home NAS, where you replace an old smaller drive with the latest and largest when you need more storage.

My raid1 currently consists of 6TB+3TB+3*2TB.

For the original OP situation, with chunks all filled op with extents and devices all filled up with chunks, 'integrating' a new 6TB drive in an 4TB+3TG+2TB raid1 array could probably be done in a bit unusual way in order to avoid immediate balancing needs:
- 'plug-in' the 6TB
- btrfs-replace 4TB by 6TB
- btrfs fi resize max 6TB_devID
- btrfs-replace 2TB by 4TB
- btrfs fi resize max 4TB_devID
- 'unplug' the 2TB

So then there would be 2 devices with roughly 2TB space available, so good for continued btrfs raid1 writes.

An offline variant with dd instead of btrfs-replace could also be done (I used to do that sometimes when btrfs-replace was not implemented). My experience is that btrfs-replace speed is roughly at max speed (so harddisk magnetic media transferspeed) during the whole replace process and it does in a more direct way what you actually want. So in total mostly way faster device replace/upgrade than with the add+delete method. And raid1 redundancy is active all the time. Of course it means first make sure the system runs up-to-date/latest kernel+tools.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html