btrfs raid1 balance slow
Hello,
I had a btrfs raid1 with 2 4TB SATA 5400RPM disks . Regular disk I/O is about 2MB/sec per drive, ~40IOPS, mostly write. I had ~150GB free and added one 2 TB disk and started the balance:
btrfs device add -f /dev/sdb /export
btrfs filesystem balance /export
It's now running for 24 hours, 70% remaining:
# btrfs balance status -v /export Balance on '/export' is running 1057 out of about 3561 chunks balanced (1058 considered), 70% left Dumping filters: flags 0x7, state 0x1, force is off DATA (flags 0x0): balancing METADATA (flags 0x0): balancing SYSTEM (flags 0x0): balancing
I have searched for similar cases, but, I do not have quotas enabled, I do not have compression enabled, and my CPU supports sse4_2 . CPU (i7-8700K) is doing fine, 80% idle (average over all threads).
Is this normal ? I have to repeat this process 2 times (adding more 2TB disks), any way I can make it faster ?
Thanks!
Klaas
Quoted 17 lines · On Mon, Jan 20, 2020 at 11:08:40PM +0100, kjansen387 wrote:
I had a btrfs raid1 with 2 4TB SATA 5400RPM disks . Regular disk I/O is about 2MB/sec per drive, ~40IOPS, mostly write. I had ~150GB free and added one 2 TB disk and started the balance:
btrfs device add -f
/dev/sdb/export
btrfs filesystem balance /exportIt's now running for 24 hours, 70% remaining:
# btrfs balance status -v /export Balance on '/export' is running 1057 out of about 3561 chunks balanced (1058 considered), 70% left Dumping filters: flags 0x7, state 0x1, force is off DATA (flags 0x0): balancing METADATA (flags 0x0): balancing SYSTEM (flags 0x0): balancing
Quoted 3 lines
I have searched for similar cases, but, I do not have quotas enabled, I do not have compression enabled, and my CPU supports
sse4_2. CPU (i7-8700K) is doing fine, 80% idle (average over all threads).
Do you have lots of snapshots? It can take a lot of time on some of
the metadata chunks if there's lots of shared extents.
Quoted 2 lines
Is this normal ? I have to repeat this process 2 times (adding more 2TB disks), any way I can make it faster ?
Cancel the current balance, add the remaining disks, and then
balance only once you've added them all.
Hugo.
Signature · 4 lines
Hugo Mills | If it's December 1941 in Casablanca, what time is it hugo@... carfax.org.uk | in New York? http://carfax.org.uk/ | PGP: E2AB1DE4 | Rick Blaine, Casablanca
Quoted 19 lines · On 2020/1/21 上午6:08, kjansen387 wrote:
Hello,
I had a btrfs raid1 with 2 4TB SATA 5400RPM disks . Regular disk I/O is about 2MB/sec per drive, ~40IOPS, mostly write. I had ~150GB free and added one 2 TB disk and started the balance:
btrfs device add -f
/dev/sdb/export
btrfs filesystem balance /exportIt's now running for 24 hours, 70% remaining:
# btrfs balance status -v /export Balance on '/export' is running 1057 out of about 3561 chunks balanced (1058 considered), 70% left Dumping filters: flags 0x7, state 0x1, force is off DATA (flags 0x0): balancing METADATA (flags 0x0): balancing SYSTEM (flags 0x0): balancing
Quoted 3 lines
I have searched for similar cases, but, I do not have quotas enabled, I do not have compression enabled, and my CPU supports
sse4_2. CPU (i7-8700K) is doing fine, 80% idle (average over all threads).
Are you using qgroup?
If so, that would be the cause.
Please use kernel newer than v5.2, which contains the optimization.
Quoted 3 lines
Is this normal ? I have to repeat this process 2 times (adding more 2TB disks), any way I can make it faster ?
Just as Hugo said, cancel the current one, add all device, then rebalance.
Thanks,
Qu
Quoted 4 lines
Thanks!
Klaas
Forgot to mention; I'm running kernel 5.4.10-200.fc31.x86_64 .
Op 20-Jan-20 om 23:08 schreef kjansen387:
Quoted 24 lines
Hello,
I had a btrfs raid1 with 2 4TB SATA 5400RPM disks . Regular disk I/O is about 2MB/sec per drive, ~40IOPS, mostly write. I had ~150GB free and added one 2 TB disk and started the balance:
btrfs device add -f
/dev/sdb/export
btrfs filesystem balance /exportIt's now running for 24 hours, 70% remaining:
# btrfs balance status -v /export Balance on '/export' is running 1057 out of about 3561 chunks balanced (1058 considered), 70% left Dumping filters: flags 0x7, state 0x1, force is off DATA (flags 0x0): balancing METADATA (flags 0x0): balancing SYSTEM (flags 0x0): balancingI have searched for similar cases, but, I do not have quotas enabled, I do not have compression enabled, and my CPU supports
sse4_2. CPU (i7-8700K) is doing fine, 80% idle (average over all threads).Is this normal ? I have to repeat this process 2 times (adding more 2TB disks), any way I can make it faster ?
Thanks!
Klaas
Hmm, for some reason, I did not get the mail from Hugo and Qu (had too look it up in the archive). I think there is a delay when subscribing to the email list.
Quoted 1 line
Do you have lots of snapshots? It can take a lot of time on some of
the metadata chunks if there's lots of shared extents.
No, I do not have any snapshots. Never made one - just got started with btrfs.
# btrfs subvolume show /export | grep Snap
Snapshot(s):Quoted 1 line
Are you using qgroup?
No, quotas are disabled:
# btrfs qgroup show /export ERROR: can't list qgroups: quotas not enabled
Quoted 1 line
Please use kernel newer than v5.2, which contains the optimization.
I'm running 5.4.10-200.fc31.x86_64, that should be good
Quoted 1 line
Just as Hugo said, cancel the current one, add all device, then rebalance.
Yes, that would be the best. I'm moving data however.. Expand btrfs > move the content of a 2TB disk > use that disk to expand btrfs > move the content of a 2TB disk etc. When this balance is done I'll move 2TB, find a temp drive to store the other 2TB and add the remaining 2 drives in 1 go.
I'm not in a hurry.. just wondering if this is normal. As I had 2x4TB with 150GB free and am adding 1x2TB (~1800GB, ~900 after raid1) I would expect that I will end up with 150+900=1050 GB free (usable), spread like (approx.):
4TB disk 1: 420GB (previously: 75GB)
4TB disk 2: 420GB (previously: 75GB)
2TB disk: 210GB
Thinking very simplisticly -and that's probably where I'm wrong, just wondering if it's a factor 8 wrong- this would mean 420-75 = 345GB read from each 4TB disks and written to the new 2TB disk. At a rate of 20MB/s (pretty conservative, but I do have a little bit of load) this should take like 10 hours. It will, however, at this rate, take approx. 82 hours. Even if all data needs to be read (does it? ~3500GB net) it should only take ~50 hours..
Klaas