Help! - "btrfs device delete missing" running out of space
Hello,
I replaced a failed 500G disk in btrfs raid1 with 2 smaller ones - 200 and 250GB. I started "btrfs device delete missing" command, which is running since friday, and it still seems far from finished.
It seems to be doing something very strange: used space on the largest drive is going down, while smaller drives are filling up to the brim.
I'm afraid it will soon run out of space and I don't know what to do.
The current situation is:
Label: none uuid: cff1e711-97fe-4c1c-b8a3-6184010b5027
Total devices 4 FS bytes used 376.66GB devid 4 size 172.81GB used 167.00GB path /dev/dm-16 devid 3 size 232.88GB used 227.00GB path /dev/dm-4 devid 2 size 452.26GB used 327.01GB path /dev/dm-0 *** Some devices missing
According to btrfs FAQ and http://carfax.org.uk/btrfs-usage, there should be around 404GB of usable space in this configuration. Assuming even distribution, used space on each device should be accordingly: 160, 216, and 405. The current situation is FAR from that.
I'd be grateful for tips what do to and how to get out of this situation.
Regards
Hello,
Quoted 2 lines
distribution, used space on each device should be accordingly: 160, 216, and 405.
The last number should be 376, I copied the wrong one. Anyway, I deleted as much data as possible, which probably won't help in the end, but at the moment it's still going. Meanwhile, I made a script to replicate this problem:
On kernel 3.12.6 the output is:
-------
WARNING! - Btrfs v0.20-rc1 IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using
adding device /dev/loop1 id 2
fs created label (null) on /dev/loop0
nodesize 4096 leafsize 4096 sectorsize 4096 size 40.00GB
Btrfs v0.20-rc1
17000+0 records in
17000+0 records out
17825792000 bytes (18 GB) copied, 533,571 s, 33,4 MB/s
Label: none uuid: f8a01060-94c2-4665-b5ff-f134f9b6ad9b
Total devices 2 FS bytes used 16.63GB devid 2 size 20.00GB used 18.01GB path /dev/loop1 devid 1 size 20.00GB used 18.03GB path /dev/loop0
Btrfs v0.20-rc1
ERROR: error removing the device 'missing' - No space left on device
Label: none uuid: f8a01060-94c2-4665-b5ff-f134f9b6ad9b
Total devices 4 FS bytes used 16.62GB devid 4 size 10.00GB used 9.03GB path /dev/loop3 devid 3 size 10.00GB used 9.25GB path /dev/loop2 devid 1 size 20.00GB used 12.31GB path /dev/loop0 *** Some devices missing
Btrfs v0.20-rc1
-------
The "delete missing" logic is pretty much broken, at least in this case. Instead of just replicating the data to other drives, it moves some of the data which fills up smaller drives and it fails with "No space left on device" error.
Regards
Quoted 10 lines · Piotr Pawłow
Hello,
Quoted 2 lines
distribution, used space on each device should be accordingly: 160, 216, and 405.
The last number should be 376, I copied the wrong one. Anyway, I deleted as much data as possible, which probably won't help in the end, but at the moment it's still going. Meanwhile, I made a script to replicate this problem:
I'm not sure what the solution is, but the issue seems to be that btrfs is laying out the RAID1 like this:
Mirror A Mirror B +------+ +------+ | 250g | | 300g | +------+ | | | | +------+ | 500g | //////// | | //////// +------+ ////////
Rather than the hoped-for:
Mirror A Mirror B +------+ +------+ | 250g | | | +------+ | 500g | | 300g | | | | | +------+ +------+ ////////
As far as I know, there's no way to tune this at the moment.
Signature · 1 line
Calvin Walton <[email protected]>
Hello,
Quoted 2 lines
I'm not sure what the solution is, but the issue seems to be that btrfs is laying out the RAID1 like this:
[snip]
Yeah, it kinda ended up like this. I think the problem stems from the fact, that restoring redundancy works by relocating block groups, which rewrites all chunks instead of restoring only the ones that are missing. And it uses the usual allocation strategy, which fails in this case. It treats unreplicated data as allocated, when in fact all of that data will be, sooner or later, freed in the process.
If I succeed deleting the missing device, then running "btrfs balance" should fix the problem.
Regards