Skip to contentSkip to conversation

btrfs balance enospc

DISCUSSION Mark Murawski 4 participants 9 messages Sep 15, 2014 – Sep 16, 2014 Btrfs file system

lore
Share & tools
lore
#0 lore raw

I should have plenty of space for this operation, but it fails....

cartman {~} root# btrfs filesystem df /
Data, RAID1: total=7.59GiB, used=7.00GiB
System, RAID1: total=32.00MiB, used=4.00KiB
Metadata, RAID1: total=1.00GiB, used=438.45MiB
unknown, single: total=148.00MiB, used=0.00

Filesystem Type Size Used Avail Use% Mounted on
/dev/disk/by-uuid/d71404d4-468e-47d5-8f06-3b65fa7776aa btrfs 19G 15G 3.1G 84% /

cartman {~} root# btrfs fi show
Label: 'Root' uuid: d71404d4-468e-47d5-8f06-3b65fa7776aa
Total devices 2 FS bytes used 7.43GiB
devid 1 size 9.31GiB used 8.06GiB path
/dev/disk/by-uuid/d71404d4-468e-47d5-8f06-3b65fa7776aa
devid 3 size 9.31GiB used 8.06GiB path /dev/sdd6

Label: 'Storage' uuid: 179749c1-1bfb-4f28-9c69-72126c96cdef

       Total devices 8 FS bytes used 1.83TiB
       devid    1 size 149.05GiB used 149.04GiB path /dev/sde
       devid    2 size 149.05GiB used 149.05GiB path /dev/sdh
       devid    3 size 465.76GiB used 465.76GiB path /dev/sdc
       devid    4 size 465.76GiB used 461.70GiB path /dev/sdf
       devid    5 size 465.76GiB used 461.00GiB path /dev/sda
       devid    6 size 465.76GiB used 461.00GiB path /dev/sdb
       devid    7 size 920.34GiB used 843.00GiB path /dev/sdg7
       devid    9 size 1.81TiB used 755.00GiB path /dev/sdd7

Btrfs v3.16
cartman {~} root# btrfs balance start /
ERROR: error during balancing '/' - No space left on device There may be more info in syslog - try dmesg | tail

Sep 13 16:35:42 localhost kernel: BTRFS info (device sdg6): found 33544 extents
Sep 13 16:35:51 localhost kernel: BTRFS info (device sdg6): found 33541 extents
Sep 13 16:35:52 localhost kernel: BTRFS info (device sdg6): relocating block group 80153214976 flags 18
Sep 13 16:35:53 localhost kernel: BTRFS info (device sdg6): found 1 extents
Sep 13 16:35:54 localhost kernel: BTRFS info (device sdg6): relocating block group 79680700416 flags 17
Sep 13 16:36:03 localhost kernel: BTRFS info (device sdg6): found 32545 extents
Sep 13 16:36:09 localhost kernel: BTRFS info (device sdg6): found 32545 extents
Sep 13 16:36:10 localhost kernel: BTRFS info (device sdg6): relocating block group 79412264960 flags 20
Sep 13 16:36:12 localhost kernel: BTRFS info (device sdg6): found 1114 extents
Sep 13 16:36:13 localhost kernel: BTRFS info (device sdg6): relocating block group 79143829504 flags 20
Sep 13 16:36:37 localhost kernel: BTRFS info (device sdg6): found 59076 extents
Sep 13 16:36:38 localhost kernel: BTRFS info (device sdg6): relocating block group 78875394048 flags 20
Sep 13 16:37:01 localhost kernel: BTRFS info (device sdg6): found 59268 extents
Sep 13 16:37:02 localhost kernel: BTRFS info (device sdg6): 6 enospc errors during balance
#2 lore raw

btrfs balance start -dusage=86 /
Done, had to relocate 1 out of 13 chunks

cartman {~} root# btrfs fi df /
Data, RAID1: total=7.03GiB, used=7.01GiB System, RAID1: total=32.00MiB, used=4.00KiB Metadata, RAID1: total=1.00GiB, used=438.88MiB unknown, single: total=148.00MiB, used=0.00

What's this 'unknown' section? Maybe this is the problem? How do I get rid of it?

I still get enospc after a balance with a filter, and then a regular balance:

cartman {~} root# btrfs balance start /
ERROR: error during balancing '/' - No space left on device

Quoted 11 lines · Leonidas Spyropoulos
Quoted 3 lines · Mark Murawski

I should have plenty of space for this operation, but it fails....

[...]

This might be useful:
https://btrfs.wiki.kernel.org/index.php/Balance_Filters

Regards,
Leonidas

#3 lore raw
Quoted 10 lines · Mark Murawski

btrfs balance start -dusage=86 /
Done, had to relocate 1 out of 13 chunks

cartman {~} root# btrfs fi df /
Data, RAID1: total=7.03GiB, used=7.01GiB System, RAID1: total=32.00MiB, used=4.00KiB Metadata, RAID1: total=1.00GiB, used=438.88MiB unknown, single: total=148.00MiB, used=0.00

What's this 'unknown' section? Maybe this is the problem? How do I get rid of it?

It's not a problem, it's cosmetic for now, something introduced in kernel 3.15 but progs doesn't yet give a label for.

Quoted 5 lines

I still get enospc after a balance with a filter, and then a regular balance:

cartman {~} root# btrfs balance start /
ERROR: error during balancing '/' - No space left on device

Maybe try mount option enospc_debug and retry, see if you get more information in dmesg.

I'm not sure if a balance in this case wants to create a new data and metadata chunk (on each device), or if it can start without creating any chunks. If it wants to create new chunks, it's 1GiB for data, and 256MiB for metadata. That's 1.256GiB but you only have 1.25GiB unallocated on each device: size 9.31GiB minus used 8.06GiB.

Chris Murphy

#4 lore raw

This is with debugging:

cartman {~} root# btrfs balance start /
ERROR: error during balancing '/' - No space left on device There may be more info in syslog - try dmesg | tail cartman {~} root#

Sep 15 18:31:47 localhost kernel: BTRFS info (device sdg6): disk space caching is enabled
Sep 15 18:31:47 localhost kernel: BTRFS info (device sdg6): disk space caching is enabled
Sep 15 18:31:47 localhost kernel: BTRFS info (device sdi): disk space caching is enabled
Sep 15 18:31:47 localhost kernel: BTRFS: bdev /dev/sdd7 errs: wr 418589, rd 444362, flush 11, corrupt 0, gen 0
Sep 15 18:31:47 localhost kernel: r8169 0000:02:00.0 eth0: link down
Sep 15 18:31:47 localhost kernel: r8169 0000:02:00.0 eth0: link down
Sep 15 18:31:49 localhost kernel: r8169 0000:02:00.0 eth0: link up
Sep 15 18:36:05 localhost kernel: BTRFS info (device sdg6): relocating block group 86631972864 flags 20
Sep 15 18:36:11 localhost kernel: BTRFS info (device sdg6): found 872 extents
Sep 15 18:36:11 localhost kernel: ------------[ cut here ]------------
Sep 15 18:36:11 localhost kernel: WARNING: CPU: 1 PID: 3763 at fs/btrfs/extent-tree.c:7273 btrfs_alloc_free_block+0x455/0x4a0()
Sep 15 18:36:11 localhost kernel: BTRFS: block rsv returned -28
Sep 15 18:36:11 localhost kernel: Modules linked in:
Sep 15 18:36:11 localhost kernel: CPU: 1 PID: 3763 Comm: tail Not tainted 3.16.1 #2
Sep 15 18:36:11 localhost kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-MA74GM-S2/GA-MA74GM-S2, BIOS F1 04/17/2008
Sep 15 18:36:11 localhost kernel: 0000000000000000 ffffffff819e3610 ffffffff817e4409 ffff88006ee2fa68
Sep 15 18:36:11 localhost kernel: ffffffff8106f6f2 ffff880073fc9e00 ffff88007525b000 0000000000001000
Sep 15 18:36:11 localhost kernel: ffff880072f58280 ffff880074196000 ffffffff8106f7d5 ffffffff819f5978
Sep 15 18:36:11 localhost kernel: Call Trace:
Sep 15 18:36:11 localhost kernel: [<ffffffff817e4409>] ? dump_stack+0x49/0x6a
Sep 15 18:36:11 localhost kernel: [<ffffffff8106f6f2>] ? warn_slowpath_common+0x82/0xb0
Sep 15 18:36:11 localhost kernel: [<ffffffff8106f7d5>] ? warn_slowpath_fmt+0x45/0x50
Sep 15 18:36:11 localhost kernel: [<ffffffff8135f074>] ? ___ratelimit+0x94/0x100
Sep 15 18:36:11 localhost kernel: [<ffffffff81296625>] ? btrfs_alloc_free_block+0x455/0x4a0
Sep 15 18:36:11 localhost kernel: [<ffffffff810992b7>] ? set_next_entity+0x37/0x80
Sep 15 18:36:11 localhost kernel: [<ffffffff812ca111>] ? read_extent_buffer+0xb1/0x110
Sep 15 18:36:11 localhost kernel: [<ffffffff81091de9>] ? finish_task_switch+0x49/0xe0
Sep 15 18:36:11 localhost kernel: [<ffffffff81280d9f>] ? btrfs_copy_root+0xef/0x2a0
Sep 15 18:36:11 localhost kernel: [<ffffffff812f1853>] ? create_reloc_root+0x1e3/0x2a0
Sep 15 18:36:11 localhost kernel: [<ffffffff812f7848>] ? btrfs_init_reloc_root+0xb8/0xd0
Sep 15 18:36:11 localhost kernel: [<ffffffff812a708f>] ? record_root_in_trans+0xaf/0x110
Sep 15 18:36:11 localhost kernel: [<ffffffff812a8496>] ? btrfs_record_root_in_trans+0x46/0x80
Sep 15 18:36:11 localhost kernel: [<ffffffff812a98fc>] ? start_transaction+0x8c/0x4f0
Sep 15 18:36:11 localhost kernel: [<ffffffff812b1168>] ? btrfs_dirty_inode+0x58/0xe0
Sep 15 18:36:11 localhost kernel: [<ffffffff8113b382>] ? touch_atime+0x152/0x160
Sep 15 18:36:11 localhost kernel: [<ffffffff810e3eb5>] ? generic_file_read_iter+0x545/0x5a0
Sep 15 18:36:11 localhost kernel: [<ffffffff810a1d49>] ? remove_wait_queue+0x19/0x60
Sep 15 18:36:11 localhost kernel: [<ffffffff810a1bc4>] ? prepare_to_wait+0x24/0x90
Sep 15 18:36:11 localhost kernel: [<ffffffff81122493>] ? new_sync_read+0x73/0xa0
Sep 15 18:36:11 localhost kernel: [<ffffffff811230ae>] ? vfs_read+0x9e/0x170
Sep 15 18:36:11 localhost kernel: [<ffffffff8112332f>] ? SyS_read+0x4f/0xd0
Sep 15 18:36:11 localhost kernel: [<ffffffff817eae12>] ? system_call_fastpath+0x16/0x1b
Sep 15 18:36:11 localhost kernel: ---[ end trace 8efb39cc34150d60 ]---
Sep 15 18:36:12 localhost kernel: BTRFS info (device sdg6): relocating block group 86598418432 flags 18
Sep 15 18:36:14 localhost kernel: BTRFS info (device sdg6): found 1 extents
Sep 15 18:36:15 localhost kernel: BTRFS info (device sdg6): relocating block group 86329982976 flags 20
Sep 15 18:36:49 localhost kernel: BTRFS info (device sdg6): found 55332 extents
Sep 15 18:36:50 localhost kernel: BTRFS info (device sdg6): relocating block group 86061547520 flags 20
Sep 15 18:37:14 localhost kernel: BTRFS info (device sdg6): found 57486 extents
Sep 15 18:37:14 localhost kernel: use_block_rsv: 2 callbacks suppressed
Sep 15 18:37:14 localhost kernel: ------------[ cut here ]------------
Sep 15 18:37:14 localhost kernel: WARNING: CPU: 1 PID: 3763 at fs/btrfs/extent-tree.c:7273 btrfs_alloc_free_block+0x455/0x4a0()
Sep 15 18:37:14 localhost kernel: BTRFS: block rsv returned -28
Sep 15 18:37:14 localhost kernel: Modules linked in:
Sep 15 18:37:14 localhost kernel: CPU: 1 PID: 3763 Comm: tail Tainted: G 
       W     3.16.1 #2
Sep 15 18:37:14 localhost kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-MA74GM-S2/GA-MA74GM-S2, BIOS F1 04/17/2008
Sep 15 18:37:14 localhost kernel: 0000000000000000 ffffffff819e3610 ffffffff817e4409 ffff88006ee2fa68
Sep 15 18:37:14 localhost kernel: ffffffff8106f6f2 ffff880073fc9da0 ffff88007525b000 0000000000001000
Sep 15 18:37:14 localhost kernel: ffff880035e393c0 ffff880074196000 ffffffff8106f7d5 ffffffff819f5978
Sep 15 18:37:14 localhost kernel: Call Trace:
Sep 15 18:37:14 localhost kernel: [<ffffffff817e4409>] ? dump_stack+0x49/0x6a
Sep 15 18:37:14 localhost kernel: [<ffffffff8106f6f2>] ? warn_slowpath_common+0x82/0xb0
Sep 15 18:37:14 localhost kernel: [<ffffffff8106f7d5>] ? warn_slowpath_fmt+0x45/0x50
Sep 15 18:37:14 localhost kernel: [<ffffffff8135f074>] ? ___ratelimit+0x94/0x100
Sep 15 18:37:14 localhost kernel: [<ffffffff81296625>] ? btrfs_alloc_free_block+0x455/0x4a0
Sep 15 18:37:14 localhost kernel: [<ffffffff810992b7>] ? set_next_entity+0x37/0x80
Sep 15 18:37:14 localhost kernel: [<ffffffff812ca111>] ? read_extent_buffer+0xb1/0x110
Sep 15 18:37:14 localhost kernel: [<ffffffff81091de9>] ? finish_task_switch+0x49/0xe0
Sep 15 18:37:14 localhost kernel: [<ffffffff81280d9f>] ? btrfs_copy_root+0xef/0x2a0
Sep 15 18:37:14 localhost kernel: [<ffffffff812a33b5>] ? btrfs_read_tree_root+0xb5/0x170
Sep 15 18:37:14 localhost kernel: [<ffffffff812f1853>] ? create_reloc_root+0x1e3/0x2a0
Sep 15 18:37:14 localhost kernel: [<ffffffff812f19e7>] ? __add_reloc_root+0x87/0x120
Sep 15 18:37:14 localhost kernel: [<ffffffff812f7848>] ? btrfs_init_reloc_root+0xb8/0xd0
Sep 15 18:37:14 localhost kernel: [<ffffffff812a708f>] ? record_root_in_trans+0xaf/0x110
Sep 15 18:37:14 localhost kernel: [<ffffffff812a8496>] ? btrfs_record_root_in_trans+0x46/0x80
Sep 15 18:37:14 localhost kernel: [<ffffffff812a98fc>] ? start_transaction+0x8c/0x4f0
Sep 15 18:37:14 localhost kernel: [<ffffffff812b1168>] ? btrfs_dirty_inode+0x58/0xe0
Sep 15 18:37:14 localhost kernel: [<ffffffff8113b382>] ? touch_atime+0x152/0x160
Sep 15 18:37:14 localhost kernel: [<ffffffff810e3eb5>] ? generic_file_read_iter+0x545/0x5a0
Sep 15 18:37:14 localhost kernel: [<ffffffff810a1d49>] ? remove_wait_queue+0x19/0x60
Sep 15 18:37:14 localhost kernel: [<ffffffff810a1bc4>] ? prepare_to_wait+0x24/0x90
Sep 15 18:37:14 localhost kernel: [<ffffffff81122493>] ? new_sync_read+0x73/0xa0
Sep 15 18:37:14 localhost kernel: [<ffffffff811230ae>] ? vfs_read+0x9e/0x170
Sep 15 18:37:14 localhost kernel: [<ffffffff8112332f>] ? SyS_read+0x4f/0xd0
Sep 15 18:37:14 localhost kernel: [<ffffffff817eae12>] ? system_call_fastpath+0x16/0x1b
Sep 15 18:37:14 localhost kernel: ---[ end trace 8efb39cc34150d61 ]---
Sep 15 18:37:14 localhost kernel: BTRFS info (device sdg6): relocating block group 84987805696 flags 17
Sep 15 18:37:24 localhost kernel: BTRFS info (device sdg6): 8 enospc errors during balance
Quoted 4 lines

Maybe try mount option enospc_debug and retry, see if you get more information in dmesg.

I'm not sure if a balance in this case wants to create a new data and metadata chunk (on each device), or if it can start without creating any chunks. If it wants to create new chunks, it's 1GiB for data, and 256MiB for metadata. That's 1.256GiB but you only have 1.25GiB unallocated on each device: size 9.31GiB minus used 8.06GiB.

#5 lore raw
In reply to Chris Murphy

Chris Murphy posted on Mon, 15 Sep 2014 14:54:57 -0600 as excerpted:

Quoted 6 lines
Quoted 5 lines

I still get enospc after a balance with a filter, and then a regular balance:

cartman {~} root# btrfs balance start /
ERROR: error during balancing '/' - No space left on device

Quoted 8 lines

Maybe try mount option enospc_debug and retry, see if you get more information in dmesg.

I'm not sure if a balance in this case wants to create a new data and metadata chunk (on each device), or if it can start without creating any chunks. If it wants to create new chunks, it's 1GiB for data, and 256MiB for metadata. That's 1.256GiB but you only have 1.25GiB unallocated on each device: size 9.31GiB minus used 8.06GiB.

Another possibility that has hit a few people:

Did you (MM/OP not CM) convert that filesystem from ext* to btrfs? If so, read on. If not, this doesn't apply so you may skip it.

Assuming such a conversion, did you delete the subvolume containing the original ext* yet, or not? If not, that may be the problem, because that subvolume must be left intact in ordered to allow rollback to ext* if desired. If you know you won't be rolling back, delete the ext* reserved subvolume as described on the wiki.

Meanwhile, after deleting that subvolume, be sure to complete the defrag and balance as suggested on the wiki, because failing to do so can lead to other problems later. Basically, the biggest extent size supported by btrfs is 1 GiB, the size of a btrfs data chunk, while ext* supports larger (unlimited size?) extents. Failing to complete the defrag in particular as suggested can mean large files with extents > 1 GiB in size, which gives btrfs balance indigestion since it expects to see only 1 GiB or smaller extents. Several folks who converted from ext3/4 have reported failed balances due to these too large extents, and fixing the problem later can require manually moving one-by-one all files large enough to be candidates for the problem (thus files > 1 GiB) out of btrfs and back in, thus resulting in properly chunk-split extents when the file is moved back to btrfs. Everyone who has reported this problem so far, has also reported that the move out and back in process solved the problem for them, but if there's lots of such files it can be a pain, and doing the defrag on the formerly ext* files before starting to use the now btrfs for other things, ESPECIALLY before trying to snapshot affected subvolumes since that locks the problem in place until those snapshots are deleted, is definitely preferred. =:^)

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
#6 lore raw
Quoted 6 lines · Duncan

Assuming such a conversion, did you delete the subvolume containing the original ext* yet, or not? If not, that may be the problem, because that subvolume must be left intact in ordered to allow rollback to ext* if desired. If you know you won't be rolling back, delete the ext* reserved subvolume as described on the wiki.

Good catch, I always forget this.

Chris Murphy

#7 lore raw
In reply to Duncan

I wish i could follow your procedure, but this wasn't an ext conversion.

I made this with mkfs for btrfs with kernel circa 3.8ish

Quoted 48 lines · Duncan

Chris Murphy posted on Mon, 15 Sep 2014 14:54:57 -0600 as excerpted:

Quoted 6 lines
Quoted 5 lines

I still get enospc after a balance with a filter, and then a regular balance:

cartman {~} root# btrfs balance start /
ERROR: error during balancing '/' - No space left on device

Quoted 8 lines

Maybe try mount option enospc_debug and retry, see if you get more information in dmesg.

I'm not sure if a balance in this case wants to create a new data and metadata chunk (on each device), or if it can start without creating any chunks. If it wants to create new chunks, it's 1GiB for data, and 256MiB for metadata. That's 1.256GiB but you only have 1.25GiB unallocated on each device: size 9.31GiB minus used 8.06GiB.

Another possibility that has hit a few people:

Did you (MM/OP not CM) convert that filesystem from ext* to btrfs? If so, read on. If not, this doesn't apply so you may skip it.

Assuming such a conversion, did you delete the subvolume containing the original ext* yet, or not? If not, that may be the problem, because that subvolume must be left intact in ordered to allow rollback to ext* if desired. If you know you won't be rolling back, delete the ext* reserved subvolume as described on the wiki.

Meanwhile, after deleting that subvolume, be sure to complete the defrag and balance as suggested on the wiki, because failing to do so can lead to other problems later. Basically, the biggest extent size supported by btrfs is 1 GiB, the size of a btrfs data chunk, while ext* supports larger (unlimited size?) extents. Failing to complete the defrag in particular as suggested can mean large files with extents > 1 GiB in size, which gives btrfs balance indigestion since it expects to see only 1 GiB or smaller extents. Several folks who converted from ext3/4 have reported failed balances due to these too large extents, and fixing the problem later can require manually moving one-by-one all files large enough to be candidates for the problem (thus files > 1 GiB) out of btrfs and back in, thus resulting in properly chunk-split extents when the file is moved back to btrfs. Everyone who has reported this problem so far, has also reported that the move out and back in process solved the problem for them, but if there's lots of such files it can be a pain, and doing the defrag on the formerly ext* files before starting to use the now btrfs for other things, ESPECIALLY before trying to snapshot affected subvolumes since that locks the problem in place until those snapshots are deleted, is definitely preferred. =:^)

#8 lore raw

Mark Murawski posted on Mon, 15 Sep 2014 22:23:47 -0400 as excerpted:

Quoted 3 lines

I wish i could follow your procedure, but this wasn't an ext conversion.

I made this with mkfs for btrfs with kernel circa 3.8ish

Good. That possible corner-case is eliminated. then. I had seen nothing indicating it might apply, but better to find that out now than to spend days tearing hair out trying to trace it down, only to discover it's that one already known corner-case! =:^)

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman