ERROR: error during balancing '.' - No space left on device

legolas:/mnt/btrfs_pool2# btrfs balance .
ERROR: error during balancing '.' - No space left on device
There may be more info in syslog - try dmesg | tail
[ 8454.159635] BTRFS info (device dm-1): relocating block group 288329039872 flags 1
[ 8590.167294] BTRFS info (device dm-1): relocating block group 232494465024 flags 1
[ 9200.801177] BTRFS info (device dm-1): relocating block group 85928706048 flags 1
[ 9533.830623] BTRFS info (device dm-1): 824 enospc errors during balance

But:

legolas:/mnt/btrfs_pool2# btrfs fi show `pwd`
Label: btrfs_pool2  uuid: 6afd4707-876c-46d6-9de2-21c4085b7bed
	Total devices 1 FS bytes used 646.41GiB
	devid    1 size 820.45GiB used 820.45GiB path /dev/mapper/disk2
Btrfs v3.12
legolas:/mnt/btrfs_pool2# 
Data, single: total=800.42GiB, used=636.91GiB
System, DUP: total=8.00MiB, used=92.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=10.00GiB, used=9.50GiB
Metadata, single: total=8.00MiB, used=0.00

I can't see how I'm full, and now that I can't run balance to fix things, this is making things worse.
Kernel is 3.14.

What am I missing?

Thanks,
Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

^^^ This is where you're full. There's a block reserve here that (should be) usable for doing a balance, and it seems to be about 500 MiB or so that's the point at which problems show up.

I think you probably shouldn't be doing a full balance, but a
filtered one:

# btrfs balance start -dusage=5 /mnt/btrfs_pool

which should only try to clean up chunks which have little usage (so it's much faster to run).

Not much. We do seem to have a problem with not being able to run
balance in recent kernels under some circumstances -- you're not the only person who's reported this kind of problem lately.

Hugo.

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
    --- Don't worry, he's not drunk. He's like that all the time. ---    

Marc MERLIN posted on Sun, 23 Mar 2014 00:01:44 -0700 as excerpted:

When it rains, it pours. What you're missing is that this is now the third thread in three days with exactly the same out-of-space-when-there- appears-to-be-plenty problem, which is well explained and a solution presented, along with further discussion, on those threads.

Evidently you haven't read the others, but rather than rewrite a similar reply here with exactly the same explanation and fix, I'll just refer you to them.

First thread, subject, original poster, message-id, gmane link:

fresh btrfs filesystem, out of disk space, hundreds of gigs free

Jon Nelson <[email protected]>

<CAKuK5J0LyagDYro6Yq7h-p-VjbwrbeuFf9AdNwRNg0jv3bq9QQ@mail.gmail.com>

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/33640

Second thread (same problem altho he chose as a subject a btrfsck error that wasn't actually related to it) ...

free space inode generation (0) did not match free space cache
generation

Hendrik Friedel <[email protected]>

<[email protected]>

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/33647

Very briefly, from your btrfs fi show:

devid 1 size 820.45GiB used 820.45GiB

That's total size and total of allocated chunks. They're equal, so there's no space left to allocate. Normally you'd use a balance, but a normal balance won't work without some space left to write the new chunks it's going to copy the old data/metadata into. There is however a solution, balance filters, using a command something like this (-d for data only since df says that's what has the extra chunks that need freed, usage=percent-full):

btrfs balance start -dusage=5 pwd

Tweak the N in usage=N as needed.

The explanation for all that is in the other threads. If that doesn't work (tho it most likely will), there's other solutions available, see the no-space discussion in the FAQ on the wiki, again as linked in the other threads.

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Both

legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=5 /mnt/btrfs_pool2
legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=0 /mnt/btrfs_pool2
failed unfortunately.

Thanks. Indeed, while I spent most of yesterday dealing with 3 btrfs filesystems, the one here that was hanging my laptop, the raid5 one that was hanging repeatedly during balance, and then my main server were one FS is so slow that it takes 8H to do an reflink copy or delete a backup with 1 million inodes, I got behind on reading the list :)

Thanks for the pointers

I had actually tried this, but it failed too:

legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=5 /mnt/btrfs_pool2
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=5
ERROR: error during balancing '/mnt/btrfs_pool2' - No space left on device

But I now just found
https://btrfs.wiki.kernel.org/index.php/Balance_Filters
and tried -dusage=0

Thanks for the other answer Hugo.

So, now I'm down to

legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=0 /mnt/btrfs_pool2
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=0
ERROR: error during balancing '/mnt/btrfs_pool2' - No space left on device

Looks like there is no good way out of this, so I'll start deleting snapshots.

Hopefully this will be handled better in later code.

Thanks for the answer,
Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Before you do this, can you take a btrfs-image of your metadata,
and add a report to bugzilla.kernel.org? You're not the only person who's had this problem recently, and I suspect there's something still lurking in there that needs attention.

With the info you can provide from a btrfs-image... let's hope so. :)

Hugo.

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
    --- Anyone using a computer to generate random numbers is, of ---    
                       course,  in a state of sin.                       

Mmmh, this may not be good :-/

legolas:/mnt/btrfs_pool2# btrfs-image -c 9 -t 6 /dev/mapper/disk2 /tmp/pool2.image
parent transid verify failed on 295965446144 wanted 51493 found 51495
parent transid verify failed on 295965446144 wanted 51493 found 51495
parent transid verify failed on 295965446144 wanted 51493 found 51495
parent transid verify failed on 295965446144 wanted 51493 found 51495
Ignoring transid failure
leaf parent key incorrect 295965446144
parent transid verify failed on 106205184 wanted 51468 found 51528
parent transid verify failed on 106205184 wanted 51468 found 51528
parent transid verify failed on 106205184 wanted 51468 found 51528
parent transid verify failed on 106205184 wanted 51468 found 51528
Ignoring transid failure
parent transid verify failed on 267162120192 wanted 51469 found 51528
parent transid verify failed on 267162120192 wanted 51469 found 51528
parent transid verify failed on 267162120192 wanted 51469 found 51528
parent transid verify failed on 267162120192 wanted 51469 found 51528
Ignoring transid failure
parent transid verify failed on 107315200 wanted 51468 found 51528
parent transid verify failed on 107315200 wanted 51468 found 51528
parent transid verify failed on 107315200 wanted 51468 found 51528
parent transid verify failed on 107315200 wanted 51468 found 51528
Ignoring transid failure
parent transid verify failed on 108081152 wanted 51468 found 51528
parent transid verify failed on 108081152 wanted 51468 found 51528
parent transid verify failed on 108081152 wanted 51468 found 51528
parent transid verify failed on 108081152 wanted 51468 found 51528
Ignoring transid failure
parent transid verify failed on 267162124288 wanted 51469 found 51528
parent transid verify failed on 267162124288 wanted 51469 found 51528
parent transid verify failed on 267162124288 wanted 51469 found 51528
parent transid verify failed on 267162124288 wanted 51469 found 51528
Ignoring transid failure
parent transid verify failed on 108351488 wanted 51468 found 51528
parent transid verify failed on 108351488 wanted 51468 found 51528
parent transid verify failed on 108351488 wanted 51468 found 51528
parent transid verify failed on 108351488 wanted 51468 found 51528
Ignoring transid failure
parent transid verify failed on 275021406208 wanted 51469 found 51529
parent transid verify failed on 275021406208 wanted 51469 found 51529
parent transid verify failed on 275021406208 wanted 51469 found 51529
parent transid verify failed on 275021406208 wanted 51469 found 51529
Ignoring transid failure
parent transid verify failed on 108466176 wanted 51468 found 51528
parent transid verify failed on 108466176 wanted 51468 found 51528
parent transid verify failed on 108466176 wanted 51468 found 51528
parent transid verify failed on 108466176 wanted 51468 found 51528
Ignoring transid failure
Segmentation fault

Is it a bug in the tool, or do I have real corruption?

Are there magic options I can give it to make it work around this?

Thanks,
Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

[snip]

xaba on IRC has just pointed out that it looks like you're running
this on a mounted filesystem -- it needs to be unmounted for btrfs-image to work reliably.

Hugo.

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
         --- Unix: For controlling fungal diseases in crops. ---         

Sorry, I didn't realize that, although it makes sense. btrfs-image really should fail though:
https://bugzilla.kernel.org/show_bug.cgi?id=72791

I got the image, but it's 1.5GB (1TB filesystem), but kernel bugzilla times out after 1% of upload (repeatedly)
Placeholder bug:
https://bugzilla.kernel.org/show_bug.cgi?id=72801

Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Here is the image:
http://marc.merlins.org/tmp/pool2.image

After taking that image for you, I got the filesysetm to recover:

legolas:/mnt/btrfs_pool2# for i in *weekly*; do btrfs subvolume delete $i;
done
Delete subvolume '/mnt/btrfs_pool2/home_weekly_20140309_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/home_weekly_20140316_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/root_weekly_20140309_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/root_weekly_20140316_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/tmp_weekly_20140309_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/tmp_weekly_20140316_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/usr_weekly_20140309_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/usr_weekly_20140316_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/var_weekly_20140309_00:06:01'
^A Delete subvolume '/mnt/btrfs_pool2/var_weekly_20140316_00:06:01'

legolas:/mnt/btrfs_pool2# btrfs fi df .
Data, single: total=800.42GiB, used=636.91GiB
System, DUP: total=8.00MiB, used=92.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=10.00GiB, used=9.50GiB
Metadata, single: total=8.00MiB, used=0.00
legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=0 /mnt/btrfs_pool2

Dumping filters: flags 0x1, state 0x0, force is off
DATA (flags 0x2): balancing, usage=0 Done, had to relocate 91 out of 823 chunks

legolas:/mnt/btrfs_pool2# btrfs fi df .
Data, single: total=709.01GiB, used=603.85GiB
System, DUP: total=8.00MiB, used=88.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=10.00GiB, used=7.39GiB
Metadata, single: total=8.00MiB, used=0.00

Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/