Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?
Hello there!
I'm stumbling over this regularly. I explicitly upgraded the kernel to avoid this, but it still occurs me every few months:
$ touch foo touch: »foo“ kann nicht berührt werden: Auf dem Gerät ist kein Speicherplatz mehr verfügbar $ sudo btrfs fi df / Data, single: total=101.14GiB, used=78.99GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=9.00GiB, used=8.48GiB unknown, single: total=512.00MiB, used=832.00KiB
Whut? Why? WTF?
$ uname -a Linux neptun 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
dmesg is empty. Resolution is to delete a whole bunch of btrfs subvolumes which are produced by apt-btrfs-snapshot. I run a
sudo btrfs balance start -dusage=5 /
afterwards as part of my regular "i have no clue what the heck is wrong with this; maybe it will help & mend my problem" process.
This is annoying. Why is even btrfs own "disk free" utility lying to me?
How to fix this? How to avoid this? How to get notified about this?
- Ben
P.S.: Just as user feedback: For /srv I'm using on the very same system ZFS since the very first day. With snapshots & all the fancy stuff like ZRAID-1, lz4, ... My number of Issues there: 0
Quoted 14 lines · On Wed, Nov 18, 2015 at 07:53:03PM +0100, [email protected] wrote:
Hello there!
I'm stumbling over this regularly. I explicitly upgraded the kernel to avoid this, but it still occurs me every few months:
$ touch foo touch: »foo“ kann nicht berührt werden: Auf dem Gerät ist kein Speicherplatz mehr verfügbar $ sudo btrfs fi df / Data, single: total=101.14GiB, used=78.99GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=9.00GiB, used=8.48GiB unknown, single: total=512.00MiB, used=832.00KiB
Quoted 1 line
Whut? Why? WTF?
Running out of metadata space, probably, although it's impossible
to tell from only this information -- see below.
Quoted 11 lines
$ uname -a Linux neptun 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linuxdmesg is empty. Resolution is to delete a whole bunch of btrfs subvolumes which are produced by apt-btrfs-snapshot. I run a
sudo btrfs balance start -dusage=5 /
afterwards as part of my regular "i have no clue what the heck is wrong with this; maybe it will help & mend my problem" process.This is annoying. Why is even btrfs own "disk free" utility lying to me?
I't snot lying to you -- you're just not seeing the whole
story. You can't tell everything about disk usage from btrfs fi df. You need to use btrfs fi show as well to get the full information (or btrfs fi usage, which combines the output of both). See the FAQ[1] for how to interpret the output.
Quoted 1 line
How to fix this? How to avoid this? How to get notified about this?
Without seeing your btrfs fi show output, I can't say for sure, but
the usual problem in this instance is that you've got all your available space allocated to either data or metadata, and the FS needs more metadata space and can't allocate it. The solution is to free up some of the data allocation, using a filtered balance. See the FAQ[2] for how to spot the problem, and what to do about it.
[1] https://btrfs.wiki.kernel.org/index.php/FAQ#Understanding_free_space.2C_using_the_original_tools
[2] https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29
Hugo.
Quoted 6 lines
- Ben
P.S.: Just as user feedback: For /srv I'm using on the very same system ZFS since the very first day. With snapshots & all the fancy stuff like ZRAID-1, lz4, ... My number of Issues there: 0
Signature · 4 lines
Hugo Mills | Close enough for government work. hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 |
Hugo Mills wrote on 2015/11/18 19:08 +0000:
Quoted 15 lines · On Wed, Nov 18, 2015 at 07:53:03PM +0100, [email protected] wrote:
Quoted 14 lines
Hello there!
I'm stumbling over this regularly. I explicitly upgraded the kernel to avoid this, but it still occurs me every few months:
$ touch foo touch: »foo“ kann nicht berührt werden: Auf dem Gerät ist kein Speicherplatz mehr verfügbar $ sudo btrfs fi df / Data, single: total=101.14GiB, used=78.99GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=9.00GiB, used=8.48GiB unknown, single: total=512.00MiB, used=832.00KiB
Quoted 4 lines
Quoted 1 line
Whut? Why? WTF?
Running out of metadata space, probably, although it's impossible
to tell from only this information -- see below.
It's almost 100% sure that you already run out of metadata space.
Although the metadata output is showing that you still have about 512M available, but the 512M is Global Reserved space, or the unknown one.
Global reserved space should not be used, until metadata is really running out. And in your case, you are already using global reserved space, see the 832KB?
You may question that why not continue using global reserved space, but that's the last method, so Btrfs will just give you ENOSPC error.
The output is really a little confusing. I'd like the change the output by adding global reserved into metadata used space and make it a sub item for metadata. This should make the output more understandable.
Thanks,
Qu
Quoted 40 lines
Quoted 5 lines
$ uname -a Linux neptun 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/LinuxQuoted 7 lines
dmesg is empty. Resolution is to delete a whole bunch of btrfs subvolumes which are produced by apt-btrfs-snapshot. I run a
sudo btrfs balance start -dusage=5 /
afterwards as part of my regular "i have no clue what the heck is wrong with this; maybe it will help & mend my problem" process.This is annoying. Why is even btrfs own "disk free" utility lying to me?
I't snot lying to you -- you're just not seeing the whole
story. You can't tell everything about disk usage from btrfs fi df. You need to use btrfs fi show as well to get the full information (or btrfs fi usage, which combines the output of both). See the FAQ[1] for how to interpret the output.Quoted 1 line
How to fix this? How to avoid this? How to get notified about this?
Without seeing your btrfs fi show output, I can't say for sure, but
the usual problem in this instance is that you've got all your available space allocated to either data or metadata, and the FS needs more metadata space and can't allocate it. The solution is to free up some of the data allocation, using a filtered balance. See the FAQ[2] for how to spot the problem, and what to do about it.[1] https://btrfs.wiki.kernel.org/index.php/FAQ#Understanding_free_space.2C_using_the_original_tools
[2] https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29Hugo.
Quoted 6 lines
- Ben
P.S.: Just as user feedback: For /srv I'm using on the very same system ZFS since the very first day. With snapshots & all the fancy stuff like ZRAID-1, lz4, ... My number of Issues there: 0
Qu Wenruo posted on Thu, 19 Nov 2015 08:42:13 +0800 as excerpted:
Quoted 2 lines
Although the metadata output is showing that you still have about 512M available, but the 512M is Global Reserved space, or the unknown one.
Unknown here, as the userspace (btrfs-progs) is evidently too old to show it as global reserve, as it does in newer versions...
Quoted 3 lines
The output is really a little confusing. I'd like the change the output by adding global reserved into metadata used space and make it a sub item for metadata.
Thanks for the clarification. It's most helpful, here. =:^)
I've at times wondered if global reserve folded into one of the other settings. Apparently it comes from the metadata allocation, but while metadata is normally dup (single-device btrfs) or raid1 (multi-device), global reserve is single.
It would have been nice if that sort of substructure was described a bit better when global reserve first made its appearance, at least in the patch descriptions and release announcement, if not then yet in btrfs fi df output, first implementations being what they are. But regardless, now at least it should be clear for list regulars who read this thread anyway, since the above reasonably clarifies things.
As for btrfs fi df, making global reserve a metadata subentry there would be one way to deal with it, preserving the exposure of the additional data provided by that line (here, the fact that global reserve is actually being used, underlining the fact that the filesystem is severely short on space).
Another way of handling it would be to simply add the global reserve into the metadata used figure before printing it, eliminating the separate global reserve line, and changing the upthread posted metadata line from 8.48 GiB of 9 GiB used, to 8.98 of 9 GiB used, which is effectively the case if the 512 MiB of global reserve indeed comes from the metadata allocation. This would more clearly show how full metadata actually is without the added complexity of an additional global reserve line, but would lose the fact that global reserve is actually in use, that the broken out global reserve line exposes.
I'd actually argue in favor of the latter, directly folding global reserve allocation into metadata used, since it'd both be simpler, and more consistent if for instance btrfs fi usage didn't report separate global reserve in the overall stats, but fail to report it in the per- device stats and in btrfs dev usage.
Either way would make much clearer that metadata is actually running out than the current report layout does, since "metadata used" would then either explicitly or implicitly include the global reserve.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 3 lines · On Wed, 18 Nov 2015 19:53:03 +0100 [email protected] wrote:
$ uname -a Linux neptun 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
For whatever unknown reason, Ubuntu repeatedly chooses to stabilize on exactly the wrong kernel versions. Their recent releases were based on kernels 3.11, 3.13 and 3.19. None of these were chosen by the upstream as the "longterm" branches, and as you can see none are listed anymore on https://www.kernel.org/
To emphasize -- the mainline Linux kernel developers (and Btrfs developers) basically do not care anymore about 3.19 at all. What you currently use, is a piecemeal of unknown quality, cobbled together by Ubuntu developers from patches lifted either from 3.18 or 4.1 series -- by taking fixes they deem important and frankensteining them together until they somehow seem to apply.
And do they back/forward-port Btrfs-related bugfixes?... all of them?... are they doing that well enough? Just nobody knows.
So my suggestion would be to try a newer kernel from www.kernel.org: if the problem disappears at 4.1 then just keep on using that, or 4.3 if you have to, but otherwise that one might be a bit too new to start using right away.
Signature · 2 lines
With respect, Roman
Quoted 7 lines · Roman Mamedov
Quoted 3 lines · On Wed, 18 Nov 2015 19:53:03 +0100 [email protected] wrote:
$ uname -a Linux neptun 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[...]
Quoted 4 lines
So my suggestion would be to try a newer kernel from www.kernel.org: if the problem disappears at 4.1 then just keep on using that, or 4.3 if you have to, but otherwise that one might be a bit too new to start using right away.
Give http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1.13-wily/ a try.
wget -e robots=off -r -l1 -np -nd -A '*all.deb','*generic*amd64.deb'
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1.13-wily/
Quoted 4 lines · On 2015-11-18 13:53, [email protected] wrote:
P.S.: Just as user feedback: For /srv I'm using on the very same system ZFS since the very first day. With snapshots & all the fancy stuff like ZRAID-1, lz4, ... My number of Issues there: 0
Since other people have adequately answered the main questions, I'll just comment on this.
In general, ZFS is not a good comparison to BTRFS for anything but features. Just the open source version of ZFS has been around for more than a decade, and it existed as proprietary code for quite a while before that. As such, it's had a lot longer to stabilize, has had exponentially more testing, and in general is a lot more reliable. As such, aside from feature comparisons, ZFS is going to win in almost all tests of performance, reliability, and usability for the foreseeable future. This doesn't mean you shouldn't use BTRFS (ZFS got to where it is now exactly because _a lot_ of people use it), it just means you need to do a lot of homework, and ideally keep you system actually up to date (having all updates installed on Ubuntu doesn't really count in this case, they're pretty bad sometimes about not properly tracking upstream development (although they are significantly better than regular Debian)).
Many thanks to everybody on the list for the awful a lot help I received. I'm impressed by the amount, quality and speed of the responses I received!
Quoted 1 line
See the FAQ[1] for how to interpret the output. ...
Very helpful, thanks.
Quoted 1 line
The solution is to free up some of the data allocation, using a filtered balance.
As noted in my original post, this is already part of my "voodoo" I'm doing in these situations. Will newer kernel relieve me from this burden?
FYI: On executing "btrfs fi show" the first time after I deleted many snapshots and did a filtered (-dusage) balance, it reported sth. like
devid 1 size 119.21GiB used 117.03GiB now (after a unfiltered balance?) it reports devid 1 size 119.21GiB used 66.06GiB
I'm really confuzzled.
Quoted 3 lines
[..] Ubuntu repeatedly chooses to stabilize on exactly the wrong kernel versions. ..
Give http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1.13-wily/ a try.
Could not agree more. Will give wily kernel a try.
Quoted 1 line
The output is really a little confusing. [..] should make the output more understandable.
+1
Considering that the status of a file systems should be understandable by regular users, it would be great if the btrfs tools could provide additional user hints like _"I seems your metadata space is running out of space. Please run foo bar to resolve this situation or consult FAQ[4] for more details"_.
Austin Hemmelgarn commented on my ZFS comparison. His argument sound very vigilant taking into account the benefit of a GPL-compliant FOSS FS like btrfs vs. a stabilized ZFS.
The thing I appreciate in ZFS as user most, is the very user friendly interface of it's tools. Just have a look at the following output:
# zpool status Permission denied the ZFS utilities must be run as root. # sudo zpool status pool: zfstank state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0 in 23h56m with 0 errors on Mon Aug 31 01:21:26 2015 config: ... errors: No known data errors
This output is totally self explaining. In case of errors, ZFS will simply lists the broken files and tell you about your options to resolve the issue (restore backup, overwrite, delete). It uses easy terms & guides me in many places.
In contrast the terminology in btrfs (DUP, Metadata, Global Reserve) is rather intimidating, and as already stated, the output to a simple command like "disk free" rather difficult to understand. But btrfs seems to make progress here.
Again thanks for all the helpful feedback here on the list!
I'm still a bit puzzled how to avoid my situation in the first place.
Should I pick "btrfs show" in favor to "btrfs fi df" to learn about an impending "disk full" situation? Will newer kernels do the balance on their own?
-Ben
Quoted 7 lines · On Thu, Nov 19, 2015 at 06:35:24PM +0100, [email protected] wrote:
Many thanks to everybody on the list for the awful a lot help I received. I'm impressed by the amount, quality and speed of the responses I received!
Quoted 1 line
See the FAQ[1] for how to interpret the output. ...
Very helpful, thanks.
Quoted 1 line
The solution is to free up some of the data allocation, using a filtered balance.
As noted in my original post, this is already part of my "voodoo" I'm doing in these situations.
Sometimes it's useful to know what the voodoo is actually doing. :)
Quoted 1 line
Will newer kernel relieve me from this burden?
Not in any current or planned kernel, to the best of my knowledge.
Quoted 6 lines
FYI: On executing "btrfs fi show" the first time after I deleted many snapshots and did a filtered (-dusage) balance, it reported sth. like
devid 1 size 119.21GiB used 117.03GiB now (after a unfiltered balance?) it reports devid 1 size 119.21GiB used 66.06GiBI'm really confuzzled.
It's just freed up lots of space. You'll probably find that your
"total" value for data in btrfs fi df is close to (but not exactly) 66 GiB now, if you've just run a full unfiltered balance. (The difference being made up of metadata).
Quoted 8 lines
Quoted 3 lines
[..] Ubuntu repeatedly chooses to stabilize on exactly the wrong kernel versions. ..
Give http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1.13-wily/ a try.Could not agree more. Will give wily kernel a try.
Quoted 1 line
The output is really a little confusing. [..] should make the output more understandable.
+1
Considering that the status of a file systems should be understandable by regular users, it would be great if the btrfs tools could provide additional user hints like _"I seems your metadata space is running out of space. Please runfoo barto resolve this situation or consult FAQ[4] for more details"_.
There's a repo of "useful btrfs scripts" that David Sterba looks
after. I think there's already a script to go in your cron.daily directory that already does the check and balances if necessary. If there isn't, there should be (and it shouldn't be that hard to write, either).
David -- where is that repo?
[snip]
Quoted 2 lines
I'm still a bit puzzled how to avoid my situation in the first place.
Should I pick "btrfs show" in favor to "btrfs fi df" to learn about an impending "disk full" situation?
In the general case, you need both of them to make sense of it. (Or
just btrfs fi usage) You're looking for the FS fully allocated, and metadata usage+block reserve very close to metadata allocation.
Quoted 1 line
Will newer kernels do the balance on their own?
I think it's on the "projects" list on the wiki, so it may get done
eventually. As I said above, I'm not aware of anyone working on it.
Hugo.
Signature · 4 lines
Hugo Mills | 2 + 2 = 5, for sufficiently large values of 2. hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 5 lines
It's just freed up lots of space. You'll probably find that your
"total" value for data in btrfs fi df is close to (but not exactly) 66 GiB now, if you've just run a full unfiltered balance. (The difference being made up of metadata).
I think i need to dive more into the details of btrfs to finally grasp the details here
Quoted 2 lines
There's a repo of "useful btrfs scripts" that David Sterba looks
after.
Great pointer!
I think you refer to https://github.com/kdave/btrfsmaintenance and it indeed looks very promising.
Quoted 2 lines
Quoted 1 line
Should I pick "btrfs show" in favor to "btrfs fi df" to learn about an impending "disk full" situation?
In the general case, you need both of them to make sense of it.
Duh.... I'll better... ehh... try to find some backported btrfs-profs supporting usage.
Quoted 2 lines
Quoted 1 line
Will newer kernels do the balance on their own?
I think it's on the "projects" list on the wiki [..] aware of anyone working on it.
Ok - which is another +1 for looking at David 's repo.
Thanks!
- Ben
Quoted 16 lines · Ben Tebulin
Quoted 5 lines
It's just freed up lots of space. You'll probably find that your
"total" value for data in btrfs fi df is close to (but not exactly) 66 GiB now, if you've just run a full unfiltered balance. (The difference being made up of metadata).I think i need to dive more into the details of btrfs to finally grasp the details here
Quoted 2 lines
There's a repo of "useful btrfs scripts" that David Sterba looks
after.Great pointer!
I think you refer to https://github.com/kdave/btrfsmaintenance and it indeed looks very promising.
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
Should I pick "btrfs show" in favor to "btrfs fi df" to learn about an impending "disk full" situation?
In the general case, you need both of them to make sense of it.
Duh.... I'll better... ehh... try to find some backported btrfs-profs supporting
usage.
It's not massively more helpful -- you still need to understand the
relationships. It just uses slightly different language, and puts all the information in one output instead of two.
You start with a load of unused space -- that's the "total" for
each device in btrfs fi show. That space is allocated to specific usages as the FS needs it. The allocated space is the "used" in btrfs fi show for each device.
Then we switch to looking at btrfs fi df. The "total" values are
the amount of the *allocated* space given to each type of storage. Within those, actual stuff is stored (like your files), which is the "used" value for each kind of storage.
If you've ever played SimCity, the allocation process is like
zoning -- you say what kind of thing can go on the space, but it's not actually used until something gets built on it.
The problem you hit is when everything has been allocated, and
there's a need for more metadata space (usually), and there's lots of unused data space. The balance operation moves some things around so that some of the unused data allocation can be freed up, giving the FS the ability to allocate more metadata space.
Hugo.
Quoted 7 lines
Quoted 2 lines
Quoted 1 line
Will newer kernels do the balance on their own?
I think it's on the "projects" list on the wiki [..] aware of anyone working on it.
Ok - which is another +1 for looking at David 's repo.
Thanks!
- Ben
Signature · 4 lines
Hugo Mills | 2 + 2 = 5, for sufficiently large values of 2. hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 |
This explanation helped a lot. Got it now!
Thank you!
Quoted 19 lines
You start with a load of unused space -- that's the "total" for
each device in btrfs fi show. That space is allocated to specific usages as the FS needs it. The allocated space is the "used" in btrfs fi show for each device.Then we switch to looking at btrfs fi df. The "total" values are
the amount of the *allocated* space given to each type of storage. Within those, actual stuff is stored (like your files), which is the "used" value for each kind of storage.If you've ever played SimCity, the allocation process is like
zoning -- you say what kind of thing can go on the space, but it's not actually used until something gets built on it.The problem you hit is when everything has been allocated, and
there's a need for more metadata space (usually), and there's lots of unused data space. The balance operation moves some things around so that some of the unused data allocation can be freed up, giving the FS the ability to allocate more metadata space.
Quoted 5 lines · Hugo Mills
Quoted 1 line · On Thu, Nov 19, 2015 at 06:35:24PM +0100, [email protected] wrote:
Will newer kernels do the balance on their own?
I think it's on the "projects" list on the wiki, so it may get done
eventually. As I said above, I'm not aware of anyone working on it.
TBH, while it would be a nice feature, it's also something that has the potential to cause issues for people if it all happens at once. IN many of the cases I've seen, the usual issue is lots of mostly empty data chunks (usually less than 20% in the two times I've run into this myself). Based on this, having something in the kernel that could scan chunk usage every now and then, and repack mostly empty chunks if there is space already allocated in existing chunks, would probably significantly reduce the chances of this happening for most people, and cause much less noticeable performance degradation than triggering a full balance on a chunk allocation failure. Since I started using a cron job to do a daily balance of chunks less than 20% full, and a weekly one for chunks less than 50% full, I've not run into these issues at all, and actually see on average better performance on my tradition hard disks.
Austin S Hemmelgarn posted on Thu, 19 Nov 2015 07:28:34 -0500 as excerpted:
Quoted 3 lines
(having all updates installed on Ubuntu doesn't really count in this case, they're pretty bad sometimes about not properly tracking upstream development[)]
No kidding. I'm involved with an upstream that had a security patch and version-bump a number of years ago. An Ubuntu bug was filed... and it sat in the bug queue IIRC untouched until it was obsoleted by newer releases, where the new version /was/ included. At least Fedora (which I remember a poster confirming the update on) and Gentoo (which I run, personally filed a bug with, and saw the security bump) made the version bump available as an update.
Apparently, as it wasn't a headline component (one would /hope/ they at least get security updates out for /them/, and they evidently do for at least some as they do publish security updates, but at this point I'd wonder how consistently they do for others), Ubuntu simply didn't care. Made /me/ glad I wasn't on Ubuntu, that's for sure!
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
linux-btrfs.tebulin posted on Thu, 19 Nov 2015 18:56:45 +0000 as excerpted:
Meta-comment:
Apparently that attribution should actually be to Hugo Mills. I've no idea what went wrong, but at least here as received from gmane.org, the from header really does say linux-btrfs.tebulin, so something obviously bugged out somewhere!
Meanwhile discussing btrfs data/metadata allocation, vs. usage of that allocation, Hugo also known here as tebulin, explained...
Quoted 3 lines
If you've ever played SimCity, the allocation process is like zoning -- you say what kind of thing can go on the space, but it's not actually used until something gets built on it.
Very nice analogy. Thanks. =:^)
Tho I'd actually put it in terms of the real thing that sim city simulates, thus eliminating the proprietary comparison. A city zones an area one way or another, restricting what can be built there -- you can't put heavy industry in a residential zone. But the lot is still empty until something's actually built there.
And if the city has all its area zoned residential, and some company wants to build a plant there (generally considered a good thing as it'll provide employment), there's a process that allows rezoning.
In btrfs, there's four types of allocations aka "zones":
1) Unallocated (unzoned)
Can be used for anything but must be allocated/zoned first
2) System
Critical but limited in size and generally only allocated at mkfs or when adding a new device.
3) Data
The actual file storage, generally the largest allocation.
4) Metadata
Information /about/ the files, where they are located (the location and size of individual extents), ownership and permissions, date stamps, checksums, and for very small files (a few KiB), sometimes the file content itself.
4a) Global reserve
A small part of metadata reserved for emergency use only. Btrfs is pretty strict about its use, and will generally error out with ENOSPC if metadata space other than the global reserve is used up, before actually using this global reserve. As a result, any of the global reserve used at all indicates a filesystem in very severe straits, crisis mode.
As it happens, btrfs in practice tends to be a bit liberal about allocating/zoning data chunks, since it's easier to find bigger blocks of space in unallocated space than it is in busy partly used data space. (Think of a big shopping center. It's easier to build it in outlying areas that haven't been built up yet, where many whole blocks worth of space can be allocated/zoned at once, than it is in the city center, where even finding a single whole block vacant, is near impossible.)
Over time, therefore, more and more space tends to be allocated to data, while existing data space, like those blocks near city center, may have most of its files/buildings deleted, but still have a couple still occupied.
Btrfs balancing, then, is comparable to the city functions of condemnation and rezoning to vacant/unallocated, forcing remaining occupants, most commonly data zoned, to move out of the way so the area can be reclaimed for other usage. Then it can be rezoned to data again, or to metadata, whatever needs it.
(FWIW, I played the original sim city, but IIRC it wasn't sophisticated enough to have zoning yet. Of course I've not played anything recent as to my knowledge it's not freedomware, and since I no longer agree to waive rights via EULA, etc, I can no longer legally install and run such things. But the real life sim city simulates, is the antitype behind both that simulation, and the analogy I drew here, based on the one you drew to the simulation.)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Am 20.11.2015 um 04:14 schrieb Duncan:
Quoted 6 lines
Meta-comment:
Apparently that attribution should actually be to Hugo Mills. I've no idea what went wrong, but at least here as received from gmane.org, the from header really does say linux-btrfs.tebulin, so something obviously bugged out somewhere!
Meta-Answer: My fault
I accidentally exposed my spamgourmet proxy address to Hugo by taking him CC. He replied to all assuming it would go directly to the list, but the proxy address masked his original address an put id in it.
Sorry for messing up!
linux-btrfs.tebulin posted on Fri, 20 Nov 2015 10:38:33 +0100 as excerpted:
Quoted 8 lines
Am 20.11.2015 um 04:14 schrieb Duncan:
Quoted 6 lines
Meta-comment:
Apparently that attribution should actually be to Hugo Mills. I've no idea what went wrong, but at least here as received from gmane.org, the from header really does say linux-btrfs.tebulin, so something obviously bugged out somewhere!
Quoted 7 lines
Meta-Answer: My fault
I accidentally exposed my spamgourmet proxy address to Hugo by taking him CC. He replied to all assuming it would go directly to the list, but the proxy address masked his original address an put id in it.
Sorry for messing up!
Makes sense.
While I suppose his reply to all went to the list and to the proxy, the proxy bounced it, presumably with the same message-id, and anything that tracks by message-id (including my client), which is supposed to be unique and thus usable for message tracking, would see only one of the two. Which one would depend on which one got to it first, and whether it took that and refused the other one, or let the second one override the first.
Either way, definitely confusing, but at least there's a reasonable explanation now. Thanks for helping to clear that up! =:^)
(FWIW, as with most linux kernel lists, replying to all is customary. I generally reply only to the list, however, because it's far easier with my specific setup, tho I'll reply to author as well if specifically requested. But Hugo actually does the list-recommended thing by replying to all by default.)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
If I may add:
Information for "System"
System, DUP: total=32.00MiB, used=16.00KiB
is also quite technical, as for end user system = metadata (one can call it "filesystem metadata" perhaps). For simplicity the numbers can be added to "Metadata" thus eliminating that line as well.
For those power users who really want to see the tiny details like "System" and "GlobalReserve" I suggest to implement "-v" flag:
# btrfs fi usage -v
Quoted 52 lines · Duncan
Qu Wenruo posted on Thu, 19 Nov 2015 08:42:13 +0800 as excerpted:
Quoted 2 lines
Although the metadata output is showing that you still have about 512M available, but the 512M is Global Reserved space, or the unknown one.
Unknown here, as the userspace (btrfs-progs) is evidently too old to show it as global reserve, as it does in newer versions...
Quoted 3 lines
The output is really a little confusing. I'd like the change the output by adding global reserved into metadata used space and make it a sub item for metadata.
Thanks for the clarification. It's most helpful, here. =:^)
I've at times wondered if global reserve folded into one of the other settings. Apparently it comes from the metadata allocation, but while metadata is normally dup (single-device btrfs) or raid1 (multi-device), global reserve is single.
It would have been nice if that sort of substructure was described a bit better when global reserve first made its appearance, at least in the patch descriptions and release announcement, if not then yet in btrfs fi df output, first implementations being what they are. But regardless, now at least it should be clear for list regulars who read this thread anyway, since the above reasonably clarifies things.
As for btrfs fi df, making global reserve a metadata subentry there would be one way to deal with it, preserving the exposure of the additional data provided by that line (here, the fact that global reserve is actually being used, underlining the fact that the filesystem is severely short on space).
Another way of handling it would be to simply add the global reserve into the metadata used figure before printing it, eliminating the separate global reserve line, and changing the upthread posted metadata line from 8.48 GiB of 9 GiB used, to 8.98 of 9 GiB used, which is effectively the case if the 512 MiB of global reserve indeed comes from the metadata allocation. This would more clearly show how full metadata actually is without the added complexity of an additional global reserve line, but would lose the fact that global reserve is actually in use, that the broken out global reserve line exposes.
I'd actually argue in favor of the latter, directly folding global reserve allocation into metadata used, since it'd both be simpler, and more consistent if for instance btrfs fi usage didn't report separate global reserve in the overall stats, but fail to report it in the per- device stats and in btrfs dev usage.
Either way would make much clearer that metadata is actually running out than the current report layout does, since "metadata used" would then either explicitly or implicitly include the global reserve.
Signature · 2 lines
With best regards, Dmitry
Quoted 15 lines · Duncan
Austin S Hemmelgarn posted on Thu, 19 Nov 2015 07:28:34 -0500 as excerpted:
Quoted 3 lines
(having all updates installed on Ubuntu doesn't really count in this case, they're pretty bad sometimes about not properly tracking upstream development[)]
No kidding. I'm involved with an upstream that had a security patch and version-bump a number of years ago. An Ubuntu bug was filed... and it sat in the bug queue IIRC untouched until it was obsoleted by newer releases, where the new version /was/ included. At least Fedora (which I remember a poster confirming the update on) and Gentoo (which I run, personally filed a bug with, and saw the security bump) made the version bump available as an update.
Quoted 6 lines
Apparently, as it wasn't a headline component (one would /hope/ they at least get security updates out for /them/, and they evidently do for at least some as they do publish security updates, but at this point I'd wonder how consistently they do for others), Ubuntu simply didn't care. Made /me/ glad I wasn't on Ubuntu, that's for sure!
Yeah, that's one of the reasons that I switched to Gentoo. For those who may be interested, the full list is:
1. Way easier to stay up to date (from an administrative perspective, not necessarily a computational time perspective (although emerge does compute dependencies noticeably faster than dnf, yum, and zypper most of the time)).
2. Exponentially more configurable than almost any other full distro (this is the big one that really made me choose Gentoo initially).
3. I build my own kernels, and Gentoo has direct integration for this (in fact, they only distribute the sources for the kernel, you build it locally regardless (which is really not hard even if you don't use genkernel to automate it)).
4. Better response to bug reports (as compared to Ubuntu, although this is on average, and not always consistent).
5. Proper support for a wide variety of desktop environments in the main project (In almost every other distro, any desktop other than the default is usually an after thought, or maintained through a separate project; and, Xfce (which is what I use) is not very popular as a default environment for some reason).
Quoted 6 lines · Dmitry Katsubo
If I may add:
Information for "System"
System, DUP: total=32.00MiB, used=16.00KiB
Quoted 8 lines
is also quite technical, as for end user system = metadata (one can call it "filesystem metadata" perhaps). For simplicity the numbers can be added to "Metadata" thus eliminating that line as well.
For those power users who really want to see the tiny details like "System" and "GlobalReserve" I suggest to implement "-v" flag:
# btrfs fi usage -v
Actually, I really like this idea, one of the questions I get asked when I show people BTRFS is the difference between System and Metadata, and it's not always easy to explain to somebody who doesn't have a background in filesystem development. For some reason, people seem to have trouble with the concept that the system tree is an index of the other trees. In general, it doesn't make sense for most non-debugging cases to have it listed separate from the Metadata (they always have the same profile unless you're part way through a conversion, and it really is metadata, just slightly higher level than the usual metadata chunks).
Quoted 56 lines · Duncan
Quoted 52 lines
Qu Wenruo posted on Thu, 19 Nov 2015 08:42:13 +0800 as excerpted:
Quoted 2 lines
Although the metadata output is showing that you still have about 512M available, but the 512M is Global Reserved space, or the unknown one.
Unknown here, as the userspace (btrfs-progs) is evidently too old to show it as global reserve, as it does in newer versions...
Quoted 3 lines
The output is really a little confusing. I'd like the change the output by adding global reserved into metadata used space and make it a sub item for metadata.
Thanks for the clarification. It's most helpful, here. =:^)
I've at times wondered if global reserve folded into one of the other settings. Apparently it comes from the metadata allocation, but while metadata is normally dup (single-device btrfs) or raid1 (multi-device), global reserve is single.
It would have been nice if that sort of substructure was described a bit better when global reserve first made its appearance, at least in the patch descriptions and release announcement, if not then yet in btrfs fi df output, first implementations being what they are. But regardless, now at least it should be clear for list regulars who read this thread anyway, since the above reasonably clarifies things.
As for btrfs fi df, making global reserve a metadata subentry there would be one way to deal with it, preserving the exposure of the additional data provided by that line (here, the fact that global reserve is actually being used, underlining the fact that the filesystem is severely short on space).
Another way of handling it would be to simply add the global reserve into the metadata used figure before printing it, eliminating the separate global reserve line, and changing the upthread posted metadata line from 8.48 GiB of 9 GiB used, to 8.98 of 9 GiB used, which is effectively the case if the 512 MiB of global reserve indeed comes from the metadata allocation. This would more clearly show how full metadata actually is without the added complexity of an additional global reserve line, but would lose the fact that global reserve is actually in use, that the broken out global reserve line exposes.
I'd actually argue in favor of the latter, directly folding global reserve allocation into metadata used, since it'd both be simpler, and more consistent if for instance btrfs fi usage didn't report separate global reserve in the overall stats, but fail to report it in the per- device stats and in btrfs dev usage.
Either way would make much clearer that metadata is actually running out than the current report layout does, since "metadata used" would then either explicitly or implicitly include the global reserve.
Quoted 14 lines · Austin S Hemmelgarn
Quoted 13 lines · Dmitry Katsubo
If I may add:
Information for "System"
System, DUP: total=32.00MiB, used=16.00KiB
is also quite technical, as for end user system = metadata (one can call it "filesystem metadata" perhaps). For simplicity the numbers can be added to "Metadata" thus eliminating that line as well.
For those power users who really want to see the tiny details like "System" and "GlobalReserve" I suggest to implement "-v" flag:
Quoted 7 lines
Quoted 1 line
# btrfs fi usage -vActually, I really like this idea, one of the questions I get asked when I show people BTRFS is the difference between System and Metadata, and it's not always easy to explain to somebody who doesn't have a background in filesystem development. For some reason, people seem to have trouble with the concept that the system tree is an index of the other trees.
Actually, it's not that in the system chunks. :)
System chunks contain the chunk tree, not the tree of tree roots.
They're special (and small) because they're listed explicitly by devid and physical offset at the end of the superblock, and allow the FS to read them first so that it can bootstrap the logical:physical mapping table before it starts reading all the other metadata like the tree of tree roots (which is "normal" metadata).
Hugo.
Quoted 63 lines
In general, it doesn't make
sense for most non-debugging cases to have it listed separate from the Metadata (they always have the same profile unless you're part way through a conversion, and it really is metadata, just slightly higher level than the usual metadata chunks).Quoted 56 lines · Duncan
Quoted 52 lines
Qu Wenruo posted on Thu, 19 Nov 2015 08:42:13 +0800 as excerpted:
Quoted 2 lines
Although the metadata output is showing that you still have about 512M available, but the 512M is Global Reserved space, or the unknown one.
Unknown here, as the userspace (btrfs-progs) is evidently too old to show it as global reserve, as it does in newer versions...
Quoted 3 lines
The output is really a little confusing. I'd like the change the output by adding global reserved into metadata used space and make it a sub item for metadata.
Thanks for the clarification. It's most helpful, here. =:^)
I've at times wondered if global reserve folded into one of the other settings. Apparently it comes from the metadata allocation, but while metadata is normally dup (single-device btrfs) or raid1 (multi-device), global reserve is single.
It would have been nice if that sort of substructure was described a bit better when global reserve first made its appearance, at least in the patch descriptions and release announcement, if not then yet in btrfs fi df output, first implementations being what they are. But regardless, now at least it should be clear for list regulars who read this thread anyway, since the above reasonably clarifies things.
As for btrfs fi df, making global reserve a metadata subentry there would be one way to deal with it, preserving the exposure of the additional data provided by that line (here, the fact that global reserve is actually being used, underlining the fact that the filesystem is severely short on space).
Another way of handling it would be to simply add the global reserve into the metadata used figure before printing it, eliminating the separate global reserve line, and changing the upthread posted metadata line from 8.48 GiB of 9 GiB used, to 8.98 of 9 GiB used, which is effectively the case if the 512 MiB of global reserve indeed comes from the metadata allocation. This would more clearly show how full metadata actually is without the added complexity of an additional global reserve line, but would lose the fact that global reserve is actually in use, that the broken out global reserve line exposes.
I'd actually argue in favor of the latter, directly folding global reserve allocation into metadata used, since it'd both be simpler, and more consistent if for instance btrfs fi usage didn't report separate global reserve in the overall stats, but fail to report it in the per- device stats and in btrfs dev usage.
Either way would make much clearer that metadata is actually running out than the current report layout does, since "metadata used" would then either explicitly or implicitly include the global reserve.
Signature · 4 lines
Hugo Mills | Startle, startle, little twink. How I wonder what hugo@... carfax.org.uk | you think. http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 23 lines · Hugo Mills
Quoted 14 lines · Austin S Hemmelgarn
Quoted 13 lines · Dmitry Katsubo
If I may add:
Information for "System"
System, DUP: total=32.00MiB, used=16.00KiBis also quite technical, as for end user system = metadata (one can call it "filesystem metadata" perhaps). For simplicity the numbers can be added to "Metadata" thus eliminating that line as well.
For those power users who really want to see the tiny details like "System" and "GlobalReserve" I suggest to implement "-v" flag:
Quoted 7 lines
Quoted 1 line
# btrfs fi usage -vActually, I really like this idea, one of the questions I get asked when I show people BTRFS is the difference between System and Metadata, and it's not always easy to explain to somebody who doesn't have a background in filesystem development. For some reason, people seem to have trouble with the concept that the system tree is an index of the other trees.
Quoted 8 lines
Actually, it's not that in the system chunks. :)
System chunks contain the chunk tree, not the tree of tree roots.
They're special (and small) because they're listed explicitly by devid and physical offset at the end of the superblock, and allow the FS to read them first so that it can bootstrap the logical:physical mapping table before it starts reading all the other metadata like the tree of tree roots (which is "normal" metadata).
I guess my understanding was wrong then. Thanks for the explanation.
Many thanks to Duncan for such a verbose clarification. I am thinking about another parallel similar to SimSity, and that is memory management in virtual machines like Java. If heap is full, it does not really mean that there is no free memory. In this case JVM forces garbage collector and if after that procedure no memory was released, then it signals this to application by raising OutOfMemoryError.
I think the similar should happen in btrfs: ENOSPC is returned to application only when there is really no space left. If no chunk can be further allocated, btrfs should check all "deleted" data chunks and and return them to unallocated pool. I would expect this automatic "cleanup" from modern filesystem.
This behaviour not necessarily should be a default one, as one can argue that:
* Such cleanup procedure may freeze the calling process for a considerable time, as btrfs would need to walk all allocated chunks to find candidates for release.
* Filesystem will perhaps anyway run of free space soon, so why not to fallback with error earlier? (for example, one process is intensively writing the log)
It would be nice to have the automatic "cleanup" function controlled by some /sys/fs/btrfs/features variable, which if set to 1, forces btrfs to do its best to allocate the chunk before giving up and returning ENOSPC, sacrificing response time of the process/application.
Quoted 89 lines · Duncan
linux-btrfs.tebulin posted on Thu, 19 Nov 2015 18:56:45 +0000 as excerpted:
Meta-comment:
Apparently that attribution should actually be to Hugo Mills. I've no idea what went wrong, but at least here as received from gmane.org, the from header really does say linux-btrfs.tebulin, so something obviously bugged out somewhere!
Meanwhile discussing btrfs data/metadata allocation, vs. usage of that allocation, Hugo also known here as tebulin, explained...
Quoted 3 lines
If you've ever played SimCity, the allocation process is like zoning -- you say what kind of thing can go on the space, but it's not actually used until something gets built on it.
Very nice analogy. Thanks. =:^)
Tho I'd actually put it in terms of the real thing that sim city simulates, thus eliminating the proprietary comparison. A city zones an area one way or another, restricting what can be built there -- you can't put heavy industry in a residential zone. But the lot is still empty until something's actually built there.
And if the city has all its area zoned residential, and some company wants to build a plant there (generally considered a good thing as it'll provide employment), there's a process that allows rezoning.
In btrfs, there's four types of allocations aka "zones":
1) Unallocated (unzoned)
Can be used for anything but must be allocated/zoned first
2) System
Critical but limited in size and generally only allocated at mkfs or when adding a new device.
3) Data
The actual file storage, generally the largest allocation.
4) Metadata
Information /about/ the files, where they are located (the location and size of individual extents), ownership and permissions, date stamps, checksums, and for very small files (a few KiB), sometimes the file content itself.
4a) Global reserve
A small part of metadata reserved for emergency use only. Btrfs is pretty strict about its use, and will generally error out with ENOSPC if metadata space other than the global reserve is used up, before actually using this global reserve. As a result, any of the global reserve used at all indicates a filesystem in very severe straits, crisis mode.
As it happens, btrfs in practice tends to be a bit liberal about allocating/zoning data chunks, since it's easier to find bigger blocks of space in unallocated space than it is in busy partly used data space. (Think of a big shopping center. It's easier to build it in outlying areas that haven't been built up yet, where many whole blocks worth of space can be allocated/zoned at once, than it is in the city center, where even finding a single whole block vacant, is near impossible.)
Over time, therefore, more and more space tends to be allocated to data, while existing data space, like those blocks near city center, may have most of its files/buildings deleted, but still have a couple still occupied.
Btrfs balancing, then, is comparable to the city functions of condemnation and rezoning to vacant/unallocated, forcing remaining occupants, most commonly data zoned, to move out of the way so the area can be reclaimed for other usage. Then it can be rezoned to data again, or to metadata, whatever needs it.
(FWIW, I played the original sim city, but IIRC it wasn't sophisticated enough to have zoning yet. Of course I've not played anything recent as to my knowledge it's not freedomware, and since I no longer agree to waive rights via EULA, etc, I can no longer legally install and run such things. But the real life sim city simulates, is the antitype behind both that simulation, and the analogy I drew here, based on the one you drew to the simulation.)
Signature · 2 lines
With best regards, Dmitry
Quoted 16 lines · Austin S Hemmelgarn
Quoted 15 lines · Hugo Mills
Quoted 11 lines · Austin S Hemmelgarn
Quoted 4 lines · Dmitry Katsubo
For those power users who really want to see the tiny details like "System" and "GlobalReserve" I suggest to implement "-v" flag:
# btrfs fi usage -vActually, I really like this idea, one of the questions I get asked when I show people BTRFS is the difference between System and Metadata, and it's not always easy to explain to somebody who doesn't have a background in filesystem development. For some reason, people seem to have trouble with the concept that the system tree is an index of the other trees.
Actually, it's not that in the system chunks. :)
Quoted 7 lines
Quoted 6 lines
System chunks contain the chunk tree, not the tree of tree roots.
They're special (and small) because they're listed explicitly by devid and physical offset at the end of the superblock, and allow the FS to read them first so that it can bootstrap the logical:physical mapping table before it starts reading all the other metadata like the tree of tree roots (which is "normal" metadata).I guess my understanding was wrong then. Thanks for the explanation.
The size of "System" is anyway very small in comparison other types of allocations. Adding it to "Metadata" or simply suppressing does not make big difference. If shown, it actually raises "dummy" questions (Is "System" area big enough? Can it run out of free space? How can I add more space to it?).
Signature · 2 lines
With best regards, Dmitry