Please review and comment, dealing with btrfs full issues
I've just written this new page:
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
First, are there problems in it?
Second, are there other FS full issues I should mention in it?
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Quoted 9 lines · Marc MERLIN
I've just written this new page:
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.htmlFirst, are there problems in it?
Second, are there other FS full issues I should mention in it?
Thanks,
Marc
"In the case above, because the filesystem is only 55% full, I can ask balance to rewrite all chunks that are more than 55% full:
legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1"
"-dusage=50" will balance all chunks that are 50% *or less* used, not more. The idea is that full chunks are better left alone while emptyish chunks are bundled together to make new full chunks, leaving big open areas for new chunks. Your process is good however - just the explanation that needs the tweak. :)
In your last example, a full rebalance is not necessary. If you want to clear all unnecessary chunks you can run the balance with -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the data chunks that are 80% and less used, which would by necessity get about ~160GB worth chunks back out of data and available for re-use.
The issue I'm not sure of how to get through is if you can't balance *because* of ENOSPC errors. I'd probably start scouring the mailing list archives if I ever come across that.
Signature · 4 lines
__________ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97
Quoted 15 lines · Brendan Hide
Quoted 10 lines · Marc MERLIN
I've just written this new page:
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.htmlFirst, are there problems in it?
Second, are there other FS full issues I should mention in it?
Thanks,
Marc"In the case above, because the filesystem is only 55% full, I can ask balance to rewrite all chunks that are more than 55% full:
legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1"
Quoted 8 lines
"-dusage=50" will balance all chunks that are 50% *or less* used, not more. The idea is that full chunks are better left alone while emptyish chunks are bundled together to make new full chunks, leaving big open areas for new chunks. Your process is good however - just the explanation that needs the tweak. :)
In your last example, a full rebalance is not necessary. If you want to clear all unnecessary chunks you can run the balance with -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the
data chunks that are 80% and less used, which would by necessity get about ~160GB worth chunks back out of data and available for re-use.The issue I'm not sure of how to get through is if you can't balance *because* of ENOSPC errors. I'd probably start scouring the mailing list archives if I ever come across that.
Forgot this part: Also in your last example, you used "-dusage=0" and it balanced 91 chunks. That means you had 91 empty or very-close-to-empty chunks. ;)
Signature · 4 lines
__________ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97
Quoted 4 lines · Brendan Hide
Quoted 3 lines · Marc MERLIN
I've just written this new page:
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
Quoted 8 lines
Quoted 4 lines
First, are there problems in it?
Second, are there other FS full issues I should mention in it?
The issue I'm not sure of how to get through is if you can't balance *because* of ENOSPC errors. I'd probably start scouring the mailing list archives if I ever come across that.
The standard response on the mailing list for this issue is to temporarily add an additional device to the filesystem (even e.g. a 4GB USB flash drive is often enough) - this will add space to allocate a few new chunks, allowing the balance to proceed. You can remove the extra device after the balance completes.
Signature · 1 line
Calvin Walton <[email protected]>
Quoted 5 lines · On Mon, 5 May 2014 14:36:09 Calvin Walton wrote:
The standard response on the mailing list for this issue is to temporarily add an additional device to the filesystem (even e.g. a 4GB USB flash drive is often enough) - this will add space to allocate a few new chunks, allowing the balance to proceed. You can remove the extra device after the balance completes.
I once had a situation where adding a new device failed due to lack of space. That was after I had tried to balance without adding a new device and to remove snapshots.
http://etbe.coker.com.au/2014/04/26/btrfs-status-april-2014/
Recently kernel 3.14 allowed fixing a metadata space error that seemed to be impossible to solve with 3.13. So it's possible that some of my other problems with a lack of metadata space could have been solved with kernel 3.14 too.
Signature · 2 lines
My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Quoted 6 lines · Brendan Hide
"In the case above, because the filesystem is only 55% full, I can ask balance to rewrite all chunks that are more than 55% full:
legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1""-dusage=50" will balance all chunks that are 50% *or less* used,
Sorry, I actually meant to write 55 there.
Quoted 4 lines
not more. The idea is that full chunks are better left alone while emptyish chunks are bundled together to make new full chunks, leaving big open areas for new chunks. Your process is good however
- just the explanation that needs the tweak. :)
Mmmh, so if I'm 55% full, should I actually use -dusage=45 or 55?
Quoted 6 lines
In your last example, a full rebalance is not necessary. If you want to clear all unnecessary chunks you can run the balance with -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the data chunks that are 80% and less used, which would by necessity get about ~160GB worth chunks back out of data and available for re-use.
So in my case when I hit that case, I had to use dusage=0 to recover. Anything above that just didn't work.
Quoted 3 lines · Brendan Hide
Forgot this part: Also in your last example, you used "-dusage=0" and it balanced 91 chunks. That means you had 91 empty or very-close-to-empty chunks. ;)
Correct. That FS was very mis-balanced.
Quoted 5 lines · Calvin Walton
The standard response on the mailing list for this issue is to temporarily add an additional device to the filesystem (even e.g. a 4GB USB flash drive is often enough) - this will add space to allocate a few new chunks, allowing the balance to proceed. You can remove the extra device after the balance completes.
I just added that tip, thank you.
Quoted 4 lines · Russell Coker
Recently kernel 3.14 allowed fixing a metadata space error that seemed to be impossible to solve with 3.13. So it's possible that some of my other problems with a lack of metadata space could have been solved with kernel 3.14 too.
Good point. I added that tip too.
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Hi, Marc. Inline below. :)
Quoted 6 lines · Marc MERLIN
Quoted 5 lines · Brendan Hide
"In the case above, because the filesystem is only 55% full, I can ask balance to rewrite all chunks that are more than 55% full:
legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1"
Quoted 8 lines
Quoted 1 line
"-dusage=50" will balance all chunks that are 50% *or less* used,
Sorry, I actually meant to write 55 there.
Quoted 4 lines
not more. The idea is that full chunks are better left alone while emptyish chunks are bundled together to make new full chunks, leaving big open areas for new chunks. Your process is good however
- just the explanation that needs the tweak. :)Mmmh, so if I'm 55% full, should I actually use -dusage=45 or 55?
As usual, it depends on what end-result you want. Paranoid rebalancing - always ensuring there are as many free chunks as possible - is totally unnecessary. There may be more good reasons to rebalance - but I'm only aware of two: a) to avoid ENOSPC due to running out of free chunks; and b) to change allocation type.
If you want all chunks either full or empty (except for that last chunk which will be somewhere inbetween), -dusage=55 will get you 99% there.
Quoted 8 lines
Quoted 6 lines
In your last example, a full rebalance is not necessary. If you want to clear all unnecessary chunks you can run the balance with -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the data chunks that are 80% and less used, which would by necessity get about ~160GB worth chunks back out of data and available for re-use.
So in my case when I hit that case, I had to use dusage=0 to recover. Anything above that just didn't work.
I suspect when using more than zero the first chunk it wanted to balance wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it didn't need a destination for the data. That is actually an interesting workaround for that case.
Quoted 23 lines · Brendan Hide
Quoted 3 lines
Forgot this part: Also in your last example, you used "-dusage=0" and it balanced 91 chunks. That means you had 91 empty or very-close-to-empty chunks. ;)
Correct. That FS was very mis-balanced.
Quoted 5 lines · Calvin Walton
The standard response on the mailing list for this issue is to temporarily add an additional device to the filesystem (even e.g. a 4GB USB flash drive is often enough) - this will add space to allocate a few new chunks, allowing the balance to proceed. You can remove the extra device after the balance completes.
I just added that tip, thank you.
Quoted 4 lines · Russell Coker
Recently kernel 3.14 allowed fixing a metadata space error that seemed to be impossible to solve with 3.13. So it's possible that some of my other problems with a lack of metadata space could have been solved with kernel 3.14 too.
Good point. I added that tip too.
Thanks,
Marc
Signature · 4 lines
__________ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97
Quoted 16 lines · Brendan Hide
Hi, Marc. Inline below. :)
Quoted 13 lines · Marc MERLIN
Quoted 6 lines · Brendan Hide
"In the case above, because the filesystem is only 55% full, I can ask balance to rewrite all chunks that are more than 55% full:
legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1""-dusage=50" will balance all chunks that are 50% *or less* used,
Sorry, I actually meant to write 55 there.
Quoted 4 lines
not more. The idea is that full chunks are better left alone while emptyish chunks are bundled together to make new full chunks, leaving big open areas for new chunks. Your process is good however
- just the explanation that needs the tweak. :)
Quoted 7 lines
Quoted 1 line
Mmmh, so if I'm 55% full, should I actually use -dusage=45 or 55?
As usual, it depends on what end-result you want. Paranoid rebalancing - always ensuring there are as many free chunks as possible - is totally unnecessary. There may be more good reasons to rebalance - but I'm only aware of two: a) to avoid ENOSPC due to running out of free chunks; and b) to change allocation type.
c) its original reason: to redistribute the data on the FS, for example in the case of a new device being added or removed.
Quoted 8 lines
If you want all chunks either full or empty (except for that last chunk which will be somewhere inbetween), -dusage=55 will get you 99% there.
Quoted 6 lines
Quoted 6 lines
In your last example, a full rebalance is not necessary. If you want to clear all unnecessary chunks you can run the balance with -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the data chunks that are 80% and less used, which would by necessity get about ~160GB worth chunks back out of data and available for re-use.
Quoted 7 lines
Quoted 2 lines
So in my case when I hit that case, I had to use dusage=0 to recover. Anything above that just didn't work.
I suspect when using more than zero the first chunk it wanted to balance wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it didn't need a destination for the data. That is actually an interesting workaround for that case.
I've actually looked into implementing a "smallest=n" filter that
would taken only the n least-full chunks (by fraction) and balance those. However, it's not entirely trivial to do efficiently with the current filtering code.
Hugo.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Hail and greetings. We are a flat-pack invasion force from ---
Planet Ikea. We come in pieces. Quoted 8 lines · Hugo Mills
Quoted 7 lines
Quoted 2 lines
So in my case when I hit that case, I had to use dusage=0 to recover. Anything above that just didn't work.
I suspect when using more than zero the first chunk it wanted to balance wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it didn't need a destination for the data. That is actually an interesting workaround for that case.
Quoted 4 lines
I've actually looked into implementing a "smallest=n" filter that
would taken only the n least-full chunks (by fraction) and balance those. However, it's not entirely trivial to do efficiently with the current filtering code.
I've prototyped something similar, to limit the number of balanced chunks by a number. To achieve "n least-full chunks" would be an iterative process of increasing the usage filter and limiting the number of chunks until the desired N is reached.
N=n
F=0
while (N > 0) {
balance -dusage=F,limit=N
N -= <number of balanced chunks>
F++
}The patch is in branch dev/balance-limit in my git repos.
We can then implement the n-least-full as a synthetic filter from userspace.
Smallest-n balance filter (was Re: Please review and comment, dealing with btrfs full issues)
Quoted 27 lines · David Sterba
Quoted 8 lines · Hugo Mills
Quoted 7 lines
Quoted 2 lines
So in my case when I hit that case, I had to use dusage=0 to recover. Anything above that just didn't work.
I suspect when using more than zero the first chunk it wanted to balance wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it didn't need a destination for the data. That is actually an interesting workaround for that case.
Quoted 4 lines
I've actually looked into implementing a "smallest=n" filter that
would taken only the n least-full chunks (by fraction) and balance those. However, it's not entirely trivial to do efficiently with the current filtering code.I've prototyped something similar, to limit the number of balanced chunks by a number. To achieve "n least-full chunks" would be an iterative process of increasing the usage filter and limiting the number of chunks until the desired N is reached.
N=n
F=0while (N > 0) { balance -dusage=F,limit=N N -= <number of balanced chunks> F++ }
Quoted 4 lines
The patch is in branch dev/balance-limit in my git repos.
We can then implement the n-least-full as a synthetic filter from userspace.
This is inefficient, because we've got an O(m) pass through all the
chunks for every call. If we reduce the number of calls by increasing the increment of F (F+=3, say), then we risk overbalancing, or missing out on smaller chunks we could have balanced earlier. From a practical point of view, it may make little difference, but the computer scientist in me is going "ewwwww".
The other method, for small n only, would be to construct the list
first, an O(m log n) operation for a filesystem of size m, requiring O(n) storage, and then iterate over just those chunks. The problem with that is the storage requirements, and keeping track of the state of the list for restart purposes. [actually, there's probably an O(m) algorithm to get the n smallest items, but those are a bit complicated]
Hugo.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- A diverse working environment: Di longer you vork here, di ---
verse it gets. Re: Smallest-n balance filter (was Re: Please review and comment, dealing with btrfs full issues)
Quoted 13 lines · Hugo Mills
Quoted 8 lines
N=n
F=0while (N > 0) { balance -dusage=F,limit=N N -= <number of balanced chunks> F++ }Quoted 4 lines
The patch is in branch dev/balance-limit in my git repos.
We can then implement the n-least-full as a synthetic filter from userspace.
Quoted 6 lines
This is inefficient, because we've got an O(m) pass through all the
chunks for every call. If we reduce the number of calls by increasing the increment of F (F+=3, say), then we risk overbalancing, or missing out on smaller chunks we could have balanced earlier. From a practical point of view, it may make little difference, but the computer scientist in me is going "ewwwww".
I'm trying to find the practical way, no doubts about the inefficiencies. The +1 increment was meant to outline the idea, I'm usually using the sequence 0, 1, 5, 10, [etc +10].
I think we can afford some inaccuracy, I as a user would not mind if there's some overbalancing (within a sane margin).
Quoted 3 lines
The other method, for small n only, would be to construct the list
first, an O(m log n) operation for a filesystem of size m, requiring O(n) storage, and then iterate over just those chunks.
The size of filesystem matters, but the scanning phase of balance uses in-memory structures and this should not be that bad for terabyte-sized filesystems (ie. number of blockgoups will be "some thousands").
Possibly we can stop looking for new chunks in the first phase of balance when there are already N candidate chunks found, and process them.
Quoted 4 lines
The problem with that is the storage requirements, and keeping track of the state of the list for restart purposes. [actually, there's probably an O(m) algorithm to get the n smallest items, but those are a bit complicated]
If the filesystem is under load, the chunks' usage may increase or decrease in time and as we know, balance takes time, so the chunk-todo-list may look different when next one is about to be processed.
But yeah, this could be a cheaper check to skip a given chunk if it's out of the filter criteria than going through the whole list again.
Brendan Hide posted on Tue, 06 May 2014 18:30:31 +0200 as excerpted:
Quoted 7 lines
Quoted 2 lines
So in my case when I hit that case, I had to use dusage=0 to recover. Anything above that just didn't work.
I suspect when using more than zero the first chunk it wanted to balance wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it didn't need a destination for the data. That is actually an interesting workaround for that case.
I've actually used -Xusage=0 (where X=m or d, obviously) for exactly that. If every last bit of filesystem is allocated so another chunk simply cannot be written in ordered to rewrite partially used chunks into, BUT the spread between allocated and actually used is quite high, there's a reasonably good chance that at least one of those allocated chunks is entirely empty, and -Xusage=0 allows returning it to the unallocated pool without actually requiring a new chunk allocation to do so.
With luck, that will free at least one zero-usage chunk (two for metadata dup, but it would both allocate and return to unallocated in pairs as so it balances out), allowing the user to rerun balance, this time with a higher -Xusage=.
The other known valid use-case for -Xusage=0 is when freeing the extraneous zero-usage single-mode chunks first created by mkfs.btrfs as part of the mkfs process, so they don't clutter up the btrfs filesystem df output. =:^)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Brendan Hide posted on Mon, 05 May 2014 19:07:29 +0200 as excerpted:
Quoted 5 lines
In your last example, a full rebalance is not necessary. If you want to clear all unnecessary chunks you can run the balance with -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the data chunks that are 80% and less used, which would by necessity get about ~160GB worth chunks back out of data and available for re-use.
I don't believe your reasoning is entirely valid. In the pathologically extreme case, all data chunks are 80% plus one byte used, save for one that's empty, and a -dusage=80 balance will return just that one empty chunk (but as a consequence return real fast as there's nothing else to do), just the same as -dusage=0, but -dusage=81 would return the 20% expected, but would take as long as -dusage=100 (or simply -d).
OTOH, if all chunks but one are 100% full, with the remaining chunks empty, a balance with -dusage=0 will have (within one chunk, depending on how full it is) the same effect as -dusage=80, with both completing at the same fast speed and returning more chunks than expected. But simple -d (or -dusage=100) would take much MUCH longer, but return nothing (well, within one chunk) more than the -dusage=0 would.
Actual results, therefore, depend on file AND chunk fragmentation and usage. Using the autodefrag mount option should increase the likelihood of better results faster, at lower -Xusage= numbers, but the only way to be sure is to do a run with numbers you think look reasonable given the situation you are in and the time you're willing to wait, and then rerun it again if the results weren't good enough.
Your algorithm is certainly a good first approximation, but it's just that, an approximation. People who want the balance over as soon as possible but are willing to retry if necessary, should probably try a lower usage= number, while those who are leaving on a trip in the morning and want it to run overnight, with little chance of redoing it if the results weren't good enough, should probably pick a higher usage= number, even perhaps usage=95 or the like, to recover as much space as possible, without wasting time rebalancing the chunks that really ARE 100% full already.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman