Removing file = quota exceded
so ive come accross the issue of being unable to remove a file when a subvolume quota is reached. This can be resolved by truncating the file first, or removing the quota temporarily.
However, it should be reasonable that you should alwasy be able to remove a file, regardless of quota limitations yes?
Upon delving into the code, I found the comment that an unlink may not always free space. This seems reasonable on a COW filesystem, however, it should not preclude a removal IMO (please correct me if i missed something)
Personally I'm looking to try and fix this issue to allow a removal of a file even when the subvol quota has been reached. I'm hoping one of the current developers may be able to assist me in where to focus my efforts, as I am still unable to follow exactly where a remove operation would check the quota limitations.
Any help is appreciated.
-Kevin Brandstatter
Quoted 7 lines · Kevin Brandstatter
so ive come accross the issue of being unable to remove a file when a subvolume quota is reached. This can be resolved by truncating the file first, or removing the quota temporarily.
However, it should be reasonable that you should alwasy be able to remove a file, regardless of quota limitations yes?
Upon delving into the code, I found the comment that an unlink may not always free space. This seems reasonable on a COW filesystem, however,
Quoted 9 lines
it should not preclude a removal IMO (please correct me if i missed something)
Personally I'm looking to try and fix this issue to allow a removal of a file even when the subvol quota has been reached. I'm hoping one of the current developers may be able to assist me in where to focus my efforts, as I am still unable to follow exactly where a remove operation would check the quota limitations.
Any help is appreciated.
For quota I think we should always allow unlink, it's not really the users fault that we sometimes won't actually remove space with unlink. The normal ENOSPC stuff still needs to take this into account of course but for quota I think it's ok to just go over quota. I'll look into this later this week. Thanks,
Josef
One thing i note is that I can unlink from a full filesystem. I tested it by writing a file until the device ran out of space, and then rm it,
the same method that i used to cause the disk quota error, and it was able to remove without
issue.
-Kevin
Quoted 27 lines · Josef Bacik
Quoted 7 lines · Kevin Brandstatter
so ive come accross the issue of being unable to remove a file when a subvolume quota is reached. This can be resolved by truncating the file first, or removing the quota temporarily.
However, it should be reasonable that you should alwasy be able to remove a file, regardless of quota limitations yes?
Upon delving into the code, I found the comment that an unlink may not always free space. This seems reasonable on a COW filesystem, however,Quoted 9 lines
it should not preclude a removal IMO (please correct me if i missed something)
Personally I'm looking to try and fix this issue to allow a removal of a file even when the subvol quota has been reached. I'm hoping one of the current developers may be able to assist me in where to focus my efforts, as I am still unable to follow exactly where a remove operation would check the quota limitations.
Any help is appreciated.For quota I think we should always allow unlink, it's not really the users fault
that we sometimes won't actually remove space with unlink. The normal ENOSPC
stuff still needs to take this into account of course but for quota I think it's
ok to just go over quota. I'll look into this later this week. Thanks,Josef
Kevin Brandstatter posted on Sun, 22 Jun 2014 12:56:30 -0500 as excerpted:
Quoted 4 lines
One thing i note is that I can unlink from a full filesystem. I tested it by writing a file until the device ran out of space, and then rm it, the same method that i used to cause the disk quota error, and it was able to remove without issue.
It's worth noting that due to the btrfs separation between data and metadata and the fact that btrfs space allocation happens in two steps but it can only automatically free one of them (with a rebalance normally used to deal with the other), there's three different kinds of "full filesystem", (1) "all space chunk allocated", which isn't yet /entirely/ full but means a significant loss of flexibility in filling up the rest,
(2) "all space chunk-allocated and metadata space ran out of room first but there's still room in the data chunks", which is what happens most of the time in normal usage, and (3) "all space chunk-allocated and data space ran out first but there's still room in the metadata chunks", which can produce decidedly non-intuitive behavior for people used to standard filesystem behavior.
Data/metadata chunk allocation is only one-way. Once a chunk is allocated to one or the other, the system cannot (yet) reallocate chunks of one type to the other without a rebalance, so once all previously unallocated space is allocated to either data or metadata chunks, it's only a matter of time until one or the other runs out.
In normal usage with a significant amount of file deletion, the spread between data chunk allocation and actual usage tends to get rather large, because file deletion normally frees much more data space than it does metadata. As such, the most common out-of-space condition is all unallocated space gone, with most of the still actually unused space allocated to data and thus not available to be used for metadata, such that metadata space runs out first.
When metadata space runs out, normal df will likely still report a decent amount of space remaining, but btrfs filesystem df combined with btrfs filesystem show will reveal that it's all locked up in data chunks -- a big spread, often multiple gigabytes between data used and total (which given the 1 GiB data chunk size means multiple data chunks could be freed), a much smaller spread between metadata used and total (the system reserves some metadata space, typically 200-ish MiB, so it should never show as entirely gone, even when it's triggering ENOSPC).
But due to COW, even file deletion requires available metadata space in ordered to create the new/modified copy of the (normally 4-16 KiB depending on mkfs.btrfs age and parameters supplied) metadata block, and if there's no metadata space left and no more unallocated space to allocate, ENOSPC even on file deletion!
OTOH, in use-cases where there is little file deletion, the spread between data chunk total and data chunk used tends to be much smaller, and it can happen that there's still free metadata chunk space when the last free data space is used and another data chunk needs allocated, but there's no more unallocated space to allocate. Of course btrfs filesystem df (to see how allocated space is used) in combination with btrfs filesystem show (to see whether all space is allocated) should tell the story, in this case, reporting all or nearly all data space used but a larger gap (> 200 MiB) between metadata total and used.
This triggers a much more interesting and non-intuitive failure mode. In particular, because there's still metadata space available, attempts to create a new file will succeed, but actually putting significant content in that file will fail, often resulting in the creation of zero-length files that won't accept data! However, because btrfs stores very small files (generally something under 16 MiB, the precise size depends on filesystem parameters) entirely within metadata without actually allocating a data extent for them, attempts to copy small enough files will generally succeed as well -- as long as they're small enough to fit in metadata only and not require a data allocation.
Now I don't deal with quotas here and thus haven't looked into how quotas account for metadata in particular, but it's worth noting that your "write a file until there's no more space" test could well have triggered the latter, all space chunk-allocated and data filled up first, condition. If that's the case, deleting a file wouldn't be a problem because there's metadata space still available to record the deletion. As I said above, another characteristic would be that attempts to create new files and fill them with data (> 16 MiB at a time) would result in zero-length files, as there's metadata space available to create them, but no data space available to fill them.
So your test may have been testing an *ENTIRELY* different failure condition!
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
PATCH 1/1 handle start_unlink_transaction the same for an exceded quota , limit as an out of space error.
--- fs/btrfs/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/btrfs/inode.c +0 -0
@@ -3751,10 +3751,10 @@ static struct btrfs_trans_handle
*__unlink_start_trans(struct inode *dir)
* 1 for the inode
*/
trans = btrfs_start_transaction(root, 5);
- if (!IS_ERR(trans) || PTR_ERR(trans) != -ENOSPC)
+ if (!IS_ERR(trans) || (PTR_ERR(trans) != -ENOSPC && PTR_ERR(trans) !=
-EDQUOT))
return trans;
.
- if (PTR_ERR(trans) == -ENOSPC) {
+ if (PTR_ERR(trans) == -ENOSPC || PTR_ERR(trans) == -EDQUOT) {
u64 num_bytes = btrfs_calc_trans_metadata_size(root, 5);
.
trans = btrfs_start_transaction(root, 0);--.
2.0.0
Quoted 83 lines · Duncan
Kevin Brandstatter posted on Sun, 22 Jun 2014 12:56:30 -0500 as excerpted:
Quoted 4 lines
One thing i note is that I can unlink from a full filesystem. I tested it by writing a file until the device ran out of space, and then rm it, the same method that i used to cause the disk quota error, and it was able to remove without issue.
It's worth noting that due to the btrfs separation between data and metadata and the fact that btrfs space allocation happens in two steps but it can only automatically free one of them (with a rebalance normally used to deal with the other), there's three different kinds of "full filesystem", (1) "all space chunk allocated", which isn't yet /entirely/ full but means a significant loss of flexibility in filling up the rest,
(2) "all space chunk-allocated and metadata space ran out of room first but there's still room in the data chunks", which is what happens most of the time in normal usage, and (3) "all space chunk-allocated and data space ran out first but there's still room in the metadata chunks", which can produce decidedly non-intuitive behavior for people used to standard filesystem behavior.Data/metadata chunk allocation is only one-way. Once a chunk is allocated to one or the other, the system cannot (yet) reallocate chunks of one type to the other without a rebalance, so once all previously unallocated space is allocated to either data or metadata chunks, it's only a matter of time until one or the other runs out.
In normal usage with a significant amount of file deletion, the spread between data chunk allocation and actual usage tends to get rather large, because file deletion normally frees much more data space than it does metadata. As such, the most common out-of-space condition is all unallocated space gone, with most of the still actually unused space allocated to data and thus not available to be used for metadata, such that metadata space runs out first.
When metadata space runs out, normal df will likely still report a decent amount of space remaining, but btrfs filesystem df combined with btrfs filesystem show will reveal that it's all locked up in data chunks -- a big spread, often multiple gigabytes between data used and total (which given the 1 GiB data chunk size means multiple data chunks could be freed), a much smaller spread between metadata used and total (the system reserves some metadata space, typically 200-ish MiB, so it should never show as entirely gone, even when it's triggering ENOSPC).
But due to COW, even file deletion requires available metadata space in ordered to create the new/modified copy of the (normally 4-16 KiB depending on mkfs.btrfs age and parameters supplied) metadata block, and if there's no metadata space left and no more unallocated space to allocate, ENOSPC even on file deletion!
OTOH, in use-cases where there is little file deletion, the spread between data chunk total and data chunk used tends to be much smaller, and it can happen that there's still free metadata chunk space when the last free data space is used and another data chunk needs allocated, but there's no more unallocated space to allocate. Of course btrfs filesystem df (to see how allocated space is used) in combination with btrfs filesystem show (to see whether all space is allocated) should tell the story, in this case, reporting all or nearly all data space used but a larger gap (> 200 MiB) between metadata total and used.
This triggers a much more interesting and non-intuitive failure mode. In particular, because there's still metadata space available, attempts to create a new file will succeed, but actually putting significant content in that file will fail, often resulting in the creation of zero-length files that won't accept data! However, because btrfs stores very small files (generally something under 16 MiB, the precise size depends on filesystem parameters) entirely within metadata without actually allocating a data extent for them, attempts to copy small enough files will generally succeed as well -- as long as they're small enough to fit in metadata only and not require a data allocation.
Now I don't deal with quotas here and thus haven't looked into how quotas account for metadata in particular, but it's worth noting that your "write a file until there's no more space" test could well have triggered the latter, all space chunk-allocated and data filled up first, condition. If that's the case, deleting a file wouldn't be a problem because there's metadata space still available to record the deletion. As I said above, another characteristic would be that attempts to create new files and fill them with data (> 16 MiB at a time) would result in zero-length files, as there's metadata space available to create them, but no data space available to fill them.
So your test may have been testing an *ENTIRELY* different failure condition!
Quoted 26 lines · Josef Bacik
Quoted 7 lines · Kevin Brandstatter
so ive come accross the issue of being unable to remove a file when a subvolume quota is reached. This can be resolved by truncating the file first, or removing the quota temporarily.
However, it should be reasonable that you should alwasy be able to remove a file, regardless of quota limitations yes?
Upon delving into the code, I found the comment that an unlink may not always free space. This seems reasonable on a COW filesystem, however,Quoted 9 lines
it should not preclude a removal IMO (please correct me if i missed something)
Personally I'm looking to try and fix this issue to allow a removal of a file even when the subvol quota has been reached. I'm hoping one of the current developers may be able to assist me in where to focus my efforts, as I am still unable to follow exactly where a remove operation would check the quota limitations.
Any help is appreciated.For quota I think we should always allow unlink, it's not really the users fault
that we sometimes won't actually remove space with unlink. The normal ENOSPC
stuff still needs to take this into account of course but for quota I think it's
ok to just go over quota. I'll look into this later this week. Thanks,
Quoted 1 line
Josef
I was able to trace the error to the __unlink_start_trans, and noticed the handling for the
ENOSPC error code. I modified this function to do the same in the case of the EDQUOT error
(the one that gets returned when the quota is exceded). When I did this i was able to remove a file even when the quota was reached. Submitted the patch to the mailing list for feedback.
-Kevin
Duncan posted on Mon, 23 Jun 2014 01:53:45 +0000 as excerpted:
Quoted 3 lines
However, because btrfs stores very small files (generally something under 16 MiB, the precise size depends on filesystem parameters) entirely within metadata
Hmm. Should be under 16 KiB I believe, not 16 MiB.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
I do recall that the issue was more specifically happening with larger files.
In another scenario i had two files, one small or even empty, and one large that filled the
subvolume quota. If i recall correctly, I was able to remove the smaller file without exceeding
the quota limit, and then remove the larger file. However, removing the larger file resulted
in the quota exceeded error. The behavior you describe with small files makes this case make
more sense.
Likewise, I found that handling an unlink with a full subvolume qgruop (EDQUOT) the same way as a full file system (ENOSPC) resolved the issue.
-Kevin
Quoted 7 lines · Duncan
Duncan posted on Mon, 23 Jun 2014 01:53:45 +0000 as excerpted:
Quoted 3 lines
However, because btrfs stores very small files (generally something under 16 MiB, the precise size depends on filesystem parameters) entirely within metadata
Hmm. Should be under 16 KiB I believe, not 16 MiB.