btrfs: qgroup: Fix the long existing regression of btrfs/153
cover btrfs: qgroup: Fix the long existing regression of btrfs/153
Since commit c6887cd11149 ("Btrfs: don't do nocow check unless we have to"), btrfs/153 always fails with early EDQUOT.
This is caused by the fact that:
- We always reserved data space for even NODATACOW buffered write This is mostly to improve performance, and not pratical to revert.
- Btrfs qgroup data and meta reserved space share the same limit So it's not ensured to return EDQUOT just for that data reservation, metadata reservation can also get EDQUOT, means we can't go the same solution as that commit.
This patchset will solve it by doing extra qgroup space flushing when EDQUOT is hit.
This is a little like what we do in ticketing space reservation system, but since there are very limited ways for qgroup to reclaim space, currently it's still handled in qgroup realm, not reusing the ticketing system yet.
By this, this patch could solve the btrfs/153 problem, while still keep btrfs qgroup space usage under the limit.
The only cost is, when we're near qgroup limit, we will cause more dirty inodes flush and transaction commit, much like what we do when the metadata space is near exhausted.
So the cost should be still acceptable.
Qu Wenruo (3):
btrfs: Introduce extent_changeset_revert() for qgroup
btrfs: qgroup: Try to flush qgroup space when we get -EDQUOT Revert "btrfs: qgroup: Commit transaction in advance to reduce early
EDQUOT"
fs/btrfs/ctree.h | 6 +- fs/btrfs/disk-io.c | 2 +- fs/btrfs/extent_io.c | 43 +++++- fs/btrfs/extent_io.h | 24 +++- fs/btrfs/qgroup.c | 310 ++++++++++++++++++++++++++++------------- fs/btrfs/transaction.c | 1 - fs/btrfs/transaction.h | 14 -- 7 files changed, 273 insertions(+), 127 deletions(-)
Signature · 1 line
2.27.0
PATCH 1/3 btrfs: Introduce extent_changeset_revert() for qgroup
[PROBLEM]
Before this patch, when btrfs_qgroup_reserve_data() fails, we free all reserved space of the changeset.
This means the following call is not possible:
ret = btrfs_qgroup_reserve_data();
if (ret == -EDQUOT) {
/* Do something to free some qgroup space */
ret = btrfs_qgroup_reserve_data();
}As if the first btrfs_qgroup_reserve_data() fails, it will free all reserved qgroup space, so the next btrfs_qgroup_reserve_data() will always success, and can go beyond qgroup limit.
[CAUSE]
This is mostly due to the fact that we don't have a good way to revert changeset modification accurately.
Currently the changeset infrastructure is implemented using ulist, which can only store two u64 values, used as start and length for each changed extent range.
So we can't record which changed extent is modified in last modification, thus unable to revert to previous status.
[FIX]
This patch will re-implement using pure rbtree, adding a new member, changed_extent::seq, so we can remove changed extents which is modified in previous modification.
This allows us to implement qgroup_revert(), which allow btrfs to revert its modification to the io_tree.
--- fs/btrfs/extent_io.c | 43 +++++++++++- fs/btrfs/extent_io.h | 24 ++++++- fs/btrfs/qgroup.c | 155 +++++++++++++++++++++++++++++-------------- 3 files changed, 168 insertions(+), 54 deletions(-)
fs/btrfs/extent_io.c +40 -3
@@ -142,6 +142,40 @@ struct extent_page_data { unsigned int sync_io:1; }; +static int insert_changed_extent (struct extent_changeset *changeset, +u64 start,u64 len) +{ +struct changed_extent *entry; +struct changed_extent *cur; +struct rb_node **p = &changeset->root.rb_node; +struct rb_node *parent =NULL ; + + entry =kzalloc (sizeof (*entry),GFP_ATOMIC ); +if (!entry) +return -ENOMEM ; + + entry->start = start; + entry->len = len; + entry->seq = changeset->seq; + +while (*p) { + parent = *p; + cur = rb_entry(parent, struct changed_extent, node); +if (cur->start < start) { + p = &(*p)->rb_right; + }else if (cur->start > start) { + p = &(*p)->rb_left; + }else { +kfree (entry); +return -EEXIST ; + } + } +rb_link_node (&entry->node, parent, p); +rb_insert_color (&entry->node, &changeset->root); + changeset->bytes_changed += len; +return 0 ; +} +static int add_extent_changeset (struct extent_state *state,unsigned bits,struct extent_changeset *changeset,int set)
@@ -154,9 +188,8 @@ static int add_extent_changeset(struct extent_state *state, unsigned bits,return 0 ;if (!set && (state->state & bits) ==0 )return 0 ; - changeset->bytes_changed += state->end - state->start +1 ; - ret =ulist_add (&changeset->range_changed, state->start, state->end, -GFP_ATOMIC ); + ret =insert_changed_extent (changeset, state->start, + state->end - state->start +1 );return ret; }
@@ -718,6 +751,8 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,if (bits & (EXTENT_LOCKED |EXTENT_BOUNDARY )) clear =1 ; +if (changeset) + changeset->seq++; again:if (!prealloc &&gfpflags_allow_blocking (mask)) {/*
@@ -980,6 +1015,8 @@ __set_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,btrfs_debug_check_extent_io_range (tree, start, end);trace_btrfs_set_extent_bit (tree, start, end - start +1 , bits); +if (changeset) + changeset->seq++; again:if (!prealloc &&gfpflags_allow_blocking (mask)) {/*
fs/btrfs/extent_io.h +21 -3
@@ -131,18 +131,29 @@ struct extent_buffer {/* * Structure to record how many bytes and which ranges are set/cleared */ +struct changed_extent { +struct rb_node node; +u64 start; +u64 len; +u64 seq;/* At which seqence number this extent is added */ +}; +struct extent_changeset { +/* Sequence number for current set/clear of this changeset */ +u64 seq; +/* How many bytes are set/cleared in this operation */ unsigned int bytes_changed;/* Changed ranges */ -struct ulist range_changed; +struct rb_root root; };static inline void extent_changeset_init (struct extent_changeset *changeset) { changeset->bytes_changed =0 ; -ulist_init (&changeset->range_changed); + changeset->seq =0 ; + changeset->root =RB_ROOT ; }static inline struct extent_changeset *extent_changeset_alloc (void )
@@ -159,10 +170,17 @@ static inline struct extent_changeset *extent_changeset_alloc(void)static inline void extent_changeset_release (struct extent_changeset *changeset) { +struct changed_extent *entry; +struct changed_extent *next; +if (!changeset)return ; changeset->bytes_changed =0 ; -ulist_release (&changeset->range_changed); + changeset->seq =0 ; +rbtree_postorder_for_each_entry_safe (entry, next, &changeset->root, + node) +kfree (entry); + changeset->root =RB_ROOT ; }static inline void extent_changeset_free (struct extent_changeset *changeset)
fs/btrfs/qgroup.c +107 -48
223 lines not on this page · raw message
Signature · 1 line
2.27.0
Re: [PATCH 1/3] btrfs: Introduce extent_changeset_revert() for qgroup
Quoted 26 lines · Qu Wenruo
[PROBLEM]
Before this patch, whenbtrfs_qgroup_reserve_data()fails, we free all reserved space of the changeset.This means the following call is not possible:
ret = btrfs_qgroup_reserve_data(); if (ret == -EDQUOT) { /* Do something to free some qgroup space */ ret = btrfs_qgroup_reserve_data(); }As if the first
btrfs_qgroup_reserve_data()fails, it will free all reserved qgroup space, so the nextbtrfs_qgroup_reserve_data()will always success, and can go beyond qgroup limit.[CAUSE]
This is mostly due to the fact that we don't have a good way to revert changeset modification accurately.Currently the changeset infrastructure is implemented using ulist, which can only store two u64 values, used as start and length for each changed extent range.
So we can't record which changed extent is modified in last modification, thus unable to revert to previous status.
Quoted 8 lines
[FIX]
This patch will re-implement using pure rbtree, adding a new member,changed_extent::seq, so we can remove changed extents which is modified in previous modification.This allows us to implement
qgroup_revert(), which allow btrfs to revert its modification to theio_tree.
I'm having a hard time groking what's going on here. These changesets are limited to a [start, end] range correct? Why can we have multiple changesets for the same range? Are we reserving doubly for modifications in the same range? Because it seems here if you find your changeset at all we'll clear the io_tree, but if you can have multiple changesets for the same range then....why even bother? Thanks,
Josef
Re: [PATCH 1/3] btrfs: Introduce extent_changeset_revert() for qgroup
Quoted 32 lines · Josef Bacik
Quoted 31 lines · Qu Wenruo
[PROBLEM]
Before this patch, whenbtrfs_qgroup_reserve_data()fails, we free all reserved space of the changeset.This means the following call is not possible:
ret =btrfs_qgroup_reserve_data();
if (ret ==-EDQUOT) {
/* Do something to free some qgroup space */ ret =btrfs_qgroup_reserve_data();
}As if the first
btrfs_qgroup_reserve_data()fails, it will free all reserved qgroup space, so the nextbtrfs_qgroup_reserve_data()will always success, and can go beyond qgroup limit.[CAUSE]
This is mostly due to the fact that we don't have a good way to revert changeset modification accurately.Currently the changeset infrastructure is implemented using ulist, which can only store two u64 values, used as start and length for each changed extent range.
So we can't record which changed extent is modified in last modification, thus unable to revert to previous status.
[FIX]
This patch will re-implement using pure rbtree, adding a new member,changed_extent::seq, so we can remove changed extents which is modified in previous modification.
Quoted 6 lines
Quoted 3 lines
This allows us to implement
qgroup_revert(), which allow btrfs to revert its modification to theio_tree.I'm having a hard time groking what's going on here. These changesets are limited to a [start, end] range correct?
Yes, but we may be only responsible for part of the changeset.
One example is we want to falloc range [0, 16K) And [0, 4K), [8K, 12K) has already one existing file extent.
Then we succeeded in allocating space for [4K, 8K), but failed to allocating space for [8K, 12K).
In that case, if we just return EDQUOT and clear the range for [4K, 8k) and [8K, 12K), everything is completely fine.
But if we want to retry, then we should only clear the range for [8K, 12K), but not to clear [4K, 8K).
That's what we need for the next patch, just revert what we did in previous set_extent_bit(), but not touching previously set bits.
Thanks,
Qu
Quoted 7 lines
Why can we have multiple
changesets for the same range? Are we reserving doubly for modifications in the same range? Because it seems here if you find your changeset at all we'll clear theio_tree, but if you can have multiple changesets for the same range then....why even bother? Thanks,Josef
Re: [PATCH 1/3] btrfs: Introduce extent_changeset_revert() for qgroup
Quoted 50 lines · Qu Wenruo
Quoted 32 lines · Josef Bacik
Quoted 31 lines · Qu Wenruo
[PROBLEM]
Before this patch, whenbtrfs_qgroup_reserve_data()fails, we free all reserved space of the changeset.This means the following call is not possible:
ret =btrfs_qgroup_reserve_data();
if (ret ==-EDQUOT) {
/* Do something to free some qgroup space */ ret =btrfs_qgroup_reserve_data();
}As if the first
btrfs_qgroup_reserve_data()fails, it will free all reserved qgroup space, so the nextbtrfs_qgroup_reserve_data()will always success, and can go beyond qgroup limit.[CAUSE]
This is mostly due to the fact that we don't have a good way to revert changeset modification accurately.Currently the changeset infrastructure is implemented using ulist, which can only store two u64 values, used as start and length for each changed extent range.
So we can't record which changed extent is modified in last modification, thus unable to revert to previous status.
[FIX]
This patch will re-implement using pure rbtree, adding a new member,changed_extent::seq, so we can remove changed extents which is modified in previous modification.Quoted 6 lines
Quoted 3 lines
This allows us to implement
qgroup_revert(), which allow btrfs to revert its modification to theio_tree.I'm having a hard time groking what's going on here. These changesets are limited to a [start, end] range correct?
Yes, but we may be only responsible for part of the changeset.
One example is we want to falloc range [0, 16K) And [0, 4K), [8K, 12K) has already one existing file extent.
Then we succeeded in allocating space for [4K, 8K), but failed to allocating space for [8K, 12K).
Quoted 9 lines
In that case, if we just return EDQUOT and clear the range for [4K, 8k) and [8K, 12K), everything is completely fine.
But if we want to retry, then we should only clear the range for [8K, 12K), but not to clear [4K, 8K).
That's what we need for the next patch, just revert what we did in previous
set_extent_bit(), but not touching previously set bits.
Ok so how do we get to this case then? The changeset already has the range we're responsible for correct? Why do we need the sequence number? Like we should have a changeset for [4k, 8k) and one for [8k, 12k) right? Or are we handed back the whole thing? If we fail _just_ for the [8k, 12k) part, do we know that? Or do we just know our whole [0k, 16k) thing failed, and so we need all this convoluted tracking to be able to retry for the range that we fucked up, and this is what you are facilitating? Thanks,
Josef
Re: [PATCH 1/3] btrfs: Introduce extent_changeset_revert() for qgroup
Quoted 46 lines · Josef Bacik
Quoted 45 lines · Qu Wenruo
Quoted 41 lines · Josef Bacik
Quoted 28 lines · Qu Wenruo
[PROBLEM]
Before this patch, whenbtrfs_qgroup_reserve_data()fails, we free all reserved space of the changeset.This means the following call is not possible:
ret =btrfs_qgroup_reserve_data();
if (ret ==-EDQUOT) {
/* Do something to free some qgroup space */ ret =btrfs_qgroup_reserve_data();
}As if the first
btrfs_qgroup_reserve_data()fails, it will free all reserved qgroup space, so the nextbtrfs_qgroup_reserve_data()will always success, and can go beyond qgroup limit.[CAUSE]
This is mostly due to the fact that we don't have a good way to revert changeset modification accurately.Currently the changeset infrastructure is implemented using ulist, which
can only store two u64 values, used as start and length for each changed
extent range.So we can't record which changed extent is modified in last modification, thus unable to revert to previous status.
Quoted 9 lines
[FIX]
This patch will re-implement using pure rbtree, adding a new member,changed_extent::seq, so we can remove changed extents which is modified in previous modification.This allows us to implement
qgroup_revert(), which allow btrfs to revert
its modification to theio_tree.I'm having a hard time groking what's going on here. These changesets are limited to a [start, end] range correct?
Quoted 7 lines
Quoted 7 lines
Yes, but we may be only responsible for part of the changeset.
One example is we want to falloc range [0, 16K) And [0, 4K), [8K, 12K) has already one existing file extent.
Then we succeeded in allocating space for [4K, 8K), but failed to allocating space for [8K, 12K).
Sorry, this should be [12K, 16K).
Quoted 11 lines
Quoted 10 lines
In that case, if we just return EDQUOT and clear the range for [4K, 8k) and [8K, 12K), everything is completely fine.
But if we want to retry, then we should only clear the range for [8K, 12K), but not to clear [4K, 8K).
That's what we need for the next patch, just revert what we did in previous
set_extent_bit(), but not touching previously set bits.
Quoted 9 lines
Ok so how do we get to this case then? The changeset already has the range we're responsible for correct?
Why do we need the sequence
number? Like we should have a changeset for [4k, 8k) and one for [8k, 12k) right? Or are we handed back the whole thing? If we fail _just_ for the [8k, 12k) part, do we know that? Or do we just know our whole [0k, 16k) thing failed, and so we need all this convoluted tracking to be able to retry for the range that we fucked up, and this is what you are facilitating? Thanks,
Oh, I got your point.
Right, when the failure happens, the range passed in is not [0, 16K), but the last failed one.
We can just check the range, free any existing entry in the failure range, and call it a day...
And I made the case over complicated them.
Thanks for the advice, I'll try to use the existing ulist mechanism to do the same thing.
Thanks,
Qu
Quoted 2 lines
Josef
PATCH 2/3 btrfs: qgroup: Try to flush qgroup space when we get -EDQUOT
In reply to Qu Wenruo[PROBLEM]
There are known problem related to how btrfs handles qgroup reserved space.
One of the most obvious case is the the test case btrfs/153, which do fallocate, then write into the preallocated range.
btrfs/153 1s ... - output mismatch (see xfstests-dev/results//btrfs/153.out.bad)
--- tests/btrfs/153.out 2019-10-22 15:18:14.068965341 +0800
+++ xfstests-dev/results//btrfs/153.out.bad 2020-07-01 20:24:40.730000089 +0800
@@ -1,2 +1,5 @@
QA output created by 153
+pwrite: Disk quota exceeded
+/mnt/scratch/testfile2: Disk quota exceeded
+/mnt/scratch/testfile2: Disk quota exceeded
Silence is golden
...
(Run 'diff -u xfstests-dev/tests/btrfs/153.out xfstests-dev/results//btrfs/153.out.bad' to see the entire diff)
[CAUSE]
Since commit c6887cd11149 ("Btrfs: don't do nocow check unless we have to"), we always reserve space no matter if it's COW or not.
Such behavior change is mostly for performance, and reverting it is not a good idea anyway.
For preallcoated extent, we reserve qgroup data space for it already, and since we also reserve data space for qgroup at buffered write time, it needs twice the space for us to write into preallocated space.
This leads to the -EDQUOT in buffered write routine.
And we can't follow the same solution, unlike data/meta space check, qgroup reserved space is shared between data/meta.
The EDQUOT can happen at the metadata reservation, so doing NODATACOW check after qgroup reservation failure is not a solution.
[FIX]
To solve the problem, we don't return -EDQUOT directly, but every time we got a -EDQUOT, we try to flush qgroup space by:
- Flush all inodes of the root
NODATACOW writes will free the qgroup reserved at run_dealloc_range(). However we don't have the infrastructure to only flush NODATACOW inodes, here we flush all inodes anyway.
- Wait ordered extents
This would convert the preallocated metadata space into per-trans metadata, which can be freed in later transaction commit.
- Commit transaction
This will free all per-trans metadata space.
Also we don't want to trigger flush too racy, so here we introduce a per-root mutex to ensure if there is a running qgroup flushing, we wait for it to end and don't start re-flush.
--- fs/btrfs/ctree.h | 1 + fs/btrfs/disk-io.c | 1 + fs/btrfs/qgroup.c | 112 ++++++++++++++++++++++++++++++++++++++++----- 3 files changed, 103 insertions(+), 11 deletions(-)
fs/btrfs/ctree.h +1 -0
@@ -1162,6 +1162,7 @@ struct btrfs_root {spinlock_t qgroup_meta_rsv_lock;u64 qgroup_meta_rsv_pertrans;u64 qgroup_meta_rsv_prealloc; +struct mutex qgroup_flushing_mutex;/* Number of active swapfiles */ atomic_t nr_swapfiles;
fs/btrfs/disk-io.c +1 -0
@@ -1116,6 +1116,7 @@ static void __setup_root(struct btrfs_root *root, struct btrfs_fs_info *fs_info,mutex_init (&root->log_mutex);mutex_init (&root->ordered_extent_mutex);mutex_init (&root->delalloc_mutex); +mutex_init (&root->qgroup_flushing_mutex);init_waitqueue_head (&root->log_writer_wait);init_waitqueue_head (&root->log_commit_wait[0 ]);init_waitqueue_head (&root->log_commit_wait[1 ]);
fs/btrfs/qgroup.c +101 -11
140 lines not on this page · raw message
Signature · 1 line
2.27.0
Re: [PATCH 2/3] btrfs: qgroup: Try to flush qgroup space when we get -EDQUOT
Quoted 29 lines · Qu Wenruo
[PROBLEM]
There are known problem related to how btrfs handles qgroup reserved space.
One of the most obvious case is the the test case btrfs/153, which do fallocate, then write into the preallocated range.btrfs/153 1s ... - output mismatch (see xfstests-dev/results//btrfs/153.out.bad)
--- tests/btrfs/153.out 2019-10-22 15:18:14.068965341 +0800
+++ xfstests-dev/results//btrfs/153.out.bad 2020-07-01 20:24:40.730000089 +0800
@@ -1,2 +1,5 @@
QA output created by 153
+pwrite: Disk quota exceeded
+/mnt/scratch/testfile2: Disk quota exceeded
+/mnt/scratch/testfile2: Disk quota exceeded
Silence is golden
...
(Run 'diff -u xfstests-dev/tests/btrfs/153.out xfstests-dev/results//btrfs/153.out.bad' to see the entire diff)[CAUSE]
Since commitc6887cd11149("Btrfs: don't do nocow check unless we have to"), we always reserve space no matter if it's COW or not.Such behavior change is mostly for performance, and reverting it is not a good idea anyway.
For preallcoated extent, we reserve qgroup data space for it already, and since we also reserve data space for qgroup at buffered write time, it needs twice the space for us to write into preallocated space.
Quoted 6 lines
This leads to the
-EDQUOTin buffered write routine.And we can't follow the same solution, unlike data/meta space check, qgroup reserved space is shared between data/meta.
The EDQUOT can happen at the metadata reservation, so doing NODATACOW check after qgroup reservation failure is not a solution.
Why not? I get that we don't know for sure how we failed, but in the case of a write we're way more likely to have failed for data reasons right? So why not just fall back to the NODATACOW check and then do the metadata reservation. Then if it fails again you know its a real EDQUOT and your done.
Or if you want to get super fancy you could even break up the metadata and data reservations here so that we only fall through to the NODATACOW check if we fail the data reservation. Thanks,
Josef
Re: [PATCH 2/3] btrfs: qgroup: Try to flush qgroup space when we get -EDQUOT
Quoted 36 lines · Josef Bacik
Quoted 35 lines · Qu Wenruo
[PROBLEM]
There are known problem related to how btrfs handles qgroup reserved space.
One of the most obvious case is the the test case btrfs/153, which do fallocate, then write into the preallocated range.btrfs/153 1s ... - output mismatch (see xfstests-dev/results//btrfs/153.out.bad)
--- tests/btrfs/153.out 2019-10-22 15:18:14.068965341 +0800 +++ xfstests-dev/results//btrfs/153.out.bad 2020-07-01 20:24:40.730000089 +0800
@@ -1,2 +1,5 @@
QA output created by 153
+pwrite: Disk quota exceeded +/mnt/scratch/testfile2: Disk quota exceeded +/mnt/scratch/testfile2: Disk quota exceeded
Silence is golden
...
(Run 'diff -u xfstests-dev/tests/btrfs/153.out xfstests-dev/results//btrfs/153.out.bad' to see the entire diff)[CAUSE]
Since commitc6887cd11149("Btrfs: don't do nocow check unless we have to"),
we always reserve space no matter if it's COW or not.Such behavior change is mostly for performance, and reverting it is not a good idea anyway.
For preallcoated extent, we reserve qgroup data space for it already, and since we also reserve data space for qgroup at buffered write time, it needs twice the space for us to write into preallocated space.
This leads to the
-EDQUOTin buffered write routine.
Quoted 8 lines
Quoted 4 lines
And we can't follow the same solution, unlike data/meta space check, qgroup reserved space is shared between data/meta.
The EDQUOT can happen at the metadata reservation, so doing NODATACOW check after qgroup reservation failure is not a solution.Why not? I get that we don't know for sure how we failed, but in the case of a write we're way more likely to have failed for data reasons right?
Nope, mostly we failed at metadata reservation, as that would return EDQUOT to user space.
We may have some cases which get EDQUOT at data reservation part, but that's what we excepted.
(And already what we're doing)
The problem is when the metadata reservation failed with EDQUOT.
Quoted 7 lines
So why not just fall back to the NODATACOW check and then do the metadata reservation. Then if it fails again you know its a real EDQUOT and your done.
Or if you want to get super fancy you could even break up the metadata and data reservations here so that we only fall through to the NODATACOW check if we fail the data reservation. Thanks,
The problem is, qgroup doesn't split metadata and data (yet). Currently data and meta shares the same limit.
So when we hit EDQUOT, you have no guarantee it would happen only in qgroup data reservation.
Thanks,
Qu
Quoted 2 lines
Josef
Re: [PATCH 2/3] btrfs: qgroup: Try to flush qgroup space when we get -EDQUOT
Quoted 65 lines · Qu Wenruo
Quoted 36 lines · Josef Bacik
Quoted 35 lines · Qu Wenruo
[PROBLEM]
There are known problem related to how btrfs handles qgroup reserved space.
One of the most obvious case is the the test case btrfs/153, which do fallocate, then write into the preallocated range.btrfs/153 1s ... - output mismatch (see
xfstests-dev/results//btrfs/153.out.bad)
--- tests/btrfs/153.out 2019-10-22 15:18:14.068965341 +0800 +++ xfstests-dev/results//btrfs/153.out.bad 2020-07-01
20:24:40.730000089 +0800
@@ -1,2 +1,5 @@
QA output created by 153
+pwrite: Disk quota exceeded +/mnt/scratch/testfile2: Disk quota exceeded +/mnt/scratch/testfile2: Disk quota exceeded
Silence is golden
...
(Run 'diff -u xfstests-dev/tests/btrfs/153.out
xfstests-dev/results//btrfs/153.out.bad' to see the entire diff)[CAUSE]
Since commitc6887cd11149("Btrfs: don't do nocow check unless we have to"),
we always reserve space no matter if it's COW or not.Such behavior change is mostly for performance, and reverting it is not a good idea anyway.
For preallcoated extent, we reserve qgroup data space for it already, and since we also reserve data space for qgroup at buffered write time, it needs twice the space for us to write into preallocated space.
This leads to the
-EDQUOTin buffered write routine.Quoted 8 lines
Quoted 4 lines
And we can't follow the same solution, unlike data/meta space check, qgroup reserved space is shared between data/meta.
The EDQUOT can happen at the metadata reservation, so doing NODATACOW check after qgroup reservation failure is not a solution.Why not? I get that we don't know for sure how we failed, but in the case of a write we're way more likely to have failed for data reasons right?
Nope, mostly we failed at metadata reservation, as that would return EDQUOT to user space.
We may have some cases which get EDQUOT at data reservation part, but that's what we excepted.
(And already what we're doing)The problem is when the metadata reservation failed with EDQUOT.
Quoted 7 lines
So why not just fall back to the NODATACOW check and then do the
metadata reservation. Then if it fails again you know its a real EDQUOT and your done.Or if you want to get super fancy you could even break up the metadata and data reservations here so that we only fall through to the NODATACOW check if we fail the data reservation. Thanks,
Quoted 6 lines
The problem is, qgroup doesn't split metadata and data (yet). Currently data and meta shares the same limit.
So when we hit EDQUOT, you have no guarantee it would happen only in qgroup data reservation.
Sure, but if you are able to do the nocow thing, then presumably your quota reservation is less now? So on failure you go do the complicated nocow check, and if it succeeds you retry your quota reservation with just the metadata portion, right? Thanks,
Josef
Re: [PATCH 2/3] btrfs: qgroup: Try to flush qgroup space when we get -EDQUOT
Quoted 69 lines · Josef Bacik
Quoted 68 lines · Qu Wenruo
Quoted 36 lines · Josef Bacik
Quoted 35 lines · Qu Wenruo
[PROBLEM]
There are known problem related to how btrfs handles qgroup reserved space.
One of the most obvious case is the the test case btrfs/153, which do fallocate, then write into the preallocated range.btrfs/153 1s ... - output mismatch (see xfstests-dev/results//btrfs/153.out.bad)
--- tests/btrfs/153.out 2019-10-22 15:18:14.068965341 +0800 +++ xfstests-dev/results//btrfs/153.out.bad 2020-07-01 20:24:40.730000089 +0800
@@ -1,2 +1,5 @@
QA output created by 153
+pwrite: Disk quota exceeded +/mnt/scratch/testfile2: Disk quota exceeded +/mnt/scratch/testfile2: Disk quota exceeded
Silence is golden
...
(Run 'diff -u xfstests-dev/tests/btrfs/153.out xfstests-dev/results//btrfs/153.out.bad' to see the entire diff)[CAUSE]
Since commitc6887cd11149("Btrfs: don't do nocow check unless we have to"),
we always reserve space no matter if it's COW or not.Such behavior change is mostly for performance, and reverting it is not a good idea anyway.
For preallcoated extent, we reserve qgroup data space for it already, and since we also reserve data space for qgroup at buffered write time, it needs twice the space for us to write into preallocated space.
This leads to the
-EDQUOTin buffered write routine.Quoted 8 lines
Quoted 4 lines
And we can't follow the same solution, unlike data/meta space check, qgroup reserved space is shared between data/meta.
The EDQUOT can happen at the metadata reservation, so doing NODATACOW check after qgroup reservation failure is not a solution.Why not? I get that we don't know for sure how we failed, but in the case of a write we're way more likely to have failed for data reasons right?
Nope, mostly we failed at metadata reservation, as that would return EDQUOT to user space.
We may have some cases which get EDQUOT at data reservation part, but that's what we excepted.
(And already what we're doing)The problem is when the metadata reservation failed with EDQUOT.
Quoted 7 lines
So why not just fall back to the NODATACOW check and then do the metadata reservation. Then if it fails again you know its a real EDQUOT and your done.
Or if you want to get super fancy you could even break up the metadata and data reservations here so that we only fall through to the NODATACOW check if we fail the data reservation. Thanks,
The problem is, qgroup doesn't split metadata and data (yet). Currently data and meta shares the same limit.
Quoted 8 lines
Quoted 3 lines
So when we hit EDQUOT, you have no guarantee it would happen only in qgroup data reservation.
Sure, but if you are able to do the nocow thing, then presumably your quota reservation is less now? So on failure you go do the complicated nocow check, and if it succeeds you retry your quota reservation with just the metadata portion, right? Thanks,
Then metadata portion can still fail, even we skipped the data reserv.
The metadata portion still needs some space, while the data rsv skip only happens after we're already near the qgroup limit, which means there are ready not much space left.
Consider this case, we have 128M limit, we fallocated 120M, then we have dirtied 7M data, plus several kilo for metadata reserved.
Then at the next 1M, we run out of qgroup limit, at whatever position. Even we skip current 4K for data, the next metadata reserve may still not be met, and still got EDQUOT at metadata reserve.
Or some other open() calls to create a new file would just get EDQUOT, without any way to free any extra space.
Instead of try to skip just several 4K for qgroup data rsv, we should flush the existing 7M, to free at least 7M data + several kilo meta space.
Thanks,
Qu
Quoted 2 lines
Josef
Re: [PATCH 2/3] btrfs: qgroup: Try to flush qgroup space when we get -EDQUOT
Quoted 94 lines · Qu Wenruo
Quoted 69 lines · Josef Bacik
Quoted 68 lines · Qu Wenruo
Quoted 36 lines · Josef Bacik
Quoted 35 lines · Qu Wenruo
[PROBLEM]
There are known problem related to how btrfs handles qgroup reserved space.
One of the most obvious case is the the test case btrfs/153, which do fallocate, then write into the preallocated range.btrfs/153 1s ... - output mismatch (see
xfstests-dev/results//btrfs/153.out.bad)
--- tests/btrfs/153.out 2019-10-22 15:18:14.068965341 +0800 +++ xfstests-dev/results//btrfs/153.out.bad 2020-07-01
20:24:40.730000089 +0800
@@ -1,2 +1,5 @@
QA output created by 153
+pwrite: Disk quota exceeded +/mnt/scratch/testfile2: Disk quota exceeded +/mnt/scratch/testfile2: Disk quota exceeded Silence is golden
...
(Run 'diff -u xfstests-dev/tests/btrfs/153.out
xfstests-dev/results//btrfs/153.out.bad' to see the entire diff)[CAUSE]
Since commitc6887cd11149("Btrfs: don't do nocow check unless we have to"),
we always reserve space no matter if it's COW or not.Such behavior change is mostly for performance, and reverting it is not a good idea anyway.
For preallcoated extent, we reserve qgroup data space for it already, and since we also reserve data space for qgroup at buffered write time, it needs twice the space for us to write into preallocated space.
This leads to the
-EDQUOTin buffered write routine.Quoted 8 lines
Quoted 4 lines
And we can't follow the same solution, unlike data/meta space check, qgroup reserved space is shared between data/meta.
The EDQUOT can happen at the metadata reservation, so doing NODATACOW check after qgroup reservation failure is not a solution.Why not? I get that we don't know for sure how we failed, but in the case of a write we're way more likely to have failed for data reasons right?
Nope, mostly we failed at metadata reservation, as that would return EDQUOT to user space.
We may have some cases which get EDQUOT at data reservation part, but that's what we excepted.
(And already what we're doing)The problem is when the metadata reservation failed with EDQUOT.
Quoted 7 lines
So why not just fall back to the NODATACOW check and then do the
metadata reservation. Then if it fails again you know its a real EDQUOT and your done.Or if you want to get super fancy you could even break up the metadata and data reservations here so that we only fall through to the NODATACOW check if we fail the data reservation. Thanks,
The problem is, qgroup doesn't split metadata and data (yet). Currently data and meta shares the same limit.
Quoted 8 lines
Quoted 3 lines
So when we hit EDQUOT, you have no guarantee it would happen only in qgroup data reservation.
Sure, but if you are able to do the nocow thing, then presumably your quota reservation is less now? So on failure you go do the complicated nocow check, and if it succeeds you retry your quota reservation with just the metadata portion, right? Thanks,
Then metadata portion can still fail, even we skipped the data reserv.
The metadata portion still needs some space, while the data rsv skip only happens after we're already near the qgroup limit, which means there are ready not much space left.
Consider this case, we have 128M limit, we fallocated 120M, then we have dirtied 7M data, plus several kilo for metadata reserved.
Then at the next 1M, we run out of qgroup limit, at whatever position. Even we skip current 4K for data, the next metadata reserve may still not be met, and still got EDQUOT at metadata reserve.
Quoted 7 lines
Or some other
open()calls to create a new file would just get EDQUOT, without any way to free any extra space.Instead of try to skip just several 4K for qgroup data rsv, we should flush the existing 7M, to free at least 7M data + several kilo meta space.
Right so I'm not against flushing in general, I just think that we can greatly improve on this particular problem without flushing. Changing how we do the NOCOW check with quota could be faster than doing the flushing.
Now as for the flushing part itself, I'd rather hook into the existing flushing infrastructure we have. Obviously the ticketing is going to be different, but the flushing part is still the same, and with data reservations now moved over to that infrastructure we finally have it all in the same place. Thanks,
Josef
Re: [PATCH 2/3] btrfs: qgroup: Try to flush qgroup space when we get -EDQUOT
Quoted 110 lines · Josef Bacik
Quoted 100 lines · Qu Wenruo
Quoted 75 lines · Josef Bacik
Quoted 74 lines · Qu Wenruo
Quoted 40 lines · Josef Bacik
Quoted 39 lines · Qu Wenruo
[PROBLEM]
There are known problem related to how btrfs handles qgroup reserved space.
One of the most obvious case is the the test case btrfs/153, which do fallocate, then write into the preallocated range.btrfs/153 1s ... - output mismatch (see xfstests-dev/results//btrfs/153.out.bad)
--- tests/btrfs/153.out 2019-10-22 15:18:14.068965341 +0800
+++ xfstests-dev/results//btrfs/153.out.bad 2020-07-01 20:24:40.730000089 +0800
@@ -1,2 +1,5 @@
QA output created by 153
+pwrite: Disk quota exceeded +/mnt/scratch/testfile2: Disk quota exceeded +/mnt/scratch/testfile2: Disk quota exceeded Silence is golden
...
(Run 'diff -u xfstests-dev/tests/btrfs/153.out xfstests-dev/results//btrfs/153.out.bad' to see the entire diff)[CAUSE]
Since commitc6887cd11149("Btrfs: don't do nocow check unless we have
to"),
we always reserve space no matter if it's COW or not.Such behavior change is mostly for performance, and reverting it is not
a good idea anyway.For preallcoated extent, we reserve qgroup data space for it already, and since we also reserve data space for qgroup at buffered write time,
it needs twice the space for us to write into preallocated space.This leads to the
-EDQUOTin buffered write routine.Quoted 8 lines
Quoted 4 lines
And we can't follow the same solution, unlike data/meta space check, qgroup reserved space is shared between data/meta.
The EDQUOT can happen at the metadata reservation, so doing NODATACOW check after qgroup reservation failure is not a solution.Why not? I get that we don't know for sure how we failed, but in the case of a write we're way more likely to have failed for data reasons right?
Nope, mostly we failed at metadata reservation, as that would return EDQUOT to user space.
We may have some cases which get EDQUOT at data reservation part, but that's what we excepted.
(And already what we're doing)The problem is when the metadata reservation failed with EDQUOT.
Quoted 9 lines
So why not just fall back to the NODATACOW check and then do the metadata reservation. Then if it fails again you know its a real EDQUOT
and your done.Or if you want to get super fancy you could even break up the metadata and data reservations here so that we only fall through to the NODATACOW
check if we fail the data reservation. Thanks,The problem is, qgroup doesn't split metadata and data (yet). Currently data and meta shares the same limit.
Quoted 8 lines
Quoted 3 lines
So when we hit EDQUOT, you have no guarantee it would happen only in qgroup data reservation.
Sure, but if you are able to do the nocow thing, then presumably your quota reservation is less now? So on failure you go do the complicated nocow check, and if it succeeds you retry your quota reservation with just the metadata portion, right? Thanks,
Then metadata portion can still fail, even we skipped the data reserv.
The metadata portion still needs some space, while the data rsv skip only happens after we're already near the qgroup limit, which means there are ready not much space left.
Consider this case, we have 128M limit, we fallocated 120M, then we have dirtied 7M data, plus several kilo for metadata reserved.
Then at the next 1M, we run out of qgroup limit, at whatever position. Even we skip current 4K for data, the next metadata reserve may still not be met, and still got EDQUOT at metadata reserve.
Quoted 8 lines
Or some other
open()calls to create a new file would just get EDQUOT, without any way to free any extra space.Instead of try to skip just several 4K for qgroup data rsv, we should flush the existing 7M, to free at least 7M data + several kilo meta space.
Quoted 4 lines
Right so I'm not against flushing in general, I just think that we can greatly improve on this particular problem without flushing. Changing how we do the NOCOW check with quota could be faster than doing the flushing.
Yep, but as mentioned, the uncertain timing of when we get the EDQUOT is really annoying and tricky to solve, thus have to go the flushing method.
The performance is definitely slower, but it's not acceptable, since we're near the limiting, slowing down is pretty common.
Quoted 3 lines
Now as for the flushing part itself, I'd rather hook into the existing flushing infrastructure we have.
That's the ultimate objective.
Quoted 4 lines
Obviously the ticketing is going to be
different, but the flushing part is still the same, and with data reservations now moved over to that infrastructure we finally have it all in the same place. Thanks,
Before the needed infrastructure get merged, I'll keep the current small retry code and look into what's needed to integrate qgroup rsv into the ticketing system.
Thanks,
Qu
Quoted 2 lines
Josef
PATCH 3/3 Revert "btrfs: qgroup: Commit transaction in advance to reduce early EDQUOT"
In reply to Qu WenruoThis reverts commit a514d63882c3d2063b21b865447266ebcb18b04c.
Since we have the ability to retry qgroup reservation, and do qgroup space flushing, there is no need for the BTRFS_FS_NEED_ASYNC_COMMIT mechanism anymore.
Just revert that commit to make the code a little simpler.
--- fs/btrfs/ctree.h | 5 ----- fs/btrfs/disk-io.c | 1 - fs/btrfs/qgroup.c | 43 ++---------------------------------------- fs/btrfs/transaction.c | 1 - fs/btrfs/transaction.h | 14 -------------- 5 files changed, 2 insertions(+), 62 deletions(-)
fs/btrfs/ctree.h +0 -5
@@ -545,11 +545,6 @@ enum {* (device replace, resize, device add/delete, balance) */ BTRFS_FS_EXCL_OP, -/* -* To info transaction_kthread we need an immediate commit so it -* doesn't need to wait for commit_interval -*/ - BTRFS_FS_NEED_ASYNC_COMMIT,/* * Indicate that balance has been set up from the ioctl and is in the * main phase. The fs_info::balance_ctl is initialized.
fs/btrfs/disk-io.c +0 -1
@@ -1746,7 +1746,6 @@ static int transaction_kthread(void *arg) now =ktime_get_seconds ();if (cur->state <TRANS_STATE_COMMIT_START && - !test_bit (BTRFS_FS_NEED_ASYNC_COMMIT , &fs_info->flags) && (now < cur->start_time || now - cur->start_time < fs_info->commit_interval)) {spin_unlock (&fs_info->trans_lock);
fs/btrfs/qgroup.c +2 -41
68 lines not on this page · raw message
fs/btrfs/transaction.c +0 -1
@@ -2351,7 +2351,6 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans)*/ cur_trans->state =TRANS_STATE_COMPLETED ;wake_up (&cur_trans->commit_wait); -clear_bit (BTRFS_FS_NEED_ASYNC_COMMIT , &fs_info->flags);spin_lock (&fs_info->trans_lock);list_del_init (&cur_trans->list);
fs/btrfs/transaction.h +0 -14
@@ -208,20 +208,6 @@ int btrfs_clean_one_deleted_snapshot(struct btrfs_root *root);int btrfs_commit_transaction (struct btrfs_trans_handle *trans);int btrfs_commit_transaction_async (struct btrfs_trans_handle *trans,int wait_for_unblock); - -/* -* Try to commit transaction asynchronously, so this is safe to call -* even holding a spinlock. -* -* It's done by informing transaction_kthread to commit transaction without -* waiting for commit interval. -*/ -static inline void btrfs_commit_transaction_locksafe ( -struct btrfs_fs_info *fs_info) -{ -set_bit (BTRFS_FS_NEED_ASYNC_COMMIT , &fs_info->flags); -wake_up_process (fs_info->transaction_kthread); -}int btrfs_end_transaction_throttle (struct btrfs_trans_handle *trans);int btrfs_should_end_transaction (struct btrfs_trans_handle *trans);void btrfs_throttle (struct btrfs_fs_info *fs_info);
Signature · 1 line
2.27.0
Re: [PATCH 3/3] Revert "btrfs: qgroup: Commit transaction in advance to reduce early EDQUOT"
Quoted 7 lines · Qu Wenruo
This reverts commit
a514d63882c3d2063b21b865447266ebcb18b04c.Since we have the ability to retry qgroup reservation, and do qgroup space flushing, there is no need for the
BTRFS_FS_NEED_ASYNC_COMMITmechanism anymore.Just revert that commit to make the code a little simpler.
Interestingly, the same comment from
https://lore.kernel.org/linux-btrfs/[email protected]/
applies to your patch sans the references to the other patch:
"
I do not recommend to do a revert here. Even if a patch reverts functionality because it's not needed anymore, then it's a "forward" change. There are also other changes that may affect the behaviour and in this case it's 47dba17171a76ea2a2a71 that removes rcu barrier, so the patch has to be put into the context of current code.
The commit is almost 2 years old, the idea of reverts is IMHO more to provide an easy way do a small step back during one devlopment cycle when the moving parts are still in sight.
Back then the commit fixed a deadlock, a revert here would read as 'ok, we want the deadlock back'.
So, the code is ok. The subject needs to drop the word 'revert' and changelg maybe mention a few more references why the logic is not needed anymore.
"
Quoted 20 lines · Qu Wenruo
Since commit
c6887cd11149("Btrfs: don't do nocow check unless we have to"), btrfs/153 always fails with early EDQUOT.This is caused by the fact that:
- We always reserved data space for even NODATACOW buffered write This is mostly to improve performance, and not pratical to revert.- Btrfs qgroup data and meta reserved space share the same limit So it's not ensured to return EDQUOT just for that data reservation, metadata reservation can also get EDQUOT, means we can't go the same solution as that commit.
This patchset will solve it by doing extra qgroup space flushing when EDQUOT is hit.
This is a little like what we do in ticketing space reservation system, but since there are very limited ways for qgroup to reclaim space, currently it's still handled in qgroup realm, not reusing the ticketing system yet.
Quoted 7 lines
By this, this patch could solve the btrfs/153 problem, while still keep btrfs qgroup space usage under the limit.
The only cost is, when we're near qgroup limit, we will cause more dirty inodes flush and transaction commit, much like what we do when the metadata space is near exhausted.
So the cost should be still acceptable.
This patchset fails to apply on misc-next as of
btrfs: allow use of global block reserve for balance item deletion
Thanks,
Josef
Quoted 24 lines · Josef Bacik
Quoted 23 lines · Qu Wenruo
Since commit
c6887cd11149("Btrfs: don't do nocow check unless we have to"), btrfs/153 always fails with early EDQUOT.This is caused by the fact that:
- We always reserved data space for even NODATACOW buffered write This is mostly to improve performance, and not pratical to revert.- Btrfs qgroup data and meta reserved space share the same limit So it's not ensured to return EDQUOT just for that data reservation, metadata reservation can also get EDQUOT, means we can't go the same solution as that commit.
This patchset will solve it by doing extra qgroup space flushing when EDQUOT is hit.
This is a little like what we do in ticketing space reservation system, but since there are very limited ways for qgroup to reclaim space, currently it's still handled in qgroup realm, not reusing the ticketing system yet.
By this, this patch could solve the btrfs/153 problem, while still keep btrfs qgroup space usage under the limit.
Quoted 8 lines
Quoted 4 lines
The only cost is, when we're near qgroup limit, we will cause more dirty inodes flush and transaction commit, much like what we do when the metadata space is near exhausted.
So the cost should be still acceptable.This patchset fails to apply on misc-next as of
btrfs: allow use of global block reserve for balance item deletion
That's about 50 patches behind some recent snapshot of misc-next, the patches apply cleanly for me here.
Quoted 30 lines · David Sterba
Quoted 29 lines · Josef Bacik
Quoted 27 lines · Qu Wenruo
Since commit
c6887cd11149("Btrfs: don't do nocow check unless we have to"), btrfs/153 always fails with early EDQUOT.This is caused by the fact that:
- We always reserved data space for even NODATACOW buffered write This is mostly to improve performance, and not pratical to revert.- Btrfs qgroup data and meta reserved space share the same limit So it's not ensured to return EDQUOT just for that data reservation, metadata reservation can also get EDQUOT, means we can't go the same solution as that commit.
This patchset will solve it by doing extra qgroup space flushing when EDQUOT is hit.
This is a little like what we do in ticketing space reservation system, but since there are very limited ways for qgroup to reclaim space, currently it's still handled in qgroup realm, not reusing the ticketing system yet.
By this, this patch could solve the btrfs/153 problem, while still keep btrfs qgroup space usage under the limit.
The only cost is, when we're near qgroup limit, we will cause more dirty inodes flush and transaction commit, much like what we do when the metadata space is near exhausted.
So the cost should be still acceptable.
Quoted 7 lines
Quoted 3 lines
This patchset fails to apply on misc-next as of
btrfs: allow use of global block reserve for balance item deletion
That's about 50 patches behind some recent snapshot of misc-next, the patches apply cleanly for me here.
I misread the date on my tree, I just saw "Thu" and assumed my cron job was doing its job correctly. Thanks,
Josef
Quoted 20 lines · Qu Wenruo
Since commit
c6887cd11149("Btrfs: don't do nocow check unless we have to"), btrfs/153 always fails with early EDQUOT.This is caused by the fact that:
- We always reserved data space for even NODATACOW buffered write This is mostly to improve performance, and not pratical to revert.- Btrfs qgroup data and meta reserved space share the same limit So it's not ensured to return EDQUOT just for that data reservation, metadata reservation can also get EDQUOT, means we can't go the same solution as that commit.
This patchset will solve it by doing extra qgroup space flushing when EDQUOT is hit.
This is a little like what we do in ticketing space reservation system, but since there are very limited ways for qgroup to reclaim space, currently it's still handled in qgroup realm, not reusing the ticketing system yet.
Quoted 7 lines
By this, this patch could solve the btrfs/153 problem, while still keep btrfs qgroup space usage under the limit.
The only cost is, when we're near qgroup limit, we will cause more dirty inodes flush and transaction commit, much like what we do when the metadata space is near exhausted.
So the cost should be still acceptable.
This sounds like a reasonable solution to me. Making the behaviour closer to ticket reservations would probably make it easier to switch some day.