btrfs: fix cases of stat(2) reporting incorrect number of used blocks
cover btrfs: fix cases of stat(2) reporting incorrect number of used blocks
From: Filipe Manana <[email protected]>
There are several cases where a stat(2) call can report an incorrect number of used blocks. In some cases can even result in reporting 0 used blocks, which is a specially bad value to report when a file is not empty, and we had user bug reports in the past for such cases (the changelogs in the patches point to one such report).
This patchset addresses all those cases. The third patch fixes a race in defrag that while it does not result in a functional problem (data loss or some corruption), it leads to unnecessary IO and space allocation, and it's necessary for the 4th and final patch to work as it is.
A couple test cases for fstests will follow.
Filipe Manana (4):
btrfs: fix missing delalloc new bit for new delalloc ranges
btrfs: refactor btrfs_drop_extents() to make it easier to extend
btrfs: fix race when defragging that leads to unnecessary IO
btrfs: update the number of bytes used by an inode atomically
fs/btrfs/btrfs_inode.h | 3 +- fs/btrfs/ctree.h | 71 ++++++++-- fs/btrfs/extent-io-tree.h | 16 ++- fs/btrfs/extent_io.c | 10 +- fs/btrfs/file.c | 246 +++++++++++++++-------------------- fs/btrfs/inode.c | 233 +++++++++++++++++++++++++++------ fs/btrfs/ioctl.c | 39 ++++++ fs/btrfs/reflink.c | 9 +- fs/btrfs/tests/inode-tests.c | 12 +- fs/btrfs/tree-log.c | 32 +++-- 10 files changed, 458 insertions(+), 213 deletions(-)
Signature · 1 line
2.28.0
PATCH 1/4 btrfs: fix missing delalloc new bit for new delalloc ranges
From: Filipe Manana <[email protected]>
When doing a buffered write, through one of the write family syscalls, we look for ranges which currenly don't have allocated extents and set the 'delalloc new' bit on them, so that we can report a correct number of used blocks to the stat(2) syscall until delalloc is flushed and ordered extents complete.
However there are a few other places where we can do a buffered write against a range that is mapped to a hole (no extent allocated) and where we do not set the 'new delalloc' bit. Those places are:
- Doing a memory mapped write against a hole;
- Cloning an inline extent into a hole starting at file offset 0;
- Calling btrfs_cont_expand() when the i_size of the file is not aligned to the sector size and is located in a hole. For example when cloning to a destination offset beyond eof.
So after such cases, until the corresponding delalloc range is flushed and the respective ordered extents complete, we can report an incorrect number of blocks used through the stat(2) syscall.
In some cases we can end up reporting 0 used blocks to stat(2), which is a particular bad value to report as it may mislead tools to think a file is completely sparse when its i_size is not zero, making them skip reading any data, an undesired consequence for tools such as archivers and other backup tools, as reported a long time ago in the following thread (and other past threads):
https://lists.gnu.org/archive/html/bug-tar/2016-07/msg00001.html
Example reproducer:
$ cat reproducer.sh #!/bin/bash
MNT=/mnt/sdi
DEV=/dev/sdi
mkfs.btrfs -f $DEV > /dev/null
# mkfs.xfs -f $DEV > /dev/null # mkfs.ext4 -F $DEV > /dev/null # mkfs.f2fs -f $DEV > /dev/null mount $DEV $MNT
xfs_io -f -c "truncate 64K" \
-c "mmap -w 0 64K" \
-c "mwrite -S 0xab 0 64K" \
-c "munmap" \
$MNT/foo
blocks_used=$(stat -c %b $MNT/foo)
echo "blocks used: $blocks_used" if [ $blocks_used -eq 0 ]; then
echo "ERROR: blocks used is 0"
fi
umount $DEV
$ ./reproducer.sh blocks used: 0 ERROR: blocks used is 0
So move the logic that decides to set the 'delalloc bit' bit into the function btrfs_set_extent_delalloc(), since that is what we use for all those missing cases as well as for the cases that currently work well.
This change is also preparatory work for an upcoming patch that fixes other problems related to tracking and reporting the number of bytes used by an inode.
--- fs/btrfs/file.c | 56 ----------------------------------- fs/btrfs/inode.c | 57 ++++++++++++++++++++++++++++++++++++ fs/btrfs/tests/inode-tests.c | 12 +++++--- 3 files changed, 65 insertions(+), 60 deletions(-)
fs/btrfs/file.c +0 -56
@@ -452,45 +452,6 @@ static void btrfs_drop_pages(struct page **pages, size_t num_pages) } } -static int btrfs_find_new_delalloc_bytes (struct btrfs_inode *inode, -const u64 start, -const u64 len, -struct extent_state **cached_state) -{ -u64 search_start = start; -const u64 end = start + len -1 ; - -while (search_start < end) { -const u64 search_len = end - search_start +1 ; -struct extent_map *em; -u64 em_len; -int ret =0 ; - - em =btrfs_get_extent (inode,NULL ,0 , search_start, search_len); -if (IS_ERR (em)) -return PTR_ERR (em); - -if (em->block_start !=EXTENT_MAP_HOLE ) - goto next; - - em_len = em->len; -if (em->start < search_start) - em_len -= search_start - em->start; -if (em_len > search_len) - em_len = search_len; - - ret =set_extent_bit (&inode->io_tree, search_start, - search_start + em_len -1 , -EXTENT_DELALLOC_NEW , cached_state,GFP_NOFS ); -next: - search_start =extent_map_end (em); -free_extent_map (em); -if (ret) -return ret; - } -return 0 ; -} -/* * after copy_from_user, pages need to be dirtied and we need to make * sure holes are created between the current EOF and the start of
@@ -533,23 +494,6 @@ int btrfs_dirty_pages(struct btrfs_inode *inode, struct page **pages,EXTENT_DELALLOC |EXTENT_DO_ACCOUNTING |EXTENT_DEFRAG , 0, 0, cached); -if (!btrfs_is_free_space_inode (inode)) { -if (start_pos >= isize && - !(inode->flags &BTRFS_INODE_PREALLOC )) { -/* -* There can't be any extents following eof in this case -* so just set the delalloc new bit for the range -* directly. -*/ - extra_bits |=EXTENT_DELALLOC_NEW ; - }else { - err =btrfs_find_new_delalloc_bytes (inode, start_pos, - num_bytes, cached); -if (err) -return err; - } - } - err =btrfs_set_extent_delalloc (inode, start_pos, end_of_last_block, extra_bits, cached);if (err)
fs/btrfs/inode.c +57 -0
68 lines not on this page · raw message
fs/btrfs/tests/inode-tests.c +8 -4
@@ -983,7 +983,8 @@ static int test_extent_accounting(u32 sectorsize, u32 nodesize) ret =clear_extent_bit (&BTRFS_I (inode)->io_tree,BTRFS_MAX_EXTENT_SIZE >>1 , (BTRFS_MAX_EXTENT_SIZE >>1 ) + sectorsize -1 , -EXTENT_DELALLOC |EXTENT_UPTODATE ,0 ,0 ,NULL ); +EXTENT_DELALLOC |EXTENT_DELALLOC_NEW | +EXTENT_UPTODATE ,0 ,0 ,NULL );if (ret) {test_err ("clear_extent_bit returned %d" , ret); goto out;
@@ -1050,7 +1051,8 @@ static int test_extent_accounting(u32 sectorsize, u32 nodesize) ret =clear_extent_bit (&BTRFS_I (inode)->io_tree,BTRFS_MAX_EXTENT_SIZE + sectorsize,BTRFS_MAX_EXTENT_SIZE +2 * sectorsize -1 , -EXTENT_DELALLOC |EXTENT_UPTODATE ,0 ,0 ,NULL ); +EXTENT_DELALLOC |EXTENT_DELALLOC_NEW | +EXTENT_UPTODATE ,0 ,0 ,NULL );if (ret) {test_err ("clear_extent_bit returned %d" , ret); goto out;
@@ -1082,7 +1084,8 @@ static int test_extent_accounting(u32 sectorsize, u32 nodesize)/* Empty */ ret =clear_extent_bit (&BTRFS_I (inode)->io_tree,0 , (u64)-1 , -EXTENT_DELALLOC |EXTENT_UPTODATE ,0 ,0 ,NULL ); +EXTENT_DELALLOC |EXTENT_DELALLOC_NEW | +EXTENT_UPTODATE ,0 ,0 ,NULL );if (ret) {test_err ("clear_extent_bit returned %d" , ret); goto out;
@@ -1097,7 +1100,8 @@ static int test_extent_accounting(u32 sectorsize, u32 nodesize) out:if (ret)clear_extent_bit (&BTRFS_I (inode)->io_tree,0 , (u64)-1 , -EXTENT_DELALLOC |EXTENT_UPTODATE ,0 ,0 ,NULL ); +EXTENT_DELALLOC |EXTENT_DELALLOC_NEW | +EXTENT_UPTODATE ,0 ,0 ,NULL );iput (inode);btrfs_free_dummy_root (root);btrfs_free_dummy_fs_info (fs_info);
Signature · 1 line
2.28.0
PATCH 2/4 btrfs: refactor btrfs_drop_extents() to make it easier to extend
In reply to [email protected]From: Filipe Manana <[email protected]>
There are many arguments for __btrfs_drop_extents() and its wrapper btrfs_drop_extents(), which makes it hard to add more arguments to it and requires changing every caller. I have added a couple myself back in 2014 commit 1acae57b161e ("Btrfs: faster file extent item replace operations") and therefore know firsthand that it is a bit cumbersome to add additional arguments to these functions.
Since I will need to add more arguments in a subsequent bug fix, this change is preparatory work and adds a data structure that holds all the arguments, for both input and output, that are passed to this function, with some comments in the structure's definition mentioning what each field is and how it relates to other fields.
Callers of this function need only to zero out the content of the structure and setup only the fields they need. This also removes the need to have both __btrfs_drop_extents() and btrfs_drop_extents(), so now we have a single function named btrfs_drop_extents() that takes a pointer to this new data structure (struct btrfs_drop_extents_args).
--- fs/btrfs/ctree.h | 63 +++++++++++++++--- fs/btrfs/file.c | 155 +++++++++++++++++++++++--------------------- fs/btrfs/inode.c | 41 +++++++----- fs/btrfs/reflink.c | 7 +- fs/btrfs/tree-log.c | 28 +++++--- 5 files changed, 187 insertions(+), 107 deletions(-)
fs/btrfs/ctree.h +54 -9
@@ -1229,6 +1229,58 @@ struct btrfs_replace_extent_info {int insertions; }; +/* Arguments for btrfs_drop_extents(). */ +struct btrfs_drop_extents_args { +/* Input parameters */ + +/* +* If NULL, btrfs_drop_extents() will allocate and free its own path. +* If 'replace_extent' is true, this must not be NULL. Also the path +* is always released except if 'replace_extent' is true and +* btrfs_drop_extents() sets 'extent_inserted' to true, in which case +* the path is kept locked. +*/ +struct btrfs_path *path; +/* Start offset of the range to drop extents from. */ +u64 start; +/* End (exclusive, last byte + 1) of the range to drop extents from. */ +u64 end; +/* If true drop all the extent maps in the range. */ +bool drop_cache; +/* +* If true it means we want to insert a new extent after dropping all +* the extents in the range. If this is true, the 'extent_item_size' +* parameter must be set as well and the 'extent_inserted' field will +* be set to true by btrfs_drop_extents() if it could insert the new +* extent. +* Note: when this is set to true the path must not be NULL. +*/ +bool replace_extent; +/* +* Used if 'replace_extent' is true. Size of the file extent item to +* insert after dropping all existing extents in the range. +*/ +u32 extent_item_size; + +/* Output parameters */ + +/* +* Set to the minimum between the input parameter 'end' and the end +* (exclusive, last byte + 1) of the last dropped extent. This is always +* set even if btrfs_drop_extents() returns an error. +*/ +u64 drop_end; +/* +* Only set if 'replace_extent' is true. Set to true if we were able +* to insert a replacement extent after dropping all extents in the +* range, otherwise set to false by btrfs_drop_extents(). +* Also, if btrfs_drop_extents() has set this to true it means it +* returned with the path locked, otherwise if it has set this to +* false it has returned with the path released. +*/ +bool extent_inserted; +}; +struct btrfs_file_private {void *filldir_buf; };
@@ -3116,16 +3168,9 @@ int btrfs_sync_file(struct file *file, loff_t start, loff_t end, int datasync);void btrfs_drop_extent_cache (struct btrfs_inode *inode,u64 start,u64 end,int skip_pinned);extern const struct file_operations btrfs_file_operations; -int __btrfs_drop_extents (struct btrfs_trans_handle *trans, -struct btrfs_root *root,struct btrfs_inode *inode, -struct btrfs_path *path,u64 start,u64 end, -u64 *drop_end,int drop_cache, -int replace_extent, -u32 extent_item_size, -int *key_inserted);int btrfs_drop_extents (struct btrfs_trans_handle *trans, -struct btrfs_root *root,struct inode *inode,u64 start, -u64 end,int drop_cache); +struct btrfs_root *root,struct btrfs_inode *inode, +struct btrfs_drop_extents_args *args); int btrfs_replace_file_extents(struct inode *inode, struct btrfs_path *path, const u64 start, const u64 end, struct btrfs_replace_extent_info *extent_info,
fs/btrfs/file.c +81 -74
372 lines not on this page · raw message
fs/btrfs/inode.c +26 -15
103 lines not on this page · raw message
fs/btrfs/reflink.c +6 -1
@@ -163,6 +163,7 @@ static int clone_copy_inline_extent(struct inode *dst,const u64 aligned_end =ALIGN (new_key->offset + datal, fs_info->sectorsize);struct btrfs_trans_handle *trans =NULL ; +struct btrfs_drop_extents_args drop_args = {0 };int ret;struct btrfs_key key;
@@ -252,7 +253,11 @@ static int clone_copy_inline_extent(struct inode *dst, trans =NULL ; goto out; } - ret =btrfs_drop_extents (trans, root, dst, drop_start, aligned_end,1 ); + drop_args.path = path; + drop_args.start = drop_start; + drop_args.end = aligned_end; + drop_args.drop_cache = true; + ret =btrfs_drop_extents (trans, root,BTRFS_I (dst), &drop_args);if (ret) goto out; ret =btrfs_insert_empty_item (trans, root, path, new_key, size);
fs/btrfs/tree-log.c +20 -8
@@ -576,6 +576,7 @@ static noinline int replay_one_extent(struct btrfs_trans_handle *trans, struct extent_buffer *eb, int slot, struct btrfs_key *key) { + struct btrfs_drop_extents_args drop_args = { 0 };struct btrfs_fs_info *fs_info = root->fs_info;int found_type;u64 extent_end;
@@ -653,7 +654,10 @@ static noinline int replay_one_extent(struct btrfs_trans_handle *trans,btrfs_release_path (path);/* drop any overlapping extents */ - ret =btrfs_drop_extents (trans, root, inode, start, extent_end,1 ); + drop_args.start = start; + drop_args.end = extent_end; + drop_args.drop_cache = true; + ret =btrfs_drop_extents (trans, root,BTRFS_I (inode), &drop_args);if (ret) goto out;
@@ -2576,6 +2580,7 @@ static int replay_one_buffer(struct btrfs_root *log, struct extent_buffer *eb,* those prealloc extents just after replaying them. */ if (S_ISREG (mode)) { +struct btrfs_drop_extents_args drop_args = {0 };struct inode *inode;u64 from;
@@ -2586,8 +2591,12 @@ static int replay_one_buffer(struct btrfs_root *log, struct extent_buffer *eb, } from =ALIGN (i_size_read (inode), root->fs_info->sectorsize); - ret =btrfs_drop_extents (wc->trans, root, inode, - from, (u64)-1 ,1 ); + drop_args.start = from; + drop_args.end = (u64)-1 ; + drop_args.drop_cache = true; + ret =btrfs_drop_extents (wc->trans, root, +BTRFS_I (inode), + &drop_args);if (!ret) {/* Update the inode's nbytes. */ ret = btrfs_update_inode(wc->trans,
@@ -4186,6 +4195,7 @@ static int log_one_extent(struct btrfs_trans_handle *trans,struct btrfs_path *path, struct btrfs_log_ctx *ctx) { + struct btrfs_drop_extents_args drop_args = { 0 };struct btrfs_root *log = root->log_root;struct btrfs_file_extent_item *fi;struct extent_buffer *leaf;
@@ -4194,19 +4204,21 @@ static int log_one_extent(struct btrfs_trans_handle *trans,u64 extent_offset = em->start - em->orig_start;u64 block_len;int ret; -int extent_inserted =0 ; ret =log_extent_csums (trans, inode, log, em, ctx);if (ret)return ret; - ret =__btrfs_drop_extents (trans, log, inode, path, em->start, - em->start + em->len,NULL ,0 ,1 , -sizeof (*fi), &extent_inserted); + drop_args.path = path; + drop_args.start = em->start; + drop_args.end = em->start + em->len; + drop_args.replace_extent = true; + drop_args.extent_item_size =sizeof (*fi); + ret =btrfs_drop_extents (trans, log, inode, &drop_args);if (ret)return ret; -if (!extent_inserted) { +if (!drop_args.extent_inserted) { key.objectid =btrfs_ino (inode); key.type =BTRFS_EXTENT_DATA_KEY ; key.offset = em->start;
Signature · 1 line
2.28.0
PATCH 3/4 btrfs: fix race when defragging that leads to unnecessary IO
In reply to [email protected]From: Filipe Manana <[email protected]>
When defragging we skip ranges that have holes or inline extents, so that we don't do unnecessary IO and waste space. We do this check when calling should_defrag_range() at btrfs_defrag_file(). However we do it without holding the inode's lock. The reason we do it like this is to avoid blocking other tasks for too long, that possibly want to operate on other file ranges, since after the call to should_defrag_range() and before locking the inode, we trigger a synchronous page cache readahead. However before we were able to lock the inode, some other task might have punched a hole in our range, or we may now have an inline extent there, in which case we should not set the range for defrag anymore since that would cause unnecessary IO and make us waste space (i.e. allocating extents to contain zeros for a hole).
So after we locked the inode and the range in the iotree, check again if we have holes or an inline extent, and if we do, just skip the range.
I hit this while testing my next patch that fixes races when updating an inode's number of bytes (subject "btrfs: update the number of bytes used by an inode atomically"), and it depends on this change in order to work correctly. Alternatively I could rework that other patch to detect holes and flag their range with the 'new delalloc' bit, but this itself fixes an efficiency problem due a race that from a functional point of view is not harmful (it could be triggered with btrfs/062 from fstests).
--- fs/btrfs/ioctl.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
fs/btrfs/ioctl.c +39 -0
@@ -1275,6 +1275,7 @@ static int cluster_pages_for_defrag(struct inode *inode,u64 page_end;u64 page_cnt;u64 start = (u64 )start_index <<PAGE_SHIFT ; +u64 search_start;int ret;int i;int i_done;
@@ -1371,6 +1372,40 @@ static int cluster_pages_for_defrag(struct inode *inode,lock_extent_bits (&BTRFS_I (inode)->io_tree, page_start, page_end -1 , &cached_state); + +/* +* When defragging we skip ranges that have holes or inline extents, +* (check should_defrag_range()), to avoid unnecessary IO and wasting +* space. At btrfs_defrag_file(), we check if a range should be defragged +* before locking the inode and then, if it should, we trigger a sync +* page cache readahead - we lock the inode only after that to avoid +* blocking for too long other tasks that possibly want to operate on +* other file ranges. But before we were able to get the inode lock, +* some other task may have punched a hole in the range, or we may have +* now an inline extent, in which case we should not defrag. So check +* for that here, where we have the inode and the range locked, and bail ++ out if that happened. +*/ + search_start = page_start; +while (search_start < page_end) { +struct extent_map *em; + + em =btrfs_get_extent (BTRFS_I (inode),NULL ,0 , search_start, + page_end - search_start); +if (IS_ERR (em)) { + ret =PTR_ERR (em); + goto out_unlock_range; + } +if (em->block_start >=EXTENT_MAP_LAST_BYTE ) { +free_extent_map (em); +/* Ok, 0 means we did not defrag anything. */ + ret =0 ; + goto out_unlock_range; + } + search_start =extent_map_end (em); +free_extent_map (em); + } +clear_extent_bit (&BTRFS_I (inode)->io_tree, page_start, page_end -1 ,EXTENT_DELALLOC |EXTENT_DO_ACCOUNTING |EXTENT_DEFRAG ,0 ,0 , &cached_state);
@@ -1401,6 +1436,10 @@ static int cluster_pages_for_defrag(struct inode *inode,btrfs_delalloc_release_extents (BTRFS_I (inode), page_cnt <<PAGE_SHIFT );extent_changeset_free (data_reserved);return i_done; + +out_unlock_range: +unlock_extent_cached (&BTRFS_I (inode)->io_tree, + page_start, page_end -1 , &cached_state); out:for (i =0 ; i < i_done; i++) {unlock_page (pages[i]);
Signature · 1 line
2.28.0
PATCH 4/4 btrfs: update the number of bytes used by an inode atomically
In reply to [email protected]From: Filipe Manana <[email protected]>
There are several occasions where we do not update the inode's number of used bytes atomically, resulting in a concurrent stat(2) syscall to report a value of used blocks that does not correspond to a valid value, that is, a value that does not match neither what we had before the operation nor what we get after the operation completes.
In extreme cases it can result in stat(2) reporting zero used blocks, which can cause problems for some userspace tools where they can consider a file with a non-zero size and zero used blocks as completely sparse and skip reading data, as reported/discussed a long time ago in some threads like the following:
https://lists.gnu.org/archive/html/bug-tar/2016-07/msg00001.html
The cases where this can happen are the following:
-> Case 1
If we do a write (buffered or direct IO) against a file region for which there is already an allocated extent (or multiple extents), then we have a short time window where we can report a number of used blocks to stat(2) that does not take into account the file region being overwritten. This short time window happens when completing the ordered extent(s).
This happens because when we drop the extents in the write range we decrement the inode's number of bytes and later on when we insert the new extent(s) we increment the number of bytes in the inode, resulting in a short time window where a stat(2) syscall can get an incorrect number of used blocks.
If we do writes that overwrite an entire file, then we have a short time window where we report 0 used blocks to stat(2).
Example reproducer:
$ cat reproducer-1.sh #!/bin/bash
MNT=/mnt/sdi
DEV=/dev/sdi
stat_loop()
{
trap "wait; exit" SIGTERM
local filepath=$1
local expected=$2
local got while :; do
got=$(stat -c %b $filepath)
if [ $got -ne $expected ]; then
echo -n "ERROR: unexpected used blocks" echo " (got: $got expected: $expected)"
fi
done
}
mkfs.btrfs -f $DEV > /dev/null
# mkfs.xfs -f $DEV > /dev/null # mkfs.ext4 -F $DEV > /dev/null # mkfs.f2fs -f $DEV > /dev/null # mkfs.reiserfs -f $DEV > /dev/null mount $DEV $MNT
xfs_io -f -s -c "pwrite -b 64K 0 64K" $MNT/foobar >/dev/null expected=$(stat -c %b $MNT/foobar)
# Create a process to keep calling stat(2) on the file and see if the
# reported number of blocks used (disk space used) changes, it should # not because we are not increasing the file size nor punching holes. stat_loop $MNT/foobar $expected & loop_pid=$!
for ((i = 0; i < 50000; i++)); do
xfs_io -s -c "pwrite -b 64K 0 64K" $MNT/foobar >/dev/null
done
kill $loop_pid &> /dev/null
wait
umount $DEV
$ ./reproducer-1.sh ERROR: unexpected used blocks (got: 0 expected: 128) ERROR: unexpected used blocks (got: 0 expected: 128) (...)
Note that since this is a short time window where the race can happen, the reproducer may not be able to always trigger the bug in one run, or it may trigger it multiple times.
-> Case 2
If we do a buffered write against a file region that does not have any allocated extents, like a hole or beyond eof, then during ordered extent completion we have a short time window where a concurrent stat(2) syscall can report a number of used blocks that does not correspond to the value before or after the write operation, a value that is actually larger than the value after the write completes.
This happens because once we start a buffered write into an unallocated file range we increment the inode's 'new_delalloc_bytes', to make sure any stat(2) call gets a correct used blocks value before delalloc is flushed and completes. However at ordered extent completion, after we inserted the new extent, we increment the inode's number of bytes used with the size of the new extent, and only later, when clearing the range in the inode's iotree, we decrement the inode's 'new_delalloc_bytes' counter with the size of the extent. So this results in a short time window where a concurrent stat(2) syscall can report a number of used blocks that accounts for the new extent twice.
Example reproducer:
$ cat reproducer-2.sh #!/bin/bash
MNT=/mnt/sdi
DEV=/dev/sdi
stat_loop()
{
trap "wait; exit" SIGTERM
local filepath=$1
local expected=$2
local got while :; do
got=$(stat -c %b $filepath)
if [ $got -ne $expected ]; then
echo -n "ERROR: unexpected used blocks" echo " (got: $got expected: $expected)"
fi
done
}
mkfs.btrfs -f $DEV > /dev/null
# mkfs.xfs -f $DEV > /dev/null # mkfs.ext4 -F $DEV > /dev/null # mkfs.f2fs -f $DEV > /dev/null # mkfs.reiserfs -f $DEV > /dev/null mount $DEV $MNT
touch $MNT/foobar
write_size=$((64 * 1024))
for ((i = 0; i < 16384; i++)); do
offset=$(($i * $write_size))
xfs_io -c "pwrite -S 0xab $offset $write_size" $MNT/foobar >/dev/null
blocks_used=$(stat -c %b $MNT/foobar) # Fsync the file to trigger writeback and keep calling stat(2) on it # to see if the number of blocks used changes.
stat_loop $MNT/foobar $blocks_used &
loop_pid=$!
xfs_io -c "fsync" $MNT/foobar
kill $loop_pid &> /dev/null
wait $loop_pid
done
umount $DEV
$ ./reproducer-2.sh ERROR: unexpected used blocks (got: 265472 expected: 265344) ERROR: unexpected used blocks (got: 284032 expected: 283904) (...)
Note that since this is a short time window where the race can happen, the reproducer may not be able to always trigger the bug in one run, or it may trigger it multiple times.
-> Case 3
Another case where such problems happen is during other operations that replace extents in a file range with other extents. Those operations are extent cloning, deduplication and fallocate's zero range operation.
The cause of the problem is similar to the first case. When we drop the extents from a range, we decrement the inode's number of bytes, and later on, after inserting the new extents we increment it. Since this is not done atomically, a concurrent stat(2) call can see and return a number of used blocks that is smaller than it should be, does not match the number of used blocks before or after the clone/deduplication/zero operation.
Like for the first case, when doing a clone, deduplication or zero range operation against an entire file, we end up having a time window where we can report 0 used blocks to a stat(2) call.
Example reproducer:
$ cat reproducer-3.sh #!/bin/bash
MNT=/mnt/sdi
DEV=/dev/sdi
mkfs.btrfs -f $DEV > /dev/null
# mkfs.xfs -f -m reflink=1 $DEV > /dev/null mount $DEV $MNT
extent_size=$((64 * 1024))
num_extents=16384
file_size=$(($extent_size * $num_extents))
# File foo has many small extents.
xfs_io -f -s -c "pwrite -S 0xab -b $extent_size 0 $file_size" $MNT/foo \
> /dev/null
# File bar has much less extents and has exactly the same data as foo.
xfs_io -f -c "pwrite -S 0xab 0 $file_size" $MNT/bar > /dev/null
expected=$(stat -c %b $MNT/foo)
# Now deduplicate bar into foo. While the deduplication is in progres,
# the number of used blocks/file size reported by stat should not change
xfs_io -c "dedupe $MNT/bar 0 0 $file_size" $MNT/foo > /dev/null &
dedupe_pid=$!
while [ -n "$(ps -p $dedupe_pid -o pid=)" ]; do
used=$(stat -c %b $MNT/foo)
if [ $used -ne $expected ]; then
echo "Unexpected blocks used: $used (expected: $expected)"
fi
doneumount $DEV
$ ./reproducer-3.sh Unexpected blocks used: 2076800 (expected: 2097152) Unexpected blocks used: 2097024 (expected: 2097152) Unexpected blocks used: 2079872 (expected: 2097152) (...)
Note that since this is a short time window where the race can happen, the reproducer may not be able to always trigger the bug in one run, or it may trigger it multiple times.
So fix this by:
1) Making btrfs_drop_extents() not decrement the vfs inode's number of bytes, and instead return the number of bytes;
2) Making any code that drops extents and adds new extents update the inode's number of bytes atomically, while holding the btrfs inode's spinlock, which is also used by the stat(2) callback to get the inode's number of bytes;
3) For ranges in the inode's iotree that are marked as 'delalloc new', corresponding to previously unallocated ranges, increment the inode's number of bytes when clearing the 'delalloc new' bit from the range, in the same critical section that decrements the inode's 'new_delalloc_bytes' counter, delimited by the btrfs inode's spinlock.
An alternative would be to have btrfs_getattr() wait for any IO (ordered extents in progress) and locking the whole range (0 to (u64)-1) while it it computes the number of blocks used. But that would mean blocking stat(2), which is a very used syscall and expected to be fast, waiting for writes, clone/dedupe, fallocate, page reads, fiemap, etc.
--- fs/btrfs/btrfs_inode.h | 3 +- fs/btrfs/ctree.h | 8 +++ fs/btrfs/extent-io-tree.h | 16 ++++- fs/btrfs/extent_io.c | 10 +-- fs/btrfs/file.c | 43 +++++++----- fs/btrfs/inode.c | 135 ++++++++++++++++++++++++++++++-------- fs/btrfs/reflink.c | 2 +- fs/btrfs/tree-log.c | 4 +- 8 files changed, 171 insertions(+), 50 deletions(-)
fs/btrfs/btrfs_inode.h +2 -1
@@ -50,7 +50,8 @@ struct btrfs_inode {/* * Lock for counters and all fields used to determine if the inode is in * the log or not (last_trans, last_sub_trans, last_log_commit, -* logged_trans). +* logged_trans), to access/update new_delalloc_bytes and to update the +* vfs' inode number of bytes used. */ spinlock_t lock;
fs/btrfs/ctree.h +8 -0
@@ -1270,6 +1270,11 @@ struct btrfs_drop_extents_args {* set even if btrfs_drop_extents() returns an error. */ u64 drop_end; +/* +* The number of allocated bytes found in the range. This can be smaller +* than the range's length when there are holes in the range. +*/ +u64 bytes_found;/* * Only set if 'replace_extent' is true. Set to true if we were able * to insert a replacement extent after dropping all extents in the
@@ -3139,6 +3144,9 @@ extern const struct iomap_dio_ops btrfs_dio_ops;int btrfs_inode_lock (struct inode *inode,unsigned int ilock_flags);void btrfs_inode_unlock (struct inode *inode,unsigned int ilock_flags); +void btrfs_update_inode_bytes (struct btrfs_inode *inode, +const u64 add_bytes, +const u64 del_bytes);/* ioctl.c */ long btrfs_ioctl (struct file *file,unsigned int cmd,unsigned long arg);
fs/btrfs/extent-io-tree.h +15 -1
@@ -21,10 +21,24 @@ struct io_failure_record;#define EXTENT_NORESERVE (1U << 11)#define EXTENT_QGROUP_RESERVED (1U << 12)#define EXTENT_CLEAR_DATA_RESV (1U << 13) +/* +* Must be cleared only during ordered extent completion or on error paths if we +* did not manage to submit bios and create the ordered extents for the range. +* Should not be cleared during page release and page invalidation (if there is +* an ordered extent in flight), that is left for the ordered extent completion. +*/ #define EXTENT_DELALLOC_NEW (1U << 14) +/* +* When an ordered extent successfully completes for a region marked as a new +* delalloc range, use this flag when clearing a new delalloc range to indicate +* that the vfs' inode number of bytes should be incremented and the inode's new +* delalloc bytes decremented, in an atomic way to prevent races with stat(2). +*/ +#define EXTENT_ADD_INODE_BYTES (1U << 15)#define EXTENT_DO_ACCOUNTING (EXTENT_CLEAR_META_RESV | \ EXTENT_CLEAR_DATA_RESV) -#define EXTENT_CTLBITS (EXTENT_DO_ACCOUNTING) +#define EXTENT_CTLBITS (EXTENT_DO_ACCOUNTING | \ + EXTENT_ADD_INODE_BYTES)/* * Redefined bits above which are used only in the device allocation tree,
fs/btrfs/extent_io.c +6 -4
@@ -4414,12 +4414,14 @@ static int try_release_extent_state(struct extent_io_tree *tree, ret =0 ; }else {/* -* at this point we can safely clear everything except the -* locked bit and the nodatasum bit +* At this point we can safely clear everything except the +* locked bit, the nodatasum bit and the delalloc new bit. +* The delalloc new bit will be cleared by ordered extent +* completion. */ ret =__clear_extent_bit (tree, start, end, - ~(EXTENT_LOCKED |EXTENT_NODATASUM ), -0 ,0 ,NULL , mask,NULL ); + ~(EXTENT_LOCKED |EXTENT_NODATASUM |EXTENT_DELALLOC_NEW ), +0 ,0 ,NULL , mask,NULL );/* if clear_extent_bit failed for enomem reasons, * we can't allow the release to continue.
fs/btrfs/file.c +28 -15
124 lines not on this page · raw message
fs/btrfs/inode.c +108 -27
284 lines not on this page · raw message
fs/btrfs/reflink.c +1 -1
@@ -268,7 +268,7 @@ static int clone_copy_inline_extent(struct inode *dst,btrfs_item_ptr_offset (path->nodes[0 ], path->slots[0 ]), size); -inode_add_bytes (dst, datal); +btrfs_update_inode_bytes (BTRFS_I (dst), datal, drop_args.bytes_found);set_bit (BTRFS_INODE_NEEDS_FULL_SYNC , &BTRFS_I (dst)->runtime_flags); ret =btrfs_inode_set_file_extent_range (BTRFS_I (dst),0 , aligned_end); out:
fs/btrfs/tree-log.c +3 -1
@@ -832,8 +832,8 @@ static noinline int replay_one_extent(struct btrfs_trans_handle *trans,if (ret) goto out; -inode_add_bytes (inode, nbytes); update_inode: +btrfs_update_inode_bytes (BTRFS_I (inode), nbytes, drop_args.bytes_found); ret =btrfs_update_inode (trans, root, inode); out:if (inode)
@@ -2598,6 +2598,8 @@ static int replay_one_buffer(struct btrfs_root *log, struct extent_buffer *eb,BTRFS_I (inode), &drop_args);if (!ret) { +inode_sub_bytes (inode, + drop_args.bytes_found);/* Update the inode's nbytes. */ ret =btrfs_update_inode (wc->trans, root, inode);
Signature · 1 line
2.28.0
Re: [PATCH 1/4] btrfs: fix missing delalloc new bit for new delalloc ranges
In reply to [email protected]Quoted 70 lines · On 11/4/20 6:07 AM, [email protected] wrote:
From: Filipe Manana <[email protected]>
When doing a buffered write, through one of the write family syscalls, we look for ranges which currenly don't have allocated extents and set the 'delalloc new' bit on them, so that we can report a correct number of used blocks to the stat(2) syscall until delalloc is flushed and ordered extents complete.
However there are a few other places where we can do a buffered write against a range that is mapped to a hole (no extent allocated) and where we do not set the 'new delalloc' bit. Those places are:
- Doing a memory mapped write against a hole;
- Cloning an inline extent into a hole starting at file offset 0;
- Calling
btrfs_cont_expand()when thei_sizeof the file is not aligned to the sector size and is located in a hole. For example when cloning to a destination offset beyond eof.So after such cases, until the corresponding delalloc range is flushed and the respective ordered extents complete, we can report an incorrect number of blocks used through the stat(2) syscall.
In some cases we can end up reporting 0 used blocks to stat(2), which is a particular bad value to report as it may mislead tools to think a file is completely sparse when its
i_sizeis not zero, making them skip reading any data, an undesired consequence for tools such as archivers and other backup tools, as reported a long time ago in the following thread (and other past threads):https://lists.gnu.org/archive/html/bug-tar/2016-07/msg00001.htmlExample reproducer:
$ cat reproducer.sh #!/bin/bashMNT=/mnt/sdi
DEV=/dev/sdimkfs.btrfs -f $DEV >
/dev/null# mkfs.xfs -f $DEV > /dev/null # mkfs.ext4 -F $DEV > /dev/null # mkfs.f2fs -f $DEV > /dev/null mount $DEV $MNTxfs_io -f -c "truncate 64K" \ -c "mmap -w 0 64K" \ -c "mwrite -S 0xab 0 64K" \ -c "munmap" \ $MNT/foo blocks_used=$(stat -c %b $MNT/foo) echo "blocks used: $blocks_used"if [ $blocks_used -eq 0 ]; then
echo "ERROR: blocks used is 0"
fiumount $DEV
$ ./reproducer.sh blocks used: 0 ERROR: blocks used is 0So move the logic that decides to set the 'delalloc bit' bit into the function
btrfs_set_extent_delalloc(), since that is what we use for all those missing cases as well as for the cases that currently work well.
Quoted 5 lines
This change is also preparatory work for an upcoming patch that fixes other problems related to tracking and reporting the number of bytes used by an inode.
Signed-off-by Filipe Manana <[email protected]>
Thanks,
Josef
Re: [PATCH 2/4] btrfs: refactor btrfs_drop_extents() to make it easier to extend
In reply to [email protected]Quoted 15 lines · On 11/4/20 6:07 AM, [email protected] wrote:
From: Filipe Manana <[email protected]>
There are many arguments for
__btrfs_drop_extents()and its wrapperbtrfs_drop_extents(), which makes it hard to add more arguments to it and requires changing every caller. I have added a couple myself back in 2014 commit1acae57b161e("Btrfs: faster file extent item replace operations") and therefore know firsthand that it is a bit cumbersome to add additional arguments to these functions.Since I will need to add more arguments in a subsequent bug fix, this change is preparatory work and adds a data structure that holds all the arguments, for both input and output, that are passed to this function, with some comments in the structure's definition mentioning what each field is and how it relates to other fields.
Quoted 7 lines
Callers of this function need only to zero out the content of the structure and setup only the fields they need. This also removes the need to have both
__btrfs_drop_extents()andbtrfs_drop_extents(), so now we have a single function namedbtrfs_drop_extents()that takes a pointer to this new data structure (structbtrfs_drop_extents_args).Signed-off-by Filipe Manana <[email protected]>
Thanks,
Josef
Re: [PATCH 3/4] btrfs: fix race when defragging that leads to unnecessary IO
In reply to [email protected]Quoted 26 lines · On 11/4/20 6:07 AM, [email protected] wrote:
From: Filipe Manana <[email protected]>
When defragging we skip ranges that have holes or inline extents, so that we don't do unnecessary IO and waste space. We do this check when calling
should_defrag_range()atbtrfs_defrag_file(). However we do it without holding the inode's lock. The reason we do it like this is to avoid blocking other tasks for too long, that possibly want to operate on other file ranges, since after the call toshould_defrag_range()and before locking the inode, we trigger a synchronous page cache readahead. However before we were able to lock the inode, some other task might have punched a hole in our range, or we may now have an inline extent there, in which case we should not set the range for defrag anymore since that would cause unnecessary IO and make us waste space (i.e. allocating extents to contain zeros for a hole).So after we locked the inode and the range in the iotree, check again if we have holes or an inline extent, and if we do, just skip the range.
I hit this while testing my next patch that fixes races when updating an inode's number of bytes (subject "btrfs: update the number of bytes used by an inode atomically"), and it depends on this change in order to work correctly. Alternatively I could rework that other patch to detect holes and flag their range with the 'new delalloc' bit, but this itself fixes an efficiency problem due a race that from a functional point of view is not harmful (it could be triggered with btrfs/062 from fstests).
Quoted 1 line
Signed-off-by Filipe Manana <[email protected]>
Thanks,
Josef
Re: [PATCH 4/4] btrfs: update the number of bytes used by an inode atomically
In reply to [email protected]Quoted 252 lines · On 11/4/20 6:07 AM, [email protected] wrote:
From: Filipe Manana <[email protected]>
There are several occasions where we do not update the inode's number of used bytes atomically, resulting in a concurrent stat(2) syscall to report a value of used blocks that does not correspond to a valid value, that is, a value that does not match neither what we had before the operation nor what we get after the operation completes.
In extreme cases it can result in stat(2) reporting zero used blocks, which can cause problems for some userspace tools where they can consider a file with a non-zero size and zero used blocks as completely sparse and skip reading data, as reported/discussed a long time ago in some threads like the following:
https://lists.gnu.org/archive/html/bug-tar/2016-07/msg00001.htmlThe cases where this can happen are the following:
-> Case 1
If we do a write (buffered or direct IO) against a file region for which there is already an allocated extent (or multiple extents), then we have a short time window where we can report a number of used blocks to stat(2) that does not take into account the file region being overwritten. This short time window happens when completing the ordered extent(s).
This happens because when we drop the extents in the write range we decrement the inode's number of bytes and later on when we insert the new extent(s) we increment the number of bytes in the inode, resulting in a short time window where a stat(2) syscall can get an incorrect number of used blocks.
If we do writes that overwrite an entire file, then we have a short time window where we report 0 used blocks to stat(2).
Example reproducer:
$ cat reproducer-1.sh #!/bin/bashMNT=/mnt/sdi
DEV=/dev/sdistat_loop() { trap "wait; exit" SIGTERM local filepath=$1 local expected=$2 local gotwhile :; do
got=$(stat -c %b $filepath)
if [ $got -ne $expected ]; then
echo -n "ERROR: unexpected used blocks" echo " (got: $got expected: $expected)"
fi
done
}mkfs.btrfs -f $DEV >
/dev/null# mkfs.xfs -f $DEV > /dev/null # mkfs.ext4 -F $DEV > /dev/null # mkfs.f2fs -f $DEV > /dev/null # mkfs.reiserfs -f $DEV > /dev/null mount $DEV $MNTxfs_io -f -s -c "pwrite -b 64K 0 64K" $MNT/foobar >/dev/null expected=$(stat -c %b $MNT/foobar)# Create a process to keep calling stat(2) on the file and see if the
# reported number of blocks used (disk space used) changes, it should # not because we are not increasing the file size nor punching holes. stat_loop $MNT/foobar $expected & loop_pid=$!for ((i = 0; i < 50000; i++)); do xfs_io -s -c "pwrite -b 64K 0 64K" $MNT/foobar >/dev/null done kill $loop_pid &> /dev/null wait umount $DEV $ ./reproducer-1.sh ERROR: unexpected used blocks (got: 0 expected: 128) ERROR: unexpected used blocks (got: 0 expected: 128) (...)Note that since this is a short time window where the race can happen, the reproducer may not be able to always trigger the bug in one run, or it may trigger it multiple times.
-> Case 2
If we do a buffered write against a file region that does not have any allocated extents, like a hole or beyond eof, then during ordered extent completion we have a short time window where a concurrent stat(2) syscall can report a number of used blocks that does not correspond to the value before or after the write operation, a value that is actually larger than the value after the write completes.
This happens because once we start a buffered write into an unallocated file range we increment the inode's '
new_delalloc_bytes', to make sure any stat(2) call gets a correct used blocks value before delalloc is flushed and completes. However at ordered extent completion, after we inserted the new extent, we increment the inode's number of bytes used with the size of the new extent, and only later, when clearing the range in the inode's iotree, we decrement the inode's 'new_delalloc_bytes' counter with the size of the extent. So this results in a short time window where a concurrent stat(2) syscall can report a number of used blocks that accounts for the new extent twice.Example reproducer:
$ cat reproducer-2.sh #!/bin/bashMNT=/mnt/sdi
DEV=/dev/sdistat_loop() { trap "wait; exit" SIGTERM local filepath=$1 local expected=$2 local gotwhile :; do
got=$(stat -c %b $filepath)
if [ $got -ne $expected ]; then
echo -n "ERROR: unexpected used blocks" echo " (got: $got expected: $expected)"
fi
done
}mkfs.btrfs -f $DEV >
/dev/null# mkfs.xfs -f $DEV > /dev/null # mkfs.ext4 -F $DEV > /dev/null # mkfs.f2fs -f $DEV > /dev/null # mkfs.reiserfs -f $DEV > /dev/null mount $DEV $MNTtouch $MNT/foobar write_size=$((64 * 1024)) for ((i = 0; i < 16384; i++)); do offset=$(($i * $write_size)) xfs_io -c "pwrite -S 0xab $offset $write_size" $MNT/foobar >/dev/null blocks_used=$(stat -c %b $MNT/foobar)# Fsync the file to trigger writeback and keep calling stat(2) on it # to see if the number of blocks used changes.
stat_loop$MNT/foobar $blocks_used &
loop_pid=$!
xfs_io-c "fsync" $MNT/foobarkill $loop_pid &>
/dev/null
wait $loop_pid
doneumount $DEV
$ ./reproducer-2.sh ERROR: unexpected used blocks (got: 265472 expected: 265344) ERROR: unexpected used blocks (got: 284032 expected: 283904) (...)Note that since this is a short time window where the race can happen, the reproducer may not be able to always trigger the bug in one run, or it may trigger it multiple times.
-> Case 3
Another case where such problems happen is during other operations that replace extents in a file range with other extents. Those operations are extent cloning, deduplication and fallocate's zero range operation.
The cause of the problem is similar to the first case. When we drop the extents from a range, we decrement the inode's number of bytes, and later on, after inserting the new extents we increment it. Since this is not done atomically, a concurrent stat(2) call can see and return a number of used blocks that is smaller than it should be, does not match the number of used blocks before or after the clone/deduplication/zero operation.
Like for the first case, when doing a clone, deduplication or zero range operation against an entire file, we end up having a time window where we can report 0 used blocks to a stat(2) call.
Example reproducer:
$ cat reproducer-3.sh #!/bin/bashMNT=/mnt/sdi DEV=/dev/sdi mkfs.btrfs -f $DEV > /dev/null # mkfs.xfs -f -m reflink=1 $DEV > /dev/null mount $DEV $MNT extent_size=$((64 * 1024)) num_extents=16384 file_size=$(($extent_size * $num_extents)) # File foo has many small extents. xfs_io -f -s -c "pwrite -S 0xab -b $extent_size 0 $file_size" $MNT/foo \ > /dev/null # File bar has much less extents and has exactly the same data as foo. xfs_io -f -c "pwrite -S 0xab 0 $file_size" $MNT/bar > /dev/null expected=$(stat -c %b $MNT/foo) # Now deduplicate bar into foo. While the deduplication is in progres, # the number of used blocks/file size reported by stat should not change xfs_io -c "dedupe $MNT/bar 0 0 $file_size" $MNT/foo > /dev/null & dedupe_pid=$! while [ -n "$(ps -p $dedupe_pid -o pid=)" ]; do used=$(stat -c %b $MNT/foo) if [ $used -ne $expected ]; then echo "Unexpected blocks used: $used (expected: $expected)" fi done umount $DEV $ ./reproducer-3.sh Unexpected blocks used: 2076800 (expected: 2097152) Unexpected blocks used: 2097024 (expected: 2097152) Unexpected blocks used: 2079872 (expected: 2097152) (...)Note that since this is a short time window where the race can happen, the reproducer may not be able to always trigger the bug in one run, or it may trigger it multiple times.
So fix this by:
1) Making
btrfs_drop_extents()not decrement the vfs inode's number ofbytes, and instead return the number of bytes;2) Making any code that drops extents and adds new extents update the inode's number of bytes atomically, while holding the btrfs inode's spinlock, which is also used by the stat(2) callback to get the inode's number of bytes;
3) For ranges in the inode's iotree that are marked as 'delalloc new', corresponding to previously unallocated ranges, increment the inode's number of bytes when clearing the 'delalloc new' bit from the range, in the same critical section that decrements the inode's '
new_delalloc_bytes' counter, delimited by the btrfs inode's spinlock.
Quoted 7 lines
An alternative would be to have
btrfs_getattr()wait for any IO (ordered extents in progress) and locking the whole range (0 to (u64)-1) while it it computes the number of blocks used. But that would mean blocking stat(2), which is a very used syscall and expected to be fast, waiting for writes, clone/dedupe, fallocate, page reads, fiemap, etc.Signed-off-by Filipe Manana <[email protected]>
Thanks,
Josef
Re: [PATCH 4/4] btrfs: update the number of bytes used by an inode atomically
In reply to [email protected]Quoted 2 lines · On 4.11.20 г. 13:07 ч., [email protected] wrote:
From: Filipe Manana <[email protected]>
<snip>
Quoted 9 lines
So fix this by:
1) Making
btrfs_drop_extents()not decrement the vfs inode's number of bytes, and instead return the number of bytes;2) Making any code that drops extents and adds new extents update the inode's number of bytes atomically, while holding the btrfs inode's spinlock, which is also used by the stat(2) callback to get the inode's number of bytes;
Since synchronization is going to be provided by the btrfs_inode's lock, then how about switching to __inode_(sub|add)_bytes functions which do not take the vfs_inode's lock? Or do we need both (btrfs' and vfs' inode locks) ?
Quoted 14 lines
3) For ranges in the inode's iotree that are marked as 'delalloc new', corresponding to previously unallocated ranges, increment the inode's number of bytes when clearing the 'delalloc new' bit from the range, in the same critical section that decrements the inode's '
new_delalloc_bytes' counter, delimited by the btrfs inode's spinlock.An alternative would be to have
btrfs_getattr()wait for any IO (ordered extents in progress) and locking the whole range (0 to (u64)-1) while it it computes the number of blocks used. But that would mean blocking stat(2), which is a very used syscall and expected to be fast, waiting for writes, clone/dedupe, fallocate, page reads, fiemap, etc.Signed-off-by Filipe Manana <[email protected]>
<snip>
Re: [PATCH 4/4] btrfs: update the number of bytes used by an inode atomically
Quoted 19 lines · Nikolay Borisov
Quoted 2 lines · On 4.11.20 г. 13:07 ч., [email protected] wrote:
From: Filipe Manana <[email protected]>
<snip>
Quoted 9 lines
So fix this by:
1) Making
btrfs_drop_extents()not decrement the vfs inode's number of bytes, and instead return the number of bytes;2) Making any code that drops extents and adds new extents update the inode's number of bytes atomically, while holding the btrfs inode's spinlock, which is also used by the stat(2) callback to get the inode's number of bytes;
Quoted 4 lines
Since synchronization is going to be provided by the btrfs_inode's lock, then how about switching to __inode_(sub|add)_bytes functions which do not take the vfs_inode's lock? Or do we need both (btrfs' and vfs' inode locks) ?
Because the synchronization is only meant for the races with stat(2), and don't want to break existing users of inode_get_bytes(), specially those outside btrfs.
Thanks.
Quoted 16 lines
Quoted 15 lines
3) For ranges in the inode's iotree that are marked as 'delalloc new', corresponding to previously unallocated ranges, increment the inode's number of bytes when clearing the 'delalloc new' bit from the range, in the same critical section that decrements the inode's '
new_delalloc_bytes' counter, delimited by the btrfs inode's spinlock.An alternative would be to have
btrfs_getattr()wait for any IO (ordered extents in progress) and locking the whole range (0 to (u64)-1) while it it computes the number of blocks used. But that would mean blocking stat(2), which is a very used syscall and expected to be fast, waiting for writes, clone/dedupe, fallocate, page reads, fiemap, etc.Signed-off-by Filipe Manana <[email protected]><snip>
Quoted 13 lines · On Wed, Nov 04, 2020 at 11:07:30AM +0000, [email protected] wrote:
From: Filipe Manana <[email protected]>
There are several cases where a stat(2) call can report an incorrect number of used blocks. In some cases can even result in reporting 0 used blocks, which is a specially bad value to report when a file is not empty, and we had user bug reports in the past for such cases (the changelogs in the patches point to one such report).
This patchset addresses all those cases. The third patch fixes a race in defrag that while it does not result in a functional problem (data loss or some corruption), it leads to unnecessary IO and space allocation, and it's necessary for the 4th and final patch to work as it is.
Quoted 7 lines
A couple test cases for fstests will follow.
Filipe Manana (4):
btrfs: fix missing delalloc new bit for new delalloc ranges
btrfs: refactorbtrfs_drop_extents()to make it easier to extend
btrfs: fix race when defragging that leads to unnecessary IO
btrfs: update the number of bytes used by an inode atomically
Thanks, now added to misc-next. As the fixes seem to be important I've checked which stable kernels are relevant, so it's 4.19 and 5.4. The code does not apply due to intermediate cleanups but it should be easy to resolve.