btrfs: DEV_STATS item related minor fixes
cover btrfs: DEV_STATS item related minor fixes
[BACKGROUND]DEV_STATS (objectid, not type) items are storing the error counts for each device. It's updated every time an error is hit, including the following types:
- READ errors
- WRITE errors
- FLUSH errors
Above alls are all errors directly returned from the device.
- CORRUPTION errors
Aka, checksum mismatch.
- GENERATION errors
Metadata generation mismatch.
[MINOR BUGS]
Recently when debugging an error reported about rejected dev-replace, the device tree dump includes a DEV_STATS item for devid 0.
I'm wondering if that DEV_STATS item will ever be deleted, but unfortunately it will never be deleted.
Normally it's not a big deal, as the DEV_STATS are normally all zeros.
But if it's not, and a new dev-replace is started and interrupted, at the next mount the replace target device will suddenly inherit all the error records, giving end users false alerts about a completely good device.
[FIX]
The first 2 are manually removing the DEV_STATS items when dev-replace and dev-removal finishes.
The 3rd patch is a more generic fix by forcing btrfs to update/create DEV_STATS item after a new device is added.
This means for patched kernels, even there is some old DEV_STATS left, the kernel will never re-use them during device add or dev-replace.
The last one is to reduce unnecessary DEV_STATS updates.
Qu Wenruo (4):
btrfs: remove the dev stats item for replace target device
btrfs: remove the dev stats item when removing a device
btrfs: always update/create the dev stats item when adding a new
device
btrfs: avoid unnecessary dev stats updates
fs/btrfs/dev-replace.c | 11 +++++++++- fs/btrfs/volumes.c | 48 ++++++++++++++++++++++++++++++++++++++++-- fs/btrfs/volumes.h | 1 + 3 files changed, 57 insertions(+), 3 deletions(-)
Signature · 1 line
2.53.0
PATCH 1/4 btrfs: remove the dev stats item for replace target device
[MINOR PROBLEM]
When a running dev-replace hits some error for the target device (devid 0), there will be a DEV_STATS with error records created at the next transaction commit.
Unfortunately that item will never to be deleted.
This means at the next dev-replace, if the replace is interrupted, then at the next mount, the target device will suddenly inherit the old error records from that DEV_STATS item, which can give some false alerts on that device.
This shouldn't affect end users that much, as it requires all the following conditions to be met, which is pretty rare:
- The initial dev-replace hits some error on the target device E.g. write errors, but those errors itself is already a big problem for a running replace.
This is required to create the DEV_STATS item in the first place.
- The next replace is interrupted
This is required to allow btrfs to read from the old records.
[CAUSE]
Btrfs just never deletes the DEV_STATS after a replace is finished.
[FIX]
Remove the DEV_STATS item for devid 0 after the replace is finished.
This is not going to completely fix the error, as we still have other error paths, e.g. by somehow the fs flips RO and can not start a new transaction for the DEV_STATS item removal.
But those corner cases will be addressed by later patches which provide a more generic fix to DEV_STATS related problems.
--- fs/btrfs/dev-replace.c | 9 ++++++++- fs/btrfs/volumes.c | 32 ++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 1 + 3 files changed, 41 insertions(+), 1 deletion(-)
fs/btrfs/dev-replace.c +8 -1
@@ -1013,8 +1013,15 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,/* write back the superblocks */ trans =btrfs_start_transaction (root,0 ); -if (!IS_ERR (trans)) +if (!IS_ERR (trans)) { +/* +* Ignore any error here, if we failed to remove the +* DEV_STATS item for devid 0, it's not a big deal. +* We have other ways to address it. +*/ +btrfs_remove_dev_stat_item (trans,BTRFS_DEV_REPLACE_DEVID );btrfs_commit_transaction (trans); + }mutex_unlock (&dev_replace->lock_finishing_cancel_unmount);
fs/btrfs/volumes.c +32 -0
@@ -2286,6 +2286,38 @@ void btrfs_scratch_superblocks(struct btrfs_fs_info *fs_info, struct btrfs_devicupdate_dev_time (rcu_dereference_raw (device->name)); } +int btrfs_remove_dev_stat_item (struct btrfs_trans_handle *trans,u64 devid) +{ +BTRFS_PATH_AUTO_RELEASE (path); +struct btrfs_fs_info *fs_info = trans->fs_info; +struct btrfs_root *dev_root = fs_info->dev_root; +struct btrfs_key key; +int ret; + + key.objectid =BTRFS_DEV_STATS_OBJECTID ; + key.type =BTRFS_PERSISTENT_ITEM_KEY ; + key.offset = devid; + + ret =btrfs_search_slot (trans, dev_root, &key, &path,-1 ,1 ); +if (ret <0 ) { +btrfs_warn (fs_info, +"error %d while searching for dev_stats item for devid %llu" , + ret, devid); +return ret; + } +/* The dev stats item is not created, nothing to bother. */ +if (ret >0 ) +return 0 ; + ret =btrfs_del_item (trans, dev_root, &path); +if (ret <0 ) { +btrfs_warn (fs_info, +"error %d while deleting dev_stats item for devid %llu" , + ret, devid); +return ret; + } +return 0 ; +} +int btrfs_rm_device (struct btrfs_fs_info *fs_info,struct btrfs_dev_lookup_args *args,struct file **bdev_file)
fs/btrfs/volumes.h +1 -0
@@ -932,6 +932,7 @@ bool btrfs_first_pending_extent(struct btrfs_device *device, u64 start, u64 len, u64 *pending_start, u64 *pending_end);bool btrfs_find_hole_in_pending_extents (struct btrfs_device *device,u64 *start,u64 *len,u64 min_hole_size); +int btrfs_remove_dev_stat_item (struct btrfs_trans_handle *trans,u64 devid);#ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS struct btrfs_io_context *alloc_btrfs_io_context(struct btrfs_fs_info *fs_info,
Signature · 1 line
2.53.0
PATCH 2/4 btrfs: remove the dev stats item when removing a device
In reply to Qu Wenruo[MINOR BUG]
The following script will cause DEV_STATS item to be left after the corresponding device is removed:
# mkfs.btrfs -f $dev1 # mount $dev1 $mnt # btrfs dev add $dev2 $mnt # umount $mnt
## Without real errors, only at mount time btrfs will update ## dev->dev_stats_ccnt, thus we need a mount cycle to create the ## DEV_STATS item for the new device.
# mount $dev1 $mnt # touch $mnt/foobar # sync # btrfs dev remove $dev2 $mnt # umount $mnt
This will result the DEV_STATS item for devid 2 still left in device tree:
device tree key (DEV_TREE ROOT_ITEM 0)
leaf 31064064 items 7 free space 15788 generation 18 owner DEV_TREE leaf 31064064 flags 0x1(WRITTEN) backref revision 1
fs uuid 4bd853ed-f6ef-45fd-bbf1-1c3a2d9987cb
chunk uuid b496eab1-ec23-46b5-81c1-2f1b3503ca07
item 0 key (DEV_STATS PERSISTENT_ITEM 1) itemoff 16243 itemsize 40
persistent item objectid DEV_STATS offset 1
device stats
write_errs 0 read_errs 0 flush_errs 0 corruption_errs 0 generation 0
item 1 key (DEV_STATS PERSISTENT_ITEM 2) itemoff 16203 itemsize 40
persistent item objectid DEV_STATS offset 2
device stats
write_errs 0 read_errs 0 flush_errs 0 corruption_errs 0 generation 0
This is not a huge problem, but if the existing DEV_STATS contains errors, and a new device is added into the fs taking the old devid, then after a mount cycle, the new device will suddenly inherit old errors which can give a false alerts.
[CAUSE]
Btrfs never has the ability to delete DEV_STATS items.
It either create a new one through update_dev_stat_item(), or read an existing one through btrfs_device_init_dev_stats().
However update_dev_stat_item() is only called lazily, if a new device is created and no new update to dev stats, then it will skip the update of the on-disk item.
So if the old DEV_STATS item exists and a new device is added, and no errors during the remaining operations, the old DEV_STATS will not be updated.
Then at the next mount cycle, btrfs_device_init_dev_stats() is called at mount time, which will read out the old records, causing false alerts to the newly added device.
[FIX]
Manually remove the DEV_STATS item during btrfs_rm_device().
--- fs/btrfs/volumes.c | 6 ++++++ 1 file changed, 6 insertions(+)
fs/btrfs/volumes.c +6 -0
@@ -2397,6 +2397,12 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info,return ret; } + ret =btrfs_remove_dev_stat_item (trans, device->devid); +if (unlikely (ret)) { +btrfs_abort_transaction (trans, ret); +btrfs_end_transaction (trans); +return ret; + }clear_bit (BTRFS_DEV_STATE_IN_FS_METADATA , &device->dev_state);btrfs_scrub_cancel_dev (device);
Signature · 1 line
2.53.0
PATCH 3/4 btrfs: always update/create the dev stats item when adding a new device
In reply to Qu Wenruo[MINOR PROBLEM]
When adding a new btrfs device, the corresponding DEV_STATS item creation can only triggered by a mount cycle if there is no other error triggered:
# mkfs.btrfs -f $dev1 $mnt # mount $dev1 $mnt # btrfs dev add $dev2 $mnt # sync # btrfs ins dump-tree -t dev $dev1 device tree key (DEV_TREE ROOT_ITEM 0) leaf 30588928 items 6 free space 15853 generation 9 owner DEV_TREE item 0 key (DEV_STATS PERSISTENT_ITEM 1) itemoff 16243 itemsize 40 <<< persistent item objectid DEV_STATS offset 1 device stats write_errs 0 read_errs 0 flush_errs 0 corruption_errs 0 generation 0 item 1 key (1 DEV_EXTENT 13631488) itemoff 16195 itemsize 48
Only after a mount cycle and a new transaction, the DEV_STATS for devid 2 can show up:
# umount $mnt # mount $dev1 $mnt # touch $mnt # sync # btrfs ins dump-tree -t dev $dev1 device tree key (DEV_TREE ROOT_ITEM 0) leaf 30605312 items 7 free space 15788 generation 10 owner DEV_TREE item 0 key (DEV_STATS PERSISTENT_ITEM 1) itemoff 16243 itemsize 40 persistent item objectid DEV_STATS offset 1 device stats write_errs 0 read_errs 0 flush_errs 0 corruption_errs 0 generation 0 item 1 key (DEV_STATS PERSISTENT_ITEM 2) itemoff 16203 itemsize 40 persistent item objectid DEV_STATS offset 2 device stats write_errs 0 read_errs 0 flush_errs 0 corruption_errs 0 generation 0
[CAUSE]
Btrfs only updates the DEV_STATS item when the device->dev_stats_ccnt counter is not 0.
This is to reduce COW for the device tree. However that dev_stats_ccnt is only increased at the following call sites:
- btrfs_dev_stat_inc()
This happens when some IO error happened.
- btrfs_dev_stat_read_and_reset()
This happens for GET_DEV_STATS ioctl with BTRFS_DEV_STATS_RESET flag.
- btrfs_dev_stat_set()
This happens inside btrfs_device_init_dev_stats().
So when a new device is added, its dev_stats_ccnt is just initialized to 0, and btrfs won't create nor update the corresponding DEV_STATS item at all.
[ENHANCEMENT]
When a new device is added, also increase the dev_stats_ccnt by one. This includes both device add ioctl and dev-replace.
This will force btrfs to create a new DEV_STATS item or update the existing one with the correct values.
This not only makes the DEV_STATS creation early, but also prevents old DEV_STATS left from older kernels to cause false alerts for the newly added device.
--- fs/btrfs/dev-replace.c | 2 ++ fs/btrfs/volumes.c | 6 ++++++ 2 files changed, 8 insertions(+)
fs/btrfs/dev-replace.c +2 -0
@@ -307,6 +307,8 @@ static int btrfs_init_dev_replace_tgtdev(struct btrfs_fs_info *fs_info, device->bdev_file = bdev_file;set_bit (BTRFS_DEV_STATE_IN_FS_METADATA , &device->dev_state);set_bit (BTRFS_DEV_STATE_REPLACE_TGT , &device->dev_state); +/* Check the comment in btrfs_init_new_device() for the reason. */ +atomic_inc (&device->dev_stats_ccnt); device->dev_stats_valid =1 ;set_blocksize (bdev_file,BTRFS_BDEV_BLOCKSIZE ); device->fs_devices = fs_devices;
fs/btrfs/volumes.c +6 -0
@@ -2927,6 +2927,12 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path device->commit_total_bytes = device->total_bytes;set_bit (BTRFS_DEV_STATE_IN_FS_METADATA , &device->dev_state);clear_bit (BTRFS_DEV_STATE_REPLACE_TGT , &device->dev_state); + +/* +* Increase dev_stats_ccnt so that corresponding DEV_STATS item +* can be created at the next transaction commit. +*/ +atomic_inc (&device->dev_stats_ccnt); device->dev_stats_valid =1 ;set_blocksize (device->bdev_file,BTRFS_BDEV_BLOCKSIZE );
Signature · 1 line
2.53.0
PATCH 4/4 btrfs: avoid unnecessary dev stats updates
In reply to Qu Wenruo[MINOR PROBLEM]
When mounting a btrfs with a valid DEV_STATS item, btrfs will always update the DEV_STATS again in the next transaction commit, even if there is no change the values.
[CAUSE]
During the mount, btrfs_device_init_dev_stats() will read out the on-disk DEV_STATS item for each device.
Then it calls btrfs_dev_stat_set() to update the in-memory structure.
However btrfs_dev_stat_set() does not only set the dev stats value, but also increase device->dev_stats_ccnt.
That member determines if we should update the device item at the next transaction commit. Since we have called btrfs_dev_stat_set() for each dev status member, dev_stats_ccnt will be non-zero and we will update the dev stats item even it doesn't change at all.
[FIX]
Instead of using btrfs_dev_stat_set() for valid on-disk DEV_STATUS values, directly call atomic_set() to set the in-memory values.
For other call sites, we still want to use btrfs_dev_stat_set() so that we will force updating/creating the dev stats item.
--- fs/btrfs/volumes.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/btrfs/volumes.c +2 -2
@@ -8181,8 +8181,8 @@ static int btrfs_device_init_dev_stats(struct btrfs_device *device,for (i =0 ; i <BTRFS_DEV_STAT_VALUES_MAX ; i++) {if (item_size >= (1 + i) *sizeof (__le64)) -btrfs_dev_stat_set (device, i, -btrfs_dev_stats_value (eb, ptr, i)); +atomic_set (device->dev_stat_values + i, +btrfs_dev_stats_value (eb, ptr, i));else btrfs_dev_stat_set (device, i,0 ); }
Signature · 1 line
2.53.0
Quoted 25 lines · Qu Wenruo
[BACKGROUND]
DEV_STATS(objectid, not type) items are storing the error counts for each device. It's updated every time an error is hit, including the following types:- READ errors
- WRITE errors
- FLUSH errors
Above alls are all errors directly returned from the device.- CORRUPTION errors
Aka, checksum mismatch.- GENERATION errors
Metadata generation mismatch.[MINOR BUGS]
Recently when debugging an error reported about rejected dev-replace, the device tree dump includes aDEV_STATSitem for devid 0.I'm wondering if that
DEV_STATSitem will ever be deleted, but unfortunately it will never be deleted.Normally it's not a big deal, as the
DEV_STATSare normally all zeros.
Quoted 8 lines
But if it's not, and a new dev-replace is started and interrupted, at the next mount the replace target device will suddenly inherit all the error records, giving end users false alerts about a completely good device.
[FIX]
The first 2 are manually removing theDEV_STATSitems when dev-replace and dev-removal finishes.
The device stat item is expected by btrfs-progs even if it's just zeros, otherwise the 'btrfs device stats' prints just 'no stats found'. This has a high potential to break any monitoring tools or scripts. So please don't delete the item, and create it empty in case it does not exist.
For the wrong stats on the replaced device the stats should be zero from the beginning of the operation.
Quoted 31 lines · David Sterba
Quoted 30 lines · Qu Wenruo
[BACKGROUND]
DEV_STATS(objectid, not type) items are storing the error counts for each device. It's updated every time an error is hit, including the following types:- READ errors
- WRITE errors
- FLUSH errors
Above alls are all errors directly returned from the device.- CORRUPTION errors
Aka, checksum mismatch.- GENERATION errors
Metadata generation mismatch.[MINOR BUGS]
Recently when debugging an error reported about rejected dev-replace, the device tree dump includes aDEV_STATSitem for devid 0.I'm wondering if that
DEV_STATSitem will ever be deleted, but unfortunately it will never be deleted.Normally it's not a big deal, as the
DEV_STATSare normally all zeros.But if it's not, and a new dev-replace is started and interrupted, at the next mount the replace target device will suddenly inherit all the error records, giving end users false alerts about a completely good device.
Quoted 6 lines
Quoted 3 lines
[FIX]
The first 2 are manually removing theDEV_STATSitems when dev-replace and dev-removal finishes.The device stat item is expected by btrfs-progs even if it's just zeros, otherwise the 'btrfs device stats' prints just 'no stats found'.
That's not true.
Firstly when btrfs is created, there is no device stats at all, it's at the first RW mount we create the dev status item when initializing the dev status.
Secondly, all device adding ioctl has the proper update to force a dev status update, thus they will properly have one no matter what. And that's what the older kernel missing.
With those features guarding the dev status, it's pretty safe to remove the old dev status.
Quoted 3 lines
This
has a high potential to break any monitoring tools or scripts. So please don't delete the item, and create it empty in case it does not exist.
Please give me an example where it doesn't work.
The only situation where no dev stats item can be found is for the unpatched kernels with offline dev stats query.
After a new device is added, no new dev stats item is added (requires an error to create), thus it requires a new RW mount to create them.
But before that, there is no such stats item, and offline dev stats call will not find the item.
And device removal/replace won't change the situation.
For kernel GET_DEV_STATS ioctl it's not a problem at all, because kernel always grab the numbers from the in-memory structure which always exists.
Thanks,
Qu
Quoted 3 lines
For the wrong stats on the replaced device the stats should be zero from the beginning of the operation.
Quoted 62 lines · Qu Wenruo
Quoted 31 lines · David Sterba
Quoted 30 lines · Qu Wenruo
[BACKGROUND]
DEV_STATS(objectid, not type) items are storing the error counts for each device. It's updated every time an error is hit, including the following types:- READ errors
- WRITE errors
- FLUSH errors
Above alls are all errors directly returned from the device.- CORRUPTION errors
Aka, checksum mismatch.- GENERATION errors
Metadata generation mismatch.[MINOR BUGS]
Recently when debugging an error reported about rejected dev-replace, the device tree dump includes aDEV_STATSitem for devid 0.I'm wondering if that
DEV_STATSitem will ever be deleted, but unfortunately it will never be deleted.Normally it's not a big deal, as the
DEV_STATSare normally all zeros.But if it's not, and a new dev-replace is started and interrupted, at the next mount the replace target device will suddenly inherit all the error records, giving end users false alerts about a completely good device.
Quoted 6 lines
Quoted 3 lines
[FIX]
The first 2 are manually removing theDEV_STATSitems when dev-replace and dev-removal finishes.The device stat item is expected by btrfs-progs even if it's just zeros, otherwise the 'btrfs device stats' prints just 'no stats found'.
That's not true.
Firstly when btrfs is created, there is no device stats at all, it's at the first RW mount we create the dev status item when initializing the dev status.
Secondly, all device adding ioctl has the proper update to force a dev status update, thus they will properly have one no matter what. And that's what the older kernel missing.
With those features guarding the dev status, it's pretty safe to remove the old dev status.
Quoted 3 lines
This
has a high potential to break any monitoring tools or scripts. So please don't delete the item, and create it empty in case it does not exist.Please give me an example where it doesn't work.
The only situation where no dev stats item can be found is for the unpatched kernels with offline dev stats query.
After a new device is added, no new dev stats item is added (requires an error to create), thus it requires a new RW mount to create them.
But before that, there is no such stats item, and offline dev stats call will not find the item.
Quoted 12 lines
And device removal/replace won't change the situation.
For kernel
GET_DEV_STATSioctl it's not a problem at all, because kernel always grab the numbers from the in-memory structure which always exists.Thanks,
QuQuoted 3 lines
For the wrong stats on the replaced device the stats should be zero from the beginning of the operation.
Nope again, as I explained in the patches and above argument, the dev stats will be read at the next mount, for dev-replace target device although at creation time it's all zeroes in-memory, it doesn't trigger a dev stats update unless there is an error hit, so the on-disk one is never updated.
In that case, if dev-replace is interrupted (e.g. unmount), and there is an old dev stats item with errors in it, then at the next mount it will load the old records, causing false alerts.
Thanks,
Qu
Quoted 39 lines · Qu Wenruo
[BACKGROUND]
DEV_STATS(objectid, not type) items are storing the error counts for each device. It's updated every time an error is hit, including the following types:- READ errors
- WRITE errors
- FLUSH errors
Above alls are all errors directly returned from the device.- CORRUPTION errors
Aka, checksum mismatch.- GENERATION errors
Metadata generation mismatch.[MINOR BUGS]
Recently when debugging an error reported about rejected dev-replace, the device tree dump includes aDEV_STATSitem for devid 0.I'm wondering if that
DEV_STATSitem will ever be deleted, but unfortunately it will never be deleted.Normally it's not a big deal, as the
DEV_STATSare normally all zeros.But if it's not, and a new dev-replace is started and interrupted, at the next mount the replace target device will suddenly inherit all the error records, giving end users false alerts about a completely good device.
[FIX]
The first 2 are manually removing theDEV_STATSitems when dev-replace and dev-removal finishes.The 3rd patch is a more generic fix by forcing btrfs to update/create
DEV_STATSitem after a new device is added.
This means for patched kernels, even there is some oldDEV_STATSleft, the kernel will never re-use them during device add or dev-replace.
Quoted 8 lines
The last one is to reduce unnecessary
DEV_STATSupdates.Qu Wenruo (4):
btrfs: remove the dev stats item for replace target device
btrfs: remove the dev stats item when removing a device
btrfs: always update/create the dev stats item when adding a new
device
btrfs: avoid unnecessary dev stats updates
Please add the series to for-next, thanks.
Quoted 9 lines · David Sterba
Quoted 6 lines · Qu Wenruo
Qu Wenruo (4):
btrfs: remove the dev stats item for replace target device
btrfs: remove the dev stats item when removing a device
btrfs: always update/create the dev stats item when adding a new
device
btrfs: avoid unnecessary dev stats updatesPlease add the series to for-next, thanks.
Now merged to for-next.