btrfs: init device stats for seed devices
PATCH 1/1 btrfs: init device stats for seed devices
We recently started recording device stats across the fleet, and noticed a large increase in messages such as this
BTRFS warning (device dm-0): get dev_stats failed, not yet valid
on our tiers that use seed devices for their root devices. This is because we do not initialize the device stats for any seed devices if we have a sprout device and mount using that sprout device. The basic steps for reproducing are
mkfs seed device
mount seed device
fill seed device
umount seed device
btrfstune -S 1 seed device
mount seed device
btrfs device add -f sprout device /mnt/wherever
umount /mnt/wherever
mount sprout device /mnt/wherever
btrfs device stats /mnt/wherever
This will fail with the above message in dmesg.
Fix this by iterating over the fs_devices->seed if they exist in btrfs_init_dev_stats. This fixed the problem and properly reports the stats for both devices.
--- fs/btrfs/volumes.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
fs/btrfs/volumes.c +7 -1
@@ -7225,7 +7225,7 @@ int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info) path =btrfs_alloc_path ();if (!path)return -ENOMEM ; - +again:mutex_lock (&fs_devices->device_list_mutex);list_for_each_entry (device, &fs_devices->devices, dev_list) {int item_size;
@@ -7263,6 +7263,12 @@ int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info) }mutex_unlock (&fs_devices->device_list_mutex); +/* If we have seed devices we need to init those stats as well. */ +if (fs_devices->seed) { + fs_devices = fs_devices->seed; + goto again; + } +btrfs_free_path (path);return ret <0 ? ret :0 ; }
Signature · 1 line
2.24.1
Quoted 21 lines · Josef Bacik
We recently started recording device stats across the fleet, and noticed a large increase in messages such as this
BTRFS warning (device dm-0): get
dev_statsfailed, not yet validon our tiers that use seed devices for their root devices. This is because we do not initialize the device stats for any seed devices if we have a sprout device and mount using that sprout device. The basic steps for reproducing are
mkfs seed device
mount seed device
fill seed device
umount seed device
btrfstune -S 1 seed device
mount seed device
btrfs device add -f sprout device/mnt/wherever
umount/mnt/wherever
mount sprout device/mnt/wherever
btrfs device stats/mnt/wherever
Quoted 5 lines
This will fail with the above message in dmesg.
Fix this by iterating over the
fs_devices->seedif they exist inbtrfs_init_dev_stats. This fixed the problem and properly reports the stats for both devices.
All devices which are anchored at fs_devices->seed are really private to that fs_info->fs_devices created during chunk tree read which happens before init_dev_stats. So that's correct, however I'd like this to be based on the new pattern of dealing with seed devices - using the list() apis. And likely the init function would need to be refactored into 2 functions:
1. Iterating over all fs_devices->seed and the main fs_devices
2. Doing the init work on every passed fs_devices.
Quoted 32 lines
Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/volumes.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)fs/btrfs/volumes.c +7
-1@@ -7225,7 +7225,7 @@ int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info) path =btrfs_alloc_path ();if (!path)return -ENOMEM ; - +again:mutex_lock (&fs_devices->device_list_mutex);list_for_each_entry (device, &fs_devices->devices, dev_list) {int item_size;@@ -7263,6 +7263,12 @@ int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info) }mutex_unlock (&fs_devices->device_list_mutex); +/* If we have seed devices we need to init those stats as well. */ +if (fs_devices->seed) { + fs_devices = fs_devices->seed; + goto again; + } +btrfs_free_path (path);return ret <0 ? ret :0 ; }
Quoted 8 lines · Josef Bacik
We recently started recording device stats across the fleet, and noticed a large increase in messages such as this
BTRFS warning (device dm-0): get
dev_statsfailed, not yet validon our tiers that use seed devices for their root devices. This is because we do not initialize the device stats for any seed devices if we have a sprout device and mount using that sprout device.
Thanks for spotting and the fix.
Quoted 14 lines
The basic
steps for reproducing aremkfs seed device
mount seed device
fill seed device
umount seed device
btrfstune -S 1 seed device
mount seed device
btrfs device add -f sprout device/mnt/wherever
umount/mnt/wherever
mount sprout device/mnt/wherever
btrfs device stats/mnt/wherever
Quoted 6 lines
This will fail with the above message in dmesg.
Fix this by iterating over the
fs_devices->seedif they exist inbtrfs_init_dev_stats. This fixed the problem and properly reports the stats for both devices.
Also btrfs_run_dev_stats() should be updated to write seed's dev stat to ondisk.
Quoted 19 lines
Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/volumes.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)fs/btrfs/volumes.c +1
-1@@ -7225,7 +7225,7 @@ int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info) path =btrfs_alloc_path ();if (!path)return -ENOMEM ; - +again:mutex_lock (&fs_devices->device_list_mutex);list_for_each_entry (device, &fs_devices->devices, dev_list) {int item_size;
fs/btrfs/volumes.c int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info) · line 7264
Quoted 2 lines
fs/btrfs/volumes.c +0
-0@@ -7263,6 +7263,12 @@ int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info) }mutex_unlock (&fs_devices->device_list_mutex);
As we loop through the seed fs_devices. %fs_devices which was pointing to the sprout fs_devices above, shall point to the seed fs_devices and so will be holding the seed::device_list_mutex. But our threads which update the seed device such as replace and delete are using the sprout::device_list_mutex.
So it should be sprout::device_list_mutex
(fs_info->fs_devices->device_list_mutex).
The loops in btrfs_init_devices_late() and __reada_start_machine() which are read only, also holds the seed device_list_mutex, I am sending a patch to fix, after which there isn't any thread which shall be holding the seed::device_list_mutex when sprout is mounted.
Thanks, Anand
Quoted 11 lines
fs/btrfs/volumes.c +6
-0+/* If we have seed devices we need to init those stats as well. */ +if (fs_devices->seed) { + fs_devices = fs_devices->seed; + goto again; + } +btrfs_free_path (path);return ret <0 ? ret :0 ; }
Quoted 21 lines · Josef Bacik
We recently started recording device stats across the fleet, and noticed a large increase in messages such as this
BTRFS warning (device dm-0): get
dev_statsfailed, not yet validon our tiers that use seed devices for their root devices. This is because we do not initialize the device stats for any seed devices if we have a sprout device and mount using that sprout device. The basic steps for reproducing are
mkfs seed device
mount seed device
fill seed device
umount seed device
btrfstune -S 1 seed device
mount seed device
btrfs device add -f sprout device/mnt/wherever
umount/mnt/wherever
mount sprout device/mnt/wherever
btrfs device stats/mnt/wherever
Quoted 7 lines
This will fail with the above message in dmesg.
Fix this by iterating over the
fs_devices->seedif they exist inbtrfs_init_dev_stats. This fixed the problem and properly reports the stats for both devices.Signed-off-by Josef Bacik <[email protected]>
Please update the patch after the device list api changes, thanks.