Skip to contentSkip to conversation

block: Optimize disk zone resource cleanup

PATCH Applied Damien Le Moal 5 participants 5 messages Jun 7, 2024 – Jun 12, 2024

Subsystems (3)
lore
Share & tools
lore

Damien Le Moal

author maintainer
#0 lore raw

PATCH 1/1 block: Optimize disk zone resource cleanup

For zoned block devices using zone write plugging, an rcu_barrier() call is needed in disk_free_zone_resources() to synchronize freeing of zone write plugs and the destrution of the mempool used to allocate the plugs. The barrier call does slow down a little teardown of zoned block devices but should not affect teardown of regular block devices or zoned block devices that do not use zone write plugging (e.g. zoned DM devices that do not require zone append emulation).

Modify disk_free_zone_resources() to return early if we do not have a mempool to start with, that is, if the device does not use zone write plugging. This avoids the costly rcu_barrier() and speeds up disk teardown.

Fixes dd291d77cc90 ("block: Introduce zone write plugging")
---
 block/blk-zoned.c | 3 +++
 1 file changed, 3 insertions(+)
block/blk-zoned.c +3 -0
index 8f89705f5e1c..137842dbb59a 100644
@@ -1552,6 +1552,9 @@ static void disk_destroy_zone_wplugs_hash_table(struct gendisk *disk)
 
 void disk_free_zone_resources(struct gendisk *disk)
 {
+	if (!disk->zone_wplugs_pool)
+		return;
+
 	cancel_work_sync(&disk->zone_wplugs_work);
 
 	if (disk->zone_wplugs_wq) {
Signature · 1 line
2.45.2
maintainer Tested-by on v1 1/1
#2 lore raw
In reply to Damien Le Moal
Quoted 8 lines · Damien Le Moal

For zoned block devices using zone write plugging, an rcu_barrier() call is needed in disk_free_zone_resources() to synchronize freeing of zone write plugs and the destrution of the mempool used to allocate the plugs. The barrier call does slow down a little teardown of zoned block devices but should not affect teardown of regular block devices or zoned block devices that do not use zone write plugging (e.g. zoned DM devices that do not require zone append emulation).

Quoted 8 lines

Modify disk_free_zone_resources() to return early if we do not have a mempool to start with, that is, if the device does not use zone write plugging. This avoids the costly rcu_barrier() and speeds up disk teardown.

Fixes dd291d77cc90 ("block: Introduce zone write plugging")

This works.

BTW. when we remove large number of DM devices, the process spends a lot of time sleeping with this stacktrace:

__schedule+0x242/0x600
schedule+0x21/0xd0
blk_mq_freeze_queue_wait+0x55/0x90
? __wake_up+0x50/0x50
del_gendisk+0x1fc/0x320
cleanup_mapped_device+0x16c/0x180 [dm_mod]
__dm_destroy+0x145/0x1f0 [dm_mod]
dm_hash_remove_all+0x5c/0x180 [dm_mod]
? dm_hash_remove_all+0x180/0x180 [dm_mod]
remove_all+0x1a/0x30 [dm_mod]
ctl_ioctl+0x1bb/0x500 [dm_mod]
dm_compat_ctl_ioctl+0x7/0x10 [dm_mod]
__x64_compat_sys_ioctl+0x133/0x160
do_syscall_64+0x17c/0x1a0
entry_SYSCALL_64_after_hwframe+0x4b/0x53

Is there some way how to remove this wait?

Mikulas

Quoted 21 lines
---
 block/blk-zoned.c | 3 +++
 1 file changed, 3 insertions(+)
block/blk-zoned.c +3 -0
index 8f89705f5e1c..137842dbb59a 100644
@@ -1552,6 +1552,9 @@ static void disk_destroy_zone_wplugs_hash_table(struct gendisk *disk)
 
 void disk_free_zone_resources(struct gendisk *disk)
 {
+	if (!disk->zone_wplugs_pool)
+		return;
+
 	cancel_work_sync(&disk->zone_wplugs_work);
 
 	if (disk->zone_wplugs_wq) {
Signature · 1 line
2.45.2
Reviewed-by on v1 1/1
#3 lore raw
In reply to Damien Le Moal
Quoted 37 lines · Damien Le Moal

For zoned block devices using zone write plugging, an rcu_barrier() call is needed in disk_free_zone_resources() to synchronize freeing of zone write plugs and the destrution of the mempool used to allocate the plugs. The barrier call does slow down a little teardown of zoned block devices but should not affect teardown of regular block devices or zoned block devices that do not use zone write plugging (e.g. zoned DM devices that do not require zone append emulation).

Modify disk_free_zone_resources() to return early if we do not have a mempool to start with, that is, if the device does not use zone write plugging. This avoids the costly rcu_barrier() and speeds up disk teardown.

Fixes dd291d77cc90 ("block: Introduce zone write plugging")
---
 block/blk-zoned.c | 3 +++
 1 file changed, 3 insertions(+)
block/blk-zoned.c +3 -0
index 8f89705f5e1c..137842dbb59a 100644
@@ -1552,6 +1552,9 @@ static void disk_destroy_zone_wplugs_hash_table(struct gendisk *disk)
 
 void disk_free_zone_resources(struct gendisk *disk)
 {
+	if (!disk->zone_wplugs_pool)
+		return;
+
 	cancel_work_sync(&disk->zone_wplugs_work);
 
 	if (disk->zone_wplugs_wq) {
Signature · 1 line
2.45.2
maintainer Applied
#4 lore raw
In reply to Damien Le Moal
Quoted 9 lines · Damien Le Moal

For zoned block devices using zone write plugging, an rcu_barrier() call is needed in disk_free_zone_resources() to synchronize freeing of zone write plugs and the destrution of the mempool used to allocate the plugs. The barrier call does slow down a little teardown of zoned block devices but should not affect teardown of regular block devices or zoned block devices that do not use zone write plugging (e.g. zoned DM devices that do not require zone append emulation).

[...]

Applied, thanks!

[1/1] block: Optimize disk zone resource cleanup

      commit: 1933192a91be0a570663a3c6310c46e4ce3b2baa

Best regards,

Signature · 1 line
Jens Axboe