block: unify elevator changing and fix lockdep warning
Subsystems (2)
cover block: unify elevator changing and fix lockdep warning
Hello Jens,
The 1st 11 patch cleans up elevator change code, and unifying it via single helper, meantime moves kobject_add/del & debugfs register/unregister out of queue freezing & elevator_lock. This way fixes many lockdep warnings reported recently, especially since fs_reclaim is connected with freeze lock manually by commit ffa1e7ada456 ("block: Make request_queue lockdep splats show up earlier").
The other 4 ones fixes lockdep warnings from bfq and updating nr_hw_queues code.
Also the 4th patch fixes kasan warning & oops report from Shinichiro by preventing elevator switching when updating nr_hw_queues is in-progress.
Thanks,
Ming Lei (15):
block: don't call freeze queue in elevator_switch() and
elevator_disable()
block: add two helpers for registering/un-registering sched debugfs
block: move sched debugfs register into elvevator_register_queue
block: prevent elevator switch during updating nr_hw_queues
block: simplify elevator reset for updating nr_hw_queues
block: add helper of elevator_change()
block: move blk_unregister_queue() & device_del() after freeze wait
block: add struct elev_change_ctx for unifying elevator change
block: unifying elevator change
block: pass elevator_queue to elv_register_queue & unregister_queue
block: move elv_register[unregister]_queue out of elevator_lock
block: move debugfs/sysfs register out of freezing queue
block: remove several ->elevator_lock
block: move hctx cpuhp add/del out of queue freezing
block: move wbt_enable_default() out of queue freezing from
scheduler's ->exit()
block/bfq-iosched.c | 2 +- block/blk-mq-debugfs.c | 13 +-- block/blk-mq-sched.c | 39 +++++--- block/blk-mq.c | 163 +++++++++--------------------- block/blk-sysfs.c | 18 ++-- block/blk.h | 10 +- block/elevator.c | 221 ++++++++++++++++++++++++++--------------- block/elevator.h | 16 +++ block/genhd.c | 25 +---- include/linux/blk-mq.h | 10 +- 10 files changed, 256 insertions(+), 261 deletions(-)
Signature · 1 line
2.47.0
PATCH 1/15 block: don't call freeze queue in elevator_switch() and elevator_disable()
Both elevator_switch() and elevator_disable() are called from sysfs store and updating nr_hw_queue code paths only.
And in the two code paths, queue has been frozen already, so don't call freeze queue in the two functions.
--- block/elevator.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
block/elevator.c +2 -7
@@ -615,12 +615,11 @@ void elevator_init_mq(struct request_queue *q)*/ int elevator_switch (struct request_queue *q,struct elevator_type *new_e) { -unsigned int memflags;int ret; +WARN_ON_ONCE (q->mq_freeze_depth ==0 );lockdep_assert_held (&q->elevator_lock); - memflags =blk_mq_freeze_queue (q);blk_mq_quiesce_queue (q);if (q->elevator) {
@@ -641,7 +640,6 @@ int elevator_switch(struct request_queue *q, struct elevator_type *new_e) out_unfreeze:blk_mq_unquiesce_queue (q); -blk_mq_unfreeze_queue (q, memflags);if (ret) { pr_warn("elv: switch to \"%s\" failed, falling back to \"none\"\n",
@@ -653,11 +651,9 @@ int elevator_switch(struct request_queue *q, struct elevator_type *new_e)void elevator_disable (struct request_queue *q) { -unsigned int memflags; - +WARN_ON_ONCE (q->mq_freeze_depth ==0 );lockdep_assert_held (&q->elevator_lock); - memflags =blk_mq_freeze_queue (q);blk_mq_quiesce_queue (q);elv_unregister_queue (q);
@@ -668,7 +664,6 @@ void elevator_disable(struct request_queue *q)blk_add_trace_msg (q,"elv switch: none" );blk_mq_unquiesce_queue (q); -blk_mq_unfreeze_queue (q, memflags); }/*
Signature · 1 line
2.47.0
PATCH 2/15 block: add two helpers for registering/un-registering sched debugfs
In reply to Ming LeiAdd blk_mq_sched_reg_debugfs()/blk_mq_sched_unreg_debugfs() to clean up sched init/exit code a bit.
--- block/blk-mq-sched.c | 42 +++++++++++++++++++++++++++--------------- 1 file changed, 27 insertions(+), 15 deletions(-)
block/blk-mq-sched.c +27 -15
@@ -436,6 +436,30 @@ static int blk_mq_init_sched_shared_tags(struct request_queue *queue)return 0 ; } +static void blk_mq_sched_reg_debugfs (struct request_queue *q) +{ +struct blk_mq_hw_ctx *hctx; +unsigned long i; + +mutex_lock (&q->debugfs_mutex); +blk_mq_debugfs_register_sched (q); +queue_for_each_hw_ctx (q, hctx, i) +blk_mq_debugfs_register_sched_hctx (q, hctx); +mutex_unlock (&q->debugfs_mutex); +} + +static void blk_mq_sched_unreg_debugfs (struct request_queue *q) +{ +struct blk_mq_hw_ctx *hctx; +unsigned long i; + +mutex_lock (&q->debugfs_mutex); +queue_for_each_hw_ctx (q, hctx, i) +blk_mq_debugfs_unregister_sched_hctx (hctx); +blk_mq_debugfs_unregister_sched (q); +mutex_unlock (&q->debugfs_mutex); +} +/* caller must have a reference to @e, will grab another one if successful */ int blk_mq_init_sched (struct request_queue *q,struct elevator_type *e) {
@@ -469,10 +493,6 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e)if (ret) goto err_free_map_and_rqs; -mutex_lock (&q->debugfs_mutex); -blk_mq_debugfs_register_sched (q); -mutex_unlock (&q->debugfs_mutex); -queue_for_each_hw_ctx (q, hctx, i) {if (e->ops.init_hctx) { ret = e->ops.init_hctx (hctx, i);
@@ -484,10 +504,8 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e)return ret; } } -mutex_lock (&q->debugfs_mutex); -blk_mq_debugfs_register_sched_hctx (q, hctx); -mutex_unlock (&q->debugfs_mutex); } +blk_mq_sched_reg_debugfs (q);return 0 ;
@@ -526,11 +544,9 @@ void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e)unsigned long i;unsigned int flags =0 ; -queue_for_each_hw_ctx (q, hctx, i) { -mutex_lock (&q->debugfs_mutex); -blk_mq_debugfs_unregister_sched_hctx (hctx); -mutex_unlock (&q->debugfs_mutex); +blk_mq_sched_unreg_debugfs (q); +queue_for_each_hw_ctx (q, hctx, i) {if (e->type->ops.exit_hctx && hctx->sched_data) { e->type->ops.exit_hctx (hctx, i); hctx->sched_data =NULL ;
@@ -538,10 +554,6 @@ void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e) flags = hctx->flags; } -mutex_lock (&q->debugfs_mutex); -blk_mq_debugfs_unregister_sched (q); -mutex_unlock (&q->debugfs_mutex); -if (e->type->ops.exit_sched) e->type->ops.exit_sched (e);blk_mq_sched_tags_teardown (q, flags);
Signature · 1 line
2.47.0
Re: [PATCH 02/15] block: add two helpers for registering/un-registering sched debugfs
Quoted 2 lines · Ming Lei
Add blk_mq_sched_reg_debugfs()/blk_mq_sched_unreg_debugfs() to clean up sched init/exit code a bit.
This not just adds the new helpers, but also changes where the hctx registration and unregistration is called. From a quick look this looks fine, but please document it and explain why it is safe and desirable.
PATCH 3/15 block: move sched debugfs register into elvevator_register_queue
In reply to Ming Leisched debugfs shares same lifetime with scheduler's kobject, and same lock(elevator lock), so move sched debugfs register/unregister into elevator_register_queue() and elevator_unregister_queue().
Then we needn't blk_mq_debugfs_register() for us to register sched debugfs any more.
--- block/blk-mq-debugfs.c | 12 ------------ block/blk-mq-sched.c | 7 ++----- block/elevator.c | 6 ++++++ block/elevator.h | 3 +++ 4 files changed, 11 insertions(+), 17 deletions(-)
block/blk-mq-debugfs.c +0 -12
@@ -624,22 +624,10 @@ void blk_mq_debugfs_register(struct request_queue *q)debugfs_create_files (q->debugfs_dir, q, blk_mq_debugfs_queue_attrs); -/* -* blk_mq_init_sched() attempted to do this already, but q->debugfs_dir -* didn't exist yet (because we don't know what to name the directory -* until the queue is registered to a gendisk). -*/ -if (q->elevator && !q->sched_debugfs_dir) -blk_mq_debugfs_register_sched (q); - -/* Similarly, blk_mq_init_hctx() couldn't do this previously. */ queue_for_each_hw_ctx (q, hctx, i) {if (!hctx->debugfs_dir)blk_mq_debugfs_register_hctx (q, hctx); -if (q->elevator && !hctx->sched_debugfs_dir) -blk_mq_debugfs_register_sched_hctx (q, hctx); } -if (q->rq_qos) {struct rq_qos *rqos = q->rq_qos;
block/blk-mq-sched.c +2 -5
@@ -436,7 +436,7 @@ static int blk_mq_init_sched_shared_tags(struct request_queue *queue)return 0 ; } -static void blk_mq_sched_reg_debugfs (struct request_queue *q) +void blk_mq_sched_reg_debugfs (struct request_queue *q) {struct blk_mq_hw_ctx *hctx;unsigned long i;
@@ -448,7 +448,7 @@ static void blk_mq_sched_reg_debugfs(struct request_queue *q)mutex_unlock (&q->debugfs_mutex); } -static void blk_mq_sched_unreg_debugfs (struct request_queue *q) +void blk_mq_sched_unreg_debugfs (struct request_queue *q) {struct blk_mq_hw_ctx *hctx;unsigned long i;
@@ -505,7 +505,6 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) } } } -blk_mq_sched_reg_debugfs (q);return 0 ;
@@ -544,8 +543,6 @@ void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e)unsigned long i;unsigned int flags =0 ; -blk_mq_sched_unreg_debugfs (q); -queue_for_each_hw_ctx (q, hctx, i) {if (e->type->ops.exit_hctx && hctx->sched_data) { e->type->ops.exit_hctx (hctx, i);
block/elevator.c +6 -0
@@ -459,6 +459,9 @@ int elv_register_queue(struct request_queue *q, bool uevent)lockdep_assert_held (&q->elevator_lock); +if (test_bit (ELEVATOR_FLAG_REGISTERED , &e->flags)) +return 0 ; + error =kobject_add (&e->kobj, &q->disk->queue_kobj,"iosched" );if (!error) {const struct elv_fs_entry *attr = e->type->elevator_attrs;
@@ -472,6 +475,7 @@ int elv_register_queue(struct request_queue *q, bool uevent)if (uevent)kobject_uevent (&e->kobj,KOBJ_ADD ); +blk_mq_sched_reg_debugfs (q);set_bit (ELEVATOR_FLAG_REGISTERED , &e->flags); }return error;
@@ -486,6 +490,8 @@ void elv_unregister_queue(struct request_queue *q)if (e &&test_and_clear_bit (ELEVATOR_FLAG_REGISTERED , &e->flags)) {kobject_uevent (&e->kobj,KOBJ_REMOVE );kobject_del (&e->kobj); + +blk_mq_sched_unreg_debugfs (q); } }
block/elevator.h +3 -0
@@ -182,4 +182,7 @@ extern struct request *elv_rb_find(struct rb_root *, sector_t);#define rq_entry_fifo (ptr) list_entry((ptr), struct request, queuelist)#define rq_fifo_clear (rq) list_del_init(&(rq)->queuelist) +void blk_mq_sched_reg_debugfs (struct request_queue *q); +void blk_mq_sched_unreg_debugfs (struct request_queue *q); +#endif /* _ELEVATOR_H */
Signature · 1 line
2.47.0
Re: [PATCH 03/15] block: move sched debugfs register into elvevator_register_queue
block/elevator.c int elv_register_queue(struct request_queue *q, bool uevent) · line 464
Quoted 5 lines
block/elevator.c +3
-0lockdep_assert_held (&q->elevator_lock); +if (test_bit (ELEVATOR_FLAG_REGISTERED , &e->flags)) +return 0 ; +
This looks unrelate to the rest of the patch, and I also don't understand why it's needed as the callers of elv_register_queue don't change.
The rest of the patch looks good to me.
Re: [PATCH 03/15] block: move sched debugfs register into elvevator_register_queue
Quoted 8 lines · Christoph Hellwig
block/elevator.c int elv_register_queue(struct request_queue *q, bool uevent) · line 464
Quoted 5 lines
block/elevator.c +3
-0lockdep_assert_held (&q->elevator_lock); +if (test_bit (ELEVATOR_FLAG_REGISTERED , &e->flags)) +return 0 ; +This looks unrelate to the rest of the patch, and I also don't understand why it's needed as the callers of
elv_register_queuedon't change.
The check is actually missed, I will move it into one single patch.
Thanks,
Ming
PATCH 4/15 block: prevent elevator switch during updating nr_hw_queues
In reply to Ming Leiupdating nr_hw_queues is usually used for error handling code, when it doesn't make sense to allow blk-mq elevator switching, since nr_hw_queues may change, and elevator tags depends on nr_hw_queues.
Prevent elevator switch during updating nr_hw_queues by setting flag of BLK_MQ_F_UPDATE_HW_QUEUES, and use srcu to fail elevator switch during the period. Here elevator switch code is srcu reader of nr_hw_queues, and blk_mq_update_nr_hw_queues() is the writer.
This way avoids lot of trouble.
--- block/blk-mq-debugfs.c | 1 + block/blk-mq.c | 19 ++++++++++++++++++- block/elevator.c | 12 +++++++++++- include/linux/blk-mq.h | 10 +++++++++- 4 files changed, 39 insertions(+), 3 deletions(-)
block/blk-mq-debugfs.c +1 -0
@@ -180,6 +180,7 @@ static const char *const hctx_flag_name[] = {HCTX_FLAG_NAME (BLOCKING ),HCTX_FLAG_NAME (TAG_RR ),HCTX_FLAG_NAME (NO_SCHED_BY_DEFAULT ), + HCTX_FLAG_NAME(UPDATE_HW_QUEUES), };#undef HCTX_FLAG_NAME
block/blk-mq.c +18 -1
@@ -4785,12 +4785,16 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) goto out_free_srcu; } + ret =init_srcu_struct (&set->update_nr_hwq_srcu); +if (ret) + goto out_cleanup_srcu; + ret = -ENOMEM ; set->tags =kcalloc_node (set->nr_hw_queues,sizeof (struct blk_mq_tags *),GFP_KERNEL , set->numa_node);if (!set->tags) - goto out_cleanup_srcu; + goto out_cleanup_hwq_srcu;for (i =0 ; i < set->nr_maps; i++) { set->map[i].mq_map = kcalloc_node(nr_cpu_ids,
@@ -4819,6 +4823,8 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) }kfree (set->tags); set->tags =NULL ; +out_cleanup_hwq_srcu: +cleanup_srcu_struct (&set->update_nr_hwq_srcu); out_cleanup_srcu:if (set->flags &BLK_MQ_F_BLOCKING )cleanup_srcu_struct (set->srcu);
@@ -5081,7 +5087,18 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,void blk_mq_update_nr_hw_queues (struct blk_mq_tag_set *set,int nr_hw_queues) {mutex_lock (&set->tag_list_lock); +/* +* Mark us in updating nr_hw_queues for preventing switching +* elevator +* +* Elevator switch code can _not_ acquire ->tag_list_lock +*/ + set->flags |=BLK_MQ_F_UPDATE_HW_QUEUES ; +synchronize_srcu (&set->update_nr_hwq_srcu); +__blk_mq_update_nr_hw_queues (set, nr_hw_queues); + + set->flags &=BLK_MQ_F_UPDATE_HW_QUEUES ;mutex_unlock (&set->tag_list_lock); }EXPORT_SYMBOL_GPL (blk_mq_update_nr_hw_queues);
block/elevator.c +11 -1
@@ -718,9 +718,10 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf, {char elevator_name[ELV_NAME_MAX ];char *name; -int ret; +int ret, idx;unsigned int memflags;struct request_queue *q = disk->queue; +struct blk_mq_tag_set *set = q->tag_set;/* * If the attribute needs to load a module, do it before freezing the
@@ -732,6 +733,13 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf,elv_iosched_load_module (name); + idx =srcu_read_lock (&set->update_nr_hwq_srcu); + +if (set->flags &BLK_MQ_F_UPDATE_HW_QUEUES ) { + ret = -EBUSY ; + goto exit; + } + memflags =blk_mq_freeze_queue (q);mutex_lock (&q->elevator_lock); ret =elevator_change (q, name);
@@ -739,6 +747,8 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf, ret = count;mutex_unlock (&q->elevator_lock);blk_mq_unfreeze_queue (q, memflags); +exit: +srcu_read_unlock (&set->update_nr_hwq_srcu, idx);return ret; }
include/linux/blk-mq.h +9 -1
@@ -527,6 +527,7 @@ struct blk_mq_tag_set {struct mutex tag_list_lock;struct list_head tag_list;struct srcu_struct *srcu; +struct srcu_struct update_nr_hwq_srcu; };/**
@@ -681,7 +682,14 @@ enum {*/ BLK_MQ_F_NO_SCHED_BY_DEFAULT =1 <<6 , - BLK_MQ_F_MAX = 1 << 7, +/* +* True when updating nr_hw_queues is in-progress +* +* tag_set only flag, not usable for hctx +*/ +BLK_MQ_F_UPDATE_HW_QUEUES =1 <<7 , + + BLK_MQ_F_MAX = 1 << 8, };#define BLK_MQ_MAX_DEPTH (10240)
Signature · 1 line
2.47.0
Re: [PATCH 04/15] block: prevent elevator switch during updating nr_hw_queues
Quoted 1 line · Ming Lei
updating
nr_hw_queuesis usually used for error handling code, when it
Capitalize the first word of each sentence, please.
Quoted 2 lines
doesn't make sense to allow blk-mq elevator switching, since
nr_hw_queuesmay change, and elevator tags depends onnr_hw_queues.
I don't think it's really updated from error handling
- nbd does it when starting a device
- nullb can do it through debugfs
- xen-blkfront does it when resuming from a suspend
- nvme does it when resetting a controller. While error handling can escalate to it¸ it's basically probing and re-probing code
Quoted 4 lines
Prevent elevator switch during updating
nr_hw_queuesby setting flag ofBLK_MQ_F_UPDATE_HW_QUEUES, and use srcu to fail elevator switch during the period. Here elevator switch code is srcu reader ofnr_hw_queues, andblk_mq_update_nr_hw_queues()is the writer.
That being said as we generally are in a setup path I think the general idea is fine. No devices should be life yet at this point and thus no udev rules changing the scheduler should run yet.
Quoted 1 line
This way avoids lot of trouble.
Can you spell that out a bit?
Quoted 2 lines
Are we using Closes for bug reports now? I haven't really seen that anywhere.
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 5095
Quoted 13 lines
out_cleanup_srcu:block/blk-mq.c +5
-0if (set->flags &BLK_MQ_F_BLOCKING )cleanup_srcu_struct (set->srcu);@@ -5081,7 +5087,18 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,void blk_mq_update_nr_hw_queues (struct blk_mq_tag_set *set,int nr_hw_queues) {mutex_lock (&set->tag_list_lock); +/* +* Mark us in updating nr_hw_queues for preventing switching +* elevator +* +* Elevator switch code can _not_ acquire ->tag_list_lock
Please add a . at the end of a sentences. Also this should probably be something like "Mark us as in.." but I'll leave more nitpicking to the native speakers.
block/elevator.c ssize_t elv_iosched_store(struct gendisk *disk, const char *buf, · line 738
Quoted 12 lines
block/elevator.c +4
-0struct request_queue *q = disk->queue; +struct blk_mq_tag_set *set = q->tag_set;/* * If the attribute needs to load a module, do it before freezing the @@ -732,6 +733,13 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf,elv_iosched_load_module (name); + idx =srcu_read_lock (&set->update_nr_hwq_srcu); + +if (set->flags &BLK_MQ_F_UPDATE_HW_QUEUES ) {
What provides atomicity for field modifications vs reading of set->flags? i.e. does this need to switch using test/set_bit?
Quoted 7 lines
include/linux/blk-mq.h +1
-0+struct srcu_struct update_nr_hwq_srcu; };/** @@ -681,7 +682,14 @@ enum {*/ BLK_MQ_F_NO_SCHED_BY_DEFAULT = 1 << 6,
include/linux/blk-mq.h enum { · line 692
Quoted 10 lines
include/linux/blk-mq.h +8
-1- BLK_MQ_F_MAX = 1 << 7, +/* +* True when updating nr_hw_queues is in-progress +* +* tag_set only flag, not usable for hctx +*/ +BLK_MQ_F_UPDATE_HW_QUEUES =1 <<7 , + + BLK_MQ_F_MAX = 1 << 8,
Also mixing internal state with driver provided flags is always a bad idea. So this should probably be a new state field in the tag_set and not reuse flags.
Re: [PATCH 04/15] block: prevent elevator switch during updating nr_hw_queues
Quoted 9 lines · Christoph Hellwig
Quoted 1 line · Ming Lei
updating
nr_hw_queuesis usually used for error handling code, when itCapitalize the first word of each sentence, please.
Quoted 2 lines
doesn't make sense to allow blk-mq elevator switching, since
nr_hw_queuesmay change, and elevator tags depends onnr_hw_queues.I don't think it's really updated from error handling
NVMe does use it in error handling. I can remove error handling words, but the trouble doesn't change.
Quoted 6 lines
- nbd does it when starting a device
- nullb can do it through debugfs
- xen-blkfront does it when resuming from a suspend
- nvme does it when resetting a controller. While error handling can escalate to it¸ it's basically probing and re-probing code
reset is part of error handling.
Quoted 6 lines
Quoted 5 lines
Prevent elevator switch during updating
nr_hw_queuesby setting flag ofBLK_MQ_F_UPDATE_HW_QUEUES, and use srcu to fail elevator switch during the period. Here elevator switch code is srcu reader ofnr_hw_queues, andblk_mq_update_nr_hw_queues()is the writer.
Quoted 7 lines
That being said as we generally are in a setup path I think the general idea is fine. No devices should be life yet at this point and thus no udev rules changing the scheduler should run yet.
Quoted 1 line
This way avoids lot of trouble.
Can you spell that out a bit?
Quoted 6 lines
Quoted 3 lines
Are we using Closes for bug reports now? I haven't really seen that anywhere.
The blktests block/039 isn't merged yet, and the patch is posted recently.
kernel panic and kasan is triggered in this test.
Quoted 12 lines
Quoted 12 lines
out_cleanup_srcu:block/blk-mq.c +3
-0if (set->flags &BLK_MQ_F_BLOCKING )cleanup_srcu_struct (set->srcu);@@ -5081,7 +5087,18 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,void blk_mq_update_nr_hw_queues (struct blk_mq_tag_set *set,int nr_hw_queues) {mutex_lock (&set->tag_list_lock); +/* +* Mark us in updating nr_hw_queues for preventing switching +* elevator
Quoted 6 lines
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 5094
Quoted 2 lines
block/blk-mq.c +2
-0+* +* Elevator switch code can _not_ acquire ->tag_list_lock Please add a . at the end of a sentences. Also this should probably be something like "Mark us as in.." but I'll leave more nitpicking to the native speakers.
OK.
Quoted 8 lines
Quoted 8 lines
block/elevator.c +1
-0struct request_queue *q = disk->queue; +struct blk_mq_tag_set *set = q->tag_set;/* * If the attribute needs to load a module, do it before freezing the @@ -732,6 +733,13 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf,
Quoted 8 lines
block/elevator.c ssize_t elv_iosched_store(struct gendisk *disk, const char *buf, · line 738
Quoted 5 lines
block/elevator.c +3
-0elv_iosched_load_module (name); + idx =srcu_read_lock (&set->update_nr_hwq_srcu); + +if (set->flags &BLK_MQ_F_UPDATE_HW_QUEUES ) {What provides atomicity for field modifications vs reading of
set->flags? i.e. does this need to switch using test/set_bit?
WRITE is serialized via tag set lock with synchronize_srcu().
READ is covered by srcu read lock.
It is typical RCU usage, one writer vs. multiple writer.
Quoted 9 lines
Quoted 9 lines
include/linux/blk-mq.h +1
-0+struct srcu_struct update_nr_hwq_srcu; };/** @@ -681,7 +682,14 @@ enum {*/ BLK_MQ_F_NO_SCHED_BY_DEFAULT = 1 << 6,
Quoted 13 lines
include/linux/blk-mq.h enum { · line 692
Quoted 9 lines
include/linux/blk-mq.h +8
-1- BLK_MQ_F_MAX = 1 << 7, +/* +* True when updating nr_hw_queues is in-progress +* +* tag_set only flag, not usable for hctx +*/ +BLK_MQ_F_UPDATE_HW_QUEUES =1 <<7 , + + BLK_MQ_F_MAX = 1 << 8,Also mixing internal state with driver provided flags is always a bad idea. So this should probably be a new state field in the
tag_setand not reuse flags.
That is fine, but BLK_MQ_F_TAG_QUEUE_SHARED is used in this way too.
thanks,
Ming
Re: [PATCH 04/15] block: prevent elevator switch during updating nr_hw_queues
Quoted 9 lines · Ming Lei
Quoted 8 lines
block/elevator.c ssize_t elv_iosched_store(struct gendisk *disk, const char *buf, · line 738
Quoted 5 lines
block/elevator.c +3
-0elv_iosched_load_module (name); + idx =srcu_read_lock (&set->update_nr_hwq_srcu); + +if (set->flags &BLK_MQ_F_UPDATE_HW_QUEUES ) {What provides atomicity for field modifications vs reading of
set->flags? i.e. does this need to switch using test/set_bit?
Quoted 5 lines
WRITE is serialized via tag set lock with
synchronize_srcu().READ is covered by srcu read lock.
It is typical RCU usage, one writer vs. multiple writer.
No, (S)RCU does not help you with atomicy of bitfields.
Quoted 5 lines
Quoted 3 lines
Also mixing internal state with driver provided flags is always a bad idea. So this should probably be a new state field in the
tag_setand not reuse flags.That is fine, but
BLK_MQ_F_TAG_QUEUE_SHAREDis used in this way too.
Yes, that should also move over to the state field. Amd rnbd needs to be fixed to not set it diretly which is a good example for why they should not be mixed..
Re: [PATCH 04/15] block: prevent elevator switch during updating nr_hw_queues
Quoted 10 lines · Christoph Hellwig
Quoted 9 lines · Ming Lei
Quoted 8 lines
block/elevator.c ssize_t elv_iosched_store(struct gendisk *disk, const char *buf, · line 738
Quoted 5 lines
block/elevator.c +3
-0elv_iosched_load_module (name); + idx =srcu_read_lock (&set->update_nr_hwq_srcu); + +if (set->flags &BLK_MQ_F_UPDATE_HW_QUEUES ) {What provides atomicity for field modifications vs reading of
set->flags? i.e. does this need to switch using test/set_bit?
Quoted 7 lines
Quoted 5 lines
WRITE is serialized via tag set lock with
synchronize_srcu().READ is covered by srcu read lock.
It is typical RCU usage, one writer vs. multiple writer.
No, (S)RCU does not help you with atomicy of bitfields.
OK, will change it into one state variable.
Thanks,
Ming
Re: [PATCH 04/15] block: prevent elevator switch during updating nr_hw_queues
In reply to Ming LeiQuoted 6 lines · Ming Lei
block/blk-mq.c +2
-0@@ -5081,7 +5087,18 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,void blk_mq_update_nr_hw_queues (struct blk_mq_tag_set *set,int nr_hw_queues) {mutex_lock (&set->tag_list_lock); +/* +* Mark us in updating nr_hw_queues for preventing switching
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 5101
Quoted 10 lines
block/blk-mq.c +9
-0+* elevator +* +* Elevator switch code can _not_ acquire ->tag_list_lock +*/ + set->flags |=BLK_MQ_F_UPDATE_HW_QUEUES ; +synchronize_srcu (&set->update_nr_hwq_srcu); +__blk_mq_update_nr_hw_queues (set, nr_hw_queues); + + set->flags &=BLK_MQ_F_UPDATE_HW_QUEUES ;
I think here we want to clear BLK_MQ_F_UPDATE_HW_QUEUES, so we need to have set->flags updated as below:
set->flags &= ~BLK_MQ_F_UPDATE_HW_QUEUES;
Thanks,
--Nilay
Re: [PATCH 04/15] block: prevent elevator switch during updating nr_hw_queues
Quoted 23 lines · Nilay Shroff
Quoted 6 lines · Ming Lei
block/blk-mq.c +2
-0@@ -5081,7 +5087,18 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,void blk_mq_update_nr_hw_queues (struct blk_mq_tag_set *set,int nr_hw_queues) {mutex_lock (&set->tag_list_lock); +/* +* Mark us in updating nr_hw_queues for preventing switching block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 5101
Quoted 10 lines
block/blk-mq.c +9
-0+* elevator +* +* Elevator switch code can _not_ acquire ->tag_list_lock +*/ + set->flags |=BLK_MQ_F_UPDATE_HW_QUEUES ; +synchronize_srcu (&set->update_nr_hwq_srcu); +__blk_mq_update_nr_hw_queues (set, nr_hw_queues); + + set->flags &=BLK_MQ_F_UPDATE_HW_QUEUES ;I think here we want to clear
BLK_MQ_F_UPDATE_HW_QUEUES, so we need to haveset->flagsupdated as below:
Quoted 1 line
set->flags&= ~BLK_MQ_F_UPDATE_HW_QUEUES;
Good catch!
Thanks,
Ming
PATCH 5/15 block: simplify elevator reset for updating nr_hw_queues
In reply to Ming LeiIn blk_mq_update_nr_hw_queues(), nr_hw_queues may change, so elevator has to be reset after nr_hw_queues is changed.
Now elevator switch isn't allowed during blk_mq_update_nr_hw_queues(), so we can simply call elevator_change() to reset elevator sched tags after nr_hw_queues is updated.
--- block/blk-mq.c | 99 +++++------------------------------------------- block/blk.h | 4 +- block/elevator.c | 12 +++--- 3 files changed, 18 insertions(+), 97 deletions(-)
block/blk-mq.c +9 -90
@@ -4930,88 +4930,10 @@ int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr)return ret; } -/* -* request_queue and elevator_type pair. -* It is just used by __blk_mq_update_nr_hw_queues to cache -* the elevator_type associated with a request_queue. -*/ -struct blk_mq_qe_pair { -struct list_head node; -struct request_queue *q; -struct elevator_type *type; -}; - -/* -* Cache the elevator_type in qe pair list and switch the -* io scheduler to 'none' -*/ -static bool blk_mq_elv_switch_none (struct list_head *head, -struct request_queue *q) -{ -struct blk_mq_qe_pair *qe; - - qe =kmalloc (sizeof (*qe),GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY); -if (!qe) -return false; - -/* Accessing q->elevator needs protection from ->elevator_lock. */ -mutex_lock (&q->elevator_lock); - -if (!q->elevator) { -kfree (qe); - goto unlock; - } - -INIT_LIST_HEAD (&qe->node); - qe->q = q; - qe->type = q->elevator->type; -/* keep a reference to the elevator module as we'll switch back */ -__elevator_get (qe->type); -list_add (&qe->node, head); -elevator_disable (q); -unlock: -mutex_unlock (&q->elevator_lock); - -return true; -} - -static struct blk_mq_qe_pair *blk_lookup_qe_pair (struct list_head *head, -struct request_queue *q) -{ -struct blk_mq_qe_pair *qe; - -list_for_each_entry (qe, head, node) -if (qe->q == q) -return qe; - -return NULL ; -} - -static void blk_mq_elv_switch_back (struct list_head *head, -struct request_queue *q) -{ -struct blk_mq_qe_pair *qe; -struct elevator_type *t; - - qe =blk_lookup_qe_pair (head, q); -if (!qe) -return ; - t = qe->type; -list_del (&qe->node); -kfree (qe); - -mutex_lock (&q->elevator_lock); -elevator_switch (q, t); -/* drop the reference acquired in blk_mq_elv_switch_none */ -elevator_put (t); -mutex_unlock (&q->elevator_lock); -} -static void __blk_mq_update_nr_hw_queues (struct blk_mq_tag_set *set,int nr_hw_queues) {struct request_queue *q; -LIST_HEAD (head);int prev_nr_hw_queues = set->nr_hw_queues;unsigned int memflags;int i;
@@ -5029,15 +4951,6 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,list_for_each_entry (q, &set->tag_list, tag_set_list)blk_mq_freeze_queue_nomemsave (q); -/* -* Switch IO scheduler to 'none', cleaning up the data associated -* with the previous scheduler. We will switch back once we are done -* updating the new sw to hw queue mappings. -*/ -list_for_each_entry (q, &set->tag_list, tag_set_list) -if (!blk_mq_elv_switch_none (&head, q)) - goto switch_back; -list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_debugfs_unregister_hctxs (q);blk_mq_sysfs_unregister_hctxs (q);
@@ -5071,9 +4984,15 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,blk_mq_debugfs_register_hctxs (q); } -switch_back: -list_for_each_entry (q, &set->tag_list, tag_set_list) -blk_mq_elv_switch_back (&head, q); +list_for_each_entry (q, &set->tag_list, tag_set_list) { +const char *name ="none" ; + +mutex_lock (&q->elevator_lock); +if (q->elevator && !blk_queue_dying (q)) + name = q->elevator->type->elevator_name; +__elevator_change (q, name, true); +mutex_unlock (&q->elevator_lock); + }list_for_each_entry (q, &set->tag_list, tag_set_list)blk_mq_unfreeze_queue_nomemrestore (q);
block/blk.h +2 -2
@@ -319,8 +319,8 @@ bool blk_bio_list_merge(struct request_queue *q, struct list_head *list,bool blk_insert_flush (struct request *rq); -int elevator_switch (struct request_queue *q,struct elevator_type *new_e); -void elevator_disable (struct request_queue *q); +int __elevator_change (struct request_queue *q,const char *elevator_name, +bool force);void elevator_exit (struct request_queue *q);int elv_register_queue (struct request_queue *q,bool uevent);void elv_unregister_queue (struct request_queue *q);
block/elevator.c +7 -5
@@ -619,7 +619,7 @@ void elevator_init_mq(struct request_queue *q)* If switching fails, we are most likely running out of memory and not able * to restore the old io scheduler, so leaving the io scheduler being none. */ -int elevator_switch (struct request_queue *q,struct elevator_type *new_e) +static int elevator_switch (struct request_queue *q,struct elevator_type *new_e) {int ret;
@@ -655,7 +655,7 @@ int elevator_switch(struct request_queue *q, struct elevator_type *new_e)return ret; } -void elevator_disable (struct request_queue *q) +static void elevator_disable (struct request_queue *q) {WARN_ON_ONCE (q->mq_freeze_depth ==0 );lockdep_assert_held (&q->elevator_lock);
@@ -675,7 +675,8 @@ void elevator_disable(struct request_queue *q)/* * Switch this queue to the given IO scheduler. */ -static int elevator_change (struct request_queue *q,const char *elevator_name) +int __elevator_change (struct request_queue *q,const char *elevator_name, +bool force) {struct elevator_type *e;int ret;
@@ -690,7 +691,8 @@ static int elevator_change(struct request_queue *q, const char *elevator_name)return 0 ; } -if (q->elevator &&elevator_match (q->elevator->type, elevator_name)) +if (!force && q->elevator && +elevator_match (q->elevator->type, elevator_name))return 0 ; e =elevator_find_get (elevator_name);
@@ -742,7 +744,7 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf, memflags =blk_mq_freeze_queue (q);mutex_lock (&q->elevator_lock); - ret =elevator_change (q, name); + ret =__elevator_change (q, name, false);if (!ret) ret = count;mutex_unlock (&q->elevator_lock);
Signature · 1 line
2.47.0
Re: [PATCH 05/15] block: simplify elevator reset for updating nr_hw_queues
Quoted 2 lines · Ming Lei
In
blk_mq_update_nr_hw_queues(),nr_hw_queuesmay change, so elevator has to be reset afternr_hw_queuesis changed.
This should be past tense I guess?
Quoted 3 lines
Now elevator switch isn't allowed during
blk_mq_update_nr_hw_queues(), so we can simply callelevator_change()to reset elevator sched tags afternr_hw_queuesis updated.
Now that elevator switches aren't allowed during blk_mq_update_nr_hw_queues(), we can simply call elevator_change() to reset elevator sched tags after nr_hw_queues is updated.
?
Anyway, glad to see this code go away!
Re: [PATCH 05/15] block: simplify elevator reset for updating nr_hw_queues
In reply to Ming LeiQuoted 6 lines · Ming Lei
block/blk-mq.c +0
-2@@ -5071,9 +4984,15 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,blk_mq_debugfs_register_hctxs (q); } -switch_back: -list_for_each_entry (q, &set->tag_list, tag_set_list)
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 4995
Quoted 10 lines
block/blk-mq.c +9
-1-blk_mq_elv_switch_back (&head, q); +list_for_each_entry (q, &set->tag_list, tag_set_list) { +const char *name ="none" ; + +mutex_lock (&q->elevator_lock); +if (q->elevator && !blk_queue_dying (q)) + name = q->elevator->type->elevator_name; +__elevator_change (q, name, true); +mutex_unlock (&q->elevator_lock); + }
Coming back to this after looking through the next patches.
Why do we even need the __elevator_change call here? We've not actually disabled the elevator, and we prevent other callers from changing it.
As you pass in the force argument this now always calls elevator_switch and thus blk_mq_init_sched. But why?
Re: [PATCH 05/15] block: simplify elevator reset for updating nr_hw_queues
Quoted 20 lines · Christoph Hellwig
Quoted 6 lines · Ming Lei
block/blk-mq.c +0
-2@@ -5071,9 +4984,15 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,blk_mq_debugfs_register_hctxs (q); } -switch_back: -list_for_each_entry (q, &set->tag_list, tag_set_list)block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 4995
Quoted 10 lines
block/blk-mq.c +9
-1-blk_mq_elv_switch_back (&head, q); +list_for_each_entry (q, &set->tag_list, tag_set_list) { +const char *name ="none" ; + +mutex_lock (&q->elevator_lock); +if (q->elevator && !blk_queue_dying (q)) + name = q->elevator->type->elevator_name; +__elevator_change (q, name, true); +mutex_unlock (&q->elevator_lock); + }Coming back to this after looking through the next patches.
Quoted 6 lines
Why do we even need the
__elevator_changecall here? We've not actually disabled the elevator, and we prevent other callers from changing it.As you pass in the force argument this now always calls
elevator_switchand thusblk_mq_init_sched. But why?
sched tags is built over hctx and depends on ->nr_hw_queues, when nr_hw_queues is changed, sched tags has to be rebuilt.
Thanks,
Ming
Re: [PATCH 05/15] block: simplify elevator reset for updating nr_hw_queues
Quoted 11 lines · Ming Lei
Quoted 8 lines
Coming back to this after looking through the next patches.
Why do we even need the
__elevator_changecall here? We've not actually disabled the elevator, and we prevent other callers from changing it.As you pass in the force argument this now always calls
elevator_switchand thusblk_mq_init_sched. But why?sched tags is built over hctx and depends on ->nr_hw_queues, when
nr_hw_queuesis changed, sched tags has to be rebuilt.
Can you add a comment explaining this?
Re: [PATCH 05/15] block: simplify elevator reset for updating nr_hw_queues
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Ming Lei
Quoted 6 lines
Coming back to this after looking through the next patches.
Why do we even need the
__elevator_changecall here? We've not actually disabled the elevator, and we prevent other callers from changing it.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
As you pass in the force argument this now always calls
elevator_switchand thusblk_mq_init_sched. But why?sched tags is built over hctx and depends on ->nr_hw_queues, when
nr_hw_queuesis changed, sched tags has to be rebuilt.Can you add a comment explaining this?
Sure.
Thanks,
Ming
PATCH 6/15 block: add helper of elevator_change()
In reply to Ming LeiAdd elevator_change() to simplify elv_iosched_store() a bit, and the new helper will be used for unifying all scheduler change.
--- block/elevator.c | 52 +++++++++++++++++++++++++++++------------------- 1 file changed, 31 insertions(+), 21 deletions(-)
block/elevator.c +31 -21
@@ -53,6 +53,8 @@ static LIST_HEAD(elv_list);*/ #define rq_hash_key (rq) (blk_rq_pos(rq) + blk_rq_sectors(rq)) +static int elevator_change (struct request_queue *q,const char *name); +/* * Query io scheduler to see if the current process issuing bio may be * merged with rq.
@@ -681,10 +683,6 @@ int __elevator_change(struct request_queue *q, const char *elevator_name,struct elevator_type *e;int ret; -/* Make sure queue is not in the middle of being removed */ -if (!blk_queue_registered (q)) -return -ENOENT ; -if (!strncmp (elevator_name,"none" ,4 )) {if (q->elevator)elevator_disable (q);
@@ -703,6 +701,29 @@ int __elevator_change(struct request_queue *q, const char *elevator_name,return ret; } +static int elevator_change (struct request_queue *q,const char *name) +{ +int ret, idx; +unsigned int memflags; +struct blk_mq_tag_set *set = q->tag_set; + + idx =srcu_read_lock (&set->update_nr_hwq_srcu); + +if (set->flags &BLK_MQ_F_UPDATE_HW_QUEUES ) { + ret = -EBUSY ; + goto exit; + } + + memflags =blk_mq_freeze_queue (q); +mutex_lock (&q->elevator_lock); + ret =__elevator_change (q, name, false); +mutex_unlock (&q->elevator_lock); +blk_mq_unfreeze_queue (q, memflags); +exit: +srcu_read_unlock (&set->update_nr_hwq_srcu, idx); +return ret; +} +static void elv_iosched_load_module (char *elevator_name) {struct elevator_type *found;
@@ -718,12 +739,10 @@ static void elv_iosched_load_module(char *elevator_name)ssize_t elv_iosched_store (struct gendisk *disk,const char *buf,size_t count) { +struct request_queue *q = disk->queue;char elevator_name[ELV_NAME_MAX ];char *name; -int ret, idx; -unsigned int memflags; -struct request_queue *q = disk->queue; -struct blk_mq_tag_set *set = q->tag_set; +int ret;/* * If the attribute needs to load a module, do it before freezing the
@@ -735,22 +754,13 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf,elv_iosched_load_module (name); - idx =srcu_read_lock (&set->update_nr_hwq_srcu); - -if (set->flags &BLK_MQ_F_UPDATE_HW_QUEUES ) { - ret = -EBUSY ; - goto exit; - } +/* Make sure queue is not in the middle of being removed */ +if (!blk_queue_registered (q)) +return -ENOENT ; - memflags =blk_mq_freeze_queue (q); -mutex_lock (&q->elevator_lock); - ret =__elevator_change (q, name, false); + ret =elevator_change (q, name);if (!ret) ret = count; -mutex_unlock (&q->elevator_lock); -blk_mq_unfreeze_queue (q, memflags); -exit: -srcu_read_unlock (&set->update_nr_hwq_srcu, idx);return ret; }
Signature · 1 line
2.47.0
PATCH 7/15 block: move blk_unregister_queue() & device_del() after freeze wait
In reply to Ming LeiMove blk_unregister_queue() & device_del() after freeze wait, and prepare for unifying elevator switch.
This way is just fine, since bdev has been unhashed at the beginning of del_gendisk(), both blk_unregister_queue() & device_del() are dealing with kobject & debugfs thing only.
--- block/genhd.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
block/genhd.c +3 -3
@@ -721,8 +721,6 @@ void del_gendisk(struct gendisk *disk)bdi_unregister (disk->bdi); } -blk_unregister_queue (disk); -kobject_put (disk->part0->bd_holder_dir);kobject_put (disk->slave_dir); disk->slave_dir =NULL ;
@@ -731,10 +729,12 @@ void del_gendisk(struct gendisk *disk) disk->part0->bd_stamp =0 ;sysfs_remove_link (block_depr,dev_name (disk_to_dev (disk)));pm_runtime_set_memalloc_noio (disk_to_dev (disk), false); -device_del (disk_to_dev (disk));blk_mq_freeze_queue_wait (q); +blk_unregister_queue (disk); +device_del (disk_to_dev (disk)); +blk_throtl_cancel_bios (disk);blk_sync_queue (q);
Signature · 1 line
2.47.0
Re: [PATCH 07/15] block: move blk_unregister_queue() & device_del() after freeze wait
Quoted 6 lines · Ming Lei
Move
blk_unregister_queue()&device_del()after freeze wait, and prepare for unifying elevator switch.This way is just fine, since bdev has been unhashed at the beginning of
del_gendisk(), bothblk_unregister_queue()&device_del()are dealing with kobject & debugfs thing only.
While I believe this is the right thing to do, moving the freeze wait caused issues in the past, so be careful. Take a look at:
commit 4c66a326b5ab784cddd72de07ac5b6210e9e1b06
Author: Christoph Hellwig <[email protected]> Date: Mon Sep 19 16:40:49 2022 +0200
Revert "block: freeze the queue earlier in del_gendisk"
Re: [PATCH 07/15] block: move blk_unregister_queue() & device_del() after freeze wait
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Ming Lei
Move
blk_unregister_queue()&device_del()after freeze wait, and prepare for unifying elevator switch.This way is just fine, since bdev has been unhashed at the beginning of
del_gendisk(), bothblk_unregister_queue()&device_del()are dealing with kobject & debugfs thing only.
Quoted 8 lines
While I believe this is the right thing to do, moving the freeze wait caused issues in the past, so be careful. Take a look at:
commit
4c66a326b5ab784cddd72de07ac5b6210e9e1b06Author: Christoph Hellwig <[email protected]> Date: Mon Sep 19 16:40:49 2022 +0200Revert "block: freeze the queue earlier in
del_gendisk"
Yeah, I know this story.
If it is triggered again with this patch, I will help to root cause.
The last thing we still can do is to just moving blk_unregister_queue() after queue is frozen, or even elevator tear down. It is allowed to delete children kobjects after their parent is removed. Just the KOBJ_REMOVE event need to be ordered.
Thanks,
Ming
PATCH 8/15 block: add `struct elev_change_ctx` for unifying elevator change
In reply to Ming LeiAdd struct elev_change_ctx and prepare for unifying elevator change, with this way, any input & output parameter can be provided & observed in top caller.
This way also helps to move kobject & debugfs things out of ->elevator_lock.
--- block/blk-mq.c | 10 +++++++--- block/blk.h | 4 ++-- block/elevator.c | 33 +++++++++++++++++++-------------- block/elevator.h | 7 +++++++ 4 files changed, 35 insertions(+), 19 deletions(-)
block/blk-mq.c +7 -3
@@ -4985,12 +4985,16 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, }list_for_each_entry (q, &set->tag_list, tag_set_list) { -const char *name ="none" ; +struct elev_change_ctx ctx = { + .name ="none" , + .force =1 , + .uevent =1 , + };mutex_lock (&q->elevator_lock);if (q->elevator && !blk_queue_dying (q)) - name = q->elevator->type->elevator_name; -__elevator_change (q, name, true); + ctx.name = q->elevator->type->elevator_name; +__elevator_change (q, &ctx);mutex_unlock (&q->elevator_lock); }
block/blk.h +2 -2
@@ -12,6 +12,7 @@#include "blk-crypto-internal.h" struct elevator_type ; +struct elev_change_ctx ;#define BLK_DEV_MAX_SECTORS (LLONG_MAX >> 9)#define BLK_MIN_SEGMENT_SIZE 4096
@@ -319,8 +320,7 @@ bool blk_bio_list_merge(struct request_queue *q, struct list_head *list,bool blk_insert_flush (struct request *rq); -int __elevator_change (struct request_queue *q,const char *elevator_name, -bool force); +int __elevator_change (struct request_queue *q,struct elev_change_ctx *ctx);void elevator_exit (struct request_queue *q);int elv_register_queue (struct request_queue *q,bool uevent);void elv_unregister_queue (struct request_queue *q);
block/elevator.c +19 -14
@@ -53,7 +53,8 @@ static LIST_HEAD(elv_list);*/ #define rq_hash_key (rq) (blk_rq_pos(rq) + blk_rq_sectors(rq)) -static int elevator_change (struct request_queue *q,const char *name); +static int elevator_change (struct request_queue *q, +struct elev_change_ctx *ctx);/* * Query io scheduler to see if the current process issuing bio may be
@@ -621,7 +622,8 @@ void elevator_init_mq(struct request_queue *q)* If switching fails, we are most likely running out of memory and not able * to restore the old io scheduler, so leaving the io scheduler being none. */ -static int elevator_switch (struct request_queue *q,struct elevator_type *new_e) +static int elevator_switch (struct request_queue *q,struct elevator_type *new_e, +struct elev_change_ctx *ctx) {int ret;
@@ -639,7 +641,7 @@ static int elevator_switch(struct request_queue *q, struct elevator_type *new_e)if (ret) goto out_unfreeze; - ret =elv_register_queue (q, true); + ret =elv_register_queue (q, ctx->uevent);if (ret) {elevator_exit (q); goto out_unfreeze;
@@ -677,9 +679,9 @@ static void elevator_disable(struct request_queue *q)/* * Switch this queue to the given IO scheduler. */ -int __elevator_change (struct request_queue *q,const char *elevator_name, -bool force) +int __elevator_change (struct request_queue *q,struct elev_change_ctx *ctx) { +const char *elevator_name = ctx->name;struct elevator_type *e;int ret;
@@ -689,19 +691,20 @@ int __elevator_change(struct request_queue *q, const char *elevator_name,return 0 ; } -if (!force && q->elevator && +if (!ctx->force && q->elevator &&elevator_match (q->elevator->type, elevator_name))return 0 ; e =elevator_find_get (elevator_name);if (!e)return -EINVAL ; - ret =elevator_switch (q, e); + ret =elevator_switch (q, e, ctx);elevator_put (e);return ret; } -static int elevator_change (struct request_queue *q,const char *name) +static int elevator_change (struct request_queue *q, +struct elev_change_ctx *ctx) {int ret, idx;unsigned int memflags;
@@ -716,7 +719,7 @@ static int elevator_change(struct request_queue *q, const char *name) memflags =blk_mq_freeze_queue (q);mutex_lock (&q->elevator_lock); - ret =__elevator_change (q, name, false); + ret =__elevator_change (q, ctx);mutex_unlock (&q->elevator_lock);blk_mq_unfreeze_queue (q, memflags); exit:
@@ -724,7 +727,7 @@ static int elevator_change(struct request_queue *q, const char *name)return ret; } -static void elv_iosched_load_module (char *elevator_name) +static void elv_iosched_load_module (const char *elevator_name) {struct elevator_type *found;
@@ -741,7 +744,9 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf, {struct request_queue *q = disk->queue;char elevator_name[ELV_NAME_MAX ]; -char *name; +struct elev_change_ctx ctx = { + .uevent =1 , + };int ret;/*
@@ -750,15 +755,15 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf,* queue is the one for the device storing the module file. */ strscpy (elevator_name, buf,sizeof (elevator_name)); - name =strstrip (elevator_name); + ctx.name =strstrip (elevator_name); -elv_iosched_load_module (name); +elv_iosched_load_module (ctx.name);/* Make sure queue is not in the middle of being removed */ if (!blk_queue_registered (q))return -ENOENT ; - ret =elevator_change (q, name); + ret =elevator_change (q, &ctx);if (!ret) ret = count;return ret;
block/elevator.h +7 -0
@@ -123,6 +123,13 @@ struct elevator_queue#define ELEVATOR_FLAG_REGISTERED 0#define ELEVATOR_FLAG_DISABLE_WBT 1 +/* Holding data for changing elevator */ +struct elev_change_ctx { +const char *name; +unsigned int force:1 ; +unsigned int uevent:1 ; +}; +/* * block elevator interface */
Signature · 1 line
2.47.0
Re: [PATCH 08/15] block: add `struct elev_change_ctx` for unifying elevator change
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 4998
Quoted 11 lines · Ming Lei
block/blk-mq.c +7
-0+struct elev_change_ctx ctx = { + .name ="none" , + .force =1 , + .uevent =1 , + };mutex_lock (&q->elevator_lock);if (q->elevator && !blk_queue_dying (q)) + ctx.name = q->elevator->type->elevator_name; +__elevator_change (q, &ctx);mutex_unlock (&q->elevator_lock);
Can we have this whole code section in a helper in elevator.c and keep the low-level __elevator_change function private there?
block/elevator.h struct elevator_queue · line 130
Quoted 5 lines
block/elevator.h +5
-0+/* Holding data for changing elevator */ +struct elev_change_ctx { +const char *name; +unsigned int force:1 ; +unsigned int uevent:1 ;
Using an integer type for flags is odd. This should a boolean, or even better aflags field with flag names.
And I think with the flag names we can just pass the name and the flags and do away with the separate structure entirely.
PATCH 9/15 block: unifying elevator change
In reply to Ming Leielevator change is one well-define behavior:
- tear down current elevator if it exists
- setup new elevator
It is supposed to cover any case for changing elevator, typically the following cases:
- setup default elevator in add_disk()
- switch to none in del_disk()
- reset elevator in blk_mq_update_nr_hw_queues()
- switch elevator in sysfs store elevator attribute
This patch uses elevator_change() to cover all above cases:
- every elevator switch is serialized with each other: add_disk/del_disk/ store elevator is serialized already, blk_mq_update_nr_hw_queues() uses srcu for syncing with the other three cases
- for both add_disk()/del_disk(), queue freeze works at atomic mode or has been froze, so the freeze in elevator_change() won't add extra delay
--- block/blk-sysfs.c | 18 ++++------- block/blk.h | 5 ++-- block/elevator.c | 76 +++++++++++++++++++++++++---------------------- block/genhd.c | 19 +----------- 4 files changed, 50 insertions(+), 68 deletions(-)
block/blk-sysfs.c +6 -12
@@ -869,14 +869,8 @@ int blk_register_queue(struct gendisk *disk)if (ret) goto out_unregister_ia_ranges; +elevator_set_default (q);mutex_lock (&q->elevator_lock); -if (q->elevator) { - ret =elv_register_queue (q, false); -if (ret) { -mutex_unlock (&q->elevator_lock); - goto out_crypto_sysfs_unregister; - } - }wbt_enable_default (disk);mutex_unlock (&q->elevator_lock);
@@ -902,8 +896,6 @@ int blk_register_queue(struct gendisk *disk)return ret; -out_crypto_sysfs_unregister: -blk_crypto_sysfs_unregister (disk); out_unregister_ia_ranges:disk_unregister_independent_access_ranges (disk); out_debugfs_remove:
@@ -949,9 +941,11 @@ void blk_unregister_queue(struct gendisk *disk)blk_mq_sysfs_unregister (disk);blk_crypto_sysfs_unregister (disk); -mutex_lock (&q->elevator_lock); -elv_unregister_queue (q); -mutex_unlock (&q->elevator_lock); +if (q->elevator) { +blk_mq_quiesce_queue (q); +elevator_set_none (q); +blk_mq_unquiesce_queue (q); + }mutex_lock (&q->sysfs_lock);disk_unregister_independent_access_ranges (disk);
block/blk.h +2 -3
@@ -321,9 +321,8 @@ bool blk_bio_list_merge(struct request_queue *q, struct list_head *list,bool blk_insert_flush (struct request *rq);int __elevator_change (struct request_queue *q,struct elev_change_ctx *ctx); -void elevator_exit (struct request_queue *q); -int elv_register_queue (struct request_queue *q,bool uevent); -void elv_unregister_queue (struct request_queue *q); +void elevator_set_default (struct request_queue *q); +void elevator_set_none (struct request_queue *q);ssize_t part_size_show (struct device *dev,struct device_attribute *attr,char *buf);
block/elevator.c +41 -35
128 lines not on this page · raw message
block/genhd.c +1 -18
@@ -416,12 +416,6 @@ int __must_check add_disk_fwnode(struct device *parent, struct gendisk *disk,*/ if (disk->fops->submit_bio || disk->fops->poll_bio)return -EINVAL ; - -/* -* Initialize the I/O scheduler code and pick a default one if -* needed. -*/ -elevator_init_mq (disk->queue); }else {if (!disk->fops->submit_bio)return -EINVAL ;
@@ -565,11 +559,7 @@ int __must_check add_disk_fwnode(struct device *parent, struct gendisk *disk,if (disk->major ==BLOCK_EXT_MAJOR )blk_free_ext_minor (disk->first_minor); out_exit_elevator: -if (disk->queue->elevator) { -mutex_lock (&disk->queue->elevator_lock); -elevator_exit (disk->queue); -mutex_unlock (&disk->queue->elevator_lock); - } +elevator_set_none (disk->queue);return ret; }EXPORT_SYMBOL_GPL (add_disk_fwnode);
@@ -743,14 +733,7 @@ void del_gendisk(struct gendisk *disk)if (queue_is_mq (q))blk_mq_cancel_work_sync (q); -blk_mq_quiesce_queue (q); -if (q->elevator) { -mutex_lock (&q->elevator_lock); -elevator_exit (q); -mutex_unlock (&q->elevator_lock); - }rq_qos_exit (q); -blk_mq_unquiesce_queue (q);/* * If the disk does not own the queue, allow using passthrough requests
Signature · 1 line
2.47.0
Re: [PATCH 09/15] block: unifying elevator change
Quoted 19 lines · Ming Lei
/*
* Use the default elevator settings. If the chosen elevator initialization
* fails, fall back to the "none" elevator (no elevator).
*/block/elevator.c +3
-6-void elevator_init_mq (struct request_queue *q) +void elevator_set_default (struct request_queue *q) { -struct elevator_type *e; -unsigned int memflags; +struct elev_change_ctx ctx = { };int err; -WARN_ON_ONCE (blk_queue_registered (q)); - -if (unlikely (q->elevator)) +if (!queue_is_mq (q))return ;
block/elevator.c EXPORT_SYMBOL_GPL(elv_unregister); · line 594
Quoted 10 lines
block/elevator.c +7
-2- e =elevator_get_default (q); -if (!e) + ctx.name =use_default_elevator (q) ?"mq-deadline" :"none" ; +if (!q->elevator && !strcmp (ctx.name,"none" ))return ; + err =elevator_change (q, &ctx); +if (err <0 ) +pr_warn ("\"%s\" set elevator failed %d, " +"falling back to \"none\"\n" , ctx.name, err); +}
If we fail to set the evator to default (mq-deadline) while registering queue, because nr_hw_queue update is simultaneously running then we may end up setting the queue elevator to none and that's not correct. Isn't it?
Quoted 15 lines
block/elevator.c +9
-3+void elevator_set_none (struct request_queue *q) +{ +struct elev_change_ctx ctx = { + .name ="none" , + .uevent =1 , + }; +int err; -blk_mq_unfreeze_queue (q, memflags); +if (!queue_is_mq (q)) +return ; -if (err) { - pr_warn("\"%s\" elevator initialization failed, "
block/elevator.c EXPORT_SYMBOL_GPL(elv_unregister); · line 613
Quoted 10 lines
block/elevator.c +5
-3- "falling back to \"none\"\n", e->elevator_name); - } +if (!q->elevator) +return ; -elevator_put (e); + err =elevator_change (q, &ctx); +if (err <0 ) +pr_warn ("%s: set none elevator failed %d\n" , __func__, err); }
Here as well if we fail to disable/exit elevator while deleting disk because nr_hw_queue update is simultaneously running then we may leak elevator resource?
block/genhd.c int __must_check add_disk_fwnode(struct device *parent, struct gendisk *disk, · line 562
Quoted 10 lines
block/genhd.c +1
-5@@ -565,11 +559,7 @@ int __must_check add_disk_fwnode(struct device *parent, struct gendisk *disk,if (disk->major ==BLOCK_EXT_MAJOR )blk_free_ext_minor (disk->first_minor); out_exit_elevator: -if (disk->queue->elevator) { -mutex_lock (&disk->queue->elevator_lock); -elevator_exit (disk->queue); -mutex_unlock (&disk->queue->elevator_lock); - } +elevator_set_none (disk->queue);
Same comment as above here as well but this is in add_disk code path.
Thanks,
--Nilay
Re: [PATCH 09/15] block: unifying elevator change
Quoted 21 lines · Nilay Shroff
Quoted 18 lines · Ming Lei
/*
* Use the default elevator settings. If the chosen elevator initialization
* fails, fall back to the "none" elevator (no elevator).
*/block/elevator.c +3
-6-void elevator_init_mq (struct request_queue *q) +void elevator_set_default (struct request_queue *q) { -struct elevator_type *e; -unsigned int memflags; +struct elev_change_ctx ctx = { };int err; -WARN_ON_ONCE (blk_queue_registered (q)); - -if (unlikely (q->elevator)) +if (!queue_is_mq (q))return ;
Quoted 14 lines
block/elevator.c EXPORT_SYMBOL_GPL(elv_unregister); · line 594
Quoted 11 lines
block/elevator.c +7
-2- e =elevator_get_default (q); -if (!e) + ctx.name =use_default_elevator (q) ?"mq-deadline" :"none" ; +if (!q->elevator && !strcmp (ctx.name,"none" ))return ; + err =elevator_change (q, &ctx); +if (err <0 ) +pr_warn ("\"%s\" set elevator failed %d, " +"falling back to \"none\"\n" , ctx.name, err); +}If we fail to set the evator to default (mq-deadline) while registering queue, because
nr_hw_queueupdate is simultaneously running then we may end up setting the queue elevator to none and that's not correct. Isn't it?
It still works with none.
I think it isn't one big deal. And if it is really one issue in future, we can set one flag in elevator_set_default(), and let blk_mq_update_nr_hw_queues set default sched for us.
Quoted 20 lines
Quoted 20 lines
block/elevator.c +11
-5+void elevator_set_none (struct request_queue *q) +{ +struct elev_change_ctx ctx = { + .name ="none" , + .uevent =1 , + }; +int err; -blk_mq_unfreeze_queue (q, memflags); +if (!queue_is_mq (q)) +return ; -if (err) { -pr_warn ("\"%s\" elevator initialization failed, " -"falling back to \"none\"\n" , e->elevator_name); - } +if (!q->elevator) +return ;
Quoted 9 lines
block/elevator.c EXPORT_SYMBOL_GPL(elv_unregister); · line 613
Quoted 6 lines
block/elevator.c +3
-1-elevator_put (e); + err =elevator_change (q, &ctx); +if (err <0 ) +pr_warn ("%s: set none elevator failed %d\n" , __func__, err); }Here as well if we fail to disable/exit elevator while deleting disk because
nr_hw_queueupdate is simultaneously running then we may leak elevator resource?
When blk_mq_update_nr_hw_queues() observes that queue is dying, it forces to change elevator to none, so there isn't elevator leak issue.
Quoted 12 lines
block/genhd.c int __must_check add_disk_fwnode(struct device *parent, struct gendisk *disk, · line 562
Quoted 11 lines
block/genhd.c +1
-5@@ -565,11 +559,7 @@ int __must_check add_disk_fwnode(struct device *parent, struct gendisk *disk,if (disk->major ==BLOCK_EXT_MAJOR )blk_free_ext_minor (disk->first_minor); out_exit_elevator: -if (disk->queue->elevator) { -mutex_lock (&disk->queue->elevator_lock); -elevator_exit (disk->queue); -mutex_unlock (&disk->queue->elevator_lock); - } +elevator_set_none (disk->queue);Same comment as above here as well but this is in
add_diskcode path.
We can avoid it by forcing to change to none in blk_mq_update_nr_hw_queues() for !blk_queue_registered()
Thanks,
Ming
Re: [PATCH 09/15] block: unifying elevator change
Quoted 73 lines · Ming Lei
Quoted 21 lines · Nilay Shroff
Quoted 18 lines · Ming Lei
/*
* Use the default elevator settings. If the chosen elevator initialization
* fails, fall back to the "none" elevator (no elevator).
*/block/elevator.c +3
-6-void elevator_init_mq (struct request_queue *q) +void elevator_set_default (struct request_queue *q) { -struct elevator_type *e; -unsigned int memflags; +struct elev_change_ctx ctx = { };int err; -WARN_ON_ONCE (blk_queue_registered (q)); - -if (unlikely (q->elevator)) +if (!queue_is_mq (q))return ;Quoted 14 lines
block/elevator.c EXPORT_SYMBOL_GPL(elv_unregister); · line 594
Quoted 11 lines
block/elevator.c +7
-2- e =elevator_get_default (q); -if (!e) + ctx.name =use_default_elevator (q) ?"mq-deadline" :"none" ; +if (!q->elevator && !strcmp (ctx.name,"none" ))return ; + err =elevator_change (q, &ctx); +if (err <0 ) +pr_warn ("\"%s\" set elevator failed %d, " +"falling back to \"none\"\n" , ctx.name, err); +}If we fail to set the evator to default (mq-deadline) while registering queue, because
nr_hw_queueupdate is simultaneously running then we may end up setting the queue elevator to none and that's not correct. Isn't it?It still works with none.
I think it isn't one big deal. And if it is really one issue in future, we can set one flag in
elevator_set_default(), and letblk_mq_update_nr_hw_queuesset default sched for us.Quoted 20 lines
Quoted 20 lines
block/elevator.c +11
-5+void elevator_set_none (struct request_queue *q) +{ +struct elev_change_ctx ctx = { + .name ="none" , + .uevent =1 , + }; +int err; -blk_mq_unfreeze_queue (q, memflags); +if (!queue_is_mq (q)) +return ; -if (err) { -pr_warn ("\"%s\" elevator initialization failed, " -"falling back to \"none\"\n" , e->elevator_name); - } +if (!q->elevator) +return ;Quoted 9 lines
block/elevator.c EXPORT_SYMBOL_GPL(elv_unregister); · line 613
Quoted 6 lines
block/elevator.c +3
-1-elevator_put (e); + err =elevator_change (q, &ctx); +if (err <0 ) +pr_warn ("%s: set none elevator failed %d\n" , __func__, err); }Here as well if we fail to disable/exit elevator while deleting disk because
nr_hw_queueupdate is simultaneously running then we may leak elevator resource?
Quoted 3 lines
When
blk_mq_update_nr_hw_queues()observes that queue is dying, it forces to change elevator to none, so there isn't elevator leak issue.
Yes if we get into blk_mq_update_nr_hw_queues after dying flag is set. But what if blk_mq_update_nr_hw_queues doesn't see dying flag and starts running __elevator_change. However later we set dying flag from del_gendisk and starts running elevator_set_none simultaneously on another cpu? In this case elevator_set_none would fail to set the elevator to "none" as blk_mq_update_nr_hw_queues is running on another cpu. Isn't it?
Quoted 13 lines
Quoted 12 lines
block/genhd.c int __must_check add_disk_fwnode(struct device *parent, struct gendisk *disk, · line 562
Quoted 11 lines
block/genhd.c +1
-5@@ -565,11 +559,7 @@ int __must_check add_disk_fwnode(struct device *parent, struct gendisk *disk,if (disk->major ==BLOCK_EXT_MAJOR )blk_free_ext_minor (disk->first_minor); out_exit_elevator: -if (disk->queue->elevator) { -mutex_lock (&disk->queue->elevator_lock); -elevator_exit (disk->queue); -mutex_unlock (&disk->queue->elevator_lock); - } +elevator_set_none (disk->queue);Same comment as above here as well but this is in
add_diskcode path.
Quoted 3 lines
We can avoid it by forcing to change to none in
blk_mq_update_nr_hw_queues()for !blk_queue_registered()
Here as well there's a thin race window possible assuming add_disk fails after we registered queue. Assuming nr_hw_queue update starts running and it sees queue is registered however on another cpu add_disk fails just after registering queue. So in this case still it might be possible that elevator_set_none might fail to set elevator to "none" just because nr_hw_queue update is running on another cpu. What do you think?
Thanks,
--Nilay
Re: [PATCH 09/15] block: unifying elevator change
Quoted 102 lines · Nilay Shroff
Quoted 73 lines · Ming Lei
Quoted 21 lines · Nilay Shroff
Quoted 18 lines · Ming Lei
/*
* Use the default elevator settings. If the chosen elevator initialization
* fails, fall back to the "none" elevator (no elevator).
*/block/elevator.c +3
-6-void elevator_init_mq (struct request_queue *q) +void elevator_set_default (struct request_queue *q) { -struct elevator_type *e; -unsigned int memflags; +struct elev_change_ctx ctx = { };int err; -WARN_ON_ONCE (blk_queue_registered (q)); - -if (unlikely (q->elevator)) +if (!queue_is_mq (q))return ;Quoted 14 lines
block/elevator.c EXPORT_SYMBOL_GPL(elv_unregister); · line 594
Quoted 11 lines
block/elevator.c +7
-2- e =elevator_get_default (q); -if (!e) + ctx.name =use_default_elevator (q) ?"mq-deadline" :"none" ; +if (!q->elevator && !strcmp (ctx.name,"none" ))return ; + err =elevator_change (q, &ctx); +if (err <0 ) +pr_warn ("\"%s\" set elevator failed %d, " +"falling back to \"none\"\n" , ctx.name, err); +}If we fail to set the evator to default (mq-deadline) while registering queue, because
nr_hw_queueupdate is simultaneously running then we may end up setting the queue elevator to none and that's not correct. Isn't it?It still works with none.
I think it isn't one big deal. And if it is really one issue in future, we can set one flag in
elevator_set_default(), and letblk_mq_update_nr_hw_queuesset default sched for us.Quoted 20 lines
Quoted 20 lines
block/elevator.c +11
-5+void elevator_set_none (struct request_queue *q) +{ +struct elev_change_ctx ctx = { + .name ="none" , + .uevent =1 , + }; +int err; -blk_mq_unfreeze_queue (q, memflags); +if (!queue_is_mq (q)) +return ; -if (err) { -pr_warn ("\"%s\" elevator initialization failed, " -"falling back to \"none\"\n" , e->elevator_name); - } +if (!q->elevator) +return ;Quoted 9 lines
block/elevator.c EXPORT_SYMBOL_GPL(elv_unregister); · line 613
Quoted 6 lines
block/elevator.c +3
-1-elevator_put (e); + err =elevator_change (q, &ctx); +if (err <0 ) +pr_warn ("%s: set none elevator failed %d\n" , __func__, err); }Here as well if we fail to disable/exit elevator while deleting disk because
nr_hw_queueupdate is simultaneously running then we may leak elevator resource?Quoted 3 lines
When
blk_mq_update_nr_hw_queues()observes that queue is dying, it forces to change elevator to none, so there isn't elevator leak issue.Yes if we get into
blk_mq_update_nr_hw_queuesafter dying flag is set. But what ifblk_mq_update_nr_hw_queuesdoesn't see dying flag and starts running__elevator_change. However later we set dying flag fromdel_gendiskand starts runningelevator_set_nonesimultaneously on another cpu? In this caseelevator_set_nonewould fail to set the elevator to "none" asblk_mq_update_nr_hw_queuesis running on another cpu. Isn't it?Quoted 13 lines
Quoted 12 lines
block/genhd.c int __must_check add_disk_fwnode(struct device *parent, struct gendisk *disk, · line 562
Quoted 11 lines
block/genhd.c +1
-5@@ -565,11 +559,7 @@ int __must_check add_disk_fwnode(struct device *parent, struct gendisk *disk,if (disk->major ==BLOCK_EXT_MAJOR )blk_free_ext_minor (disk->first_minor); out_exit_elevator: -if (disk->queue->elevator) { -mutex_lock (&disk->queue->elevator_lock); -elevator_exit (disk->queue); -mutex_unlock (&disk->queue->elevator_lock); - } +elevator_set_none (disk->queue);Same comment as above here as well but this is in
add_diskcode path.Quoted 3 lines
We can avoid it by forcing to change to none in
blk_mq_update_nr_hw_queues()for !blk_queue_registered()
Quoted 6 lines
Here as well there's a thin race window possible assuming
add_diskfails after we registered queue. Assumingnr_hw_queueupdate starts running and it sees queue is registered however on another cpuadd_diskfails just after registering queue. So in this case still it might be possible thatelevator_set_nonemight fail to set elevator to "none" just becausenr_hw_queueupdate is running on another cpu. What do you think?
Yeah.
It isn't hard to solve, but just don't want to make the whole implementation too complicated.
Another way is to prevent add_disk & del_disk from happening during updating nr_hw_queues, and this way is reasonable too because both blk_mq debugfs & sysfs registering depends on nr_hw_queues.
Meantime we can retry add_disk/del_disk until updating_nr_hw_queues are finished, and one waitqueue can be added, so the wait can be:
add_disk():
while (true) {
srcu_read_lock()
if (set->is_updating_nr_hw_queus) {
srcu_read_unlock();
goto wait;
}
__add_disk();
srcu_read_unlock()
break;
wait:
wait_event(set->wq, !set->is_updating_nr_hw_queus);
}Thanks,
Ming
PATCH 10/15 block: pass elevator_queue to elv_register_queue & unregister_queue
In reply to Ming LeiPass elevator_queue reference to elv_register_queue() & elv_unregister_queue().
No functional change, and prepare for moving the two out of elevator lock.
--- block/elevator.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
block/elevator.c +8 -8
@@ -455,9 +455,10 @@ static const struct kobj_type elv_ktype = { .release = elevator_release, }; -static int elv_register_queue (struct request_queue *q,bool uevent) +static int elv_register_queue (struct request_queue *q, +struct elevator_queue *e, +bool uevent) { -struct elevator_queue *e = q->elevator;int error;lockdep_assert_held (&q->elevator_lock);
@@ -484,10 +485,9 @@ static int elv_register_queue(struct request_queue *q, bool uevent)return error; } -static void elv_unregister_queue (struct request_queue *q) +static void elv_unregister_queue (struct request_queue *q, +struct elevator_queue *e) { -struct elevator_queue *e = q->elevator; -lockdep_assert_held (&q->elevator_lock);if (e &&test_and_clear_bit (ELEVATOR_FLAG_REGISTERED , &e->flags)) {
@@ -629,7 +629,7 @@ static int elevator_switch(struct request_queue *q, struct elevator_type *new_e,blk_mq_quiesce_queue (q);if (q->elevator) { -elv_unregister_queue (q); +elv_unregister_queue (q, q->elevator);elevator_exit (q); }
@@ -637,7 +637,7 @@ static int elevator_switch(struct request_queue *q, struct elevator_type *new_e,if (ret) goto out_unfreeze; - ret =elv_register_queue (q, ctx->uevent); + ret =elv_register_queue (q, q->elevator, ctx->uevent);if (ret) {elevator_exit (q); goto out_unfreeze;
@@ -662,7 +662,7 @@ static void elevator_disable(struct request_queue *q)blk_mq_quiesce_queue (q); -elv_unregister_queue (q); +elv_unregister_queue (q, q->elevator);elevator_exit (q);blk_queue_flag_clear (QUEUE_FLAG_SQ_SCHED , q); q->elevator =NULL ;
Signature · 1 line
2.47.0
Re: [PATCH 10/15] block: pass elevator_queue to elv_register_queue & unregister_queue
Quoted 5 lines · Ming Lei
Pass
elevator_queuereference toelv_register_queue()&elv_unregister_queue().No functional change, and prepare for moving the two out of elevator lock.
Can you explain a bit more why passing a seemlingly duplicate argument helps further down the series?
Re: [PATCH 10/15] block: pass elevator_queue to elv_register_queue & unregister_queue
Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Ming Lei
Pass
elevator_queuereference toelv_register_queue()&elv_unregister_queue().No functional change, and prepare for moving the two out of elevator lock.
Can you explain a bit more why passing a seemlingly duplicate argument helps further down the series?
Please see the following patch, especially elevator_change_done() in which the old elevator queue is retrieved from the context structure, then its unregistering can be moved out of queue freezing & elevator lock.
Same with registering of the new added elevator queue.
I will add more words in commit log for this motivation in next version.
Thanks,
Ming
Re: [PATCH 10/15] block: pass elevator_queue to elv_register_queue & unregister_queue
Quoted 7 lines · Ming Lei
Please see the following patch, especially
elevator_change_done()in which the old elevator queue is retrieved from the context structure, then its unregistering can be moved out of queue freezing & elevator lock.Same with registering of the new added elevator queue.
I will add more words in commit log for this motivation in next version.
Thanks. When bisecting or using git-blame finding a following patch for reference is really hard, so please make sure commit logs are useful when standing alone.
PATCH 11/15 block: move elv_register[unregister]_queue out of elevator_lock
In reply to Ming LeiMove elv_register[unregister]_queue out of ->elevator_lock & queue freezing, so we can kill many lockdep warnings.
elv_register[unregister]_queue() is serialized, and just dealing with sysfs/debugfs things, no need to be done with queue frozen.
--- block/blk-mq.c | 9 ++++---- block/blk.h | 1 + block/elevator.c | 58 ++++++++++++++++++++++++++++++++++-------------- block/elevator.h | 5 +++++ 4 files changed, 52 insertions(+), 21 deletions(-)
block/blk-mq.c +5 -4
@@ -4990,16 +4990,17 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, .force = 1, .uevent = 1, }; +int ret = -ENODEV ;mutex_lock (&q->elevator_lock);if (q->elevator && !blk_queue_dying (q)) ctx.name = q->elevator->type->elevator_name; -__elevator_change (q, &ctx); + ret =__elevator_change (q, &ctx);mutex_unlock (&q->elevator_lock); - } - -list_for_each_entry (q, &set->tag_list, tag_set_list)blk_mq_unfreeze_queue_nomemrestore (q); +if (!ret) +WARN_ON_ONCE (elevator_change_done (q, &ctx)); + }memalloc_noio_restore (memflags);/* Free the excess tags when nr_hw_queues shrink. */
block/blk.h +1 -0
@@ -321,6 +321,7 @@ bool blk_bio_list_merge(struct request_queue *q, struct list_head *list,bool blk_insert_flush (struct request *rq);int __elevator_change (struct request_queue *q,struct elev_change_ctx *ctx); +int elevator_change_done (struct request_queue *q,struct elev_change_ctx *ctx);void elevator_set_default (struct request_queue *q);void elevator_set_none (struct request_queue *q);
block/elevator.c +41 -17
128 lines not on this page · raw message
block/elevator.h +5 -0
@@ -128,6 +128,11 @@ struct elev_change_ctx {const char *name; unsigned int force:1; unsigned int uevent:1; + +/* for unregistering old elevator */ +struct elevator_queue *old; +/* for registering new elevator */ +struct elevator_queue *new; };/*
Signature · 1 line
2.47.0
Re: [PATCH 11/15] block: move elv_register[unregister]_queue out of elevator_lock
Quoted 12 lines · Ming Lei
block/elevator.c +12
-0+int elevator_change_done (struct request_queue *q,struct elev_change_ctx *ctx) +{ +int ret =0 ; + +if (ctx->old) { +elv_unregister_queue (q, ctx->old); +kobject_put (&ctx->old->kobj); + } +if (ctx->new) { + ret =elv_register_queue (q, ctx->new, ctx->uevent); +if (ret) { +unsigned memflags =blk_mq_freeze_queue (q);
block/elevator.c static void elevator_disable(struct request_queue *q) · line 695
Quoted 10 lines
block/elevator.c +10
-0+ +mutex_lock (&q->elevator_lock); +elevator_exit (q); +mutex_unlock (&q->elevator_lock); +blk_mq_unfreeze_queue (q, memflags); + } + } +return 0 ; +} +
We could have sysf elevator attributes simultaneously accessed while this function adds/removes sysfs elevator attributes without any protection. In fact, the show/store methods of elevator attributes runs with e->sysfs_lock held. So it seems moving the above function out of lock protection might cause crash or other side effects?
Thanks,
--Nilay
Re: [PATCH 11/15] block: move elv_register[unregister]_queue out of elevator_lock
Quoted 21 lines · Nilay Shroff
Quoted 18 lines · Ming Lei
block/elevator.c +18
-0+int elevator_change_done (struct request_queue *q,struct elev_change_ctx *ctx) +{ +int ret =0 ; + +if (ctx->old) { +elv_unregister_queue (q, ctx->old); +kobject_put (&ctx->old->kobj); + } +if (ctx->new) { + ret =elv_register_queue (q, ctx->new, ctx->uevent); +if (ret) { +unsigned memflags =blk_mq_freeze_queue (q); + +mutex_lock (&q->elevator_lock); +elevator_exit (q); +mutex_unlock (&q->elevator_lock); +blk_mq_unfreeze_queue (q, memflags); + }
Quoted 8 lines
block/elevator.c static void elevator_disable(struct request_queue *q) · line 695
Quoted 4 lines
block/elevator.c +4
-0+ } +return 0 ; +} +We could have sysf elevator attributes simultaneously accessed while this function adds/removes sysfs elevator attributes without any protection. In fact, the show/store methods of elevator attributes runs with
e->sysfs_lockheld. So it seems moving the above function out of lock protection might cause crash or other side effects?
sysfs/kobject provides such protection, and kobject_del() will drain any in-flight attribute access.
Thanks,
Ming
Re: [PATCH 11/15] block: move elv_register[unregister]_queue out of elevator_lock
Quoted 31 lines · Ming Lei
Quoted 21 lines · Nilay Shroff
Quoted 18 lines · Ming Lei
block/elevator.c +18
-0+int elevator_change_done (struct request_queue *q,struct elev_change_ctx *ctx) +{ +int ret =0 ; + +if (ctx->old) { +elv_unregister_queue (q, ctx->old); +kobject_put (&ctx->old->kobj); + } +if (ctx->new) { + ret =elv_register_queue (q, ctx->new, ctx->uevent); +if (ret) { +unsigned memflags =blk_mq_freeze_queue (q); + +mutex_lock (&q->elevator_lock); +elevator_exit (q); +mutex_unlock (&q->elevator_lock); +blk_mq_unfreeze_queue (q, memflags); + }Quoted 8 lines
block/elevator.c static void elevator_disable(struct request_queue *q) · line 695
Quoted 4 lines
block/elevator.c +4
-0+ } +return 0 ; +} +We could have sysf elevator attributes simultaneously accessed while this function adds/removes sysfs elevator attributes without any protection. In fact, the show/store methods of elevator attributes runs with
e->sysfs_lockheld. So it seems moving the above function out of lock protection might cause crash or other side effects?
Quoted 3 lines
sysfs/kobject provides such protection, and
kobject_del()will drain any in-flight attribute access.
Okay, so in that case do we now really need e->sysfs_lock protection while accessing elevator attributes?
Thanks,
--Nilay
Re: [PATCH 11/15] block: move elv_register[unregister]_queue out of elevator_lock
Quoted 34 lines · Nilay Shroff
Quoted 31 lines · Ming Lei
Quoted 29 lines · Nilay Shroff
Quoted 12 lines · Ming Lei
block/elevator.c +12
-0+int elevator_change_done (struct request_queue *q,struct elev_change_ctx *ctx) +{ +int ret =0 ; + +if (ctx->old) { +elv_unregister_queue (q, ctx->old); +kobject_put (&ctx->old->kobj); + } +if (ctx->new) { + ret =elv_register_queue (q, ctx->new, ctx->uevent); +if (ret) { +unsigned memflags =blk_mq_freeze_queue (q);block/elevator.c static void elevator_disable(struct request_queue *q) · line 695
Quoted 10 lines
block/elevator.c +10
-0+ +mutex_lock (&q->elevator_lock); +elevator_exit (q); +mutex_unlock (&q->elevator_lock); +blk_mq_unfreeze_queue (q, memflags); + } + } +return 0 ; +} +We could have sysf elevator attributes simultaneously accessed while this function adds/removes sysfs elevator attributes without any protection. In fact, the show/store methods of elevator attributes runs with
e->sysfs_lockheld. So it seems moving the above function out of lock protection might cause crash or other side effects?
Quoted 5 lines
Quoted 3 lines
sysfs/kobject provides such protection, and
kobject_del()will drain any in-flight attribute access.Okay, so in that case do we now really need
e->sysfs_lockprotection while accessing elevator attributes?
Yeah, I think so, elevator_exit() is always called after elevator kobject is deleted.
However, this patchset moves elv_unregister_queue() after blk_mq_exit_sched(), we may need this lock for failing elevator's attribute ->show() & ->store() by adding one '->exiting' flag to 'struct elevator_queue'.
Thanks,
Ming
PATCH 12/15 block: move debugfs/sysfs register out of freezing queue
In reply to Ming LeiMove debugfs/sysfs register out of freezing queue in __blk_mq_update_nr_hw_queues(), so that the following lockdep dependency can be killed:
#2 (&q->q_usage_counter(io)#16){++++}-{0:0}:
#1 (fs_reclaim){+.+.}-{0:0}:
#0 (&sb->s_type->i_mutex_key#3){+.+.}-{4:4}: //debugfsAnd registering/un-registering debugfs/sysfs does not require queue to be frozen.
--- block/blk-mq.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
block/blk-mq.c +10 -10
@@ -4947,15 +4947,15 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,if (set->nr_maps ==1 && nr_hw_queues == set->nr_hw_queues)return ; - memflags =memalloc_noio_save (); -list_for_each_entry (q, &set->tag_list, tag_set_list) -blk_mq_freeze_queue_nomemsave (q); -list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_debugfs_unregister_hctxs (q);blk_mq_sysfs_unregister_hctxs (q); } + memflags =memalloc_noio_save (); +list_for_each_entry (q, &set->tag_list, tag_set_list) +blk_mq_freeze_queue_nomemsave (q); +if (blk_mq_realloc_tag_set_tags (set, nr_hw_queues) <0 ) goto reregister;
@@ -4978,12 +4978,6 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,blk_mq_map_swqueue (q); } -reregister: -list_for_each_entry (q, &set->tag_list, tag_set_list) { -blk_mq_sysfs_register_hctxs (q); -blk_mq_debugfs_register_hctxs (q); - } -list_for_each_entry (q, &set->tag_list, tag_set_list) {struct elev_change_ctx ctx = { .name ="none" ,
@@ -5003,6 +4997,12 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, }memalloc_noio_restore (memflags); +reregister: +list_for_each_entry (q, &set->tag_list, tag_set_list) { +blk_mq_sysfs_register_hctxs (q); +blk_mq_debugfs_register_hctxs (q); + } +/* Free the excess tags when nr_hw_queues shrink. */ for (i = set->nr_hw_queues; i < prev_nr_hw_queues; i++)__blk_mq_free_map_and_rqs (set, i);
Signature · 1 line
2.47.0
Re: [PATCH 12/15] block: move debugfs/sysfs register out of freezing queue
Quoted 22 lines · Ming Lei
Move debugfs/sysfs register out of freezing queue in
__blk_mq_update_nr_hw_queues(), so that the following lockdep dependency can be killed:#2 (&q->q_usage_counter(io)#16){++++}-{0:0}: #1 (fs_reclaim){+.+.}-{0:0}: #0 (&sb->s_type->i_mutex_key#3){+.+.}-{4:4}: //debugfsAnd registering/un-registering debugfs/sysfs does not require queue to be frozen.
Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)block/blk-mq.c +0
-0@@ -4947,15 +4947,15 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,if (set->nr_maps ==1 && nr_hw_queues == set->nr_hw_queues)
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 4953
Quoted 10 lines
block/blk-mq.c +0
-4return ; - memflags =memalloc_noio_save (); -list_for_each_entry (q, &set->tag_list, tag_set_list) -blk_mq_freeze_queue_nomemsave (q); -list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_debugfs_unregister_hctxs (q);blk_mq_sysfs_unregister_hctxs (q); }
As we removed hctx sysfs protection while un-registering it, this might cause crash or other side-effect if simultaneously these sysfs attributes are accessed. The read access of these attributes are still protected using ->elevator_lock.
Thanks,
--Nilay
Re: [PATCH 12/15] block: move debugfs/sysfs register out of freezing queue
Quoted 27 lines · Nilay Shroff
Quoted 24 lines · Ming Lei
Move debugfs/sysfs register out of freezing queue in
__blk_mq_update_nr_hw_queues(), so that the following lockdep dependency can be killed:#2 (&q->q_usage_counter(io)#16){++++}-{0:0}: #1 (fs_reclaim){+.+.}-{0:0}: #0 (&sb->s_type->i_mutex_key#3){+.+.}-{4:4}: //debugfsAnd registering/un-registering debugfs/sysfs does not require queue to be frozen.
Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)block/blk-mq.c +0
-0@@ -4947,15 +4947,15 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,if (set->nr_maps ==1 && nr_hw_queues == set->nr_hw_queues)return ;
Quoted 12 lines
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 4953
Quoted 8 lines
block/blk-mq.c +0
-4- memflags =memalloc_noio_save (); -list_for_each_entry (q, &set->tag_list, tag_set_list) -blk_mq_freeze_queue_nomemsave (q); -list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_debugfs_unregister_hctxs (q);blk_mq_sysfs_unregister_hctxs (q); }As we removed hctx sysfs protection while un-registering it, this might cause crash or other side-effect if simultaneously these sysfs attributes are accessed. The read access of these attributes are still protected using ->elevator_lock.
The ->elevator_lock in ->show() is useless except for reading the elevator internal data(sched tags, requests, ...), even for reading elevator data, it should have been relying on elevator reference, instead of lock, but that is another topic & improvement in future.
Also this patch does _not_ change ->elevator_lock for above debugfs/sysfs unregistering, does it? It is always done without holding ->elevator_lock. Also ->show() does not require ->q_usage_counter too.
As I mentioned, kobject/sysfs provides protection between ->show()/->store() and kobject_del(), isn't it the reason why you want to remove ->sys_lock?
https://lore.kernel.org/linux-block/[email protected]/
Thanks,
Ming
Re: [PATCH 12/15] block: move debugfs/sysfs register out of freezing queue
Quoted 46 lines · Ming Lei
Quoted 27 lines · Nilay Shroff
Quoted 24 lines · Ming Lei
Move debugfs/sysfs register out of freezing queue in
__blk_mq_update_nr_hw_queues(), so that the following lockdep dependency can be killed:#2 (&q->q_usage_counter(io)#16){++++}-{0:0}: #1 (fs_reclaim){+.+.}-{0:0}: #0 (&sb->s_type->i_mutex_key#3){+.+.}-{4:4}: //debugfsAnd registering/un-registering debugfs/sysfs does not require queue to be frozen.
Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)block/blk-mq.c +0
-0@@ -4947,15 +4947,15 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,if (set->nr_maps ==1 && nr_hw_queues == set->nr_hw_queues)return ;Quoted 12 lines
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 4953
Quoted 8 lines
block/blk-mq.c +0
-4- memflags =memalloc_noio_save (); -list_for_each_entry (q, &set->tag_list, tag_set_list) -blk_mq_freeze_queue_nomemsave (q); -list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_debugfs_unregister_hctxs (q);blk_mq_sysfs_unregister_hctxs (q); }As we removed hctx sysfs protection while un-registering it, this might cause crash or other side-effect if simultaneously these sysfs attributes are accessed. The read access of these attributes are still protected using ->elevator_lock.
The ->elevator_lock in ->show() is useless except for reading the elevator internal
data(sched tags, requests, ...), even for reading elevator data, it should have been relying on elevator reference, instead of lock, but that is another topic & improvement in future.
Quoted 9 lines
Also this patch does _not_ change ->elevator_lock for above debugfs/sysfs unregistering, does it? It is always done without holding ->elevator_lock. Also ->show() does not require ->q_usage_counter too.
As I mentioned, kobject/sysfs provides protection between ->show()/->store() and
kobject_del(), isn't it the reason why you want to remove ->sys_lock?
Yes you were correct, that was the reason we wanted to remove ->sysfs_lock. However for these particular hctx sysfs attributes (nr_tags and nr_reserved_tags) could be updated simultaneously from another blk-mq sysfs attribute named nr_requests. Hence IMO, the default protection provided by sysfs/kernfs may not be sufficient and so we need to protect those attributes using ->elevator_lock.
Consider this case: While blk_mq_update_nr_hw_queues removes hctx attributes, and simultaneously if nr_requests is also updating num of tags, would that not cause any side effect? Maybe we also want to protect blk_mq_update_nr_requests with srcu read lock (set->update_nr_hwq_srcu) so that it couldn't run while blk_mq_update_nr_hw_queues is in progress?
Thanks,
--Nilay
Re: [PATCH 12/15] block: move debugfs/sysfs register out of freezing queue
Quoted 56 lines · Nilay Shroff
Quoted 53 lines · Ming Lei
Quoted 27 lines · Nilay Shroff
Quoted 24 lines · Ming Lei
Move debugfs/sysfs register out of freezing queue in
__blk_mq_update_nr_hw_queues(), so that the following lockdep dependency can be killed:#2 (&q->q_usage_counter(io)#16){++++}-{0:0}: #1 (fs_reclaim){+.+.}-{0:0}: #0 (&sb->s_type->i_mutex_key#3){+.+.}-{4:4}: //debugfsAnd registering/un-registering debugfs/sysfs does not require queue to be frozen.
Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)block/blk-mq.c +0
-0@@ -4947,15 +4947,15 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,if (set->nr_maps ==1 && nr_hw_queues == set->nr_hw_queues)return ;Quoted 12 lines
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 4953
Quoted 8 lines
block/blk-mq.c +0
-4- memflags =memalloc_noio_save (); -list_for_each_entry (q, &set->tag_list, tag_set_list) -blk_mq_freeze_queue_nomemsave (q); -list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_debugfs_unregister_hctxs (q);blk_mq_sysfs_unregister_hctxs (q); }As we removed hctx sysfs protection while un-registering it, this might cause crash or other side-effect if simultaneously these sysfs attributes are accessed. The read access of these attributes are still protected using ->elevator_lock.
The ->elevator_lock in ->show() is useless except for reading the elevator internal
data(sched tags, requests, ...), even for reading elevator data, it should have been relying on elevator reference, instead of lock, but that is another topic & improvement in future.Also this patch does _not_ change ->elevator_lock for above debugfs/sysfs unregistering, does it? It is always done without holding ->elevator_lock. Also ->show() does not require ->q_usage_counter too.
As I mentioned, kobject/sysfs provides protection between ->show()/->store() and
kobject_del(), isn't it the reason why you want to remove ->sys_lock?
Quoted 7 lines
Quoted 2 lines
Yes you were correct, that was the reason we wanted to remove ->sysfs_lock. However for these particular hctx sysfs attributes (
nr_tagsandnr_reserved_tags) could be updated simultaneously from another blk-mq sysfs attribute namednr_requests. Hence IMO, the default protection provided by sysfs/kernfs may not be sufficient and so we need to protect those attributes using ->elevator_lock.
Yes, what is why this patchset doesn't kill more ->elevator_lock uses, such as, the uses in blk-mq-debugs, update_nr_requests, but many of them can be replaced with grabbing elevator reference.
But with/without this patch, the touched register/unregisger code does not require ->elevator_lock:
blk_mq_debugfs_unregister_hctxs(q);
blk_mq_sysfs_unregister_hctxs(q);so I don't understand why you argue here about ->elevator_lock use?
Quoted 4 lines
Consider this case: While
blk_mq_update_nr_hw_queuesremoves hctx attributes, and simultaneously ifnr_requestsis also updating num of tags, would that not cause any side effect?
Why is updating nr_requests related with removing hctx attributes?
Can you explain the side effect in details?
Quoted 3 lines
Maybe we also want to protect
blk_mq_update_nr_requests
with srcu read lock (set->update_nr_hwq_srcu) so that it couldn't run whileblk_mq_update_nr_hw_queuesis in progress?
Yeah, agree, and it can be one new patch for covering race between blk_mq_update_nr_requests and blk_mq_update_nr_hw_queues, the point is just that nr_hw_queues is being changed, and not related with removing hctx attributes, IMO.
Thanks,
Ming
Re: [PATCH 12/15] block: move debugfs/sysfs register out of freezing queue
Quoted 69 lines · Ming Lei
Quoted 56 lines · Nilay Shroff
Quoted 53 lines · Ming Lei
Quoted 27 lines · Nilay Shroff
Quoted 24 lines · Ming Lei
Move debugfs/sysfs register out of freezing queue in
__blk_mq_update_nr_hw_queues(), so that the following lockdep dependency can be killed:#2 (&q->q_usage_counter(io)#16){++++}-{0:0}: #1 (fs_reclaim){+.+.}-{0:0}: #0 (&sb->s_type->i_mutex_key#3){+.+.}-{4:4}: //debugfsAnd registering/un-registering debugfs/sysfs does not require queue to be frozen.
Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)block/blk-mq.c +0
-0@@ -4947,15 +4947,15 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,if (set->nr_maps ==1 && nr_hw_queues == set->nr_hw_queues)return ;Quoted 12 lines
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 4953
Quoted 8 lines
block/blk-mq.c +0
-4- memflags =memalloc_noio_save (); -list_for_each_entry (q, &set->tag_list, tag_set_list) -blk_mq_freeze_queue_nomemsave (q); -list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_debugfs_unregister_hctxs (q);blk_mq_sysfs_unregister_hctxs (q); }As we removed hctx sysfs protection while un-registering it, this might cause crash or other side-effect if simultaneously these sysfs attributes are accessed. The read access of these attributes are still protected using ->elevator_lock.
The ->elevator_lock in ->show() is useless except for reading the elevator internal
data(sched tags, requests, ...), even for reading elevator data, it should have been relying on elevator reference, instead of lock, but that is another topic & improvement in future.Also this patch does _not_ change ->elevator_lock for above debugfs/sysfs unregistering, does it? It is always done without holding ->elevator_lock. Also ->show() does not require ->q_usage_counter too.
As I mentioned, kobject/sysfs provides protection between ->show()/->store() and
kobject_del(), isn't it the reason why you want to remove ->sys_lock?Quoted 7 lines
Quoted 2 lines
Yes you were correct, that was the reason we wanted to remove ->sysfs_lock. However for these particular hctx sysfs attributes (
nr_tagsandnr_reserved_tags) could be updated simultaneously from another blk-mq sysfs attribute namednr_requests. Hence IMO, the default protection provided by sysfs/kernfs may not be sufficient and so we need to protect those attributes using ->elevator_lock.Yes, what is why this patchset doesn't kill more ->elevator_lock uses, such as, the uses in blk-mq-debugs,
update_nr_requests, but many of them can be replaced with grabbing elevator reference.
Quoted 8 lines
But with/without this patch, the touched register/unregisger code does not require ->elevator_lock:
blk_mq_debugfs_unregister_hctxs(q); blk_mq_sysfs_unregister_hctxs(q);so I don't understand why you argue here about ->elevator_lock use?
I am not arguing using ->elevator_lock wrt removal of hctx sysfs attributes as you explained that sysfs/kernfs already provides the needed protection. But please see below my explanation.
Quoted 8 lines
Quoted 4 lines
Consider this case: While
blk_mq_update_nr_hw_queuesremoves hctx attributes, and simultaneously ifnr_requestsis also updating num of tags, would that not cause any side effect?Why is updating
nr_requestsrelated with removing hctx attributes?Can you explain the side effect in details?
Thread 1:
writing-to-blk-mq-sysfs-attribute-nr_requests
-> queue_requests_store ==> freezes queue and acquires ->elevator_lock
-> blk_mq_update_nr_requests
-> blk_mq_tag_update_depth
-> blk_mq_alloc_map_and_rqs
-> blk_mq_alloc_rq_map
-> blk_mq_init_tags ==> updates ->nr_tags and ->nr_reserved_tags Thread2:blk_mq_update_nr_hw_queues
-> __blk_mq_update_nr_hw_queues
-> blk_mq_realloc_tag_set_tags
-> __blk_mq_alloc_map_and_rqs
-> blk_mq_alloc_map_and_rqs
-> blk_mq_alloc_rq_map
-> blk_mq_init_tags ==> updates ->nr_tags and ->nr_reserved_tags Thread 3:
reading-hctx-sysfs-attribute-nr_tags
-> blk_mq_hw_sysfs_show ==> acquires ->elevaor_lock
-> blk_mq_hw_sysfs_nr_tags_show ==> access nr_tags
Thread 4:
reading-hctx-sysfs-attribute-nr_reserved_tags
-> blk_mq_hw_sysfs_show ==> acquires ->elevaor_lock
-> blk_mq_hw_sysfs_nr_reserved_tags_show ==> access nr_reserved_tags
As we can see above, ->nr_tags and ->nr_reserved_tags are also exported to userspace using hctx sysfs attributes (nr_tags and nr_reserved_tags).
So my point was,
#1 For alleviating race between nr_hw_queues and nr_requests update,
we need protection (probably using srcu lock) so that ->nr_tags and ->nr_reserved_tags are not updated simultaneously.
#2 How could we protect race between thread 3 and thread 2 above or
race between thread 4 and thread 2 above?
Quoted 10 lines
Quoted 4 lines
Maybe we also want to protect
blk_mq_update_nr_requests
with srcu read lock (set->update_nr_hwq_srcu) so that it couldn't run whileblk_mq_update_nr_hw_queuesis in progress?Yeah, agree, and it can be one new patch for covering race between
blk_mq_update_nr_requestsandblk_mq_update_nr_hw_queues, the point is just thatnr_hw_queuesis being changed, and not related with removing hctx attributes, IMO.
Please note that blk_mq_update_nr_requests also updates q->nr_requests, however looking at all code paths which updates this value is already protected with ->elevator_lock. So the only thing which worries me about updates of ->nr_tags and ->nr_reserved tags as shown above.
Thanks,
--Nilay
Re: [PATCH 12/15] block: move debugfs/sysfs register out of freezing queue
Quoted 113 lines · Nilay Shroff
Quoted 69 lines · Ming Lei
Quoted 56 lines · Nilay Shroff
Quoted 53 lines · Ming Lei
Quoted 27 lines · Nilay Shroff
Quoted 24 lines · Ming Lei
Move debugfs/sysfs register out of freezing queue in
__blk_mq_update_nr_hw_queues(), so that the following lockdep dependency can be killed:#2 (&q->q_usage_counter(io)#16){++++}-{0:0}: #1 (fs_reclaim){+.+.}-{0:0}: #0 (&sb->s_type->i_mutex_key#3){+.+.}-{4:4}: //debugfsAnd registering/un-registering debugfs/sysfs does not require queue to be frozen.
Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)block/blk-mq.c +0
-0@@ -4947,15 +4947,15 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,if (set->nr_maps ==1 && nr_hw_queues == set->nr_hw_queues)return ;Quoted 12 lines
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 4953
Quoted 8 lines
block/blk-mq.c +0
-4- memflags =memalloc_noio_save (); -list_for_each_entry (q, &set->tag_list, tag_set_list) -blk_mq_freeze_queue_nomemsave (q); -list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_debugfs_unregister_hctxs (q);blk_mq_sysfs_unregister_hctxs (q); }As we removed hctx sysfs protection while un-registering it, this might cause crash or other side-effect if simultaneously these sysfs attributes are accessed. The read access of these attributes are still protected using ->elevator_lock.
The ->elevator_lock in ->show() is useless except for reading the elevator internal
data(sched tags, requests, ...), even for reading elevator data, it should have been relying on elevator reference, instead of lock, but that is another topic & improvement in future.Also this patch does _not_ change ->elevator_lock for above debugfs/sysfs unregistering, does it? It is always done without holding ->elevator_lock. Also ->show() does not require ->q_usage_counter too.
As I mentioned, kobject/sysfs provides protection between ->show()/->store() and
kobject_del(), isn't it the reason why you want to remove ->sys_lock?Quoted 7 lines
Quoted 2 lines
Yes you were correct, that was the reason we wanted to remove ->sysfs_lock. However for these particular hctx sysfs attributes (
nr_tagsandnr_reserved_tags) could be updated simultaneously from another blk-mq sysfs attribute namednr_requests. Hence IMO, the default protection provided by sysfs/kernfs may not be sufficient and so we need to protect those attributes using ->elevator_lock.Yes, what is why this patchset doesn't kill more ->elevator_lock uses, such as, the uses in blk-mq-debugs,
update_nr_requests, but many of them can be replaced with grabbing elevator reference.Quoted 8 lines
But with/without this patch, the touched register/unregisger code does not require ->elevator_lock:
blk_mq_debugfs_unregister_hctxs(q); blk_mq_sysfs_unregister_hctxs(q);so I don't understand why you argue here about ->elevator_lock use?
I am not arguing using ->elevator_lock wrt removal of hctx sysfs attributes as you explained that sysfs/kernfs already provides the needed protection. But please see below my explanation.
Quoted 8 lines
Quoted 4 lines
Consider this case: While
blk_mq_update_nr_hw_queuesremoves hctx attributes, and simultaneously ifnr_requestsis also updating num of tags, would that not cause any side effect?Why is updating
nr_requestsrelated with removing hctx attributes?Can you explain the side effect in details?
Thread 1:
writing-to-blk-mq-sysfs-attribute-nr_requests
->queue_requests_store==> freezes queue and acquires ->elevator_lock-> blk_mq_update_nr_requests -> blk_mq_tag_update_depth -> blk_mq_alloc_map_and_rqs -> blk_mq_alloc_rq_map -> blk_mq_init_tags ==> updates ->nr_tags and ->nr_reserved_tagsThread2:
blk_mq_update_nr_hw_queues
->__blk_mq_update_nr_hw_queues-> blk_mq_realloc_tag_set_tags -> __blk_mq_alloc_map_and_rqs -> blk_mq_alloc_map_and_rqs -> blk_mq_alloc_rq_map -> blk_mq_init_tags ==> updates ->nr_tags and ->nr_reserved_tagsThread 3:
reading-hctx-sysfs-attribute-nr_tags
->blk_mq_hw_sysfs_show==> acquires ->elevaor_lock
Quoted 6 lines
-> blk_mq_hw_sysfs_nr_tags_show ==> access nr_tagsThread 4:
reading-hctx-sysfs-attribute-nr_reserved_tags
->blk_mq_hw_sysfs_show==> acquires ->elevaor_lock-> blk_mq_hw_sysfs_nr_reserved_tags_show ==> access nr_reserved_tags
hctx->tags is guaranteed to be live if above ->show() method, and the elevator lock is actually not needed, which isn't supposed to protect hctx->tags too.
Quoted 11 lines
As we can see above, ->nr_tags and ->nr_reserved_tags are also exported to userspace using hctx sysfs attributes (
nr_tagsandnr_reserved_tags).So my point was,
#1 For alleviating race betweennr_hw_queuesandnr_requestsupdate,
we need protection (probably using srcu lock) so that ->nr_tags and ->nr_reserved_tags are not updated simultaneously.#2 How could we protect race between thread 3 and thread 2 above or
race between thread 4 and thread 2 above?
blk_mq_update_nr_hw_queues() calls blk_mq_sysfs_unregister_hctxs() first, then user can not see the above attributes before calling blk_mq_sysfs_register_hctxs().
So there isn't the race.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Maybe we also want to protect
blk_mq_update_nr_requests
with srcu read lock (set->update_nr_hwq_srcu) so that it couldn't run whileblk_mq_update_nr_hw_queuesis in progress?
Quoted 6 lines
Quoted 5 lines
Yeah, agree, and it can be one new patch for covering race between
blk_mq_update_nr_requestsandblk_mq_update_nr_hw_queues, the point is just thatnr_hw_queuesis being changed, and not related with removing hctx attributes, IMO.Please note that
blk_mq_update_nr_requestsalso updatesq->nr_requests,
blk_mq_update_nr_requests() uses nr_hw_queues, so there is race between blk_mq_update_nr_requests() and blk_mq_update_nr_hw_queues().
Quoted 3 lines
however looking at all code paths which updates this value is already protected with ->elevator_lock. So the only thing which worries me about updates of ->nr_tags and ->nr_reserved tags as shown above.
As I mentioned, there isn't such race.
Thanks,
Ming
Re: [PATCH 12/15] block: move debugfs/sysfs register out of freezing queue
Quoted 27 lines · Ming Lei
Quoted 27 lines
Quoted 4 lines
Why is updating
nr_requestsrelated with removing hctx attributes?Can you explain the side effect in details?
Thread 1:
writing-to-blk-mq-sysfs-attribute-nr_requests
->queue_requests_store==> freezes queue and acquires ->elevator_lock-> blk_mq_update_nr_requests -> blk_mq_tag_update_depth -> blk_mq_alloc_map_and_rqs -> blk_mq_alloc_rq_map -> blk_mq_init_tags ==> updates ->nr_tags and ->nr_reserved_tagsThread2:
blk_mq_update_nr_hw_queues
->__blk_mq_update_nr_hw_queues-> blk_mq_realloc_tag_set_tags -> __blk_mq_alloc_map_and_rqs -> blk_mq_alloc_map_and_rqs -> blk_mq_alloc_rq_map -> blk_mq_init_tags ==> updates ->nr_tags and ->nr_reserved_tagsThread 3:
reading-hctx-sysfs-attribute-nr_tags
->blk_mq_hw_sysfs_show==> acquires ->elevaor_lock-> blk_mq_hw_sysfs_nr_tags_show ==> access nr_tags
Quoted 9 lines
Quoted 4 lines
Thread 4:
reading-hctx-sysfs-attribute-nr_reserved_tags
->blk_mq_hw_sysfs_show==> acquires ->elevaor_lock-> blk_mq_hw_sysfs_nr_reserved_tags_show ==> access nr_reserved_tags
hctx->tagsis guaranteed to be live if above ->show() method, and the elevator lock is actually not needed, which isn't supposed to protecthctx->tagstoo.
I think, the ->elavtor_lock would still be needed for protecting updates to hctx->tags from thread # 1 above and simultaneously reading the hctx->tags from thread #3 and #4 above.
Quoted 29 lines
Quoted 11 lines
As we can see above, ->nr_tags and ->nr_reserved_tags are also exported to userspace using hctx sysfs attributes (
nr_tagsandnr_reserved_tags).So my point was,
#1 For alleviating race betweennr_hw_queuesandnr_requestsupdate,
we need protection (probably using srcu lock) so that ->nr_tags and ->nr_reserved_tags are not updated simultaneously.#2 How could we protect race between thread 3 and thread 2 above or
race between thread 4 and thread 2 above?
blk_mq_update_nr_hw_queues()callsblk_mq_sysfs_unregister_hctxs()first, then user can not see the above attributes before callingblk_mq_sysfs_register_hctxs().So there isn't the race.
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Maybe we also want to protect
blk_mq_update_nr_requests
with srcu read lock (set->update_nr_hwq_srcu) so that it couldn't run whileblk_mq_update_nr_hw_queuesis in progress?Yeah, agree, and it can be one new patch for covering race between
blk_mq_update_nr_requestsandblk_mq_update_nr_hw_queues, the point is just thatnr_hw_queuesis being changed, and not related with removing hctx attributes, IMO.
Quoted 5 lines
Quoted 1 line
Please note that
blk_mq_update_nr_requestsalso updatesq->nr_requests,
blk_mq_update_nr_requests()usesnr_hw_queues, so there is race betweenblk_mq_update_nr_requests()andblk_mq_update_nr_hw_queues().
Okay so I believe, this could be protected using srcu lock.
Quoted 5 lines
Quoted 3 lines
however looking at all code paths which updates this value is already protected with ->elevator_lock. So the only thing which worries me about updates of ->nr_tags and ->nr_reserved tags as shown above.
As I mentioned, there isn't such race.
Yes agreed, there's no race between thread 3 and thread 2 or thread 4 and thread 2.
Thanks,
--Nilay
Re: [PATCH 12/15] block: move debugfs/sysfs register out of freezing queue
Quoted 35 lines · Nilay Shroff
Quoted 32 lines · Ming Lei
Quoted 31 lines
Quoted 4 lines
Why is updating
nr_requestsrelated with removing hctx attributes?Can you explain the side effect in details?
Thread 1:
writing-to-blk-mq-sysfs-attribute-nr_requests
->queue_requests_store==> freezes queue and acquires ->elevator_lock-> blk_mq_update_nr_requests -> blk_mq_tag_update_depth -> blk_mq_alloc_map_and_rqs -> blk_mq_alloc_rq_map -> blk_mq_init_tags ==> updates ->nr_tags and ->nr_reserved_tagsThread2:
blk_mq_update_nr_hw_queues
->__blk_mq_update_nr_hw_queues-> blk_mq_realloc_tag_set_tags -> __blk_mq_alloc_map_and_rqs -> blk_mq_alloc_map_and_rqs -> blk_mq_alloc_rq_map -> blk_mq_init_tags ==> updates ->nr_tags and ->nr_reserved_tagsThread 3:
reading-hctx-sysfs-attribute-nr_tags
->blk_mq_hw_sysfs_show==> acquires ->elevaor_lock-> blk_mq_hw_sysfs_nr_tags_show ==> access nr_tagsThread 4:
reading-hctx-sysfs-attribute-nr_reserved_tags
->blk_mq_hw_sysfs_show==> acquires ->elevaor_lock-> blk_mq_hw_sysfs_nr_reserved_tags_show ==> access nr_reserved_tags
Quoted 7 lines
Quoted 4 lines
hctx->tagsis guaranteed to be live if above ->show() method, and the elevator lock is actually not needed, which isn't supposed to protecthctx->tagstoo.I think, the ->elavtor_lock would still be needed for protecting updates to
hctx->tagsfrom thread # 1 above and simultaneously reading thehctx->tagsfrom thread #3 and #4 above.
update_nr_requests() can just reduce ->tags.sbitmap size, please see blk_mq_tag_update_depth(), even it won't change hctx->tags->nr_tags, so there isn't race.
More importantly ->elevator_lock shouldn't cover ->tags which isn't related with scheduler, one exception is blk-mq debugfs, in which request may be allocated from ->sched_tags, even ->tags is visiting.
Thanks,
Ming
PATCH 13/15 block: remove several ->elevator_lock
In reply to Ming LeiBoth blk_mq_map_swqueue() and blk_mq_realloc_hw_ctxs() are only called from queue initialization or updating nr_hw_queues code, in which elevator switch can't happen any more.
So remove these ->elevator_lock uses.
--- block/blk-mq.c | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-)
block/blk-mq.c +4 -15
@@ -4095,8 +4095,6 @@ static void blk_mq_map_swqueue(struct request_queue *q)struct blk_mq_ctx *ctx;struct blk_mq_tag_set *set = q->tag_set; -mutex_lock (&q->elevator_lock); -queue_for_each_hw_ctx (q, hctx, i) {cpumask_clear (hctx->cpumask); hctx->nr_ctx =0 ;
@@ -4201,8 +4199,6 @@ static void blk_mq_map_swqueue(struct request_queue *q) hctx->next_cpu =blk_mq_first_mapped_cpu (hctx); hctx->next_cpu_batch =BLK_MQ_CPU_WORK_BATCH ; } - -mutex_unlock (&q->elevator_lock); }/*
@@ -4506,16 +4502,9 @@ static void __blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, }static void blk_mq_realloc_hw_ctxs (struct blk_mq_tag_set *set, -struct request_queue *q,bool lock) +struct request_queue *q) { -if (lock) { -/* protect against switching io scheduler */ -mutex_lock (&q->elevator_lock); -__blk_mq_realloc_hw_ctxs (set, q); -mutex_unlock (&q->elevator_lock); - }else { -__blk_mq_realloc_hw_ctxs (set, q); - } +__blk_mq_realloc_hw_ctxs (set, q);/* unregister cpuhp callbacks for exited hctxs */ blk_mq_remove_hw_queues_cpuhp (q);
@@ -4547,7 +4536,7 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,xa_init (&q->hctx_table); -blk_mq_realloc_hw_ctxs (set, q, false); +blk_mq_realloc_hw_ctxs (set, q);if (!q->nr_hw_queues) goto err_hctxs;
@@ -4962,7 +4951,7 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, fallback:blk_mq_update_queue_map (set);list_for_each_entry (q, &set->tag_list, tag_set_list) { -blk_mq_realloc_hw_ctxs (set, q, true); +blk_mq_realloc_hw_ctxs (set, q);if (q->nr_hw_queues != set->nr_hw_queues) {int i = prev_nr_hw_queues;
Signature · 1 line
2.47.0
Re: [PATCH 13/15] block: remove several ->elevator_lock
Quoted 6 lines · Ming Lei
Both
blk_mq_map_swqueue()andblk_mq_realloc_hw_ctxs()are only called from queue initialization or updatingnr_hw_queuescode, in which elevator switch can't happen any more.So remove these ->elevator_lock uses.
But what if blk_mq_map_swqueue runs in parallel, one context from blk_mq_init_allocated_queue and another from blk_mq_update_nr_hw_queues? It seems this is possible due to blk_mq_map_swqueue is invoked right after queue is added in tag-set from blk_mq_init_allocated_queue.
Thanks,
--Nilay
Re: [PATCH 13/15] block: remove several ->elevator_lock
Quoted 7 lines · Nilay Shroff
Quoted 4 lines · Ming Lei
Both
blk_mq_map_swqueue()andblk_mq_realloc_hw_ctxs()are only called from queue initialization or updatingnr_hw_queuescode, in which elevator switch can't happen any more.
Quoted 6 lines
Quoted 2 lines
So remove these ->elevator_lock uses.
But what if
blk_mq_map_swqueueruns in parallel, one context fromblk_mq_init_allocated_queueand another fromblk_mq_update_nr_hw_queues? It seems this is possible due toblk_mq_map_swqueueis invoked right after queue is added in tag-set fromblk_mq_init_allocated_queue.
Good catch, one simple fix is to swap blk_mq_add_queue_tag_set() with blk_mq_map_swqueue() in blk_mq_init_allocated_queue() since blk_mq_map_swqueue doesn't rely on BLK_MQ_F_TAG_QUEUE_SHARED.
Thanks,
Ming
PATCH 14/15 block: move hctx cpuhp add/del out of queue freezing
In reply to Ming LeiMove hctx cpuhp add/del out of queue freezing for not connecting freeze lock with cpuhp locks, then lockdep warning can be avoided.
This way is safe because both needn't queue to be frozen and scheduler switch isn't allowed.
--- block/blk-mq.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
block/blk-mq.c +4 -1
@@ -4951,7 +4951,7 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, fallback:blk_mq_update_queue_map (set);list_for_each_entry (q, &set->tag_list, tag_set_list) { -blk_mq_realloc_hw_ctxs (set, q); +__blk_mq_realloc_hw_ctxs (set, q);if (q->nr_hw_queues != set->nr_hw_queues) {int i = prev_nr_hw_queues;
@@ -4990,6 +4990,9 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_sysfs_register_hctxs (q);blk_mq_debugfs_register_hctxs (q); + +blk_mq_remove_hw_queues_cpuhp (q); +blk_mq_add_hw_queues_cpuhp (q); }/* Free the excess tags when nr_hw_queues shrink. */
Signature · 1 line
2.47.0
PATCH 15/15 block: move wbt_enable_default() out of queue freezing from scheduler's ->exit()
In reply to Ming Leischeduler's ->exit() is called with queue frozen and elevator lock is held, and wbt_enable_default() can't be called with queue frozen, otherwise the following lockdep warning is triggered:
#6 (&q->rq_qos_mutex){+.+.}-{4:4}:
#5 (&eq->sysfs_lock){+.+.}-{4:4}:
#4 (&q->elevator_lock){+.+.}-{4:4}:
#3 (&q->q_usage_counter(io)#3){++++}-{0:0}:
#2 (fs_reclaim){+.+.}-{0:0}:
#1 (&sb->s_type->i_mutex_key#3){+.+.}-{4:4}:
#0 (&q->debugfs_mutex){+.+.}-{4:4}:Fix the issue by moving wbt_enable_default() out of bfq's exit(), and call it from elevator_change_done().
--- block/bfq-iosched.c | 2 +- block/elevator.c | 5 +++++ block/elevator.h | 1 + 3 files changed, 7 insertions(+), 1 deletion(-)
block/bfq-iosched.c +1 -1
@@ -7211,7 +7211,7 @@ static void bfq_exit_queue(struct elevator_queue *e)blk_stat_disable_accounting (bfqd->queue);clear_bit (ELEVATOR_FLAG_DISABLE_WBT , &e->flags); -wbt_enable_default (bfqd->queue->disk); +set_bit (ELEVATOR_FLAG_ENABLE_WBT_ON_EXIT , &e->flags);kfree (bfqd); }
block/elevator.c +5 -0
@@ -676,8 +676,13 @@ int elevator_change_done(struct request_queue *q, struct elev_change_ctx *ctx)int ret =0 ;if (ctx->old) { +bool enable_wbt =test_bit (ELEVATOR_FLAG_ENABLE_WBT_ON_EXIT , + &ctx->old->flags); +elv_unregister_queue (q, ctx->old);kobject_put (&ctx->old->kobj); +if (enable_wbt) +wbt_enable_default (q->disk); }if (ctx->new) { ret =elv_register_queue (q, ctx->new, ctx->uevent);
block/elevator.h +1 -0
@@ -122,6 +122,7 @@ struct elevator_queue#define ELEVATOR_FLAG_REGISTERED 0#define ELEVATOR_FLAG_DISABLE_WBT 1 +#define ELEVATOR_FLAG_ENABLE_WBT_ON_EXIT 2/* Holding data for changing elevator */ struct elev_change_ctx {
Signature · 1 line
2.47.0
Re: [PATCH 15/15] block: move wbt_enable_default() out of queue freezing from scheduler's ->exit()
Quoted 14 lines · Ming Lei
scheduler's ->exit() is called with queue frozen and elevator lock is held, and
wbt_enable_default()can't be called with queue frozen, otherwise the following lockdep warning is triggered:#6 (&q->rq_qos_mutex){+.+.}-{4:4}: #5 (&eq->sysfs_lock){+.+.}-{4:4}: #4 (&q->elevator_lock){+.+.}-{4:4}: #3 (&q->q_usage_counter(io)#3){++++}-{0:0}: #2 (fs_reclaim){+.+.}-{0:0}: #1 (&sb->s_type->i_mutex_key#3){+.+.}-{4:4}: #0 (&q->debugfs_mutex){+.+.}-{4:4}:Fix the issue by moving
wbt_enable_default()out of bfq'sexit(), and call it fromelevator_change_done().
[...]
Quoted 6 lines
block/elevator.c +0
-0@@ -676,8 +676,13 @@ int elevator_change_done(struct request_queue *q, struct elev_change_ctx *ctx)int ret =0 ;
block/elevator.c int elevator_change_done(struct request_queue *q, struct elev_change_ctx *ctx) · line 686
Quoted 10 lines
block/elevator.c +5
-0if (ctx->old) { +bool enable_wbt =test_bit (ELEVATOR_FLAG_ENABLE_WBT_ON_EXIT , + &ctx->old->flags); +elv_unregister_queue (q, ctx->old);kobject_put (&ctx->old->kobj); +if (enable_wbt) +wbt_enable_default (q->disk); }
wbt_enable_default is also called from ioc_qos_write and blk_register_queue with ->elevator_lock protection. So avoiding protection here doesn't seem correct.
Thanks,
--Nilay
Re: [PATCH 15/15] block: move wbt_enable_default() out of queue freezing from scheduler's ->exit()
Quoted 27 lines · Nilay Shroff
Quoted 14 lines · Ming Lei
scheduler's ->exit() is called with queue frozen and elevator lock is held, and
wbt_enable_default()can't be called with queue frozen, otherwise the following lockdep warning is triggered:#6 (&q->rq_qos_mutex){+.+.}-{4:4}: #5 (&eq->sysfs_lock){+.+.}-{4:4}: #4 (&q->elevator_lock){+.+.}-{4:4}: #3 (&q->q_usage_counter(io)#3){++++}-{0:0}: #2 (fs_reclaim){+.+.}-{0:0}: #1 (&sb->s_type->i_mutex_key#3){+.+.}-{4:4}: #0 (&q->debugfs_mutex){+.+.}-{4:4}:Fix the issue by moving
wbt_enable_default()out of bfq'sexit(), and call it fromelevator_change_done().[...]
Quoted 7 lines
block/elevator.c +0
-0@@ -676,8 +676,13 @@ int elevator_change_done(struct request_queue *q, struct elev_change_ctx *ctx)int ret =0 ;
Quoted 12 lines
block/elevator.c int elevator_change_done(struct request_queue *q, struct elev_change_ctx *ctx) · line 686
Quoted 9 lines
block/elevator.c +5
-0if (ctx->old) { +bool enable_wbt =test_bit (ELEVATOR_FLAG_ENABLE_WBT_ON_EXIT , + &ctx->old->flags); +elv_unregister_queue (q, ctx->old);kobject_put (&ctx->old->kobj); +if (enable_wbt) +wbt_enable_default (q->disk); }
wbt_enable_defaultis also called fromioc_qos_writeandblk_register_queuewith ->elevator_lock protection. So avoiding protection here doesn't seem correct.
The only code which needs the protection should be:
``` wbt_enable_default(): ... if (q->elevator && test_bit(ELEVATOR_FLAG_DISABLE_WBT, &q->elevator->flags)) enable = false; ```
The flag can be moved to request queue from elevator queue, will do it in V2.
Thanks,
Ming