blk-ioc: don't hold queue_lock for ioc_lookup_icq()
Subsystems (2)
PATCH v2 1/1 blk-ioc: don't hold queue_lock for ioc_lookup_icq()
From: Yu Kuai <[email protected]>
Currently issue io can grab queue_lock three times from bfq_bio_merge(), bfq_limit_depth() and bfq_prepare_request(), the queue_lock is not necessary if icq is already created because both queue and ioc can't be freed before io issuing is done, hence remove the unnecessary queue_lock and use rcu to protect radix tree lookup.
Noted this is also a prep patch to support request batch dispatching[1].
[1] https://lore.kernel.org/all/[email protected]/
---
changes from v1:
- modify ioc_lookup_icq() directly to get rid of queue_lock
block/bfq-iosched.c | 18 ++---------------- block/blk-ioc.c | 10 +++------- 2 files changed, 5 insertions(+), 23 deletions(-)
block/bfq-iosched.c +2 -16
@@ -454,17 +454,10 @@ static struct bfq_io_cq *icq_to_bic(struct io_cq *icq)*/ static struct bfq_io_cq *bfq_bic_lookup (struct request_queue *q) { -struct bfq_io_cq *icq; -unsigned long flags; -if (!current->io_context)return NULL ; -spin_lock_irqsave (&q->queue_lock, flags); - icq =icq_to_bic (ioc_lookup_icq (q)); -spin_unlock_irqrestore (&q->queue_lock, flags); - -return icq; +return icq_to_bic (ioc_lookup_icq (q)); }/*
@@ -2457,15 +2450,8 @@ static bool bfq_bio_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs) {struct bfq_data *bfqd = q->elevator->elevator_data; -struct request *free =NULL ; -/* -* bfq_bic_lookup grabs the queue_lock: invoke it now and -* store its return value for later use, to avoid nesting -* queue_lock inside the bfqd->lock. We assume that the bic -* returned by bfq_bic_lookup does not go away before -* bfqd->lock is taken. -*/ struct bfq_io_cq *bic =bfq_bic_lookup (q); +struct request *free =NULL ;bool ret;spin_lock_irq (&bfqd->lock);
block/blk-ioc.c +3 -7
@@ -308,19 +308,18 @@ int __copy_io(unsigned long clone_flags, struct task_struct *tsk)#ifdef CONFIG_BLK_ICQ /** -* ioc_lookup_icq - lookup io_cq from ioc +* ioc_lookup_icq - lookup io_cq from ioc in io issue path * @q: the associated request_queue * * Look up io_cq associated with @ioc - @q pair from @ioc. Must be called -* with @q->queue_lock held. +* from io issue path, either return NULL if current issue io to @q for the +* first time, or return a valid icq. */ struct io_cq *ioc_lookup_icq (struct request_queue *q) {struct io_context *ioc = current->io_context;struct io_cq *icq; -lockdep_assert_held (&q->queue_lock); -/* * icq's are indexed from @ioc using radix tree and hint pointer, * both of which are protected with RCU. All removals are done
@@ -419,10 +418,7 @@ struct io_cq *ioc_find_get_icq(struct request_queue *q)task_unlock (current); }else {get_io_context (ioc); - -spin_lock_irq (&q->queue_lock); icq =ioc_lookup_icq (q); -spin_unlock_irq (&q->queue_lock); }if (!icq) {
Signature · 1 line
2.43.0
Quoted 8 lines · Yu Kuai
From: Yu Kuai <[email protected]>
Currently issue io can grab
queue_lockthree times frombfq_bio_merge(),bfq_limit_depth()andbfq_prepare_request(), thequeue_lockis not necessary if icq is already created because both queue and ioc can't be freed before io issuing is done, hence remove the unnecessaryqueue_lockand use rcu to protect radix tree lookup.
Quoted 4 lines
Noted this is also a prep patch to support request batch dispatching[1].
[1] https://lore.kernel.org/all/[email protected]/
Signed-off-by Yu Kuai <[email protected]>
Look OK.
Signature · 2 lines
Damien Le Moal Western Digital Research
Quoted 8 lines · Yu Kuai
From: Yu Kuai <[email protected]>
Currently issue io can grab
queue_lockthree times frombfq_bio_merge(),bfq_limit_depth()andbfq_prepare_request(), thequeue_lockis not necessary if icq is already created because both queue and ioc can't be freed before io issuing is done, hence remove the unnecessaryqueue_lockand use rcu to protect radix tree lookup.
Quoted 4 lines
Noted this is also a prep patch to support request batch dispatching[1].
[1] https://lore.kernel.org/all/[email protected]/
Signed-off-by Yu Kuai <[email protected]>
Looks good! Just one small comment below. With that fixed feel free to add:
Quoted 17 lines
block/blk-ioc.c +3
-2@@ -308,19 +308,18 @@ int __copy_io(unsigned long clone_flags, struct task_struct *tsk)#ifdef CONFIG_BLK_ICQ /** -* ioc_lookup_icq - lookup io_cq from ioc +* ioc_lookup_icq - lookup io_cq from ioc in io issue path * @q: the associated request_queue * * Look up io_cq associated with @ioc - @q pair from @ioc. Must be called -* with @q->queue_lock held. +* from io issue path, either return NULL if current issue io to @q for the +* first time, or return a valid icq. */
block/blk-ioc.c int __copy_io(unsigned long clone_flags, struct task_struct *tsk) · line 325
Quoted 10 lines
block/blk-ioc.c +0
-2struct io_cq *ioc_lookup_icq (struct request_queue *q) {struct io_context *ioc = current->io_context;struct io_cq *icq; -lockdep_assert_held (&q->queue_lock); -/* * icq's are indexed from @ioc using radix tree and hint pointer, * both of which are protected with RCU. All removals are done
In this comment there's still reference to holding 'q lock'. I think you should replace that with justification why when called from IO issue path we are guaranteed found pointer is safe to use.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Hi,
Quoted 39 lines · Jan Kara
Quoted 8 lines · Yu Kuai
From: Yu Kuai <[email protected]>
Currently issue io can grab
queue_lockthree times frombfq_bio_merge(),bfq_limit_depth()andbfq_prepare_request(), thequeue_lockis not necessary if icq is already created because both queue and ioc can't be freed before io issuing is done, hence remove the unnecessaryqueue_lockand use rcu to protect radix tree lookup.Quoted 4 lines
Noted this is also a prep patch to support request batch dispatching[1].
[1] https://lore.kernel.org/all/[email protected]/
Signed-off-by Yu Kuai <[email protected]>Looks good! Just one small comment below. With that fixed feel free to add:
Reviewed-by Jan Kara <[email protected]>Quoted 22 lines
block/blk-ioc.c +3
-2@@ -308,19 +308,18 @@ int __copy_io(unsigned long clone_flags, struct task_struct *tsk)#ifdef CONFIG_BLK_ICQ /** -* ioc_lookup_icq - lookup io_cq from ioc +* ioc_lookup_icq - lookup io_cq from ioc in io issue path * @q: the associated request_queue * * Look up io_cq associated with @ioc - @q pair from @ioc. Must be called -* with @q->queue_lock held. +* from io issue path, either return NULL if current issue io to @q for the +* first time, or return a valid icq. */ struct io_cq *ioc_lookup_icq (struct request_queue *q) {struct io_context *ioc = current->io_context;struct io_cq *icq;
Quoted 8 lines
block/blk-ioc.c int __copy_io(unsigned long clone_flags, struct task_struct *tsk) · line 325
Quoted 5 lines
block/blk-ioc.c +0
-2-lockdep_assert_held (&q->queue_lock); -/* * icq's are indexed from @ioc using radix tree and hint pointer, * both of which are protected with RCU. All removals are done In this comment there's still reference to holding 'q lock'. I think you should replace that with justification why when called from IO issue path we are guaranteed found pointer is safe to use.
Thanks for the review! How about:
block/blk-ioc.c +0 -0
@@ -322,9 +322,9 @@ struct io_cq *ioc_lookup_icq(struct request_queue *q)
/*
* icq's are indexed from @ioc using radix tree and hint pointer, - * both of which are protected with RCU. All removals are done - * holding both q and ioc locks, and we're holding q lock - if we - * find a icq which points to us, it's guaranteed to be valid. + * both of which are protected with RCU, io issue path ensures that + * both request_queue and current task are valid, the founded icq + * is guaranteed to be valid until the io is done.
*/
rcu_read_lock();
icq = rcu_dereference(ioc->icq_hint);
Quoted 2 lines
Honza
Quoted 48 lines · Yu Kuai
Hi,
Quoted 37 lines · Jan Kara
Quoted 8 lines · Yu Kuai
From: Yu Kuai <[email protected]>
Currently issue io can grab
queue_lockthree times frombfq_bio_merge(),bfq_limit_depth()andbfq_prepare_request(), thequeue_lockis not necessary if icq is already created because both queue and ioc can't be freed before io issuing is done, hence remove the unnecessaryqueue_lockand use rcu to protect radix tree lookup.Quoted 4 lines
Noted this is also a prep patch to support request batch dispatching[1].
[1] https://lore.kernel.org/all/[email protected]/
Signed-off-by Yu Kuai <[email protected]>Looks good! Just one small comment below. With that fixed feel free to add:
Reviewed-by Jan Kara <[email protected]>Quoted 20 lines
block/blk-ioc.c +3
-2@@ -308,19 +308,18 @@ int __copy_io(unsigned long clone_flags, struct task_struct *tsk)#ifdef CONFIG_BLK_ICQ /** -* ioc_lookup_icq - lookup io_cq from ioc +* ioc_lookup_icq - lookup io_cq from ioc in io issue path * @q: the associated request_queue * * Look up io_cq associated with @ioc - @q pair from @ioc. Must be called -* with @q->queue_lock held. +* from io issue path, either return NULL if current issue io to @q for the +* first time, or return a valid icq. */ struct io_cq *ioc_lookup_icq (struct request_queue *q) {struct io_context *ioc = current->io_context;struct io_cq *icq;Quoted 8 lines
block/blk-ioc.c int __copy_io(unsigned long clone_flags, struct task_struct *tsk) · line 323
Quoted 5 lines
block/blk-ioc.c +0
-2-lockdep_assert_held (&q->queue_lock); -/* * icq's are indexed from @ioc using radix tree and hint pointer, * both of which are protected with RCU. All removals are done In this comment there's still reference to holding 'q lock'. I think you should replace that with justification why when called from IO issue path we are guaranteed found pointer is safe to use.
Quoted 1 line
Thanks for the review! How about:
Just one spelling correction. Otherwise looks great to me.
block/blk-ioc.c struct io_cq *ioc_lookup_icq(struct request_queue *q) · line 326
Quoted 13 lines
block/blk-ioc.c +2
-3@@ -322,9 +322,9 @@ struct io_cq *ioc_lookup_icq(struct request_queue *q) /** icq's are indexed from @ioc using radix tree and hint pointer, -* both of which are protected with RCU. All removals are done -* holding both q and ioc locks, and we're holding q lock - if we -* find a icq which points to us, it's guaranteed to be valid. +* both of which are protected with RCU, io issue path ensures that +* both request_queue and current task are valid, the founded icq
^^^^ thus the
found icq
block/blk-ioc.c struct io_cq *ioc_lookup_icq(struct request_queue *q) · line 330
Quoted 4 lines
block/blk-ioc.c +1
-0+* is guaranteed to be valid until the io is done. */ rcu_read_lock(); icq = rcu_dereference(ioc->icq_hint);
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR