blk-mq vs cpu hotplug performance (due to percpu_ref_put performance)
Subsystems (2)
Tejun,
when going from 3.17 to 3.18-rc2 cpu hotplug become horrible slow on some KVM guests on s390
I was able to bisect this to
commit 9eca80461a45177e456219a9cd944c27675d6512
("Revert "blk-mq, percpu_ref: implement a kludge for SCSI blk-mq stall during probe")
Seems that this is due to all the rcu grace periods on percpu_ref_put during the cpu hotplug notifiers.
This is barely noticable on small guests (lets say 1 virtio disk), but on guests with 20 disks a hotplug takes 2 or 3 instead of around 0.1 sec. There are three things that make this especially noticably on s390:
- s390 has 100HZ which makes grace period waiting slower
- s390 does not yet implement context tracking which would speed up RCU
- s390 systems usually have a bigger amount of disk (e.g. 20 7GB disks instead of one 140GB disks)
Any idea how to improve the situation? I think we could accept an expedited variant on cpu hotplug, since stop_machine_run will cause hickups anyway, but there are probably other callers.
Christian
PS: on the plus side, this makes CPU hotplug races less likely....
Hello,
Quoted 6 lines · Christian Borntraeger
when going from 3.17 to 3.18-rc2 cpu hotplug become horrible slow on some KVM guests on s390
I was able to bisect this to
commit
9eca80461a45177e456219a9cd944c27675d6512
("Revert "blk-mq,percpu_ref: implement a kludge for SCSI blk-mq stall during probe")
That removes the earlier kludge to avoid the RCU delay so RCU latencies are expected to show up right after; however, the following patches implement proper fix for the problem and the latencies shouldn't be visible afterwards.
So, 17497acbdce9 ("blk-mq, percpu_ref: start q->mq_usage_counter in atomic mode") should remove the latencies again. It doesn't?
Thanks.
Signature · 1 line
tejun
Am 28.10.2014 21:00, schrieb Tejun Heo:
Quoted 10 lines
Hello,
Quoted 6 lines · Christian Borntraeger
when going from 3.17 to 3.18-rc2 cpu hotplug become horrible slow on some KVM guests on s390
I was able to bisect this to
commit
9eca80461a45177e456219a9cd944c27675d6512
("Revert "blk-mq,percpu_ref: implement a kludge for SCSI blk-mq stall during probe")
Quoted 7 lines
That removes the earlier kludge to avoid the RCU delay so RCU latencies are expected to show up right after; however, the following patches implement proper fix for the problem and the latencies shouldn't be visible afterwards.
So,
17497acbdce9("blk-mq,percpu_ref: startq->mq_usage_counterin atomic mode") should remove the latencies again. It doesn't?
I have not verified this, but I guess what happens is:
hotplug
-> notify
-> blk_mq_queue_reinit_notify
-> blk_mq_queue_reinit
-> blk_mq_freeze_queue
-> percpu_ref_kill
-> percpu_ref_kill_and_confirm
-> __percpu_ref_switch_to_atomic
-> call_rcu_sched
for every request queue.
Christian
Quoted 10 lines · Christian Borntraeger
I have not verified this, but I guess what happens is:
hotplug
-> notify
->blk_mq_queue_reinit_notify
->blk_mq_queue_reinit
->blk_mq_freeze_queue
->percpu_ref_kill
->percpu_ref_kill_and_confirm
->__percpu_ref_switch_to_atomic
->call_rcu_sched
But call_rcu_sched() wouldn't show up as latency. It's an async call unlike synchronize_*().
Thanks.
Signature · 1 line
tejun
Quoted 12 lines · Tejun Heo
Quoted 10 lines · Christian Borntraeger
I have not verified this, but I guess what happens is:
hotplug
-> notify
->blk_mq_queue_reinit_notify
->blk_mq_queue_reinit
->blk_mq_freeze_queue
->percpu_ref_kill
->percpu_ref_kill_and_confirm
->__percpu_ref_switch_to_atomic
->call_rcu_sched
Quoted 2 lines
But
call_rcu_sched()wouldn't show up as latency. It's an async call unlike synchronize_*().
I got confused, so perpcu_ref does wait for the async grace period making it synchronous. I see what you mean. This isn't during init but freezing itself being slow. Hmmmm... so are you seeing multiple queues doing that back-to-back? If so, the right thing to do would be making the freezing take place in parallel. I'll look into it.
Thanks.
Signature · 1 line
tejun
Am 28.10.2014 21:22, schrieb Tejun Heo:
Quoted 12 lines · Christian Borntraeger
Quoted 10 lines
I have not verified this, but I guess what happens is:
hotplug
-> notify
->blk_mq_queue_reinit_notify
->blk_mq_queue_reinit
->blk_mq_freeze_queue
->percpu_ref_kill
->percpu_ref_kill_and_confirm
->__percpu_ref_switch_to_atomic
->call_rcu_sched
Quoted 2 lines
But
call_rcu_sched()wouldn't show up as latency. It's an async call unlike synchronize_*().
Right, but
blk_mq_freeze_queue
also contains
wait_event(q->mq_freeze_wq, percpu_ref_is_zero(&q->mq_usage_counter));
Isnt that wait_event woken up at the end of the call_rcu_sched?
Quoted 18 lines · Christian Borntraeger
Am 28.10.2014 21:22, schrieb Tejun Heo:
Quoted 12 lines · Christian Borntraeger
Quoted 10 lines
I have not verified this, but I guess what happens is:
hotplug
-> notify
->blk_mq_queue_reinit_notify
->blk_mq_queue_reinit
->blk_mq_freeze_queue
->percpu_ref_kill
->percpu_ref_kill_and_confirm
->__percpu_ref_switch_to_atomic
->call_rcu_schedQuoted 2 lines
But
call_rcu_sched()wouldn't show up as latency. It's an async call unlike synchronize_*().Right, but
Quoted 7 lines
blk_mq_freeze_queuealso contains
wait_event(q->mq_freeze_wq, percpu_ref_is_zero(&q->mq_usage_counter));Isnt that
wait_eventwoken up at the end of thecall_rcu_sched?
Yeah, yeah, I was confused. We just need to initiate the killing for all mqs at once and then wait for the completions. Shouldn't be too difficult to fix. Will get to it soon.
Thanks.
Signature · 1 line
tejun
PATCH 1/1 blk-mq: make mq_queue_reinit_notify() freeze queues in parallel
q->mq_usage_counter is a percpu_ref which is killed and drained when the queue is frozen. On a CPU hotplug event, blk_mq_queue_reinit() which involves freezing the queue is invoked on all existing queues. Because percpu_ref killing and draining involve a RCU grace period, doing the above on one queue after another may take a long time if there are many queues on the system.
This patch splits out initiation of freezing and waiting for its completion, and updates blk_mq_queue_reinit_notify() so that the queues are frozen in parallel instead of one after another. Note that freezing and unfreezing are moved from blk_mq_queue_reinit() to blk_mq_queue_reinit_notify().
---
Christian, can you please verify that this resolves the latency issue that you're seeing? Jens, can you please route this patch once Christian confirms it?
Thanks!
block/blk-mq.c | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-)
block/blk-mq.c +33 -8
@@ -107,11 +107,7 @@ static void blk_mq_usage_counter_releasewake_up_all (&q->mq_freeze_wq); } -/* -* Guarantee no request is in use, so we can change any data structure of -* the queue afterward. -*/ -void blk_mq_freeze_queue (struct request_queue *q) +static void blk_mq_freeze_queue_start (struct request_queue *q) {bool freeze;
@@ -123,9 +119,23 @@ void blk_mq_freeze_queue(struct request_percpu_ref_kill (&q->mq_usage_counter);blk_mq_run_queues (q, false); } +} + +static void blk_mq_freeze_queue_wait (struct request_queue *q) +{wait_event (q->mq_freeze_wq,percpu_ref_is_zero (&q->mq_usage_counter)); } +/* +* Guarantee no request is in use, so we can change any data structure of +* the queue afterward. +*/ +void blk_mq_freeze_queue (struct request_queue *q) +{ +blk_mq_freeze_queue_start (q); +blk_mq_freeze_queue_wait (q); +} +static void blk_mq_unfreeze_queue (struct request_queue *q) {bool wake;
@@ -1921,7 +1931,7 @@ void blk_mq_free_queue(struct request_qu/* Basically redo blk_mq_init_queue with queue frozen */ static void blk_mq_queue_reinit (struct request_queue *q) { -blk_mq_freeze_queue (q); +WARN_ON_ONCE (!q->mq_freeze_depth);blk_mq_sysfs_unregister (q);
@@ -1936,8 +1946,6 @@ static void blk_mq_queue_reinit(struct rblk_mq_map_swqueue (q);blk_mq_sysfs_register (q); - -blk_mq_unfreeze_queue (q); } static int blk_mq_queue_reinit_notify(struct notifier_block *nb,
@@ -1956,8 +1964,25 @@ static int blk_mq_queue_reinit_notify(streturn NOTIFY_OK ;mutex_lock (&all_q_mutex); + +/* +* We need to freeze and reinit all existing queues. Freezing +* involves synchronous wait for an RCU grace period and doing it +* one by one may take a long time. Start freezing all queues in +* one swoop and then wait for the completions so that freezing can +* take place in parallel. +*/ +list_for_each_entry (q, &all_q_list, all_q_node) +blk_mq_freeze_queue_start (q); +list_for_each_entry (q, &all_q_list, all_q_node) +blk_mq_freeze_queue_wait (q); +list_for_each_entry (q, &all_q_list, all_q_node)blk_mq_queue_reinit (q); + +list_for_each_entry (q, &all_q_list, all_q_node) +blk_mq_unfreeze_queue (q); +mutex_unlock (&all_q_mutex);return NOTIFY_OK ; }
Re: [PATCH block/for-linus] blk-mq: make mq_queue_reinit_notify() freeze queues in parallel
Am 04.11.2014 19:52, schrieb Tejun Heo:
Quoted 7 lines
q->mq_usage_counteris apercpu_refwhich is killed and drained when the queue is frozen. On a CPU hotplug event,blk_mq_queue_reinit()which involves freezing the queue is invoked on all existing queues. Becausepercpu_refkilling and draining involve a RCU grace period, doing the above on one queue after another may take a long time if there are many queues on the system.
Quoted 8 lines
This patch splits out initiation of freezing and waiting for its completion, and updates
blk_mq_queue_reinit_notify()so that the queues are frozen in parallel instead of one after another. Note that freezing and unfreezing are moved fromblk_mq_queue_reinit()toblk_mq_queue_reinit_notify().Signed-off-by Tejun Heo <[email protected]>Reported-by Christian Borntraeger <[email protected]>
Thanks.
Quoted 94 lines
---
Christian, can you please verify that this resolves the latency issue that you're seeing? Jens, can you please route this patch once Christian confirms it?Thanks!
block/blk-mq.c | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-)block/blk-mq.c +33
-8@@ -107,11 +107,7 @@ static void blk_mq_usage_counter_releasewake_up_all (&q->mq_freeze_wq); } -/* -* Guarantee no request is in use, so we can change any data structure of -* the queue afterward. -*/ -void blk_mq_freeze_queue (struct request_queue *q) +static void blk_mq_freeze_queue_start (struct request_queue *q) {bool freeze;@@ -123,9 +119,23 @@ void blk_mq_freeze_queue(struct request_percpu_ref_kill (&q->mq_usage_counter);blk_mq_run_queues (q, false); } +} + +static void blk_mq_freeze_queue_wait (struct request_queue *q) +{wait_event (q->mq_freeze_wq,percpu_ref_is_zero (&q->mq_usage_counter)); } +/* +* Guarantee no request is in use, so we can change any data structure of +* the queue afterward. +*/ +void blk_mq_freeze_queue (struct request_queue *q) +{ +blk_mq_freeze_queue_start (q); +blk_mq_freeze_queue_wait (q); +} +static void blk_mq_unfreeze_queue (struct request_queue *q) {bool wake;@@ -1921,7 +1931,7 @@ void blk_mq_free_queue(struct request_qu/* Basically redo blk_mq_init_queue with queue frozen */ static void blk_mq_queue_reinit (struct request_queue *q) { -blk_mq_freeze_queue (q); +WARN_ON_ONCE (!q->mq_freeze_depth);blk_mq_sysfs_unregister (q);@@ -1936,8 +1946,6 @@ static void blk_mq_queue_reinit(struct rblk_mq_map_swqueue (q);blk_mq_sysfs_register (q); - -blk_mq_unfreeze_queue (q); } static int blk_mq_queue_reinit_notify(struct notifier_block *nb,@@ -1956,8 +1964,25 @@ static int blk_mq_queue_reinit_notify(streturn NOTIFY_OK ;mutex_lock (&all_q_mutex); + +/* +* We need to freeze and reinit all existing queues. Freezing +* involves synchronous wait for an RCU grace period and doing it +* one by one may take a long time. Start freezing all queues in +* one swoop and then wait for the completions so that freezing can +* take place in parallel. +*/ +list_for_each_entry (q, &all_q_list, all_q_node) +blk_mq_freeze_queue_start (q); +list_for_each_entry (q, &all_q_list, all_q_node) +blk_mq_freeze_queue_wait (q); +list_for_each_entry (q, &all_q_list, all_q_node)blk_mq_queue_reinit (q); + +list_for_each_entry (q, &all_q_list, all_q_node) +blk_mq_unfreeze_queue (q); +mutex_unlock (&all_q_mutex);return NOTIFY_OK ; }
Jens Axboe
maintainerRe: [PATCH block/for-linus] blk-mq: make mq_queue_reinit_notify() freeze queues in parallel
In reply to Tejun HeoQuoted 13 lines · Tejun Heo
q->mq_usage_counteris apercpu_refwhich is killed and drained when the queue is frozen. On a CPU hotplug event,blk_mq_queue_reinit()which involves freezing the queue is invoked on all existing queues. Becausepercpu_refkilling and draining involve a RCU grace period, doing the above on one queue after another may take a long time if there are many queues on the system.This patch splits out initiation of freezing and waiting for its completion, and updates
blk_mq_queue_reinit_notify()so that the queues are frozen in parallel instead of one after another. Note that freezing and unfreezing are moved fromblk_mq_queue_reinit()toblk_mq_queue_reinit_notify().
Quoted 6 lines
Signed-off-by Tejun Heo <[email protected]>Reported-by Christian Borntraeger <[email protected]>---
Christian, can you please verify that this resolves the latency issue that you're seeing? Jens, can you please route this patch once Christian confirms it?
Will queue up for 3.18.
Signature · 1 line
Jens Axboe