PATCH: softlockup: Fix hung_task_check_count sysctl
I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting hung_task_check_count to 0:
# echo 0 > /proc/sys/kernel/hung_task_check_count
which didn't have the intended response. Change to a post increment of max_count, so a value of 0 means check 0 tasks.
---
kernel/hung_task.c +1 -1
@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_trcu_read_lock (); do_each_thread(g, t) { -if (!--max_count) +if (!max_count--) goto unlock;if (!--batch_count) { batch_count =HUNG_TASK_BATCHING ;
Frederic Weisbecker
Acked-byQuoted 10 lines · Anton Blanchard
I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting
hung_task_check_countto 0:# echo 0 > /proc/sys/kernel/hung_task_check_countwhich didn't have the intended response. Change to a post increment of
max_count, so a value of 0 means check 0 tasks.Signed-off-by Anton Blanchard <[email protected]>
Quoted 15 lines
---
Index:
linux.trees.git/kernel/hung_task.c
===================================================================kernel/hung_task.c +1
-1@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_trcu_read_lock (); do_each_thread(g, t) { -if (!--max_count) +if (!max_count--) goto unlock;if (!--batch_count) { batch_count =HUNG_TASK_BATCHING ;
Quoted 10 lines · Anton Blanchard
I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting
hung_task_check_countto 0:# echo 0 > /proc/sys/kernel/hung_task_check_countwhich didn't have the intended response. Change to a post increment of
max_count, so a value of 0 means check 0 tasks.Signed-off-by Anton Blanchard <[email protected]>
Ack.
I would also suggest to make 'max_count' as unsigned long, since sysctl_hung_task_check_count is.
Thanks.
Quoted 15 lines
---
Index:
linux.trees.git/kernel/hung_task.c
===================================================================kernel/hung_task.c 2009-11-27 13:11:57.000000000 +1100 renamed from kernel/hung_task.c 2009-11-27 13:11:46.000000000 +1100 +1
-1@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t rcu_read_lock(); do_each_thread(g, t) { - if (!--max_count) + if (!max_count--) goto unlock; if (!--batch_count) { batch_count = HUNG_TASK_BATCHING;
Quoted 13 lines · Américo Wang
Quoted 10 lines · Anton Blanchard
I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting
hung_task_check_countto 0:# echo 0 > /proc/sys/kernel/hung_task_check_countwhich didn't have the intended response. Change to a post increment of
max_count, so a value of 0 means check 0 tasks.Signed-off-by Anton Blanchard <[email protected]>
Quoted 6 lines
Ack.
I would also suggest to make '
max_count' as unsigned long, sincesysctl_hung_task_check_countis.Thanks.
Also, the batch_count thing should be dropped I think. This is a hardcoded, not overridable pause after 1024 threads checks to avoid latencies caused by rcu_read_lock. But now we have PREEMPT_RCU so people can enable it if they care about latency. We should remove it as it adds unnecessary complexity.
I'm preparing a patch for that, on top of Anton patch.
Quoted 22 lines · Frederic Weisbecker
Quoted 13 lines · Américo Wang
Quoted 10 lines · Anton Blanchard
I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting
hung_task_check_countto 0:# echo 0 > /proc/sys/kernel/hung_task_check_countwhich didn't have the intended response. Change to a post increment of
max_count, so a value of 0 means check 0 tasks.Signed-off-by Anton Blanchard <[email protected]>Quoted 6 lines
Ack.
I would also suggest to make '
max_count' as unsigned long, sincesysctl_hung_task_check_countis.Thanks.
Quoted 6 lines
Also, the
batch_countthing should be dropped I think. This is a hardcoded, not overridable pause after 1024 threads checks to avoid latencies caused byrcu_read_lock. But now we havePREEMPT_RCUso people can enable it if they care about latency. We should remove it as it adds unnecessary complexity.
This sounds OK for me.
Quoted 3 lines
I'm preparing a patch for that, on top of Anton patch.
Great!
Thanks.
Quoted 34 lines · Américo Wang
Quoted 22 lines · Frederic Weisbecker
Quoted 13 lines · Américo Wang
Quoted 10 lines · Anton Blanchard
I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting
hung_task_check_countto 0:# echo 0 > /proc/sys/kernel/hung_task_check_countwhich didn't have the intended response. Change to a post increment of
max_count, so a value of 0 means check 0 tasks.Signed-off-by Anton Blanchard <[email protected]>Quoted 6 lines
Ack.
I would also suggest to make '
max_count' as unsigned long, sincesysctl_hung_task_check_countis.Thanks.
Quoted 6 lines
Also, the
batch_countthing should be dropped I think. This is a hardcoded, not overridable pause after 1024 threads checks to avoid latencies caused byrcu_read_lock. But now we havePREEMPT_RCUso people can enable it if they care about latency. We should remove it as it adds unnecessary complexity.This sounds OK for me.
Quoted 1 line
Quoted 6 lines
Quoted 2 lines
I'm preparing a patch for that, on top of Anton patch.
Great!
Thanks.
Actually it looks like this check is not only there to take care of latency but also to avoid too much waiting before a grace period.
Hm, well, I'm not sure what to do.
[tip:core/softlockup] softlockup: Fix hung_task_check_count sysctl
In reply to Anton BlanchardCommit-ID: e5af02261668350b43eb7381648930bde8e872f7 Gitweb: http://git.kernel.org/tip/e5af02261668350b43eb7381648930bde8e872f7 Author: Anton Blanchard <[email protected]> AuthorDate: Fri, 27 Nov 2009 13:28:20 +1100 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 27 Nov 2009 06:21:57 +0100
softlockup: Fix hung_task_check_count sysctl
I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting hung_task_check_count to 0:
# echo 0 > /proc/sys/kernel/hung_task_check_count
which didn't have the intended response. Change to a post increment of max_count, so a value of 0 means check 0 tasks.
LKML-Reference: <20091127022820.GU32182@kryten>
--- kernel/hung_task.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/hung_task.c +1 -1
@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)rcu_read_lock (); do_each_thread(g, t) { -if (!--max_count) +if (!max_count--) goto unlock;if (!--batch_count) { batch_count =HUNG_TASK_BATCHING ;
To disable hung_task at runtime:
# echo 0 > /proc/sys/kernel/hung_task_check_count
33 /*
34 * Zero means infinite timeout - no checking done:
35 */
36 unsigned long __read_mostly sysctl_hung_task_timeout_secs = 120;
For hung_task_check_count, 0 means there is no maximum so all tasks are checked.
Quoted 4 lines · Anton Blanchard
I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting
hung_task_check_countto 0:
Hmm, maybe khungtaskd should be run using SCHED_IDLE.
Quoted 21 lines
# echo 0 > /proc/sys/kernel/hung_task_check_countwhich didn't have the intended response. Change to a post increment of
max_count, so a value of 0 means check 0 tasks.Signed-off-by Anton Blanchard <[email protected]>---
Index:
linux.trees.git/kernel/hung_task.c
===================================================================kernel/hung_task.c +1
-1@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_trcu_read_lock (); do_each_thread(g, t) { -if (!--max_count) +if (!max_count--) goto unlock;if (!--batch_count) { batch_count =HUNG_TASK_BATCHING ;
Quoted 4 lines · Mandeep Singh Baines
To disable
hung_taskat runtime:# echo 0 > /proc/sys/kernel/hung_task_check_count
D'oh. Copy and paste error. I meant
# echo 0 > /proc/sys/kernel/hung_task_timeout_secs
The nice thing about hung_task_timeout_secs=0 is that hungtaskd will no longer run. Implementing
no checking done with hung_task_check_count=0 would result in the task getting scheduled,
running, and then doing nothing.
Quoted 38 lines
33 /*
34 * Zero means infinite timeout - no checking done:
35 */
36 unsigned long__read_mostlysysctl_hung_task_timeout_secs= 120;For
hung_task_check_count, 0 means there is no maximum so all tasks are checked.Quoted 4 lines · Anton Blanchard
I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting
hung_task_check_countto 0:Hmm, maybe khungtaskd should be run using
SCHED_IDLE.Quoted 21 lines
# echo 0 > /proc/sys/kernel/hung_task_check_countwhich didn't have the intended response. Change to a post increment of
max_count, so a value of 0 means check 0 tasks.Signed-off-by Anton Blanchard <[email protected]>---
Index:
linux.trees.git/kernel/hung_task.c
===================================================================kernel/hung_task.c 2009-11-27 13:11:57.000000000 +1100 renamed from kernel/hung_task.c 2009-11-27 13:11:46.000000000 +1100 +1
-1@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t rcu_read_lock(); do_each_thread(g, t) { - if (!--max_count) + if (!max_count--) goto unlock; if (!--batch_count) { batch_count = HUNG_TASK_BATCHING;