Skip to contentSkip to conversation

PATCH: softlockup: Fix hung_task_check_count sysctl

PATCH Anton Blanchard 4 participants 9 messages Nov 27, 2009 – Nov 30, 2009 Hung task detector

lore
Share & tools
lore
#0 lore raw

I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting hung_task_check_count to 0:

# echo 0 > /proc/sys/kernel/hung_task_check_count

which didn't have the intended response. Change to a post increment of max_count, so a value of 0 means check 0 tasks.

---

kernel/hung_task.c +1 -1
Index: linux.trees.git/kernel/hung_task.c
===================================================================
@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
 
 	rcu_read_lock();
 	do_each_thread(g, t) {
-		if (!--max_count)
+		if (!max_count--)
 			goto unlock;
 		if (!--batch_count) {
 			batch_count = HUNG_TASK_BATCHING;
#1 lore raw
Quoted 10 lines · Anton Blanchard

I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting hung_task_check_count to 0:

# echo 0 > /proc/sys/kernel/hung_task_check_count

which didn't have the intended response. Change to a post increment of max_count, so a value of 0 means check 0 tasks.

Quoted 15 lines

---

Index: linux.trees.git/kernel/hung_task.c
===================================================================

kernel/hung_task.c +1 -1
@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
 
 	rcu_read_lock();
 	do_each_thread(g, t) {
-		if (!--max_count)
+		if (!max_count--)
 			goto unlock;
 		if (!--batch_count) {
 			batch_count = HUNG_TASK_BATCHING;
#2 lore raw
In reply to Anton Blanchard
Quoted 10 lines · Anton Blanchard

I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting hung_task_check_count to 0:

# echo 0 > /proc/sys/kernel/hung_task_check_count

which didn't have the intended response. Change to a post increment of max_count, so a value of 0 means check 0 tasks.

Ack.

I would also suggest to make 'max_count' as unsigned long, since sysctl_hung_task_check_count is.

Thanks.

Quoted 15 lines

---

Index: linux.trees.git/kernel/hung_task.c
===================================================================

kernel/hung_task.c  2009-11-27 13:11:57.000000000 +1100 renamed from kernel/hung_task.c     2009-11-27 13:11:46.000000000 +1100 +1 -1
@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
 
        rcu_read_lock();
        do_each_thread(g, t) {
-               if (!--max_count)
+               if (!max_count--)
                        goto unlock;
                if (!--batch_count) {
                        batch_count = HUNG_TASK_BATCHING;
#3 lore raw
Quoted 13 lines · Américo Wang
Quoted 10 lines · Anton Blanchard

I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting hung_task_check_count to 0:

# echo 0 > /proc/sys/kernel/hung_task_check_count

which didn't have the intended response. Change to a post increment of max_count, so a value of 0 means check 0 tasks.

Quoted 6 lines

Ack.

I would also suggest to make 'max_count' as unsigned long, since sysctl_hung_task_check_count is.

Thanks.

Also, the batch_count thing should be dropped I think. This is a hardcoded, not overridable pause after 1024 threads checks to avoid latencies caused by rcu_read_lock. But now we have PREEMPT_RCU so people can enable it if they care about latency. We should remove it as it adds unnecessary complexity.

I'm preparing a patch for that, on top of Anton patch.

#4 lore raw
Quoted 22 lines · Frederic Weisbecker
Quoted 13 lines · Américo Wang
Quoted 10 lines · Anton Blanchard

I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting hung_task_check_count to 0:

# echo 0 > /proc/sys/kernel/hung_task_check_count

which didn't have the intended response. Change to a post increment of max_count, so a value of 0 means check 0 tasks.

Quoted 6 lines

Ack.

I would also suggest to make 'max_count' as unsigned long, since sysctl_hung_task_check_count is.

Thanks.

Quoted 6 lines

Also, the batch_count thing should be dropped I think. This is a hardcoded, not overridable pause after 1024 threads checks to avoid latencies caused by rcu_read_lock. But now we have PREEMPT_RCU so people can enable it if they care about latency. We should remove it as it adds unnecessary complexity.

This sounds OK for me.

Quoted 3 lines

I'm preparing a patch for that, on top of Anton patch.

Great!

Thanks.

#5 lore raw
Quoted 34 lines · Américo Wang
Quoted 22 lines · Frederic Weisbecker
Quoted 13 lines · Américo Wang
Quoted 10 lines · Anton Blanchard

I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting hung_task_check_count to 0:

# echo 0 > /proc/sys/kernel/hung_task_check_count

which didn't have the intended response. Change to a post increment of max_count, so a value of 0 means check 0 tasks.

Quoted 6 lines

Ack.

I would also suggest to make 'max_count' as unsigned long, since sysctl_hung_task_check_count is.

Thanks.

Quoted 6 lines

Also, the batch_count thing should be dropped I think. This is a hardcoded, not overridable pause after 1024 threads checks to avoid latencies caused by rcu_read_lock. But now we have PREEMPT_RCU so people can enable it if they care about latency. We should remove it as it adds unnecessary complexity.

This sounds OK for me.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

I'm preparing a patch for that, on top of Anton patch.

Great!

Thanks.

Actually it looks like this check is not only there to take care of latency but also to avoid too much waiting before a grace period.

Hm, well, I'm not sure what to do.

#6 lore raw

[tip:core/softlockup] softlockup: Fix hung_task_check_count sysctl

In reply to Anton Blanchard
Commit-ID:  e5af02261668350b43eb7381648930bde8e872f7
Gitweb:     http://git.kernel.org/tip/e5af02261668350b43eb7381648930bde8e872f7
Author:     Anton Blanchard <[email protected]>
AuthorDate: Fri, 27 Nov 2009 13:28:20 +1100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 27 Nov 2009 06:21:57 +0100

softlockup: Fix hung_task_check_count sysctl

I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting hung_task_check_count to 0:

 # echo 0 > /proc/sys/kernel/hung_task_check_count

which didn't have the intended response. Change to a post increment of max_count, so a value of 0 means check 0 tasks.

LKML-Reference: <20091127022820.GU32182@kryten>

---
 kernel/hung_task.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/hung_task.c +1 -1
index d4e8417..0c642d5 100644
@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 
 	rcu_read_lock();
 	do_each_thread(g, t) {
-		if (!--max_count)
+		if (!max_count--)
 			goto unlock;
 		if (!--batch_count) {
 			batch_count = HUNG_TASK_BATCHING;
#7 lore raw
In reply to Anton Blanchard

To disable hung_task at runtime:

# echo 0 > /proc/sys/kernel/hung_task_check_count

33 /*
34 * Zero means infinite timeout - no checking done:
35 */
36 unsigned long __read_mostly sysctl_hung_task_timeout_secs = 120;

For hung_task_check_count, 0 means there is no maximum so all tasks are checked.

Quoted 4 lines · Anton Blanchard

I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting hung_task_check_count to 0:

Hmm, maybe khungtaskd should be run using SCHED_IDLE.

Quoted 21 lines
# echo 0 > /proc/sys/kernel/hung_task_check_count

which didn't have the intended response. Change to a post increment of max_count, so a value of 0 means check 0 tasks.

---

Index: linux.trees.git/kernel/hung_task.c
===================================================================

kernel/hung_task.c +1 -1
@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
 
 	rcu_read_lock();
 	do_each_thread(g, t) {
-		if (!--max_count)
+		if (!max_count--)
 			goto unlock;
 		if (!--batch_count) {
 			batch_count = HUNG_TASK_BATCHING;
#8 lore raw
Quoted 4 lines · Mandeep Singh Baines

To disable hung_task at runtime:

# echo 0 > /proc/sys/kernel/hung_task_check_count

D'oh. Copy and paste error. I meant

# echo 0 > /proc/sys/kernel/hung_task_timeout_secs

The nice thing about hung_task_timeout_secs=0 is that hungtaskd will no longer run. Implementing
no checking done with hung_task_check_count=0 would result in the task getting scheduled,
running, and then doing nothing.

Quoted 38 lines

33 /*
34  * Zero means infinite timeout - no checking done:
35  */
36 unsigned long __read_mostly sysctl_hung_task_timeout_secs = 120;

For hung_task_check_count, 0 means there is no maximum so all tasks are checked.

Quoted 4 lines · Anton Blanchard

I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce this source of jitter I tried setting hung_task_check_count to 0:

Hmm, maybe khungtaskd should be run using SCHED_IDLE.

Quoted 21 lines
# echo 0 > /proc/sys/kernel/hung_task_check_count

which didn't have the intended response. Change to a post increment of max_count, so a value of 0 means check 0 tasks.

---

Index: linux.trees.git/kernel/hung_task.c
===================================================================

kernel/hung_task.c        2009-11-27 13:11:57.000000000 +1100 renamed from kernel/hung_task.c   2009-11-27 13:11:46.000000000 +1100 +1 -1
@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
 
       rcu_read_lock();
       do_each_thread(g, t) {
-             if (!--max_count)
+             if (!max_count--)
                       goto unlock;
               if (!--batch_count) {
                       batch_count = HUNG_TASK_BATCHING;