hung_task checking and sys_sync

DISCUSSION Daniel Walker 4 participants 8 messages Jun 12, 2012 – Jun 13, 2012

lore

Share & tools

lore

Daniel Walker

author

2012-06-12 22:09 UTC #0 lore raw

I found this commit which was a while ago,

commit fb822db465bd9fd4208eef1af4490539b236c54e

Author: Ingo Molnar <[email protected]>
Date:   Wed Aug 20 11:17:40 2008 +0200

softlockup: increase hung tasks check from 2 minutes to 8 minutes

Andrew says:

> Seems that about 100% of the reports we get of this warning triggering > are sys_sync, transaction commit, etc.

increase the timeout. If it still triggers for people, we can kill it.

Signed-off-by Ingo Molnar <[email protected]>

We're seeing these messages on an older kernel (montavista) but the code areas appear similar to current kernels. The issue is that we're doing a file copy which takes 10-15minutes, and in the background there is a "df --sync" happening (which is calling sys_sync). We end up getting a hung task message like below,

INFO: task df:1778 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ffffffff81578d40 0000000000000086 ffff8801f6135b00 ffff880269a91800 ffff880269a91800 ffff8802702be000 ffff8801f602a080 0000000000000000 ffff8801f602a440 ffffffff8109c166 ffff8801e863de18 0000000000000004

Call Trace:
  [<ffffffff8109c166>] ? sync_page+0x0/0x49
  [<ffffffff81320de2>] ? __schedule+0x3c/0x57
  [<ffffffff810ea3c7>] ? bdi_sched_wait+0x0/0xe
  [<ffffffff81320de2>] ? __schedule+0x3c/0x57
  [<ffffffff81320e0d>] ? schedule+0x10/0x1e
  [<ffffffff810ea3d0>] ? bdi_sched_wait+0x9/0xe

There some variation in the stack trace , but always thru bdi_sched_wait().

These don't seem like valid warnings, since the copy happening is know to take a long time. Has there been any commit that disable these messages bdi_sched_wait?

Daniel

Mandeep Singh Baines

2012-06-12 22:29 UTC #1 lore raw

Quoted 1 line · Daniel Walker

Hi Daniel,

Quoted 37 lines

I found this commit which was a while ago,
commit fb822db465bd9fd4208eef1af4490539b236c54e
Author: Ingo Molnar <[email protected]>
Date:   Wed Aug 20 11:17:40 2008 +0200
softlockup: increase hung tasks check from 2 minutes to 8 minutes
Andrew says:
> Seems that about 100% of the reports we get of this warning triggering > are sys_sync, transaction commit, etc.
increase the timeout. If it still triggers for people, we can kill it.
Signed-off-by Ingo Molnar <[email protected]>
We're seeing these messages on an older kernel (montavista) but the code areas appear similar to current kernels. The issue is that we're doing a file copy which takes 10-15minutes, and in the background there is a "df --sync" happening (which is calling sys_sync). We end up getting a hung task message like below,
INFO: task df:1778 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ffffffff81578d40 0000000000000086 ffff8801f6135b00 ffff880269a91800 ffff880269a91800 ffff8802702be000 ffff8801f602a080 0000000000000000 ffff8801f602a440 ffffffff8109c166 ffff8801e863de18 0000000000000004
Call Trace:
  [<ffffffff8109c166>] ? sync_page+0x0/0x49
  [<ffffffff81320de2>] ? __schedule+0x3c/0x57
  [<ffffffff810ea3c7>] ? bdi_sched_wait+0x0/0xe
  [<ffffffff81320de2>] ? __schedule+0x3c/0x57
  [<ffffffff81320e0d>] ? schedule+0x10/0x1e
  [<ffffffff810ea3d0>] ? bdi_sched_wait+0x9/0xe

Quoted 5 lines

There some variation in the stack trace , but always thru bdi_sched_wait().
These don't seem like valid warnings, since the copy happening is know to take a long time.

But the time is not unbounded. You could mask the hung_task_detector for this case but then you lose the ability to catch bugs in this code path.

The timeout is configurable via /proc/sys/kernel/hung_task_timeout_secs. Can you bump up the value at boot via sysctl.conf?

Quoted 2 lines

Has there been any commit that disable these messages bdi_sched_wait?

No. There is no mechanism to disable hung_task for a specific code path. We do skip processes if PF_PROZEN or PF_FROZEN_SKIP is set but that is really a different situation where the wait is unbounded.

Regards,
Mandeep

Quoted 1 line

Daniel

Daniel Walker

author

2012-06-12 22:34 UTC #2 lore raw

Quoted 6 lines · Mandeep Singh Baines

But the time is not unbounded. You could mask the hung_task_detector for this case but then you lose the ability to catch bugs in this code path.
The timeout is configurable via /proc/sys/kernel/hung_task_timeout_secs. Can you bump up the value at boot via sysctl.conf?

Maybe, but I'm wondering if these types should just be stopped because Andrew had complained about them already.

Quoted 6 lines

Quoted 2 lines
Has there been any commit that disable these messages bdi_sched_wait?
No. There is no mechanism to disable hung_task for a specific code path. We do skip processes if PF_PROZEN or PF_FROZEN_SKIP is set but that is really a different situation where the wait is unbounded.

There is presidence for this type of change,

Author: Mark Lord <[email protected]>
Date:   Fri Sep 24 09:51:13 2010 -0400

block: Prevent hang_check firing during long I/O

During long I/O operations, the hang_check timer may fire, trigger stack dumps that unnecessarily alarm the user.

Eg. hdparm --security-erase NULL /dev/sdb ## can take *hours* to complete

So, if hang_check is armed, we should wake up periodically
to prevent it from triggering. This patch uses a wake-up interval equal to half the hang_check timer period, which keeps overhead low enough.

Signed-off-by Mark Lord <[email protected]>

Signed-off-by Jens Axboe <[email protected]>

Mandeep Baines

2012-06-12 22:45 UTC #3 lore raw

Quoted 11 lines · Daniel Walker

Quoted 6 lines · Mandeep Singh Baines
But the time is not unbounded. You could mask the hung_task_detector for this case but then you lose the ability to catch bugs in this code path.
The timeout is configurable via /proc/sys/kernel/hung_task_timeout_secs. Can you bump up the value at boot via sysctl.conf?
Maybe, but I'm wondering if these types should just be stopped because Andrew had complained about them already.

Fair enough. Actually, internally I had a patch where we'd use a task flag to disable and enable the hang check but the approach in the patch you pointed me to seems better.

Quoted 17 lines

Quoted 6 lines
Quoted 2 lines
Has there been any commit that disable these messages bdi_sched_wait?
No. There is no mechanism to disable hung_task for a specific code path. We do skip processes if PF_PROZEN or PF_FROZEN_SKIP is set but that is really a different situation where the wait is unbounded.
There is presidence for this type of change,
Author: Mark Lord <[email protected]>
Date: Fri Sep 24 09:51:13 2010 -0400
block: Prevent hang_check firing during long I/O
During long I/O operations, the hang_check timer may fire, trigger stack dumps that unnecessarily alarm the user.

Quoted 9 lines

Eg. hdparm --security-erase NULL /dev/sdb ## can take *hours* to complete
So, if hang_check is armed, we should wake up periodically
to prevent it from triggering. This patch uses a wake-up interval equal to half the hang_check timer period, which keeps overhead low enough.
Signed-off-by Mark Lord <[email protected]>
Signed-off-by Jens Axboe <[email protected]>

Interesting. I wasn't aware of this patch. Maybe we could abstract this approach via wait_for_completion_no_hang_check().

Regards,
Mandeep

Daniel Walker

author

2012-06-12 22:57 UTC #4 lore raw

Quoted 9 lines · Mandeep Baines

Quoted 8 lines · Daniel Walker
Quoted 6 lines · Mandeep Singh Baines
But the time is not unbounded. You could mask the hung_task_detector for this case but then you lose the ability to catch bugs in this code path.
The timeout is configurable via /proc/sys/kernel/hung_task_timeout_secs. Can you bump up the value at boot via sysctl.conf?

Quoted 7 lines

Quoted 3 lines
Maybe, but I'm wondering if these types should just be stopped because Andrew had complained about them already.
Fair enough. Actually, internally I had a patch where we'd use a task flag to disable and enable the hang check but the approach in the patch you pointed me to seems better.

I'm not really in love with it actually.. It's not ifdef'd for one, but it's also changing potentially good kernel behavior to avoid warnings.

Quoted 23 lines

Quoted 23 lines
Quoted 6 lines
Quoted 2 lines
Has there been any commit that disable these messages bdi_sched_wait?
No. There is no mechanism to disable hung_task for a specific code path. We do skip processes if PF_PROZEN or PF_FROZEN_SKIP is set but that is really a different situation where the wait is unbounded.
There is presidence for this type of change,
Author: Mark Lord <[email protected]>
Date: Fri Sep 24 09:51:13 2010 -0400
block: Prevent hang_check firing during long I/O
During long I/O operations, the hang_check timer may fire, trigger stack dumps that unnecessarily alarm the user.
Eg. hdparm --security-erase NULL /dev/sdb ## can take *hours* to complete
So, if hang_check is armed, we should wake up periodically
to prevent it from triggering. This patch uses a wake-up interval equal to half the hang_check timer period, which keeps overhead low enough.

Quoted 6 lines

Quoted 3 lines
Signed-off-by Mark Lord <[email protected]>
Signed-off-by Jens Axboe <[email protected]>
Interesting. I wasn't aware of this patch. Maybe we could abstract this approach via wait_for_completion_no_hang_check().

Could be .. You could put a stack structure into a list of tasks that should be ignored prior to the task sleeping. Then when the thread wakes the stack structure could be removed. Then that list get checked during the hung task checking.

Daniel

Muthu Kumar

2012-06-13 01:03 UTC #5 lore raw

Quoted 14 lines · Daniel Walker

Quoted 13 lines · Mandeep Baines
Quoted 11 lines · Daniel Walker
Quoted 6 lines · Mandeep Singh Baines
But the time is not unbounded. You could mask the hung_task_detector for this case but then you lose the ability to catch bugs in this code path.
The timeout is configurable via /proc/sys/kernel/hung_task_timeout_secs. Can you bump up the value at boot via sysctl.conf?
Maybe, but I'm wondering if these types should just be stopped because Andrew had complained about them already.

Quoted 7 lines

Quoted 3 lines
Fair enough. Actually, internally I had a patch where we'd use a task flag to disable and enable the hang check but the approach in the patch you pointed me to seems better.
I'm not really in love with it actually.. It's not ifdef'd for one, but it's also changing potentially good kernel behavior to avoid warnings.

I totally agree with you (but, not the ifdef part :). The mentioned change actually was masking a potential problem - see
https://lkml.org/lkml/2012/6/6/483. If not for that change, we would have got hung task message for the case where blk_execute_req() would have stuck forever without the completion being called.

Quoted 41 lines

Quoted 23 lines
Quoted 23 lines
Quoted 6 lines
Quoted 2 lines
Has there been any commit that disable these messages bdi_sched_wait?
No. There is no mechanism to disable hung_task for a specific code path. We do skip processes if PF_PROZEN or PF_FROZEN_SKIP is set but that is really a different situation where the wait is unbounded.
There is presidence for this type of change,
Author: Mark Lord <[email protected]>
Date: Fri Sep 24 09:51:13 2010 -0400
block: Prevent hang_check firing during long I/O
During long I/O operations, the hang_check timer may fire, trigger stack dumps that unnecessarily alarm the user.
Eg. hdparm --security-erase NULL /dev/sdb ## can take *hours* to complete
So, if hang_check is armed, we should wake up periodically
to prevent it from triggering. This patch uses a wake-up interval equal to half the hang_check timer period, which keeps overhead low enough.
Quoted 6 lines
Quoted 3 lines
Signed-off-by Mark Lord <[email protected]>
Signed-off-by Jens Axboe <[email protected]>
Interesting. I wasn't aware of this patch. Maybe we could abstract this approach via wait_for_completion_no_hang_check().
Could be .. You could put a stack structure into a list of tasks that should be ignored prior to the task sleeping. Then when the thread wakes the stack structure could be removed. Then that list get checked during the hung task checking.
Daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Daniel Walker

author

2012-06-13 15:44 UTC #6 lore raw

Quoted 19 lines · Muthu Kumar

Quoted 18 lines · Daniel Walker
Quoted 16 lines · Mandeep Baines
Quoted 11 lines · Daniel Walker
Quoted 6 lines · Mandeep Singh Baines
But the time is not unbounded. You could mask the hung_task_detector for this case but then you lose the ability to catch bugs in this code path.
The timeout is configurable via /proc/sys/kernel/hung_task_timeout_secs. Can you bump up the value at boot via sysctl.conf?
Maybe, but I'm wondering if these types should just be stopped because Andrew had complained about them already.
Fair enough. Actually, internally I had a patch where we'd use a task flag to disable and enable the hang check but the approach in the patch you pointed me to seems better.

Quoted 8 lines

Quoted 3 lines
I'm not really in love with it actually.. It's not ifdef'd for one, but it's also changing potentially good kernel behavior to avoid warnings.
I totally agree with you (but, not the ifdef part :). The mentioned change actually was masking a potential problem - see
https://lkml.org/lkml/2012/6/6/483. If not for that change, we would have got hung task message for the case where blk_execute_req() would have stuck forever without the completion being called.

Not sure how the link you gave relates here.

The hang checker isn't always part of the kernel i.e. it's configurable. So this fix doesn't always need to exist, which is what I mean by ifdef'd ..

Daniel

Sadasivan Shaiju

2012-06-13 03:03 UTC #7 lore raw

In reply to Mandeep Baines

There was another patch addressing these type of issue .

https://lkml.org/lkml/2009/1/12/18

regards,
shaiju.

-----Original Message-----
From: Mandeep Baines [mailto:[email protected]]
Sent: Tuesday, June 12, 2012 3:45 PM
To: Daniel Walker
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: hung_task checking and sys_sync

Quoted 4 lines · Daniel Walker

Quoted 3 lines · Mandeep Singh Baines
But the time is not unbounded. You could mask the hung_task_detector for this case but then you lose the ability to catch bugs in this code

path.

Quoted 2 lines

Quoted 2 lines
The timeout is configurable via

/proc/sys/kernel/hung_task_timeout_secs.

Quoted 5 lines

Quoted 1 line
Can you bump up the value at boot via sysctl.conf?
Maybe, but I'm wondering if these types should just be stopped because Andrew had complained about them already.

Fair enough. Actually, internally I had a patch where we'd use a task flag to disable and enable the hang check but the approach in the patch you pointed me to seems better.

Quoted 4 lines

Quoted 4 lines
Quoted 2 lines
Has there been any commit that disable these messages bdi_sched_wait?
No. There is no mechanism to disable hung_task for a specific code

path.

Quoted 13 lines

Quoted 2 lines
We do skip processes if PF_PROZEN or PF_FROZEN_SKIP is set but that is really a different situation where the wait is unbounded.
There is presidence for this type of change,
Author: Mark Lord <[email protected]>
Date: Fri Sep 24 09:51:13 2010 -0400
block: Prevent hang_check firing during long I/O
During long I/O operations, the hang_check timer may fire, trigger stack dumps that unnecessarily alarm the user.

Quoted 6 lines

Eg. hdparm --security-erase NULL /dev/sdb ## can take *hours* to complete
So, if hang_check is armed, we should wake up periodically to prevent it from triggering. This patch uses a wake-up interval equal to half the hang_check timer period, which keeps overhead low

enough.

Quoted 4 lines

Signed-off-by Mark Lord <[email protected]>
Signed-off-by Jens Axboe <[email protected]>

Interesting. I wasn't aware of this patch. Maybe we could abstract this approach via wait_for_completion_no_hang_check().

Regards,
Mandeep