writeback: fix hung_task alarm when sync block

PATCH Superseded Wanpeng Li 2 participants 5 messages Jun 13, 2012 VFS

lore

Share & tools

lore

Wanpeng Li

author

2012-06-13 00:52 UTC #0 lore raw

PATCH 1/1 writeback: fix hung_task alarm when sync block

From: Wanpeng Li <[email protected]>

I use several dd processes to write a slow SD card dd if=/dev/sda1 of=/dev/sdc4 bs=1M count=4000 and several sync commands(maybe > 10),dmesg show this:

[  366.888741] INFO: task sync:3518 blocked for more than 120 seconds.
[  366.888742] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  366.888746] sync            D 00000201     0  3518   3462 0x00000000
[  366.888752]  dcde5e54 00000082 00000000 00000201 00000000 c180d220 c1933d00 c1933d00
[  366.888758]  614a7ae7 00000023 f6bc0d00 ebecb280 c180d220 f6bc0d6c f04acbfc 00000000
[  366.888786]  f6bc0d6c dcde5e2c 00000023 f6bc0d44 00000000 00000000 ebd8b2b4 dcde5e6c
[  366.888788] Call Trace:
[  366.888792]  [<c107a0ee>] ? enqueue_entity+0xee/0x5a0
[  366.888795]  [<c15abc83>] schedule+0x23/0x60
[  366.888798]  [<c15aa5ad>] schedule_timeout+0x22d/0x2a0
[  366.888801]  [<c1071ce2>] ? check_preempt_curr+0x72/0x90
[  366.888804]  [<c1071d28>] ? ttwu_do_wakeup+0x28/0x130
[  366.888807]  [<c1037f28>] ? default_spin_lock_flags+0x8/0x10
[  366.888810]  [<c15acaad>] ? _raw_spin_lock_irqsave+0x2d/0x40
[  366.888813]  [<c15abb17>] wait_for_common+0xa7/0x110
[  366.888816]  [<c1074690>] ? try_to_wake_up+0x220/0x220
[  366.888819]  [<c15abc57>] wait_for_completion+0x17/0x20
[  366.888822]  [<c116ad80>] writeback_inodes_sb_nr+0x70/0x90
[  366.888825]  [<c116af75>] writeback_inodes_sb+0x25/0x30
[  366.888828]  [<c117159f>] __sync_filesystem+0x4f/0x90
[  366.888831]  [<c11715f7>] sync_one_sb+0x17/0x20
[  366.888834]  [<c114c430>] iterate_supers+0xc0/0xd0
[  366.888837]  [<c11715e0>] ? __sync_filesystem+0x90/0x90
[  366.888840]  [<c117167b>] sys_sync+0x2b/0x60
[  366.888842]  [<c15b385f>] sysenter_do_call+0x12/0x28

Too many similar messages flood the logs. So I use a present method to fix this issue.
------------------------------------------

Author: Mark Lord <[email protected]>
Date:   Fri Sep 24 09:51:13 2010 -0400

block: Prevent hang_check firing during long I/O
------------------------------------------

Signed-off-by Wanpeng Li <[email protected]>

---
 fs/fs-writeback.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

fs/fs-writeback.c +7 -1

index f2d0109..5d403a1 100644
@@ -1300,6 +1300,7 @@ void writeback_inodes_sb_nr(struct super_block *sb,
 			    enum wb_reason reason)
 {
 	DECLARE_COMPLETION_ONSTACK(done);
+	unsigned long hangcheck;
 	struct wb_writeback_work work = {
 		.sb			= sb,
 		.sync_mode		= WB_SYNC_NONE,

@@ -1311,7 +1312,12 @@ void writeback_inodes_sb_nr(struct super_block *sb,
 
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 	bdi_queue_work(sb->s_bdi, &work);
-	wait_for_completion(&done);
+	hangcheck = sysctl_hung_task_timeout_secs;
+	if (hangcheck)
+		while (!wait_for_completion_timeout(&done, HZ/2))
+			;
+	else
+		wait_for_completion(&done);
 }
 EXPORT_SYMBOL(writeback_inodes_sb_nr);

Signature · 1 line

1.7.9.5

Fengguang Wu

2012-06-13 00:59 UTC #1 lore raw

Quoted 13 lines · Wanpeng Li

From: Wanpeng Li <[email protected]>
I use several dd processes to write a slow SD card dd if=/dev/sda1 of=/dev/sdc4 bs=1M count=4000 and several sync commands(maybe > 10),dmesg show this:
[  366.888741] INFO: task sync:3518 blocked for more than 120 seconds.
[  366.888742] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  366.888746] sync            D 00000201     0  3518   3462 0x00000000
Too many similar messages flood the logs. So I use a present method to fix this issue.
------------------------------------------

Quoted 7 lines

Author: Mark Lord <[email protected]>
Date:   Fri Sep 24 09:51:13 2010 -0400
block: Prevent hang_check firing during long I/O
------------------------------------------
Signed-off-by Wanpeng Li <[email protected]>

Yeah that's also what I'd like to do. So you've tested it?

Quoted 17 lines

---
 fs/fs-writeback.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

fs/fs-writeback.c +1 -0

index f2d0109..5d403a1 100644
@@ -1300,6 +1300,7 @@ void writeback_inodes_sb_nr(struct super_block *sb,
 			    enum wb_reason reason)
 {
 	DECLARE_COMPLETION_ONSTACK(done);
+	unsigned long hangcheck;
 	struct wb_writeback_work work = {
 		.sb			= sb,
 		.sync_mode		= WB_SYNC_NONE,

fs/fs-writeback.c void writeback_inodes_sb_nr(struct super_block *sb, · line 1316

Quoted 6 lines

fs/fs-writeback.c +2 -1

@@ -1311,7 +1312,12 @@ void writeback_inodes_sb_nr(struct super_block *sb,
 
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 	bdi_queue_work(sb->s_bdi, &work);
-	wait_for_completion(&done);
+	hangcheck = sysctl_hung_task_timeout_secs;
+	if (hangcheck)

The hangcheck variable looks redundant.

Quoted 9 lines

fs/fs-writeback.c +4 -0

+		while (!wait_for_completion_timeout(&done, HZ/2))
+			;
+	else
+		wait_for_completion(&done);
 }
 EXPORT_SYMBOL(writeback_inodes_sb_nr);

Signature · 1 line

1.7.9.5

Wanpeng Li

author

2012-06-13 01:12 UTC #2 lore raw

Quoted 17 lines · Fengguang Wu

Quoted 16 lines · Wanpeng Li
From: Wanpeng Li <[email protected]>
I use several dd processes to write a slow SD card dd if=/dev/sda1 of=/dev/sdc4 bs=1M count=4000 and several sync commands(maybe > 10),dmesg show this:
[  366.888741] INFO: task sync:3518 blocked for more than 120 seconds.
[  366.888742] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  366.888746] sync            D 00000201     0  3518   3462 0x00000000
Too many similar messages flood the logs. So I use a present method to fix this issue.
------------------------------------------
Author: Mark Lord <[email protected]>
Date:   Fri Sep 24 09:51:13 2010 -0400

Quoted 6 lines

Quoted 4 lines
block: Prevent hang_check firing during long I/O
------------------------------------------
Signed-off-by Wanpeng Li <[email protected]>
Yeah that's also what I'd like to do. So you've tested it?

Not yet, I will test it today.

Quoted 19 lines

---
 fs/fs-writeback.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

fs/fs-writeback.c +1 -0

index f2d0109..5d403a1 100644
@@ -1300,6 +1300,7 @@ void writeback_inodes_sb_nr(struct super_block *sb,
 			    enum wb_reason reason)
 {
 	DECLARE_COMPLETION_ONSTACK(done);
+	unsigned long hangcheck;
 	struct wb_writeback_work work = {
 		.sb			= sb,
 		.sync_mode		= WB_SYNC_NONE,

@@ -1311,7 +1312,12 @@ void writeback_inodes_sb_nr(struct super_block *sb,

Quoted 7 lines

fs/fs-writeback.c void writeback_inodes_sb_nr(struct super_block *sb, · line 1316
Quoted 5 lines
fs/fs-writeback.c +2 -1
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 	bdi_queue_work(sb->s_bdi, &work);
-	wait_for_completion(&done);
+	hangcheck = sysctl_hung_task_timeout_secs;
+	if (hangcheck)
The hangcheck variable looks redundant.

if sysctl_hung_task_timeout_secs is equal to ZERO, it means infinite timeout -- no checking done. So I think wait_for_completion_timeout makes no sense this time.

Regards,
Wanpeng Li

Quoted 10 lines

fs/fs-writeback.c +4 -0

+		while (!wait_for_completion_timeout(&done, HZ/2))
+			;
+	else
+		wait_for_completion(&done);
 }
 EXPORT_SYMBOL(writeback_inodes_sb_nr);

Signature · 1 line

1.7.9.5

Fengguang Wu

2012-06-13 01:18 UTC #3 lore raw

Quoted 10 lines

Quoted 9 lines
fs/fs-writeback.c void writeback_inodes_sb_nr(struct super_block *sb, · line 1316
Quoted 7 lines
fs/fs-writeback.c +2 -1
@@ -1311,7 +1312,12 @@ void writeback_inodes_sb_nr(struct super_block *sb,
 
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 	bdi_queue_work(sb->s_bdi, &work);
-	wait_for_completion(&done);
+	hangcheck = sysctl_hung_task_timeout_secs;
+	if (hangcheck)
The hangcheck variable looks redundant.

Quoted 3 lines

if sysctl_hung_task_timeout_secs is equal to ZERO, it means infinite timeout -- no checking done. So I think wait_for_completion_timeout makes no sense this time.

I mean, you can test sysctl_hung_task_timeout_secs directly? It's a one shot test anyway.

Quoted 9 lines

fs/fs-writeback.c +4 -0

+		while (!wait_for_completion_timeout(&done, HZ/2))
+			;
+	else
+		wait_for_completion(&done);
 }
 EXPORT_SYMBOL(writeback_inodes_sb_nr);

Signature · 1 line

1.7.9.5

Wanpeng Li

author

2012-06-13 01:30 UTC #4 lore raw

Quoted 10 lines · Fengguang Wu

Quoted 10 lines
Quoted 9 lines
fs/fs-writeback.c void writeback_inodes_sb_nr(struct super_block *sb, · line 1316
Quoted 7 lines
fs/fs-writeback.c +2 -1
@@ -1311,7 +1312,12 @@ void writeback_inodes_sb_nr(struct super_block *sb,
 
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 	bdi_queue_work(sb->s_bdi, &work);
-	wait_for_completion(&done);
+	hangcheck = sysctl_hung_task_timeout_secs;
+	if (hangcheck)
The hangcheck variable looks redundant.

Quoted 6 lines

Quoted 3 lines
if sysctl_hung_task_timeout_secs is equal to ZERO, it means timeout -- no checking done. So I think wait_for_completion_timeout makes no sense this time.
I mean, you can test sysctl_hung_task_timeout_secs directly? It's a one shot test anyway.

/*
* Zero means infinite timeout - no checking done:
*/
unsigned long __read_mostly sysctl_hung_task_timeout_secs = CONFIG_DEFAULT_HUNG_TASK_TIMEOUT;

The comment in kernel/hung_task.c says "Zero means infinite timeout - no cheking done". Maybe I can just test if this time alarm doesn't flood my logs. Do you have more suggestion. :-)

Regards,
Wanpeng Li

Quoted 10 lines

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
fs/fs-writeback.c +4 -0
+		while (!wait_for_completion_timeout(&done, HZ/2))
+			;
+	else
+		wait_for_completion(&done);
 }
 EXPORT_SYMBOL(writeback_inodes_sb_nr);
 
Signature · 1 line
1.7.9.5