Skip to contentSkip to conversation

block: Prevent hung_check firing during long sync IO

PATCH Applied Ming Lei 3 participants 4 messages Mar 18, 2020 Block layer

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH v2 1/1 block: Prevent hung_check firing during long sync IO

submit_bio_wait() can be called from ioctl(BLKSECDISCARD), which may take long time to complete, as Salman mentioned, 4K BLKSECDISCARD takes up to 100 second on some devices. Also any block I/O operation that occurs after the BLKSECDISCARD is submitted will also potentially be affected by the hung task timeouts.

Another report is that task hang can be observed when running mkfs over raid10 which takes a small max discard sectors limit because of chunk size.

So prevent hung_check from firing by taking same approach used in blk_execute_rq(), and the wake-up interval is set as half the hung_check timer period, which keeps overhead low enough.

---
V2:
- fix checkpatch warning
- add reviewed-by
- add comment log for covering one recent report on task hung on raid10

 block/bio.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
block/bio.c +11 -1
index 94d697217887..0985f3422556 100644
@@ -17,6 +17,7 @@
 #include <linux/cgroup.h>
 #include <linux/blk-cgroup.h>
 #include <linux/highmem.h>
+#include <linux/sched/sysctl.h>
 
 #include <trace/events/block.h>
 #include "blk.h"
@@ -1019,12 +1020,21 @@ static void submit_bio_wait_endio(struct bio *bio)
 int submit_bio_wait(struct bio *bio)
 {
 	DECLARE_COMPLETION_ONSTACK_MAP(done, bio->bi_disk->lockdep_map);
+	unsigned long hang_check;
 
 	bio->bi_private = &done;
 	bio->bi_end_io = submit_bio_wait_endio;
 	bio->bi_opf |= REQ_SYNC;
 	submit_bio(bio);
-	wait_for_completion_io(&done);
+
+	/* Prevent hang_check timer from firing at us during very long I/O */
+	hang_check = sysctl_hung_task_timeout_secs;
+	if (hang_check)
+		while (!wait_for_completion_io_timeout(&done,
+					hang_check * (HZ/2)))
+			;
+	else
+		wait_for_completion_io(&done);
 
 	return blk_status_to_errno(bio->bi_status);
 }
Signature · 1 line
2.20.1
Acked-by on v2 1/1
#1 lore raw

Hi Ming,

Quoted 9 lines · Ming Lei

submit_bio_wait() can be called from ioctl(BLKSECDISCARD), which may take long time to complete, as Salman mentioned, 4K BLKSECDISCARD takes up to 100 second on some devices. Also any block I/O operation that occurs after the BLKSECDISCARD is submitted will also potentially be affected by the hung task timeouts.

Another report is that task hang can be observed when running mkfs over raid10 which takes a small max discard sectors limit because of chunk size.

Could you point the link about the raid10 task hang? And we have observed task hang with raid5, not sure it is related or not.

Our set up is md/raid5 -> 100+ LVs (fs is created on top of one LV), run heavy IOs on LVs and dbench on the LV with fs, then dbench hangs with 'D' state.

Quoted 49 lines

So prevent hung_check from firing by taking same approach used in blk_execute_rq(), and the wake-up interval is set as half the hung_check timer period, which keeps overhead low enough.

---
V2:
- fix checkpatch warning
- add reviewed-by
- add comment log for covering one recent report on task hung on raid10

  block/bio.c | 12 +++++++++++-
  1 file changed, 11 insertions(+), 1 deletion(-)
block/bio.c +4 -1
index 94d697217887..0985f3422556 100644
@@ -17,6 +17,7 @@
  #include <linux/cgroup.h>
  #include <linux/blk-cgroup.h>
  #include <linux/highmem.h>
+#include <linux/sched/sysctl.h>
 
  #include <trace/events/block.h>
  #include "blk.h"
@@ -1019,12 +1020,21 @@ static void submit_bio_wait_endio(struct bio *bio)
  int submit_bio_wait(struct bio *bio)
  {
  	DECLARE_COMPLETION_ONSTACK_MAP(done, bio->bi_disk->lockdep_map);
+	unsigned long hang_check;
 
  	bio->bi_private = &done;
  	bio->bi_end_io = submit_bio_wait_endio;
  	bio->bi_opf |= REQ_SYNC;
  	submit_bio(bio);
-	wait_for_completion_io(&done);
+
+	/* Prevent hang_check timer from firing at us during very long I/O */

block/bio.c static void submit_bio_wait_endio(struct bio *bio) · line 1040

Quoted 10 lines
block/bio.c +7 -0
+	hang_check = sysctl_hung_task_timeout_secs;
+	if (hang_check)
+		while (!wait_for_completion_io_timeout(&done,
+					hang_check * (HZ/2)))
+			;
+	else
+		wait_for_completion_io(&done);
 
  	return blk_status_to_errno(bio->bi_status);
  }

I hope the change could resolve our issue as well.

Thanks,
Guoqing

author
#2 lore raw
Quoted 13 lines · Guoqing Jiang

Hi Ming,

Quoted 9 lines · Ming Lei

submit_bio_wait() can be called from ioctl(BLKSECDISCARD), which may take long time to complete, as Salman mentioned, 4K BLKSECDISCARD takes up to 100 second on some devices. Also any block I/O operation that occurs after the BLKSECDISCARD is submitted will also potentially be affected by the hung task timeouts.

Another report is that task hang can be observed when running mkfs over raid10 which takes a small max discard sectors limit because of chunk size.

Quoted 5 lines

Could you point the link about the raid10 task hang? And we have observed task hang with raid5, not sure it is related or not.

Our set up is md/raid5 -> 100+ LVs (fs is created on top of one LV), run heavy IOs on LVs and dbench on the LV with fs, then dbench hangs with 'D' state.

Red Hat Bugzilla 1813383.

Thanks,
Ming

maintainer Applied
#3 lore raw
In reply to Ming Lei
Quoted 6 lines · Ming Lei

submit_bio_wait() can be called from ioctl(BLKSECDISCARD), which may take long time to complete, as Salman mentioned, 4K BLKSECDISCARD takes up to 100 second on some devices. Also any block I/O operation that occurs after the BLKSECDISCARD is submitted will also potentially be affected by the hung task timeouts.

Quoted 7 lines

Another report is that task hang can be observed when running mkfs over raid10 which takes a small max discard sectors limit because of chunk size.

So prevent hung_check from firing by taking same approach used in blk_execute_rq(), and the wake-up interval is set as half the hung_check timer period, which keeps overhead low enough.

Applied, thanks.

We have two of these similar constructs now in block alone, I wonder if there are others in other sub-systems. Might make sense to make this a generic helper at some point.

Signature · 1 line
Jens Axboe