block: Prevent hung_check firing during long sync IO
PATCH v2 1/1 block: Prevent hung_check firing during long sync IO
submit_bio_wait() can be called from ioctl(BLKSECDISCARD), which may take long time to complete, as Salman mentioned, 4K BLKSECDISCARD takes up to 100 second on some devices. Also any block I/O operation that occurs after the BLKSECDISCARD is submitted will also potentially be affected by the hung task timeouts.
Another report is that task hang can be observed when running mkfs over raid10 which takes a small max discard sectors limit because of chunk size.
So prevent hung_check from firing by taking same approach used in blk_execute_rq(), and the wake-up interval is set as half the hung_check timer period, which keeps overhead low enough.
---
V2:
- fix checkpatch warning
- add reviewed-by
- add comment log for covering one recent report on task hung on raid10
block/bio.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
block/bio.c +11 -1
@@ -17,6 +17,7 @@#include <linux/cgroup.h> #include <linux/blk-cgroup.h> #include <linux/highmem.h> +#include <linux/sched/sysctl.h> #include <trace/events/block.h> #include "blk.h"
@@ -1019,12 +1020,21 @@ static void submit_bio_wait_endio(struct bio *bio)int submit_bio_wait (struct bio *bio) {DECLARE_COMPLETION_ONSTACK_MAP (done, bio->bi_disk->lockdep_map); +unsigned long hang_check; bio->bi_private = &done; bio->bi_end_io = submit_bio_wait_endio; bio->bi_opf |=REQ_SYNC ;submit_bio (bio); -wait_for_completion_io (&done); + +/* Prevent hang_check timer from firing at us during very long I/O */ + hang_check = sysctl_hung_task_timeout_secs; +if (hang_check) +while (!wait_for_completion_io_timeout (&done, + hang_check * (HZ /2 ))) + ; +else +wait_for_completion_io (&done);return blk_status_to_errno (bio->bi_status); }
Signature · 1 line
2.20.1
Hi Ming,
Quoted 9 lines · Ming Lei
submit_bio_wait()can be called from ioctl(BLKSECDISCARD), which may take long time to complete, as Salman mentioned, 4K BLKSECDISCARD takes up to 100 second on some devices. Also any block I/O operation that occurs after the BLKSECDISCARD is submitted will also potentially be affected by the hung task timeouts.Another report is that task hang can be observed when running mkfs over raid10 which takes a small max discard sectors limit because of chunk size.
Could you point the link about the raid10 task hang? And we have observed task hang with raid5, not sure it is related or not.
Our set up is md/raid5 -> 100+ LVs (fs is created on top of one LV), run heavy IOs on LVs and dbench on the LV with fs, then dbench hangs with 'D' state.
Quoted 49 lines
So prevent
hung_checkfrom firing by taking same approach used inblk_execute_rq(), and the wake-up interval is set as half thehung_checktimer period, which keeps overhead low enough.Cc Salman Qazi <[email protected]>Cc Jesse Barnes <[email protected]>Cc Bart Van Assche <[email protected]>Reported-by Salman Qazi <[email protected]>Reviewed-by Jesse Barnes <[email protected]>Reviewed-by Salman Qazi <[email protected]>Signed-off-by Ming Lei <[email protected]>---
V2:
- fix checkpatch warning
- add reviewed-by
- add comment log for covering one recent report on task hung on raid10block/bio.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)block/bio.c +4
-1@@ -17,6 +17,7 @@#include <linux/cgroup.h> #include <linux/blk-cgroup.h> #include <linux/highmem.h> +#include <linux/sched/sysctl.h> #include <trace/events/block.h> #include "blk.h" @@ -1019,12 +1020,21 @@ static void submit_bio_wait_endio(struct bio *bio)int submit_bio_wait (struct bio *bio) {DECLARE_COMPLETION_ONSTACK_MAP (done, bio->bi_disk->lockdep_map); +unsigned long hang_check; bio->bi_private = &done; bio->bi_end_io = submit_bio_wait_endio; bio->bi_opf |=REQ_SYNC ;submit_bio (bio); -wait_for_completion_io (&done); + +/* Prevent hang_check timer from firing at us during very long I/O */
block/bio.c static void submit_bio_wait_endio(struct bio *bio) · line 1040
Quoted 10 lines
block/bio.c +7
-0+ hang_check = sysctl_hung_task_timeout_secs; +if (hang_check) +while (!wait_for_completion_io_timeout (&done, + hang_check * (HZ /2 ))) + ; +else +wait_for_completion_io (&done);return blk_status_to_errno (bio->bi_status); }
I hope the change could resolve our issue as well.
Thanks,
Guoqing
Quoted 13 lines · Guoqing Jiang
Hi Ming,
Quoted 9 lines · Ming Lei
submit_bio_wait()can be called from ioctl(BLKSECDISCARD), which may take long time to complete, as Salman mentioned, 4K BLKSECDISCARD takes up to 100 second on some devices. Also any block I/O operation that occurs after the BLKSECDISCARD is submitted will also potentially be affected by the hung task timeouts.Another report is that task hang can be observed when running mkfs over raid10 which takes a small max discard sectors limit because of chunk size.
Quoted 5 lines
Could you point the link about the raid10 task hang? And we have observed task hang with raid5, not sure it is related or not.
Our set up is md/raid5 -> 100+ LVs (fs is created on top of one LV), run heavy IOs on LVs and dbench on the LV with fs, then dbench hangs with 'D' state.
Red Hat Bugzilla 1813383.
Thanks,
Ming
Quoted 6 lines · Ming Lei
submit_bio_wait()can be called from ioctl(BLKSECDISCARD), which may take long time to complete, as Salman mentioned, 4K BLKSECDISCARD takes up to 100 second on some devices. Also any block I/O operation that occurs after the BLKSECDISCARD is submitted will also potentially be affected by the hung task timeouts.
Quoted 7 lines
Another report is that task hang can be observed when running mkfs over raid10 which takes a small max discard sectors limit because of chunk size.
So prevent
hung_checkfrom firing by taking same approach used inblk_execute_rq(), and the wake-up interval is set as half thehung_checktimer period, which keeps overhead low enough.
Applied, thanks.
We have two of these similar constructs now in block alone, I wonder if there are others in other sub-systems. Might make sense to make this a generic helper at some point.
Signature · 1 line
Jens Axboe