fio: fix hangs due to iodepth_low
PATCH 1/1 fio: fix hangs due to iodepth_low
With some combinations of iodepth, iodepth_batch, iodepth_batch_complete, and io_depth_low, do_io hangs after reaping the first set of completions since io_u_queued_complete is called requesting more completions than td->cur_depth.
Example printing min_evts and td->cur_depth in the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:09s]
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 2863d:18h:28m:38s] <fio never exits>
Fix this by adjusting min_evts to the current_depth if that is smaller.
Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15time_based
Made the same change to do_verify, but not tested there.
--- backend.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)
backend.c +4 -0
@@ -606,6 +606,8 @@ reap:* and do the verification on them through * the callback handler */ +if (min_events < td->cur_depth) + min_events = td->cur_depth;if (io_u_queued_complete (td, min_events, bytes_done) <0 ) { ret =-1 ;break ;
@@ -873,6 +875,8 @@ reap:fio_gettime (&comp_time,NULL ); do { + if (min_evts < td->cur_depth) + min_evts = td->cur_depth; ret = io_u_queued_complete(td, min_evts, bytes_done); if (ret < 0) break;
Huh. Wonder if that's what I hit a week or two ago.
-- steve
Quoted 67 lines · Robert Elliott
With some combinations of iodepth,
iodepth_batch,iodepth_batch_complete, andio_depth_low,do_iohangs after reaping the first set of completions sinceio_u_queued_completeis called requesting more completions thantd->cur_depth.Example printing
min_evtsandtd->cur_depthin the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:09s]
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 2863d:18h:28m:38s]
<fio never exits>Fix this by adjusting
min_evtsto thecurrent_depthif that is smaller.Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15time_basedMade the same change to
do_verify, but not tested there.Signed-off-by Robert Elliott <[email protected]>--- backend.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)backend.c +2
-0@@ -606,6 +606,8 @@ reap:* and do the verification on them through * the callback handler */ +if (min_events < td->cur_depth) + min_events = td->cur_depth;if (io_u_queued_complete (td, min_events, bytes_done) <0 ) { ret =-1 ;break;backend.c +2
-0@@ -873,6 +875,8 @@ reap:fio_gettime (&comp_time,NULL ); do { + if (min_evts < td->cur_depth) + min_evts = td->cur_depth; ret = io_u_queued_complete(td, min_evts, bytes_done); if (ret < 0)break;--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 40 lines · Robert Elliott
With some combinations of iodepth,
iodepth_batch,iodepth_batch_complete, andio_depth_low,do_iohangs after reaping the first set of completions sinceio_u_queued_completeis called requesting more completions thantd->cur_depth.Example printing
min_evtsandtd->cur_depthin the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:09s]
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 2863d:18h:28m:38s] <fio never exits>Fix this by adjusting
min_evtsto thecurrent_depthif that is smaller.Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15time_basedMade the same change to
do_verify, but not tested there.Signed-off-by Robert Elliott <[email protected]>--- backend.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)backend.c +0
-0
backend.c reap: · line 610
Quoted 5 lines
backend.c +2
-0@@ -606,6 +606,8 @@ reap:* and do the verification on them through * the callback handler */ +if (min_events < td->cur_depth) + min_events = td->cur_depth;
Did you reverse these? From the description and debug output, seems it should be:
if (min_events > td->cur_depth) min_events = td->cur_depth;
and we should probably put this logic in io_u_queued_complete(), I think that would be a safer alternative instead of near the callers.
Signature · 1 line
Jens Axboe
Quoted 52 lines · Jens Axboe
Quoted 42 lines · Robert Elliott
With some combinations of iodepth,
iodepth_batch,iodepth_batch_complete, andio_depth_low,do_iohangs after reaping the first set of completions sinceio_u_queued_completeis called requesting more completions thantd->cur_depth.Example printing
min_evtsandtd->cur_depthin the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:09s]
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 2863d:18h:28m:38s]
<fio never exits>Fix this by adjusting
min_evtsto thecurrent_depthif that is smaller.Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15time_basedMade the same change to
do_verify, but not tested there.Signed-off-by Robert Elliott <[email protected]>--- backend.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)backend.c +0
-0backend.c reap: · line 610
Quoted 5 lines
backend.c +2
-0@@ -606,6 +606,8 @@ reap:* and do the verification on them through * the callback handler */ +if (min_events < td->cur_depth) + min_events = td->cur_depth;Did you reverse these? From the description and debug output, seems it should be:
Quoted 5 lines
if (min_events > td->cur_depth) min_events = td->cur_depth;and we should probably put this logic in
io_u_queued_complete(), I think that would be a safer alternative instead of near the callers.
Ala the attached.
Signature · 1 line
Jens Axboe
io_u.c +2 -0
@@ -1792,6 +1792,8 @@ int io_u_queued_complete(struct thread_data *td, int min_evts,if (!min_evts) tvp = &ts; +else if (min_evts > td->cur_depth) + min_evts = td->cur_depth; ret =td_io_getevents (td, min_evts, td->o.iodepth_batch_complete, tvp);if (ret <0 ) {
Quoted 8 lines
Quoted 6 lines
should be:
if (min_events > td->cur_depth) min_events = td->cur_depth;and we should probably put this logic in
io_u_queued_complete(), Ithink
Quoted 1 line
that would be a safer alternative instead of near the callers.
io_u_quisce, which calls io_u_queued_complete with an argument of 1, includes this comment:
* and cur_depth is meaningless for sync engines.
If that is invoked during sync traffic too, then putting this change inside io_u_queued_complete might not work right.
---
Rob Elliott HP Server Storage
Quoted 7 lines · Robert (Server Storage)
Quoted 7 lines
Quoted 6 lines
should be:
if (min_events > td->cur_depth) min_events = td->cur_depth;and we should probably put this logic in
io_u_queued_complete(), Ithink
Quoted 8 lines
Quoted 1 line
Quoted 1 line
that would be a safer alternative instead of near the callers.
io_u_quisce, which callsio_u_queued_completewith an argument of 1, includes this comment:
* andcur_depthis meaningless for sync engines.If that is invoked during sync traffic too, then putting this change inside
io_u_queued_completemight not work right.
It's good enough for this use case. And it doesn't really matter if it's near the caller or in the function from this point of view, since there will be nothing (0) to complete for sync engines. By their very nature, they have nothing pending at that point. So I think the patch is fine as I sent out.
Signature · 1 line
Jens Axboe
Quoted 27 lines
-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Thursday, 04 September, 2014 9:39 AM
To: Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH] fio: fix hangs due toiodepth_lowQuoted 6 lines · Robert (Server Storage)
Quoted 6 lines
Quoted 6 lines
should be:
if (min_events > td->cur_depth) min_events = td->cur_depth;and we should probably put this logic in
io_u_queued_complete(),I
Quoted 9 lines
Quoted 2 lines
think
Quoted 1 line
that would be a safer alternative instead of near the callers.
io_u_quisce, which callsio_u_queued_completewith an argument of 1, includes this comment:
* andcur_depthis meaningless for sync engines.If that is invoked during sync traffic too, then putting this change inside
io_u_queued_completemight not work right.
Quoted 9 lines
It's good enough for this use case. And it doesn't really matter if it's
near the caller or in the function from this point of view, since there
will be nothing (0) to complete for sync engines. By their very nature,
they have nothing pending at that point. So I think the patch is fine as
I sent out.
I ran this version through similar tests and it works as well.
Quoted 37 lines · Robert (Server Storage)
Quoted 27 lines
-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Thursday, 04 September, 2014 9:39 AM
To: Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH] fio: fix hangs due toiodepth_lowQuoted 6 lines · Robert (Server Storage)
Quoted 6 lines
Quoted 6 lines
should be:
if (min_events > td->cur_depth) min_events = td->cur_depth;and we should probably put this logic in
io_u_queued_complete(),I
Quoted 9 lines
Quoted 2 lines
think
Quoted 1 line
that would be a safer alternative instead of near the callers.
io_u_quisce, which callsio_u_queued_completewith an argument of 1, includes this comment:
* andcur_depthis meaningless for sync engines.If that is invoked during sync traffic too, then putting this change inside
io_u_queued_completemight not work right.Quoted 9 lines
It's good enough for this use case. And it doesn't really matter if it's
near the caller or in the function from this point of view, since there
will be nothing (0) to complete for sync engines. By their very nature,
they have nothing pending at that point. So I think the patch is fine as
I sent out.
Quoted 1 line
I ran this version through similar tests and it works as well.
Great thanks Rob, I committed that fix.
Signature · 1 line
Jens Axboe
Quoted 64 lines
-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Wednesday, September 03, 2014 8:34 PM
To: Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH] fio: fix hangs due toiodepth_lowQuoted 1 line · Robert Elliott
With some combinations of iodepth,
iodepth_batch,
iodepth_batch_complete,Quoted 1 line
and
io_depth_low,do_iohangs after reaping the first set ofcompletions
Quoted 12 lines
since
io_u_queued_completeis called requesting more completions thantd->cur_depth.Example printing
min_evtsandtd->cur_depthin the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops][eta 00m:09s]
Quoted 2 lines
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops][eta 2863d:18h:28m:38s]
Quoted 3 lines
<fio never exits>
Fix this by adjusting
min_evtsto thecurrent_depthif that issmaller.
Quoted 21 lines
Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15time_basedMade the same change to
do_verify, but not tested there.Signed-off-by Robert Elliott <[email protected]>--- backend.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)backend.c +0
-0backend.c reap: · line 610
Quoted 5 lines
backend.c +2
-0@@ -606,6 +606,8 @@ reap:* and do the verification on them through * the callback handler */ +if (min_events < td->cur_depth) + min_events = td->cur_depth;Did you reverse these? From the description and debug output, seems it should be:
Quoted 8 lines
if (min_events > td->cur_depth) min_events = td->cur_depth;and we should probably put this logic in
io_u_queued_complete(), I think that would be a safer alternative instead of near the callers.--
Jens Axboe
Sorry, yes - I didn't put it back right after adding code to inject the error.
I will send an updated patch tomorrow putting the check into io_u_queued_complete (if that has access to td).
Quoted 70 lines · Robert (Server Storage)
Quoted 70 lines
-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Wednesday, September 03, 2014 8:34 PM
To: Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH] fio: fix hangs due toiodepth_lowQuoted 1 line · Robert Elliott
With some combinations of iodepth,
iodepth_batch,
iodepth_batch_complete,Quoted 1 line
and
io_depth_low,do_iohangs after reaping the first set ofcompletions
Quoted 12 lines
since
io_u_queued_completeis called requesting more completions thantd->cur_depth.Example printing
min_evtsandtd->cur_depthin the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops][eta 00m:09s]
Quoted 2 lines
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops][eta 2863d:18h:28m:38s]
Quoted 3 lines
<fio never exits>
Fix this by adjusting
min_evtsto thecurrent_depthif that issmaller.
Quoted 21 lines
Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15time_basedMade the same change to
do_verify, but not tested there.Signed-off-by Robert Elliott <[email protected]>--- backend.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)backend.c +0
-0backend.c reap: · line 610
Quoted 5 lines
backend.c +2
-0@@ -606,6 +606,8 @@ reap:* and do the verification on them through * the callback handler */ +if (min_events < td->cur_depth) + min_events = td->cur_depth;Did you reverse these? From the description and debug output, seems it should be:
if (min_events > td->cur_depth) min_events = td->cur_depth;and we should probably put this logic in
io_u_queued_complete(), I think that would be a safer alternative instead of near the callers.
Quoted 8 lines
Quoted 2 lines
--
Jens AxboeSorry, yes - I didn't put it back right after adding code to inject the error.
I will send an updated patch tomorrow putting the check into
io_u_queued_complete(if that has access to td).
Thanks, much appreciated! The fix is the easy part, the diagnosing was the hard part.
Signature · 1 line
Jens Axboe