Skip to contentSkip to conversation

fio: fix hangs due to iodepth_low

PATCH In review Robert Elliott 3 participants 10 messages Sep 4, 2014

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fio: fix hangs due to iodepth_low

With some combinations of iodepth, iodepth_batch, iodepth_batch_complete, and io_depth_low, do_io hangs after reaping the first set of completions since io_u_queued_complete is called requesting more completions than td->cur_depth.

Example printing min_evts and td->cur_depth in the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:09s]
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 2863d:18h:28m:38s] <fio never exits>

Fix this by adjusting min_evts to the current_depth if that is smaller.

Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15
time_based

Made the same change to do_verify, but not tested there.

---
 backend.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
backend.c +4 -0
index 7cb0a39..ce97f6d 100644
@@ -606,6 +606,8 @@ reap:
 				 * and do the verification on them through
 				 * the callback handler
 				 */
+				if (min_events < td->cur_depth)
+					min_events = td->cur_depth;
 				if (io_u_queued_complete(td, min_events, bytes_done) < 0) {
 					ret = -1;
 					break;
@@ -873,6 +875,8 @@ reap:
 				fio_gettime(&comp_time, NULL);
 
 			do {
+				if (min_evts < td->cur_depth)
+					min_evts = td->cur_depth;
 				ret = io_u_queued_complete(td, min_evts, bytes_done);
 				if (ret < 0)
 					break;
#1 lore raw

Huh. Wonder if that's what I hit a week or two ago.

-- steve

Quoted 67 lines · Robert Elliott

With some combinations of iodepth, iodepth_batch, iodepth_batch_complete, and io_depth_low, do_io hangs after reaping the first set of completions since io_u_queued_complete is called requesting more completions than td->cur_depth.

Example printing min_evts and td->cur_depth in the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:09s]
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 2863d:18h:28m:38s]
<fio never exits>

Fix this by adjusting min_evts to the current_depth if that is smaller.

Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15
time_based

Made the same change to do_verify, but not tested there.

---
 backend.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
backend.c +2 -0
index 7cb0a39..ce97f6d 100644
@@ -606,6 +606,8 @@ reap:
                                 * and do the verification on them through
                                 * the callback handler
                                 */
+                               if (min_events < td->cur_depth)
+                                       min_events = td->cur_depth;
                                if (io_u_queued_complete(td, min_events,
 bytes_done) < 0) {
                                        ret = -1;
                                        break;
backend.c +2 -0
@@ -873,6 +875,8 @@ reap:
                                fio_gettime(&comp_time, NULL);
 
                        do {
+                               if (min_evts < td->cur_depth)
+                                       min_evts = td->cur_depth;
                                ret = io_u_queued_complete(td, min_evts,
 bytes_done);
                                if (ret < 0)
                                        break;

--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
In reply to Robert Elliott
Quoted 40 lines · Robert Elliott

With some combinations of iodepth, iodepth_batch, iodepth_batch_complete, and io_depth_low, do_io hangs after reaping the first set of completions since io_u_queued_complete is called requesting more completions than td->cur_depth.

Example printing min_evts and td->cur_depth in the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:09s]
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 2863d:18h:28m:38s] <fio never exits>

Fix this by adjusting min_evts to the current_depth if that is smaller.

Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15
time_based

Made the same change to do_verify, but not tested there.

---
  backend.c |    4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)
backend.c +0 -0
index 7cb0a39..ce97f6d 100644

backend.c reap: · line 610

Quoted 5 lines
backend.c +2 -0
@@ -606,6 +606,8 @@ reap:
  				 * and do the verification on them through
  				 * the callback handler
  				 */
+				if (min_events < td->cur_depth)
+					min_events = td->cur_depth;

Did you reverse these? From the description and debug output, seems it should be:

	if (min_events > td->cur_depth)
		min_events = td->cur_depth;

and we should probably put this logic in io_u_queued_complete(), I think that would be a safer alternative instead of near the callers.

Signature · 1 line
Jens Axboe
#3 lore raw
Quoted 52 lines · Jens Axboe
Quoted 42 lines · Robert Elliott

With some combinations of iodepth, iodepth_batch, iodepth_batch_complete, and io_depth_low, do_io hangs after reaping the first set of completions since io_u_queued_complete is called requesting more completions than td->cur_depth.

Example printing min_evts and td->cur_depth in the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:09s]
...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 2863d:18h:28m:38s]
<fio never exits>

Fix this by adjusting min_evts to the current_depth if that is smaller.

Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15
time_based

Made the same change to do_verify, but not tested there.

---
  backend.c |    4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)
backend.c +0 -0
index 7cb0a39..ce97f6d 100644

backend.c reap: · line 610

Quoted 5 lines
backend.c +2 -0
@@ -606,6 +606,8 @@ reap:
                   * and do the verification on them through
                   * the callback handler
                   */
+                if (min_events < td->cur_depth)
+                    min_events = td->cur_depth;

Did you reverse these? From the description and debug output, seems it should be:

Quoted 5 lines
     if (min_events > td->cur_depth)
         min_events = td->cur_depth;

and we should probably put this logic in io_u_queued_complete(), I think that would be a safer alternative instead of near the callers.

Ala the attached.

Signature · 1 line
Jens Axboe
io_u.c +2 -0
index ba192a32a985..be2f242a6e2b 100644
@@ -1792,6 +1792,8 @@ int io_u_queued_complete(struct thread_data *td, int min_evts,
 
 	if (!min_evts)
 		tvp = &ts;
+	else if (min_evts > td->cur_depth)
+		min_evts = td->cur_depth;
 
 	ret = td_io_getevents(td, min_evts, td->o.iodepth_batch_complete, tvp);
 	if (ret < 0) {
#4 lore raw
Quoted 8 lines
Quoted 6 lines

should be:

     if (min_events > td->cur_depth)
         min_events = td->cur_depth;

and we should probably put this logic in io_u_queued_complete(), I

think

Quoted 1 line

that would be a safer alternative instead of near the callers.

io_u_quisce, which calls io_u_queued_complete with an argument of 1, includes this comment:
* and cur_depth is meaningless for sync engines.

If that is invoked during sync traffic too, then putting this change inside io_u_queued_complete might not work right.

---
Rob Elliott HP Server Storage

#5 lore raw
Quoted 7 lines · Robert (Server Storage)
Quoted 7 lines
Quoted 6 lines

should be:

     if (min_events > td->cur_depth)
         min_events = td->cur_depth;

and we should probably put this logic in io_u_queued_complete(), I

think

Quoted 8 lines
Quoted 1 line
Quoted 1 line

that would be a safer alternative instead of near the callers.

io_u_quisce, which calls io_u_queued_complete with an argument of 1, includes this comment:
* and cur_depth is meaningless for sync engines.

If that is invoked during sync traffic too, then putting this change inside io_u_queued_complete might not work right.

It's good enough for this use case. And it doesn't really matter if it's near the caller or in the function from this point of view, since there will be nothing (0) to complete for sync engines. By their very nature, they have nothing pending at that point. So I think the patch is fine as I sent out.

Signature · 1 line
Jens Axboe
#6 lore raw
Quoted 27 lines

-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Thursday, 04 September, 2014 9:39 AM
To: Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH] fio: fix hangs due to iodepth_low

Quoted 6 lines · Robert (Server Storage)
Quoted 6 lines
Quoted 6 lines

should be:

     if (min_events > td->cur_depth)
         min_events = td->cur_depth;

and we should probably put this logic in io_u_queued_complete(),

I

Quoted 9 lines
Quoted 2 lines

think

Quoted 1 line

that would be a safer alternative instead of near the callers.

io_u_quisce, which calls io_u_queued_complete with an argument of 1, includes this comment:
* and cur_depth is meaningless for sync engines.

If that is invoked during sync traffic too, then putting this change inside io_u_queued_complete might not work right.

Quoted 9 lines

It's good enough for this use case. And it doesn't really matter if it's
near the caller or in the function from this point of view, since there
will be nothing (0) to complete for sync engines. By their very nature,
they have nothing pending at that point. So I think the patch is fine as
I sent out.

I ran this version through similar tests and it works as well.

#7 lore raw
Quoted 37 lines · Robert (Server Storage)
Quoted 27 lines

-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Thursday, 04 September, 2014 9:39 AM
To: Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH] fio: fix hangs due to iodepth_low

Quoted 6 lines · Robert (Server Storage)
Quoted 6 lines
Quoted 6 lines

should be:

     if (min_events > td->cur_depth)
         min_events = td->cur_depth;

and we should probably put this logic in io_u_queued_complete(),

I

Quoted 9 lines
Quoted 2 lines

think

Quoted 1 line

that would be a safer alternative instead of near the callers.

io_u_quisce, which calls io_u_queued_complete with an argument of 1, includes this comment:
* and cur_depth is meaningless for sync engines.

If that is invoked during sync traffic too, then putting this change inside io_u_queued_complete might not work right.

Quoted 9 lines

It's good enough for this use case. And it doesn't really matter if it's
near the caller or in the function from this point of view, since there
will be nothing (0) to complete for sync engines. By their very nature,
they have nothing pending at that point. So I think the patch is fine as
I sent out.

Quoted 1 line

I ran this version through similar tests and it works as well.

Great thanks Rob, I committed that fix.

Signature · 1 line
Jens Axboe
#8 lore raw
In reply to Jens Axboe
Quoted 64 lines

-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Wednesday, September 03, 2014 8:34 PM
To: Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH] fio: fix hangs due to iodepth_low

Quoted 1 line · Robert Elliott

With some combinations of iodepth, iodepth_batch,

iodepth_batch_complete,

Quoted 1 line

and io_depth_low, do_io hangs after reaping the first set of

completions

Quoted 12 lines

since io_u_queued_complete is called requesting more completions than td->cur_depth.

Example printing min_evts and td->cur_depth in the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops]

[eta 00m:09s]

Quoted 2 lines

...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops]

[eta 2863d:18h:28m:38s]

Quoted 3 lines

<fio never exits>

Fix this by adjusting min_evts to the current_depth if that is

smaller.

Quoted 21 lines

Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15
time_based

Made the same change to do_verify, but not tested there.

---
  backend.c |    4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)
backend.c +0 -0
index 7cb0a39..ce97f6d 100644

backend.c reap: · line 610

Quoted 5 lines
backend.c +2 -0
@@ -606,6 +606,8 @@ reap:
  				 * and do the verification on them through
  				 * the callback handler
  				 */
+				if (min_events < td->cur_depth)
+					min_events = td->cur_depth;

Did you reverse these? From the description and debug output, seems it should be:

Quoted 8 lines
	if (min_events > td->cur_depth)
		min_events = td->cur_depth;

and we should probably put this logic in io_u_queued_complete(), I think that would be a safer alternative instead of near the callers.

--
Jens Axboe

Sorry, yes - I didn't put it back right after adding code to inject the error.

I will send an updated patch tomorrow putting the check into io_u_queued_complete (if that has access to td).

#9 lore raw
Quoted 70 lines · Robert (Server Storage)
Quoted 70 lines

-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Wednesday, September 03, 2014 8:34 PM
To: Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH] fio: fix hangs due to iodepth_low

Quoted 1 line · Robert Elliott

With some combinations of iodepth, iodepth_batch,

iodepth_batch_complete,

Quoted 1 line

and io_depth_low, do_io hangs after reaping the first set of

completions

Quoted 12 lines

since io_u_queued_complete is called requesting more completions than td->cur_depth.

Example printing min_evts and td->cur_depth in the do/while loop:
waiting on min=96 cd=627
waiting on min=96 cd=531
waiting on min=96 cd=435
waiting on min=96 cd=339
waiting on min=96 cd=243
waiting on min=96 cd=147
waiting on min=96 cd=51
Jobs: 12 (f=12): [r(12)] [43.8% done] [0KB/0KB/0KB /s] [0/0/0 iops]

[eta 00m:09s]

Quoted 2 lines

...
Jobs: 12 (f=12): [r(12)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops]

[eta 2863d:18h:28m:38s]

Quoted 3 lines

<fio never exits>

Fix this by adjusting min_evts to the current_depth if that is

smaller.

Quoted 21 lines

Tested with a jobfile including:
iodepth=1011
iodepth_batch=96
iodepth_batch_complete=96
iodepth_low=1
runtime=15
time_based

Made the same change to do_verify, but not tested there.

---
   backend.c |    4 ++++
   1 files changed, 4 insertions(+), 0 deletions(-)
backend.c +0 -0
index 7cb0a39..ce97f6d 100644

backend.c reap: · line 610

Quoted 5 lines
backend.c +2 -0
@@ -606,6 +606,8 @@ reap:
   				 * and do the verification on them through
   				 * the callback handler
   				 */
+				if (min_events < td->cur_depth)
+					min_events = td->cur_depth;

Did you reverse these? From the description and debug output, seems it should be:

	if (min_events > td->cur_depth)
		min_events = td->cur_depth;

and we should probably put this logic in io_u_queued_complete(), I think that would be a safer alternative instead of near the callers.

Quoted 8 lines
Quoted 2 lines

--
Jens Axboe

Sorry, yes - I didn't put it back right after adding code to inject the error.

I will send an updated patch tomorrow putting the check into io_u_queued_complete (if that has access to td).

Thanks, much appreciated! The fix is the easy part, the diagnosing was the hard part.

Signature · 1 line
Jens Axboe