Skip to contentSkip to conversation

Question about poll_multi_file

PATCH Hao Xu 3 participants 4 messages Jun 3, 2021 – Jun 7, 2021

Subsystems (2)
lore
Share & tools
lore

Hao Xu

author
#0 lore raw

Hi Jens,
I've a question about poll_multi_file in io_do_iopoll(). It keeps spinning in f_op->iopoll() if poll_multi_file is true (and we're under the requested amount). But in my understanding, reqs may be in different hardware queues for blk-mq device even in this situation.
Should we consider the hardware queue number as well? Some thing like below:

fs/io_uring.c +1 -0
index 7b0e6bc9ea3d..479a75022449 100644
@@ -2533,10 +2533,13 @@ static void io_iopoll_req_issued(struct io_kiocb *req, bool in_async)
                ctx->poll_multi_file = false;
        } else if (!ctx->poll_multi_file) {
                struct io_kiocb *list_req;
+               unsigned int queue_num0, queue_num1;
 
                list_req = list_first_entry(&ctx->iopoll_list, struct 

io_kiocb,

                                                inflight_entry);
-               if (list_req->file != req->file)
+               queue_num0 = blk_qc_t_to_queue_num(list_req->rw.kiocb.ki_cookie);
+               queue_num1 = blk_qc_t_to_queue_num(req->rw.kiocb.ki_cookie);
+               if (list_req->file != req->file || queue_num0 != queue_num1)
                        ctx->poll_multi_file = true;
        }
maintainer
#1 lore raw
Quoted 8 lines · Hao Xu

Hi Jens,
I've a question about poll_multi_file in io_do_iopoll(). It keeps spinning in f_op->iopoll() if poll_multi_file is true (and we're under the requested amount). But in my understanding, reqs may be in different hardware queues for blk-mq device even in this situation.
Should we consider the hardware queue number as well? Some thing like below:

That looks reasonable to me - do you have any performance numbers to go with it?

Signature · 1 line
Jens Axboe
author
#2 lore raw
Quoted 10 lines · Jens Axboe
Quoted 8 lines · Hao Xu

Hi Jens,
I've a question about poll_multi_file in io_do_iopoll(). It keeps spinning in f_op->iopoll() if poll_multi_file is true (and we're under the requested amount). But in my understanding, reqs may be in different hardware queues for blk-mq device even in this situation.
Should we consider the hardware queue number as well? Some thing like below:

Quoted 2 lines

That looks reasonable to me - do you have any performance numbers to go with it?

Not very easy for me to construct a good case. I'm trying to mock the below situation:
manully control uring reqs to go to 2 hardware queues, like:

   hw_queue0     hw_queue1
   heavy_req     simple_req
   heavy_req     simple_req
     ...            ...

heavy_req is some request that needs more time to complete, while simple_req takes less time. And make the io_do_iopoll() alway first spin on hw_queue0.
any ideas?

Thanks,
Hao

Quoted 1 line
#3 lore raw
Quoted 22 lines · Hao Xu
Quoted 10 lines · Jens Axboe
Quoted 8 lines · Hao Xu

Hi Jens,
I've a question about poll_multi_file in io_do_iopoll(). It keeps spinning in f_op->iopoll() if poll_multi_file is true (and we're under the requested amount). But in my understanding, reqs may be in different hardware queues for blk-mq device even in this situation.
Should we consider the hardware queue number as well? Some thing like below:

Quoted 2 lines

That looks reasonable to me - do you have any performance numbers to go with it?

Not very easy for me to construct a good case. I'm trying to mock the below situation:
manully control uring reqs to go to 2 hardware queues, like:
   hw_queue0     hw_queue1
   heavy_req     simple_req
   heavy_req     simple_req
     ...            ...

Quoted 4 lines

heavy_req is some request that needs more time to complete, while simple_req takes less time. And make the io_do_iopoll() alway first spin on hw_queue0.
any ideas?

- NVMe with #HW qs >= #CPUs, so HW to SW qs are 1-to-1.
- 2 threads pinned to different CPUs, so they submit to different qs.

Then one thread is doing 512B rand reads, and the second is doing 64-128 KB rand reads. So, I'd expect a latency spike on some nine. Not tested, so just a suggestion.

The second can also be doing writes, but that would need
1) waiting for steady state
2) higher QD/load for writes because otherwise SSD caches might hide waiting.

Signature · 1 line
Pavel Begunkov