Question about poll_multi_file
Hi Jens,
I've a question about poll_multi_file in io_do_iopoll(). It keeps spinning in f_op->iopoll() if poll_multi_file is true (and we're under the requested amount). But in my understanding, reqs may be in different hardware queues for blk-mq device even in this situation.
Should we consider the hardware queue number as well? Some thing like below:
fs/io_uring.c +1 -0
@@ -2533,10 +2533,13 @@ static void io_iopoll_req_issued(struct io_kiocb *req, bool in_async) ctx->poll_multi_file = false; }else if (!ctx->poll_multi_file) {struct io_kiocb *list_req; +unsigned int queue_num0, queue_num1; list_req = list_first_entry(&ctx->iopoll_list, struct
io_kiocb,
inflight_entry);
- if (list_req->file != req->file)
+ queue_num0 = blk_qc_t_to_queue_num(list_req->rw.kiocb.ki_cookie);
+ queue_num1 = blk_qc_t_to_queue_num(req->rw.kiocb.ki_cookie);
+ if (list_req->file != req->file || queue_num0 != queue_num1)
ctx->poll_multi_file = true;
}Jens Axboe
maintainerQuoted 8 lines · Hao Xu
Hi Jens,
I've a question aboutpoll_multi_fileinio_do_iopoll(). It keeps spinning inf_op->iopoll()ifpoll_multi_fileis true (and we're under the requested amount). But in my understanding, reqs may be in different hardware queues for blk-mq device even in this situation.
Should we consider the hardware queue number as well? Some thing like below:
That looks reasonable to me - do you have any performance numbers to go with it?
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
Quoted 8 lines · Hao Xu
Hi Jens,
I've a question aboutpoll_multi_fileinio_do_iopoll(). It keeps spinning inf_op->iopoll()ifpoll_multi_fileis true (and we're under the requested amount). But in my understanding, reqs may be in different hardware queues for blk-mq device even in this situation.
Should we consider the hardware queue number as well? Some thing like below:
Quoted 2 lines
That looks reasonable to me - do you have any performance numbers to go with it?
Not very easy for me to construct a good case. I'm trying to mock the below situation:
manully control uring reqs to go to 2 hardware queues, like:
hw_queue0 hw_queue1
heavy_req simple_req
heavy_req simple_req
... ...heavy_req is some request that needs more time to complete, while simple_req takes less time. And make the io_do_iopoll() alway first spin on hw_queue0.
any ideas?
Thanks,
Hao
Quoted 1 line
Quoted 22 lines · Hao Xu
Quoted 10 lines · Jens Axboe
Quoted 8 lines · Hao Xu
Hi Jens,
I've a question aboutpoll_multi_fileinio_do_iopoll(). It keeps spinning inf_op->iopoll()ifpoll_multi_fileis true (and we're under the requested amount). But in my understanding, reqs may be in different hardware queues for blk-mq device even in this situation.
Should we consider the hardware queue number as well? Some thing like below:Quoted 2 lines
That looks reasonable to me - do you have any performance numbers to go with it?
Not very easy for me to construct a good case. I'm trying to mock the below situation:
manully control uring reqs to go to 2 hardware queues, like:
hw_queue0hw_queue1
heavy_reqsimple_req
heavy_reqsimple_req
... ...
Quoted 4 lines
heavy_reqis some request that needs more time to complete, whilesimple_reqtakes less time. And make theio_do_iopoll()alway first spin onhw_queue0.
any ideas?
- NVMe with #HW qs >= #CPUs, so HW to SW qs are 1-to-1.
- 2 threads pinned to different CPUs, so they submit to different qs.
Then one thread is doing 512B rand reads, and the second is doing 64-128 KB rand reads. So, I'd expect a latency spike on some nine. Not tested, so just a suggestion.
The second can also be doing writes, but that would need
1) waiting for steady state
2) higher QD/load for writes because otherwise SSD caches might hide waiting.
Signature · 1 line
Pavel Begunkov