virtio-blk: support polling I/O and mq_ops->queue_rqs()
Subsystems (3)
v3 cover virtio-blk: support polling I/O and mq_ops->queue_rqs()
This patch serise adds support for polling I/O and mq_ops->queue_rqs() to virtio-blk driver.
Changes
v2 -> v3
- Fix warning by kernel test robot
static int virtblk_poll()
...
if (!blk_mq_add_to_batch(req, iob, virtblk_result(vbr),
-> vbr->status,
v1 -> v2
- To receive the number of poll queues from user, use module parameter instead of QEMU uapi change.
- Add the comment about virtblk_map_queues().
- Add support for mq_ops->queue_rqs() to implement submit side batch.
Suwan Kim (2):
virtio-blk: support polling I/O
virtio-blk: support mq_ops->queue_rqs()
drivers/block/virtio_blk.c | 194 ++++++++++++++++++++++++++++++++++--- 1 file changed, 181 insertions(+), 13 deletions(-)
Signature · 1 line
2.26.3
PATCH v3 1/2 virtio-blk: support polling I/O
This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.
The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.
virtio-blk polling is implemented upon 'batched completion' of block layer. virtblk_poll() queues completed request to io_comp_batch->req_list and later, virtblk_complete_batch() calls unmap function and ends the requests in batch.
virtio-blk reads the number of poll queues from module parameter "num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.
Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.
For verifying the improvement, I did Fio polling I/O performance test with io_uring engine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.
As a result, IOPS and average latency improved about 10%.
Test result:
- Fio io_uring poll without virtio-blk poll support
-- numjobs=1 : IOPS = 339K, avg latency = 188.33us -- numjobs=2 : IOPS = 367K, avg latency = 347.33us -- numjobs=4 : IOPS = 383K, avg latency = 682.06us
- Fio io_uring poll with virtio-blk poll support
-- numjobs=1 : IOPS = 380K, avg latency = 167.87us -- numjobs=2 : IOPS = 409K, avg latency = 312.6us -- numjobs=4 : IOPS = 413K, avg latency = 619.72us
--- drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++-- 1 file changed, 97 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +97 -4
@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,"0 for no limit. " "Values > nr_cpu_ids truncated to nr_cpu_ids."); +static unsigned int num_poll_queues; +module_param (num_poll_queues, uint,0644 ); +MODULE_PARM_DESC (num_poll_queues,"The number of dedicated virtqueues for polling I/O" ); +static int major;static DEFINE_IDA (vd_index_ida);
@@ -81,6 +85,7 @@ struct virtio_blk {/* num of vqs */ int num_vqs; +int io_queues[HCTX_MAX_TYPES ];struct virtio_blk_vq *vqs; };
@@ -548,6 +553,7 @@ static int init_vq(struct virtio_blk *vblk)const char **names;struct virtqueue **vqs;unsigned short num_vqs; +unsigned int num_poll_vqs;struct virtio_device *vdev = vblk->vdev;struct irq_affinity desc = {0 , };
@@ -556,6 +562,7 @@ static int init_vq(struct virtio_blk *vblk) &num_vqs);if (err) num_vqs =1 ; +if (!err && !num_vqs) {dev_err (&vdev->dev,"MQ advertised but zero queues reported\n" );return -EINVAL ;
@@ -565,6 +572,13 @@ static int init_vq(struct virtio_blk *vblk)min_not_zero (num_request_queues, nr_cpu_ids), num_vqs); + num_poll_vqs = min_t(unsigned int, num_poll_queues, num_vqs - 1); + +memset (vblk->io_queues,0 ,sizeof (int ) *HCTX_MAX_TYPES ); + vblk->io_queues[HCTX_TYPE_DEFAULT ] = num_vqs - num_poll_vqs; + vblk->io_queues[HCTX_TYPE_READ ] =0 ; + vblk->io_queues[HCTX_TYPE_POLL ] = num_poll_vqs; + vblk->vqs =kmalloc_array (num_vqs,sizeof (*vblk->vqs),GFP_KERNEL );if (!vblk->vqs)return -ENOMEM ;
@@ -578,8 +592,13 @@ static int init_vq(struct virtio_blk *vblk) }for (i =0 ; i < num_vqs; i++) { - callbacks[i] = virtblk_done; -snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req.%d" , i); +if (i < num_vqs - num_poll_vqs) { + callbacks[i] = virtblk_done; +snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req.%d" , i); + }else { + callbacks[i] =NULL ; +snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req_poll.%d" , i); + } names[i] = vblk->vqs[i].name; }
@@ -728,16 +747,87 @@ static const struct attribute_group *virtblk_attr_groups[] = {static int virtblk_map_queues (struct blk_mq_tag_set *set) {struct virtio_blk *vblk = set->driver_data; +int i, qoff; + +for (i =0 , qoff =0 ; i < set->nr_maps; i++) { +struct blk_mq_queue_map *map = &set->map[i]; + + map->nr_queues = vblk->io_queues[i]; + map->queue_offset = qoff; + qoff += map->nr_queues; + +if (map->nr_queues ==0 ) +continue ; + +/* +* Regular queues have interrupts and hence CPU affinity is +* defined by the core virtio code, but polling queues have +* no interrupts so we let the block layer assign CPU affinity. +*/ +if (i ==HCTX_TYPE_DEFAULT ) +blk_mq_virtio_map_queues (&set->map[i], vblk->vdev,0 ); +else +blk_mq_map_queues (&set->map[i]); + } + +return 0 ; +} + +static void virtblk_complete_batch (struct io_comp_batch *iob) +{ +struct request *req; +struct virtblk_req *vbr; -return blk_mq_virtio_map_queues (&set->map[HCTX_TYPE_DEFAULT ], - vblk->vdev,0 ); +rq_list_for_each (&iob->req_list, req) { + vbr =blk_mq_rq_to_pdu (req); +virtblk_unmap_data (req, vbr); +virtblk_cleanup_cmd (req); + } +blk_mq_end_request_batch (iob); +} + +static int virtblk_poll (struct blk_mq_hw_ctx *hctx,struct io_comp_batch *iob) +{ +struct virtio_blk_vq *vq = hctx->driver_data; +struct virtblk_req *vbr; +unsigned long flags; +unsigned int len; +int found =0 ; + +spin_lock_irqsave (&vq->lock, flags); + +while ((vbr =virtqueue_get_buf (vq->vq, &len)) !=NULL ) { +struct request *req =blk_mq_rq_from_pdu (vbr); + + found++; +if (!blk_mq_add_to_batch (req, iob, vbr->status, + virtblk_complete_batch)) +blk_mq_complete_request (req); + } + +spin_unlock_irqrestore (&vq->lock, flags); + +return found; +} + +static int virtblk_init_hctx (struct blk_mq_hw_ctx *hctx,void *data, +unsigned int hctx_idx) +{ +struct virtio_blk *vblk = data; +struct virtio_blk_vq *vq = &vblk->vqs[hctx_idx]; + +WARN_ON (vblk->tag_set.tags[hctx_idx] != hctx->tags); + hctx->driver_data = vq; +return 0 ; }static const struct blk_mq_ops virtio_mq_ops = { .queue_rq = virtio_queue_rq, .commit_rqs = virtio_commit_rqs, + .init_hctx = virtblk_init_hctx, .complete = virtblk_request_done, .map_queues = virtblk_map_queues, + .poll = virtblk_poll, };static unsigned int virtblk_queue_depth;
@@ -816,6 +906,9 @@ static int virtblk_probe(struct virtio_device *vdev)sizeof (struct scatterlist ) *VIRTIO_BLK_INLINE_SG_CNT ; vblk->tag_set.driver_data = vblk; vblk->tag_set.nr_hw_queues = vblk->num_vqs; + vblk->tag_set.nr_maps =1 ; +if (vblk->io_queues[HCTX_TYPE_POLL ]) + vblk->tag_set.nr_maps =3 ; err =blk_mq_alloc_tag_set (&vblk->tag_set);if (err)
Signature · 1 line
2.26.3
Michael S. Tsirkin
maintainerRe: [PATCH v3 1/2] virtio-blk: support polling I/O
Quoted 56 lines · Suwan Kim
This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "
num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.
virtio-blk polling is implemented upon 'batched completion' of block layer.
virtblk_poll()queues completed request toio_comp_batch->req_listand later,virtblk_complete_batch()calls unmap function and ends the requests in batch.virtio-blk reads the number of poll queues from module parameter "
num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.
For verifying the improvement, I did Fio polling I/O performance test with
io_uringengine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.As a result, IOPS and average latency improved about 10%.
Test result:
- Fio
io_uringpoll without virtio-blk poll support-- numjobs=1 : IOPS = 339K, avg latency = 188.33us -- numjobs=2 : IOPS = 367K, avg latency = 347.33us -- numjobs=4 : IOPS = 383K, avg latency = 682.06us- Fio
io_uringpoll with virtio-blk poll support-- numjobs=1 : IOPS = 380K, avg latency = 167.87us -- numjobs=2 : IOPS = 409K, avg latency = 312.6us -- numjobs=4 : IOPS = 413K, avg latency = 619.72usReported-by kernel test robot <[email protected]>Signed-off-by Suwan Kim <[email protected]>--- drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++-- 1 file changed, 97 insertions(+), 4 deletions(-)drivers/block/virtio_blk.c +0
-0
drivers/block/virtio_blk.c MODULE_PARM_DESC(num_request_queues, · line 46
Quoted 10 lines
drivers/block/virtio_blk.c +4
-0@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,"0 for no limit. " "Values > nr_cpu_ids truncated to nr_cpu_ids."); +static unsigned int num_poll_queues; +module_param (num_poll_queues, uint,0644 ); +MODULE_PARM_DESC (num_poll_queues,"The number of dedicated virtqueues for polling I/O" ); +static int major;static DEFINE_IDA (vd_index_ida);
Is there some way to make it work reasonably without need to set module parameters? I don't see any other devices with a num_poll_queues parameter - how do they handle this?
Quoted 156 lines
drivers/block/virtio_blk.c +93
-4@@ -81,6 +85,7 @@ struct virtio_blk {/* num of vqs */ int num_vqs; +int io_queues[HCTX_MAX_TYPES ];struct virtio_blk_vq *vqs; };@@ -548,6 +553,7 @@ static int init_vq(struct virtio_blk *vblk)const char **names;struct virtqueue **vqs;unsigned short num_vqs; +unsigned int num_poll_vqs;struct virtio_device *vdev = vblk->vdev;struct irq_affinity desc = {0 , };@@ -556,6 +562,7 @@ static int init_vq(struct virtio_blk *vblk) &num_vqs);if (err) num_vqs =1 ; +if (!err && !num_vqs) {dev_err (&vdev->dev,"MQ advertised but zero queues reported\n" );return -EINVAL ;@@ -565,6 +572,13 @@ static int init_vq(struct virtio_blk *vblk)min_not_zero (num_request_queues, nr_cpu_ids), num_vqs); + num_poll_vqs = min_t(unsigned int, num_poll_queues, num_vqs - 1); + +memset (vblk->io_queues,0 ,sizeof (int ) *HCTX_MAX_TYPES ); + vblk->io_queues[HCTX_TYPE_DEFAULT ] = num_vqs - num_poll_vqs; + vblk->io_queues[HCTX_TYPE_READ ] =0 ; + vblk->io_queues[HCTX_TYPE_POLL ] = num_poll_vqs; + vblk->vqs =kmalloc_array (num_vqs,sizeof (*vblk->vqs),GFP_KERNEL );if (!vblk->vqs)return -ENOMEM ;@@ -578,8 +592,13 @@ static int init_vq(struct virtio_blk *vblk) }for (i =0 ; i < num_vqs; i++) { - callbacks[i] = virtblk_done; -snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req.%d" , i); +if (i < num_vqs - num_poll_vqs) { + callbacks[i] = virtblk_done; +snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req.%d" , i); + }else { + callbacks[i] =NULL ; +snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req_poll.%d" , i); + } names[i] = vblk->vqs[i].name; }@@ -728,16 +747,87 @@ static const struct attribute_group *virtblk_attr_groups[] = {static int virtblk_map_queues (struct blk_mq_tag_set *set) {struct virtio_blk *vblk = set->driver_data; +int i, qoff; + +for (i =0 , qoff =0 ; i < set->nr_maps; i++) { +struct blk_mq_queue_map *map = &set->map[i]; + + map->nr_queues = vblk->io_queues[i]; + map->queue_offset = qoff; + qoff += map->nr_queues; + +if (map->nr_queues ==0 ) +continue ; + +/* +* Regular queues have interrupts and hence CPU affinity is +* defined by the core virtio code, but polling queues have +* no interrupts so we let the block layer assign CPU affinity. +*/ +if (i ==HCTX_TYPE_DEFAULT ) +blk_mq_virtio_map_queues (&set->map[i], vblk->vdev,0 ); +else +blk_mq_map_queues (&set->map[i]); + } + +return 0 ; +} + +static void virtblk_complete_batch (struct io_comp_batch *iob) +{ +struct request *req; +struct virtblk_req *vbr; -return blk_mq_virtio_map_queues (&set->map[HCTX_TYPE_DEFAULT ], - vblk->vdev,0 ); +rq_list_for_each (&iob->req_list, req) { + vbr =blk_mq_rq_to_pdu (req); +virtblk_unmap_data (req, vbr); +virtblk_cleanup_cmd (req); + } +blk_mq_end_request_batch (iob); +} + +static int virtblk_poll (struct blk_mq_hw_ctx *hctx,struct io_comp_batch *iob) +{ +struct virtio_blk_vq *vq = hctx->driver_data; +struct virtblk_req *vbr; +unsigned long flags; +unsigned int len; +int found =0 ; + +spin_lock_irqsave (&vq->lock, flags); + +while ((vbr =virtqueue_get_buf (vq->vq, &len)) !=NULL ) { +struct request *req =blk_mq_rq_from_pdu (vbr); + + found++; +if (!blk_mq_add_to_batch (req, iob, vbr->status, + virtblk_complete_batch)) +blk_mq_complete_request (req); + } + +spin_unlock_irqrestore (&vq->lock, flags); + +return found; +} + +static int virtblk_init_hctx (struct blk_mq_hw_ctx *hctx,void *data, +unsigned int hctx_idx) +{ +struct virtio_blk *vblk = data; +struct virtio_blk_vq *vq = &vblk->vqs[hctx_idx]; + +WARN_ON (vblk->tag_set.tags[hctx_idx] != hctx->tags); + hctx->driver_data = vq; +return 0 ; }static const struct blk_mq_ops virtio_mq_ops = { .queue_rq = virtio_queue_rq, .commit_rqs = virtio_commit_rqs, + .init_hctx = virtblk_init_hctx, .complete = virtblk_request_done, .map_queues = virtblk_map_queues, + .poll = virtblk_poll, };static unsigned int virtblk_queue_depth;@@ -816,6 +906,9 @@ static int virtblk_probe(struct virtio_device *vdev)sizeof (struct scatterlist ) *VIRTIO_BLK_INLINE_SG_CNT ; vblk->tag_set.driver_data = vblk; vblk->tag_set.nr_hw_queues = vblk->num_vqs; + vblk->tag_set.nr_maps =1 ; +if (vblk->io_queues[HCTX_TYPE_POLL ]) + vblk->tag_set.nr_maps =3 ; err =blk_mq_alloc_tag_set (&vblk->tag_set);if (err)Signature · 1 line
2.26.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 1/2] virtio-blk: support polling I/O
Quoted 60 lines · Michael S. Tsirkin
Quoted 59 lines · Suwan Kim
This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "
num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.
virtio-blk polling is implemented upon 'batched completion' of block layer.
virtblk_poll()queues completed request toio_comp_batch->req_listand later,virtblk_complete_batch()calls unmap function and ends the requests in batch.virtio-blk reads the number of poll queues from module parameter "
num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.
For verifying the improvement, I did Fio polling I/O performance test with
io_uringengine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.As a result, IOPS and average latency improved about 10%.
Test result:
- Fio
io_uringpoll without virtio-blk poll support-- numjobs=1 : IOPS = 339K, avg latency = 188.33us -- numjobs=2 : IOPS = 367K, avg latency = 347.33us -- numjobs=4 : IOPS = 383K, avg latency = 682.06us- Fio
io_uringpoll with virtio-blk poll support-- numjobs=1 : IOPS = 380K, avg latency = 167.87us -- numjobs=2 : IOPS = 409K, avg latency = 312.6us -- numjobs=4 : IOPS = 413K, avg latency = 619.72usReported-by kernel test robot <[email protected]>Signed-off-by Suwan Kim <[email protected]>--- drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++-- 1 file changed, 97 insertions(+), 4 deletions(-)drivers/block/virtio_blk.c +0
-0@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,"0 for no limit. " "Values > nr_cpu_ids truncated to nr_cpu_ids.");
Quoted 11 lines
drivers/block/virtio_blk.c MODULE_PARM_DESC(num_request_queues, · line 46
Quoted 7 lines
drivers/block/virtio_blk.c +4
-0+static unsigned int num_poll_queues; +module_param (num_poll_queues, uint,0644 ); +MODULE_PARM_DESC (num_poll_queues,"The number of dedicated virtqueues for polling I/O" ); +static int major;static DEFINE_IDA (vd_index_ida);Is there some way to make it work reasonably without need to set module parameters? I don't see any other devices with a
num_poll_queuesparameter - how do they handle this?
Hi Michael,
NVMe driver uses module parameter.
Please refer to this.
-----drivers/nvme/host/pci.c
static unsigned int poll_queues; module_param_cb(poll_queues, &io_queue_count_ops, &poll_queues, 0644); MODULE_PARM_DESC(poll_queues, "Number of queues to use for polled IO.");
-----
Regards,
Suwan Kim
Michael S. Tsirkin
maintainerRe: [PATCH v3 1/2] virtio-blk: support polling I/O
Quoted 81 lines · Suwan Kim
Quoted 60 lines · Michael S. Tsirkin
Quoted 59 lines · Suwan Kim
This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "
num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.
virtio-blk polling is implemented upon 'batched completion' of block layer.
virtblk_poll()queues completed request toio_comp_batch->req_listand later,virtblk_complete_batch()calls unmap function and ends the requests in batch.virtio-blk reads the number of poll queues from module parameter "
num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.
For verifying the improvement, I did Fio polling I/O performance test with
io_uringengine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.As a result, IOPS and average latency improved about 10%.
Test result:
- Fio
io_uringpoll without virtio-blk poll support-- numjobs=1 : IOPS = 339K, avg latency = 188.33us -- numjobs=2 : IOPS = 367K, avg latency = 347.33us -- numjobs=4 : IOPS = 383K, avg latency = 682.06us- Fio
io_uringpoll with virtio-blk poll support-- numjobs=1 : IOPS = 380K, avg latency = 167.87us -- numjobs=2 : IOPS = 409K, avg latency = 312.6us -- numjobs=4 : IOPS = 413K, avg latency = 619.72usReported-by kernel test robot <[email protected]>Signed-off-by Suwan Kim <[email protected]>--- drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++-- 1 file changed, 97 insertions(+), 4 deletions(-)drivers/block/virtio_blk.c +0
-0@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,"0 for no limit. " "Values > nr_cpu_ids truncated to nr_cpu_ids.");Quoted 11 lines
drivers/block/virtio_blk.c MODULE_PARM_DESC(num_request_queues, · line 46
Quoted 7 lines
drivers/block/virtio_blk.c +4
-0+static unsigned int num_poll_queues; +module_param (num_poll_queues, uint,0644 ); +MODULE_PARM_DESC (num_poll_queues,"The number of dedicated virtqueues for polling I/O" ); +static int major;static DEFINE_IDA (vd_index_ida);Is there some way to make it work reasonably without need to set module parameters? I don't see any other devices with a
num_poll_queuesparameter - how do they handle this?Hi Michael,
NVMe driver uses module parameter.
Please refer to this.
-----drivers/nvme/host/pci.c
Quoted 7 lines
static unsigned int
poll_queues;module_param_cb(poll_queues, &io_queue_count_ops, &poll_queues, 0644);MODULE_PARM_DESC(poll_queues, "Number of queues to use for polled IO.");
-----Regards,
Suwan Kim
OK then. Let's maybe be consistent wrt parameter naming?
Signature · 1 line
MST
Re: [PATCH v3 1/2] virtio-blk: support polling I/O
Quoted 87 lines · Michael S. Tsirkin
Quoted 86 lines · Suwan Kim
Quoted 60 lines · Michael S. Tsirkin
Quoted 59 lines · Suwan Kim
This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "
num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.
virtio-blk polling is implemented upon 'batched completion' of block layer.
virtblk_poll()queues completed request toio_comp_batch->req_listand later,virtblk_complete_batch()calls unmap function and ends the requests in batch.virtio-blk reads the number of poll queues from module parameter "
num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.
For verifying the improvement, I did Fio polling I/O performance test with
io_uringengine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.As a result, IOPS and average latency improved about 10%.
Test result:
- Fio
io_uringpoll without virtio-blk poll support-- numjobs=1 : IOPS = 339K, avg latency = 188.33us -- numjobs=2 : IOPS = 367K, avg latency = 347.33us -- numjobs=4 : IOPS = 383K, avg latency = 682.06us- Fio
io_uringpoll with virtio-blk poll support-- numjobs=1 : IOPS = 380K, avg latency = 167.87us -- numjobs=2 : IOPS = 409K, avg latency = 312.6us -- numjobs=4 : IOPS = 413K, avg latency = 619.72usReported-by kernel test robot <[email protected]>Signed-off-by Suwan Kim <[email protected]>--- drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++-- 1 file changed, 97 insertions(+), 4 deletions(-)drivers/block/virtio_blk.c +0
-0@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,"0 for no limit. " "Values > nr_cpu_ids truncated to nr_cpu_ids.");Quoted 11 lines
drivers/block/virtio_blk.c MODULE_PARM_DESC(num_request_queues, · line 46
Quoted 7 lines
drivers/block/virtio_blk.c +4
-0+static unsigned int num_poll_queues; +module_param (num_poll_queues, uint,0644 ); +MODULE_PARM_DESC (num_poll_queues,"The number of dedicated virtqueues for polling I/O" ); +static int major;static DEFINE_IDA (vd_index_ida);Is there some way to make it work reasonably without need to set module parameters? I don't see any other devices with a
num_poll_queuesparameter - how do they handle this?Hi Michael,
NVMe driver uses module parameter.
Please refer to this.
-----drivers/nvme/host/pci.cstatic unsigned int
poll_queues;module_param_cb(poll_queues, &io_queue_count_ops, &poll_queues, 0644);MODULE_PARM_DESC(poll_queues, "Number of queues to use for polled IO.");
-----
Quoted 4 lines
Quoted 2 lines
Regards,
Suwan KimOK then. Let's maybe be consistent wrt parameter naming?
Ok. Consistent naming scheme seems to be better for code readability. I will rename it to 'poll_queues' in next version.
Regards,
Suwan Kim
Re: [PATCH v3 1/2] virtio-blk: support polling I/O
In reply to Suwan KimHi Suwan,
The NVMe prints something like below by nvme_setup_io_queues() to confirm if the configuration takes effect.
"[ 0.620458] nvme nvme0: 4/0/0 default/read/poll queues".
How about to print in virtio-blk as well?
Thank you very much!
Dongli Zhang
Quoted 211 lines · Suwan Kim
This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "
num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.
virtio-blk polling is implemented upon 'batched completion' of block layer.
virtblk_poll()queues completed request toio_comp_batch->req_listand later,virtblk_complete_batch()calls unmap function and ends the requests in batch.virtio-blk reads the number of poll queues from module parameter "
num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.
For verifying the improvement, I did Fio polling I/O performance test with
io_uringengine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.As a result, IOPS and average latency improved about 10%.
Test result:
- Fio
io_uringpoll without virtio-blk poll support-- numjobs=1 : IOPS = 339K, avg latency = 188.33us -- numjobs=2 : IOPS = 367K, avg latency = 347.33us -- numjobs=4 : IOPS = 383K, avg latency = 682.06us- Fio
io_uringpoll with virtio-blk poll support-- numjobs=1 : IOPS = 380K, avg latency = 167.87us -- numjobs=2 : IOPS = 409K, avg latency = 312.6us -- numjobs=4 : IOPS = 413K, avg latency = 619.72usReported-by kernel test robot <[email protected]>Signed-off-by Suwan Kim <[email protected]>--- drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++-- 1 file changed, 97 insertions(+), 4 deletions(-)drivers/block/virtio_blk.c +94
-4@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,"0 for no limit. " "Values > nr_cpu_ids truncated to nr_cpu_ids."); +static unsigned int num_poll_queues; +module_param (num_poll_queues, uint,0644 ); +MODULE_PARM_DESC (num_poll_queues,"The number of dedicated virtqueues for polling I/O" ); +static int major;static DEFINE_IDA (vd_index_ida);@@ -81,6 +85,7 @@ struct virtio_blk {/* num of vqs */ int num_vqs; +int io_queues[HCTX_MAX_TYPES ];struct virtio_blk_vq *vqs; };@@ -548,6 +553,7 @@ static int init_vq(struct virtio_blk *vblk)const char **names;struct virtqueue **vqs;unsigned short num_vqs; +unsigned int num_poll_vqs;struct virtio_device *vdev = vblk->vdev;struct irq_affinity desc = {0 , };@@ -556,6 +562,7 @@ static int init_vq(struct virtio_blk *vblk) &num_vqs);if (err) num_vqs =1 ; +if (!err && !num_vqs) {dev_err (&vdev->dev,"MQ advertised but zero queues reported\n" );return -EINVAL ;@@ -565,6 +572,13 @@ static int init_vq(struct virtio_blk *vblk)min_not_zero (num_request_queues, nr_cpu_ids), num_vqs); + num_poll_vqs = min_t(unsigned int, num_poll_queues, num_vqs - 1); + +memset (vblk->io_queues,0 ,sizeof (int ) *HCTX_MAX_TYPES ); + vblk->io_queues[HCTX_TYPE_DEFAULT ] = num_vqs - num_poll_vqs; + vblk->io_queues[HCTX_TYPE_READ ] =0 ; + vblk->io_queues[HCTX_TYPE_POLL ] = num_poll_vqs; + vblk->vqs =kmalloc_array (num_vqs,sizeof (*vblk->vqs),GFP_KERNEL );if (!vblk->vqs)return -ENOMEM ;@@ -578,8 +592,13 @@ static int init_vq(struct virtio_blk *vblk) }for (i =0 ; i < num_vqs; i++) { - callbacks[i] = virtblk_done; -snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req.%d" , i); +if (i < num_vqs - num_poll_vqs) { + callbacks[i] = virtblk_done; +snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req.%d" , i); + }else { + callbacks[i] =NULL ; +snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req_poll.%d" , i); + } names[i] = vblk->vqs[i].name; }@@ -728,16 +747,87 @@ static const struct attribute_group *virtblk_attr_groups[] = {static int virtblk_map_queues (struct blk_mq_tag_set *set) {struct virtio_blk *vblk = set->driver_data; +int i, qoff; + +for (i =0 , qoff =0 ; i < set->nr_maps; i++) { +struct blk_mq_queue_map *map = &set->map[i]; + + map->nr_queues = vblk->io_queues[i]; + map->queue_offset = qoff; + qoff += map->nr_queues; + +if (map->nr_queues ==0 ) +continue ; + +/* +* Regular queues have interrupts and hence CPU affinity is +* defined by the core virtio code, but polling queues have +* no interrupts so we let the block layer assign CPU affinity. +*/ +if (i ==HCTX_TYPE_DEFAULT ) +blk_mq_virtio_map_queues (&set->map[i], vblk->vdev,0 ); +else +blk_mq_map_queues (&set->map[i]); + } + +return 0 ; +} + +static void virtblk_complete_batch (struct io_comp_batch *iob) +{ +struct request *req; +struct virtblk_req *vbr; -return blk_mq_virtio_map_queues (&set->map[HCTX_TYPE_DEFAULT ], - vblk->vdev,0 ); +rq_list_for_each (&iob->req_list, req) { + vbr =blk_mq_rq_to_pdu (req); +virtblk_unmap_data (req, vbr); +virtblk_cleanup_cmd (req); + } +blk_mq_end_request_batch (iob); +} + +static int virtblk_poll (struct blk_mq_hw_ctx *hctx,struct io_comp_batch *iob) +{ +struct virtio_blk_vq *vq = hctx->driver_data; +struct virtblk_req *vbr; +unsigned long flags; +unsigned int len; +int found =0 ; + +spin_lock_irqsave (&vq->lock, flags); + +while ((vbr =virtqueue_get_buf (vq->vq, &len)) !=NULL ) { +struct request *req =blk_mq_rq_from_pdu (vbr); + + found++; +if (!blk_mq_add_to_batch (req, iob, vbr->status, + virtblk_complete_batch)) +blk_mq_complete_request (req); + } + +spin_unlock_irqrestore (&vq->lock, flags); + +return found; +} + +static int virtblk_init_hctx (struct blk_mq_hw_ctx *hctx,void *data, +unsigned int hctx_idx) +{ +struct virtio_blk *vblk = data; +struct virtio_blk_vq *vq = &vblk->vqs[hctx_idx]; + +WARN_ON (vblk->tag_set.tags[hctx_idx] != hctx->tags); + hctx->driver_data = vq; +return 0 ; }static const struct blk_mq_ops virtio_mq_ops = { .queue_rq = virtio_queue_rq, .commit_rqs = virtio_commit_rqs, + .init_hctx = virtblk_init_hctx, .complete = virtblk_request_done, .map_queues = virtblk_map_queues, + .poll = virtblk_poll, };static unsigned int virtblk_queue_depth;
drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 914
Quoted 9 lines
drivers/block/virtio_blk.c +3
-0@@ -816,6 +906,9 @@ static int virtblk_probe(struct virtio_device *vdev)sizeof (struct scatterlist ) *VIRTIO_BLK_INLINE_SG_CNT ; vblk->tag_set.driver_data = vblk; vblk->tag_set.nr_hw_queues = vblk->num_vqs; + vblk->tag_set.nr_maps =1 ; +if (vblk->io_queues[HCTX_TYPE_POLL ]) + vblk->tag_set.nr_maps =3 ; err =blk_mq_alloc_tag_set (&vblk->tag_set);if (err)
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 1/2] virtio-blk: support polling I/O
Quoted 8 lines · Dongli Zhang
Hi Suwan,
The NVMe prints something like below by
nvme_setup_io_queues()to confirm if the configuration takes effect."[ 0.620458] nvme nvme0: 4/0/0 default/read/poll queues".
How about to print in virtio-blk as well?
Hi Dongli,
Thansk for your feedback. It is good idea.
I will add it in next version.
Regards,
Suwan Kim
Michael S. Tsirkin
maintainerRe: [PATCH v3 1/2] virtio-blk: support polling I/O
In reply to Suwan KimQuoted 211 lines · Suwan Kim
This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "
num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.
virtio-blk polling is implemented upon 'batched completion' of block layer.
virtblk_poll()queues completed request toio_comp_batch->req_listand later,virtblk_complete_batch()calls unmap function and ends the requests in batch.virtio-blk reads the number of poll queues from module parameter "
num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.
For verifying the improvement, I did Fio polling I/O performance test with
io_uringengine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.As a result, IOPS and average latency improved about 10%.
Test result:
- Fio
io_uringpoll without virtio-blk poll support-- numjobs=1 : IOPS = 339K, avg latency = 188.33us -- numjobs=2 : IOPS = 367K, avg latency = 347.33us -- numjobs=4 : IOPS = 383K, avg latency = 682.06us- Fio
io_uringpoll with virtio-blk poll support-- numjobs=1 : IOPS = 380K, avg latency = 167.87us -- numjobs=2 : IOPS = 409K, avg latency = 312.6us -- numjobs=4 : IOPS = 413K, avg latency = 619.72usReported-by kernel test robot <[email protected]>Signed-off-by Suwan Kim <[email protected]>--- drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++-- 1 file changed, 97 insertions(+), 4 deletions(-)drivers/block/virtio_blk.c +94
-4@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,"0 for no limit. " "Values > nr_cpu_ids truncated to nr_cpu_ids."); +static unsigned int num_poll_queues; +module_param (num_poll_queues, uint,0644 ); +MODULE_PARM_DESC (num_poll_queues,"The number of dedicated virtqueues for polling I/O" ); +static int major;static DEFINE_IDA (vd_index_ida);@@ -81,6 +85,7 @@ struct virtio_blk {/* num of vqs */ int num_vqs; +int io_queues[HCTX_MAX_TYPES ];struct virtio_blk_vq *vqs; };@@ -548,6 +553,7 @@ static int init_vq(struct virtio_blk *vblk)const char **names;struct virtqueue **vqs;unsigned short num_vqs; +unsigned int num_poll_vqs;struct virtio_device *vdev = vblk->vdev;struct irq_affinity desc = {0 , };@@ -556,6 +562,7 @@ static int init_vq(struct virtio_blk *vblk) &num_vqs);if (err) num_vqs =1 ; +if (!err && !num_vqs) {dev_err (&vdev->dev,"MQ advertised but zero queues reported\n" );return -EINVAL ;@@ -565,6 +572,13 @@ static int init_vq(struct virtio_blk *vblk)min_not_zero (num_request_queues, nr_cpu_ids), num_vqs); + num_poll_vqs = min_t(unsigned int, num_poll_queues, num_vqs - 1); + +memset (vblk->io_queues,0 ,sizeof (int ) *HCTX_MAX_TYPES ); + vblk->io_queues[HCTX_TYPE_DEFAULT ] = num_vqs - num_poll_vqs; + vblk->io_queues[HCTX_TYPE_READ ] =0 ; + vblk->io_queues[HCTX_TYPE_POLL ] = num_poll_vqs; + vblk->vqs =kmalloc_array (num_vqs,sizeof (*vblk->vqs),GFP_KERNEL );if (!vblk->vqs)return -ENOMEM ;@@ -578,8 +592,13 @@ static int init_vq(struct virtio_blk *vblk) }for (i =0 ; i < num_vqs; i++) { - callbacks[i] = virtblk_done; -snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req.%d" , i); +if (i < num_vqs - num_poll_vqs) { + callbacks[i] = virtblk_done; +snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req.%d" , i); + }else { + callbacks[i] =NULL ; +snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req_poll.%d" , i); + } names[i] = vblk->vqs[i].name; }@@ -728,16 +747,87 @@ static const struct attribute_group *virtblk_attr_groups[] = {static int virtblk_map_queues (struct blk_mq_tag_set *set) {struct virtio_blk *vblk = set->driver_data; +int i, qoff; + +for (i =0 , qoff =0 ; i < set->nr_maps; i++) { +struct blk_mq_queue_map *map = &set->map[i]; + + map->nr_queues = vblk->io_queues[i]; + map->queue_offset = qoff; + qoff += map->nr_queues; + +if (map->nr_queues ==0 ) +continue ; + +/* +* Regular queues have interrupts and hence CPU affinity is +* defined by the core virtio code, but polling queues have +* no interrupts so we let the block layer assign CPU affinity. +*/ +if (i ==HCTX_TYPE_DEFAULT ) +blk_mq_virtio_map_queues (&set->map[i], vblk->vdev,0 ); +else +blk_mq_map_queues (&set->map[i]); + } + +return 0 ; +} + +static void virtblk_complete_batch (struct io_comp_batch *iob) +{ +struct request *req; +struct virtblk_req *vbr; -return blk_mq_virtio_map_queues (&set->map[HCTX_TYPE_DEFAULT ], - vblk->vdev,0 ); +rq_list_for_each (&iob->req_list, req) { + vbr =blk_mq_rq_to_pdu (req); +virtblk_unmap_data (req, vbr); +virtblk_cleanup_cmd (req); + } +blk_mq_end_request_batch (iob); +} + +static int virtblk_poll (struct blk_mq_hw_ctx *hctx,struct io_comp_batch *iob) +{ +struct virtio_blk_vq *vq = hctx->driver_data; +struct virtblk_req *vbr; +unsigned long flags; +unsigned int len; +int found =0 ; + +spin_lock_irqsave (&vq->lock, flags); + +while ((vbr =virtqueue_get_buf (vq->vq, &len)) !=NULL ) { +struct request *req =blk_mq_rq_from_pdu (vbr); + + found++; +if (!blk_mq_add_to_batch (req, iob, vbr->status, + virtblk_complete_batch)) +blk_mq_complete_request (req); + } + +spin_unlock_irqrestore (&vq->lock, flags); + +return found; +} + +static int virtblk_init_hctx (struct blk_mq_hw_ctx *hctx,void *data, +unsigned int hctx_idx) +{ +struct virtio_blk *vblk = data; +struct virtio_blk_vq *vq = &vblk->vqs[hctx_idx]; + +WARN_ON (vblk->tag_set.tags[hctx_idx] != hctx->tags); + hctx->driver_data = vq; +return 0 ; }static const struct blk_mq_ops virtio_mq_ops = { .queue_rq = virtio_queue_rq, .commit_rqs = virtio_commit_rqs, + .init_hctx = virtblk_init_hctx, .complete = virtblk_request_done, .map_queues = virtblk_map_queues, + .poll = virtblk_poll, };static unsigned int virtblk_queue_depth;
drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 914
Quoted 9 lines
drivers/block/virtio_blk.c +3
-0@@ -816,6 +906,9 @@ static int virtblk_probe(struct virtio_device *vdev)sizeof (struct scatterlist ) *VIRTIO_BLK_INLINE_SG_CNT ; vblk->tag_set.driver_data = vblk; vblk->tag_set.nr_hw_queues = vblk->num_vqs; + vblk->tag_set.nr_maps =1 ; +if (vblk->io_queues[HCTX_TYPE_POLL ]) + vblk->tag_set.nr_maps =3 ; err =blk_mq_alloc_tag_set (&vblk->tag_set);if (err)
So wrt cleanup, does something poll for all buffers to be used when device is removed?
Quoted 2 lines
Signature · 1 line
2.26.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 1/2] virtio-blk: support polling I/O
Quoted 219 lines · Michael S. Tsirkin
Quoted 218 lines · Suwan Kim
This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "
num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.
virtio-blk polling is implemented upon 'batched completion' of block layer.
virtblk_poll()queues completed request toio_comp_batch->req_listand later,virtblk_complete_batch()calls unmap function and ends the requests in batch.virtio-blk reads the number of poll queues from module parameter "
num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.
For verifying the improvement, I did Fio polling I/O performance test with
io_uringengine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.As a result, IOPS and average latency improved about 10%.
Test result:
- Fio
io_uringpoll without virtio-blk poll support-- numjobs=1 : IOPS = 339K, avg latency = 188.33us -- numjobs=2 : IOPS = 367K, avg latency = 347.33us -- numjobs=4 : IOPS = 383K, avg latency = 682.06us- Fio
io_uringpoll with virtio-blk poll support-- numjobs=1 : IOPS = 380K, avg latency = 167.87us -- numjobs=2 : IOPS = 409K, avg latency = 312.6us -- numjobs=4 : IOPS = 413K, avg latency = 619.72usReported-by kernel test robot <[email protected]>Signed-off-by Suwan Kim <[email protected]>--- drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++-- 1 file changed, 97 insertions(+), 4 deletions(-)drivers/block/virtio_blk.c +97
-4@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,"0 for no limit. " "Values > nr_cpu_ids truncated to nr_cpu_ids."); +static unsigned int num_poll_queues; +module_param (num_poll_queues, uint,0644 ); +MODULE_PARM_DESC (num_poll_queues,"The number of dedicated virtqueues for polling I/O" ); +static int major;static DEFINE_IDA (vd_index_ida);@@ -81,6 +85,7 @@ struct virtio_blk {/* num of vqs */ int num_vqs; +int io_queues[HCTX_MAX_TYPES ];struct virtio_blk_vq *vqs; };@@ -548,6 +553,7 @@ static int init_vq(struct virtio_blk *vblk)const char **names;struct virtqueue **vqs;unsigned short num_vqs; +unsigned int num_poll_vqs;struct virtio_device *vdev = vblk->vdev;struct irq_affinity desc = {0 , };@@ -556,6 +562,7 @@ static int init_vq(struct virtio_blk *vblk) &num_vqs);if (err) num_vqs =1 ; +if (!err && !num_vqs) {dev_err (&vdev->dev,"MQ advertised but zero queues reported\n" );return -EINVAL ;@@ -565,6 +572,13 @@ static int init_vq(struct virtio_blk *vblk)min_not_zero (num_request_queues, nr_cpu_ids), num_vqs); + num_poll_vqs = min_t(unsigned int, num_poll_queues, num_vqs - 1); + +memset (vblk->io_queues,0 ,sizeof (int ) *HCTX_MAX_TYPES ); + vblk->io_queues[HCTX_TYPE_DEFAULT ] = num_vqs - num_poll_vqs; + vblk->io_queues[HCTX_TYPE_READ ] =0 ; + vblk->io_queues[HCTX_TYPE_POLL ] = num_poll_vqs; + vblk->vqs =kmalloc_array (num_vqs,sizeof (*vblk->vqs),GFP_KERNEL );if (!vblk->vqs)return -ENOMEM ;@@ -578,8 +592,13 @@ static int init_vq(struct virtio_blk *vblk) }for (i =0 ; i < num_vqs; i++) { - callbacks[i] = virtblk_done; -snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req.%d" , i); +if (i < num_vqs - num_poll_vqs) { + callbacks[i] = virtblk_done; +snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req.%d" , i); + }else { + callbacks[i] =NULL ; +snprintf (vblk->vqs[i].name,VQ_NAME_LEN ,"req_poll.%d" , i); + } names[i] = vblk->vqs[i].name; }@@ -728,16 +747,87 @@ static const struct attribute_group *virtblk_attr_groups[] = {static int virtblk_map_queues (struct blk_mq_tag_set *set) {struct virtio_blk *vblk = set->driver_data; +int i, qoff; + +for (i =0 , qoff =0 ; i < set->nr_maps; i++) { +struct blk_mq_queue_map *map = &set->map[i]; + + map->nr_queues = vblk->io_queues[i]; + map->queue_offset = qoff; + qoff += map->nr_queues; + +if (map->nr_queues ==0 ) +continue ; + +/* +* Regular queues have interrupts and hence CPU affinity is +* defined by the core virtio code, but polling queues have +* no interrupts so we let the block layer assign CPU affinity. +*/ +if (i ==HCTX_TYPE_DEFAULT ) +blk_mq_virtio_map_queues (&set->map[i], vblk->vdev,0 ); +else +blk_mq_map_queues (&set->map[i]); + } + +return 0 ; +} + +static void virtblk_complete_batch (struct io_comp_batch *iob) +{ +struct request *req; +struct virtblk_req *vbr; -return blk_mq_virtio_map_queues (&set->map[HCTX_TYPE_DEFAULT ], - vblk->vdev,0 ); +rq_list_for_each (&iob->req_list, req) { + vbr =blk_mq_rq_to_pdu (req); +virtblk_unmap_data (req, vbr); +virtblk_cleanup_cmd (req); + } +blk_mq_end_request_batch (iob); +} + +static int virtblk_poll (struct blk_mq_hw_ctx *hctx,struct io_comp_batch *iob) +{ +struct virtio_blk_vq *vq = hctx->driver_data; +struct virtblk_req *vbr; +unsigned long flags; +unsigned int len; +int found =0 ; + +spin_lock_irqsave (&vq->lock, flags); + +while ((vbr =virtqueue_get_buf (vq->vq, &len)) !=NULL ) { +struct request *req =blk_mq_rq_from_pdu (vbr); + + found++; +if (!blk_mq_add_to_batch (req, iob, vbr->status, + virtblk_complete_batch)) +blk_mq_complete_request (req); + } + +spin_unlock_irqrestore (&vq->lock, flags); + +return found; +} + +static int virtblk_init_hctx (struct blk_mq_hw_ctx *hctx,void *data, +unsigned int hctx_idx) +{ +struct virtio_blk *vblk = data; +struct virtio_blk_vq *vq = &vblk->vqs[hctx_idx]; + +WARN_ON (vblk->tag_set.tags[hctx_idx] != hctx->tags); + hctx->driver_data = vq; +return 0 ; }static const struct blk_mq_ops virtio_mq_ops = { .queue_rq = virtio_queue_rq, .commit_rqs = virtio_commit_rqs, + .init_hctx = virtblk_init_hctx, .complete = virtblk_request_done, .map_queues = virtblk_map_queues, + .poll = virtblk_poll, };static unsigned int virtblk_queue_depth;@@ -816,6 +906,9 @@ static int virtblk_probe(struct virtio_device *vdev)sizeof (struct scatterlist ) *VIRTIO_BLK_INLINE_SG_CNT ; vblk->tag_set.driver_data = vblk; vblk->tag_set.nr_hw_queues = vblk->num_vqs; + vblk->tag_set.nr_maps =1 ; +if (vblk->io_queues[HCTX_TYPE_POLL ]) + vblk->tag_set.nr_maps =3 ;
Quoted 7 lines
drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 914
Quoted 2 lines
drivers/block/virtio_blk.c +0
-0err =blk_mq_alloc_tag_set (&vblk->tag_set);if (err)So wrt cleanup, does something poll for all buffers to be used when device is removed?
Sorry for late reply.
Maybe below function calls iterate each HW queue and flush requests before device is removed?
-----virtblk_remove() -> blk_cleanup_disk()/blk_cleanup_queue() -> blk_queue_start_drain()/blk_freeze_queue()
-----
Regards,
Suwan Kim
Stefan Hajnoczi
reviewerRe: [PATCH v3 1/2] virtio-blk: support polling I/O
In reply to Suwan KimQuoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10
-0+static int virtblk_poll (struct blk_mq_hw_ctx *hctx,struct io_comp_batch *iob) +{ +struct virtio_blk_vq *vq = hctx->driver_data; +struct virtblk_req *vbr; +unsigned long flags; +unsigned int len; +int found =0 ; + +spin_lock_irqsave (&vq->lock, flags); +
drivers/block/virtio_blk.c static const struct attribute_group *virtblk_attr_groups[] = { · line 808
Quoted 10 lines
drivers/block/virtio_blk.c +10
-0+while ((vbr =virtqueue_get_buf (vq->vq, &len)) !=NULL ) { +struct request *req =blk_mq_rq_from_pdu (vbr); + + found++; +if (!blk_mq_add_to_batch (req, iob, vbr->status, + virtblk_complete_batch)) +blk_mq_complete_request (req); + } + +spin_unlock_irqrestore (&vq->lock, flags);
virtblk_done() does:
/* In case queue is stopped waiting for more buffers. */ if (req_done)
blk_mq_start_stopped_hw_queues(vblk->disk->queue, true);
Is the same thing needed here in virtblk_poll() so that stopped queues are restarted when requests complete?
Re: [PATCH v3 1/2] virtio-blk: support polling I/O
Quoted 24 lines · Stefan Hajnoczi
Quoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10
-0+static int virtblk_poll (struct blk_mq_hw_ctx *hctx,struct io_comp_batch *iob) +{ +struct virtio_blk_vq *vq = hctx->driver_data; +struct virtblk_req *vbr; +unsigned long flags; +unsigned int len; +int found =0 ; + +spin_lock_irqsave (&vq->lock, flags); +drivers/block/virtio_blk.c static const struct attribute_group *virtblk_attr_groups[] = { · line 808
Quoted 10 lines
drivers/block/virtio_blk.c +10
-0+while ((vbr =virtqueue_get_buf (vq->vq, &len)) !=NULL ) { +struct request *req =blk_mq_rq_from_pdu (vbr); + + found++; +if (!blk_mq_add_to_batch (req, iob, vbr->status, + virtblk_complete_batch)) +blk_mq_complete_request (req); + } + +spin_unlock_irqrestore (&vq->lock, flags);
virtblk_done()does:
Quoted 6 lines
/* In case queue is stopped waiting for more buffers. */ if (
req_done)blk_mq_start_stopped_hw_queues(vblk->disk->queue, true);Is the same thing needed here in
virtblk_poll()so that stopped queues are restarted when requests complete?
I think you are right. I missed that.
I just added blk_mq_start_stopped_hw_queues() to virtblk_poll as you commented and did performance test again.
It showed higher peak performance than virtblk_poll without blk_mq_start_stopped_hw_queues().
I will add it in next version.
Thanks for the comment!
Regards,
Suwan Kim
PATCH v3 2/2 virtio-blk: support mq_ops->queue_rqs()
In reply to Suwan KimThis patch supports mq_ops->queue_rqs() hook. It has an advantage of batch submission to virtio-blk driver. It also helps polling I/O because polling uses batched completion of block layer. Batch submission in queue_rqs() can boost polling performance.
In queue_rqs(), it iterates plug->mq_list, collects requests that belong to same HW queue and adds requests into virtqueue until it encounters a request from other HW queue or sees the end of the list. Then, virtio-blk kicks virtqueue to submit requests.
If there is an error, it inserts error request to requeue_list and passes it to ordinary block layer path.
For verification, I did fio test.
(io_uring, randread, direct=1, bs=4K, iodepth=64 numjobs=N) I set 4 vcpu and 2 virtio-blk queues for VM and run fio test 5 times. It shows about 1-4% improvement.
| numjobs=2 | numjobs=4
-----------------------------------------------------------
fio without queue_rqs() | 282K IOPS | 245K IOPS
-----------------------------------------------------------
fio with queue_rqs() | 294K IOPS | 249K IOPS
For polling I/O performance, I also did fio test as below. (io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=4) I set 4 vcpu and 2 poll queues for VM.
It shows upto 7% improvement in polling I/O.
| IOPS | avg latency
-----------------------------------------------------------
fio poll without queue_rqs() | 413K | 619.72 usec
-----------------------------------------------------------
fio poll with queue_rqs() | 445K | 581.2 usec
--- drivers/block/virtio_blk.c | 93 ++++++++++++++++++++++++++++++++++---- 1 file changed, 84 insertions(+), 9 deletions(-)
drivers/block/virtio_blk.c +84 -9
@@ -311,6 +311,28 @@ static void virtio_commit_rqs(struct blk_mq_hw_ctx *hctx)virtqueue_notify (vq->vq); } +static blk_status_t virtblk_prep_rq (struct blk_mq_hw_ctx *hctx, +struct virtio_blk *vblk, +struct request *req, +struct virtblk_req *vbr,int *num) +{ +blk_status_t status; + + status =virtblk_setup_cmd (vblk->vdev, req, vbr); +if (unlikely (status)) +return status; + +blk_mq_start_request (req); + + *num =virtblk_map_data (hctx, req, vbr); +if (unlikely (*num <0 )) { +virtblk_cleanup_cmd (req); +return BLK_STS_RESOURCE ; + } + +return BLK_STS_OK ; +} +static blk_status_t virtio_queue_rq (struct blk_mq_hw_ctx *hctx,const struct blk_mq_queue_data *bd) {
@@ -324,18 +346,10 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,blk_status_t status;int err; - status =virtblk_setup_cmd (vblk->vdev, req, vbr); + status =virtblk_prep_rq (hctx, vblk, req, vbr, &num);if (unlikely (status))return status; -blk_mq_start_request (req); - - num =virtblk_map_data (hctx, req, vbr); -if (unlikely (num <0 )) { -virtblk_cleanup_cmd (req); -return BLK_STS_RESOURCE ; - } -spin_lock_irqsave (&vblk->vqs[qid].lock, flags); err =virtblk_add_req (vblk->vqs[qid].vq, vbr, vbr->sg_table.sgl, num);if (err) {
@@ -367,6 +381,66 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,return BLK_STS_OK ; } +static bool virtblk_prep_rq_batch (struct virtio_blk_vq *vq,struct request *req) +{ +struct virtio_blk *vblk = req->mq_hctx->queue->queuedata; +struct virtblk_req *vbr =blk_mq_rq_to_pdu (req); +unsigned long flags; +int num, err; + + req->mq_hctx->tags->rqs[req->tag] = req; + +if (virtblk_prep_rq (req->mq_hctx, vblk, req, vbr, &num) !=BLK_STS_OK ) +return false; + +spin_lock_irqsave (&vq->lock, flags); + err =virtblk_add_req (vq->vq, vbr, vbr->sg_table.sgl, num); +if (err) { +spin_unlock_irqrestore (&vq->lock, flags); +virtblk_unmap_data (req, vbr); +virtblk_cleanup_cmd (req); +return false; + } +spin_unlock_irqrestore (&vq->lock, flags); + +return true; +} + +static void virtio_queue_rqs (struct request **rqlist) +{ +struct request *req, *next, *prev =NULL ; +struct request *requeue_list =NULL ; + +rq_list_for_each_safe (rqlist, req, next) { +struct virtio_blk_vq *vq = req->mq_hctx->driver_data; +unsigned long flags; +bool kick; + +if (!virtblk_prep_rq_batch (vq, req)) { +rq_list_move (rqlist, &requeue_list, req, prev); + req = prev; + +if (!req) +continue ; + } + +if (!next || req->mq_hctx != next->mq_hctx) { +spin_lock_irqsave (&vq->lock, flags); + kick =virtqueue_kick_prepare (vq->vq); +spin_unlock_irqrestore (&vq->lock, flags); +if (kick) +virtqueue_notify (vq->vq); + + req->rq_next =NULL ; + *rqlist = next; + prev =NULL ; + }else + prev = req; + } + + *rqlist = requeue_list; +} +/* return id (s/n) string for *disk to *id_str */ static int virtblk_get_id (struct gendisk *disk,char *id_str)
@@ -823,6 +897,7 @@ static int virtblk_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,static const struct blk_mq_ops virtio_mq_ops = { .queue_rq = virtio_queue_rq, + .queue_rqs = virtio_queue_rqs, .commit_rqs = virtio_commit_rqs, .init_hctx = virtblk_init_hctx, .complete = virtblk_request_done,
Signature · 1 line
2.26.3
Stefan Hajnoczi
reviewerRe: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()
Quoted 15 lines · Suwan Kim
drivers/block/virtio_blk.c +11
-0@@ -367,6 +381,66 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,return BLK_STS_OK ; } +static bool virtblk_prep_rq_batch (struct virtio_blk_vq *vq,struct request *req) +{ +struct virtio_blk *vblk = req->mq_hctx->queue->queuedata; +struct virtblk_req *vbr =blk_mq_rq_to_pdu (req); +unsigned long flags; +int num, err; + + req->mq_hctx->tags->rqs[req->tag] = req; + +if (virtblk_prep_rq (req->mq_hctx, vblk, req, vbr, &num) !=BLK_STS_OK ) +return false;
drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 404
Quoted 10 lines
drivers/block/virtio_blk.c +10
-0+ +spin_lock_irqsave (&vq->lock, flags); + err =virtblk_add_req (vq->vq, vbr, vbr->sg_table.sgl, num); +if (err) { +spin_unlock_irqrestore (&vq->lock, flags); +virtblk_unmap_data (req, vbr); +virtblk_cleanup_cmd (req); +return false; + } +spin_unlock_irqrestore (&vq->lock, flags);
Simplification:
spin_lock_irqsave(&vq->lock, flags);
err = virtblk_add_req(vq->vq, vbr, vbr->sg_table.sgl, num);
spin_unlock_irqrestore(&vq->lock, flags);
if (err) {
virtblk_unmap_data(req, vbr);
virtblk_cleanup_cmd(req);
return false;
}Quoted 14 lines
drivers/block/virtio_blk.c +14
-0+ +return true; +} + +static void virtio_queue_rqs (struct request **rqlist) +{ +struct request *req, *next, *prev =NULL ; +struct request *requeue_list =NULL ; + +rq_list_for_each_safe (rqlist, req, next) { +struct virtio_blk_vq *vq = req->mq_hctx->driver_data; +unsigned long flags; +bool kick; +
drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 428
Quoted 10 lines
drivers/block/virtio_blk.c +10
-0+if (!virtblk_prep_rq_batch (vq, req)) { +rq_list_move (rqlist, &requeue_list, req, prev); + req = prev; + +if (!req) +continue ; + } + +if (!next || req->mq_hctx != next->mq_hctx) { +spin_lock_irqsave (&vq->lock, flags);
Did you try calling virtblk_add_req() here to avoid acquiring and releasing the lock multiple times? In other words, do virtblk_prep_rq() but wait until we get here to do virtblk_add_req().
I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.
drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 438
Quoted 10 lines
drivers/block/virtio_blk.c +10
-0+ kick =virtqueue_kick_prepare (vq->vq); +spin_unlock_irqrestore (&vq->lock, flags); +if (kick) +virtqueue_notify (vq->vq); + + req->rq_next =NULL ; + *rqlist = next; + prev =NULL ; + }else + prev = req;
What guarantees that req is still alive after we called virtblk_add_req()? The device may have seen it and completed it already by the time we get here.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()
Quoted 29 lines · Stefan Hajnoczi
Quoted 15 lines · Suwan Kim
(quoted diff) +11
-0@@ -367,6 +381,66 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, return BLK_STS_OK; } +static bool virtblk_prep_rq_batch(struct virtio_blk_vq *vq, struct request *req) +{ + struct virtio_blk *vblk = req->mq_hctx->queue->queuedata; + struct virtblk_req *vbr = blk_mq_rq_to_pdu(req); + unsigned long flags; + int num, err; + + req->mq_hctx->tags->rqs[req->tag] = req; + + if (virtblk_prep_rq(req->mq_hctx, vblk, req, vbr, &num) != BLK_STS_OK) + return false;Review static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 404
Quoted 10 lines
(quoted diff) +10
-0+ + spin_lock_irqsave(&vq->lock, flags); + err = virtblk_add_req(vq->vq, vbr, vbr->sg_table.sgl, num); + if (err) { + spin_unlock_irqrestore(&vq->lock, flags); + virtblk_unmap_data(req, vbr); + virtblk_cleanup_cmd(req); + return false; + } + spin_unlock_irqrestore(&vq->lock, flags);Simplification:
Quoted 9 lines
spin_lock_irqsave(&vq->lock, flags);
err = virtblk_add_req(vq->vq, vbr, vbr->sg_table.sgl, num);
spin_unlock_irqrestore(&vq->lock, flags);
if (err) {
virtblk_unmap_data(req, vbr);
virtblk_cleanup_cmd(req);
return false;
}Thanks! I will fix it.
Quoted 29 lines
Quoted 14 lines
(quoted diff) +14
-0+ + return true; +} + +static void virtio_queue_rqs(struct request **rqlist) +{ + struct request *req, *next, *prev = NULL; + struct request *requeue_list = NULL; + + rq_list_for_each_safe(rqlist, req, next) { + struct virtio_blk_vq *vq = req->mq_hctx->driver_data; + unsigned long flags; + bool kick; +Review static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 428
Quoted 10 lines
(quoted diff) +10
-0+ if (!virtblk_prep_rq_batch(vq, req)) { + rq_list_move(rqlist, &requeue_list, req, prev); + req = prev; + + if (!req) + continue; + } + + if (!next || req->mq_hctx != next->mq_hctx) { + spin_lock_irqsave(&vq->lock, flags);Did you try calling
virtblk_add_req()here to avoid acquiring and releasing the lock multiple times? In other words, dovirtblk_prep_rq()but wait until we get here to dovirtblk_add_req().
Quoted 3 lines
I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.
I tried as you said. I called virtlblk_add_req() and added requests of rqlist to virtqueue in this if statement with holding the lock only once.
I attach the code at the end of this mail.
Please refer the code.
But I didn't see improvement. It showed slightly worse performance than the current patch.
Quoted 6 lines
Review static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 434
Quoted 6 lines
(quoted diff) +6
-0+ kick = virtqueue_kick_prepare(vq->vq); + spin_unlock_irqrestore(&vq->lock, flags); + if (kick) + virtqueue_notify(vq->vq); + + req->rq_next = NULL;
Did you ask this part?
Quoted 8 lines
Review static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 438
Quoted 4 lines
(quoted diff) +4
-0+ *rqlist = next; + prev = NULL; + } else + prev = req;What guarantees that req is still alive after we called
virtblk_add_req()? The device may have seen it and completed it already by the time we get here.
Isn't request completed after the kick?
If you asked about "req->rq_next = NULL",
I think it should be placed before
"kick = virtqueue_kick_prepare(vq->vq);"
-----------
req->rq_next = NULL; kick = virtqueue_kick_prepare(vq->vq); spin_unlock_irqrestore(&vq->lock, flags); if (kick) virtqueue_notify(vq->vq);
-----------
Regards,
Suwan Kim
---
drivers/block/virtio_blk.c +99 -11
@@ -92,6 +92,7 @@ struct virtio_blk {struct virtblk_req {struct virtio_blk_outhdr out_hdr;u8 status; +int sg_num;struct sg_table sg_table;struct scatterlist sg[]; };
@@ -311,18 +312,13 @@ static void virtio_commit_rqs(struct blk_mq_hw_ctx *hctx)virtqueue_notify (vq->vq); } -static blk_status_t virtio_queue_rq (struct blk_mq_hw_ctx *hctx, -const struct blk_mq_queue_data *bd) +static blk_status_t virtblk_prep_rq (struct blk_mq_hw_ctx *hctx, +struct virtio_blk *vblk, +struct request *req, +struct virtblk_req *vbr) { -struct virtio_blk *vblk = hctx->queue->queuedata; -struct request *req = bd->rq; -struct virtblk_req *vbr =blk_mq_rq_to_pdu (req); -unsigned long flags; -int num; -int qid = hctx->queue_num; -bool notify = false;blk_status_t status; -int err; +int num; status =virtblk_setup_cmd (vblk->vdev, req, vbr);if (unlikely (status))
@@ -335,9 +331,30 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,virtblk_cleanup_cmd (req);return BLK_STS_RESOURCE ; } + vbr->sg_num = num; + +return BLK_STS_OK ; +} + +static blk_status_t virtio_queue_rq (struct blk_mq_hw_ctx *hctx, +const struct blk_mq_queue_data *bd) +{ +struct virtio_blk *vblk = hctx->queue->queuedata; +struct request *req = bd->rq; +struct virtblk_req *vbr =blk_mq_rq_to_pdu (req); +unsigned long flags; +int qid = hctx->queue_num; +bool notify = false; +blk_status_t status; +int err; + + status =virtblk_prep_rq (hctx, vblk, req, vbr); +if (unlikely (status)) +return status;spin_lock_irqsave (&vblk->vqs[qid].lock, flags); - err =virtblk_add_req (vblk->vqs[qid].vq, vbr, vbr->sg_table.sgl, num); + err =virtblk_add_req (vblk->vqs[qid].vq, vbr, + vbr->sg_table.sgl, vbr->sg_num);if (err) {virtqueue_kick (vblk->vqs[qid].vq);/* Don't stop the queue if -ENOMEM: we may have failed to
@@ -367,6 +384,76 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,return BLK_STS_OK ; } +static bool virtblk_prep_rq_batch (struct virtio_blk_vq *vq,struct request *req) +{ +struct virtio_blk *vblk = req->mq_hctx->queue->queuedata; +struct virtblk_req *vbr =blk_mq_rq_to_pdu (req); + + req->mq_hctx->tags->rqs[req->tag] = req; + +return virtblk_prep_rq (req->mq_hctx, vblk, req, vbr) ==BLK_STS_OK ; +} + +static bool virtblk_add_req_batch (struct virtio_blk_vq *vq, +struct request **rqlist, +struct request **requeue_list) +{ +unsigned long flags; +int err; +bool kick; + +spin_lock_irqsave (&vq->lock, flags); +while (!rq_list_empty (*rqlist)) { +struct request *req =rq_list_pop (rqlist); +struct virtblk_req *vbr =blk_mq_rq_to_pdu (req); + + err =virtblk_add_req (vq->vq, vbr, + vbr->sg_table.sgl, vbr->sg_num); +if (err) { +virtblk_unmap_data (req, vbr); +virtblk_cleanup_cmd (req); +rq_list_add (requeue_list, req); + } + } + + kick =virtqueue_kick_prepare (vq->vq); +spin_unlock_irqrestore (&vq->lock, flags); + +return kick; +} + +static void virtio_queue_rqs (struct request **rqlist) +{ +struct request *req, *next, *prev =NULL ; +struct request *requeue_list =NULL ; + +rq_list_for_each_safe (rqlist, req, next) { +struct virtio_blk_vq *vq = req->mq_hctx->driver_data; +bool kick; + +if (!virtblk_prep_rq_batch (vq, req)) { +rq_list_move (rqlist, &requeue_list, req, prev); + req = prev; + +if (!req) +continue ; + } + +if (!next || req->mq_hctx != next->mq_hctx) { + kick =virtblk_add_req_batch (vq, rqlist, &requeue_list); +if (kick) +virtqueue_notify (vq->vq); + + req->rq_next =NULL ; + *rqlist = next; + prev =NULL ; + }else + prev = req; + } + + *rqlist = requeue_list; +} +/* return id (s/n) string for *disk to *id_str */ static int virtblk_get_id (struct gendisk *disk,char *id_str)
@@ -818,6 +905,7 @@ static int virtblk_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,static const struct blk_mq_ops virtio_mq_ops = { .queue_rq = virtio_queue_rq, + .queue_rqs = virtio_queue_rqs, .commit_rqs = virtio_commit_rqs, .init_hctx = virtblk_init_hctx, .complete = virtblk_request_done,
Stefan Hajnoczi
reviewerRe: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()
Quoted 35 lines · Suwan Kim
Quoted 26 lines · Stefan Hajnoczi
Quoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10
-0+static void virtio_queue_rqs (struct request **rqlist) +{ +struct request *req, *next, *prev =NULL ; +struct request *requeue_list =NULL ; + +rq_list_for_each_safe (rqlist, req, next) { +struct virtio_blk_vq *vq = req->mq_hctx->driver_data; +unsigned long flags; +bool kick; +drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
Quoted 10 lines
drivers/block/virtio_blk.c +10
-0+if (!virtblk_prep_rq_batch (vq, req)) { +rq_list_move (rqlist, &requeue_list, req, prev); + req = prev; + +if (!req) +continue ; + } + +if (!next || req->mq_hctx != next->mq_hctx) { +spin_lock_irqsave (&vq->lock, flags);Did you try calling
virtblk_add_req()here to avoid acquiring and releasing the lock multiple times? In other words, dovirtblk_prep_rq()but wait until we get here to dovirtblk_add_req().Quoted 3 lines
I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.
I tried as you said. I called
virtlblk_add_req()and added requests of rqlist to virtqueue in this if statement with holding the lock only once.
Quoted 5 lines
I attach the code at the end of this mail.
Please refer the code.But I didn't see improvement. It showed slightly worse performance than the current patch.
Okay, thanks for trying it!
Quoted 24 lines
Quoted 6 lines
drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 447
Quoted 6 lines
drivers/block/virtio_blk.c +6
-0+ kick =virtqueue_kick_prepare (vq->vq); +spin_unlock_irqrestore (&vq->lock, flags); +if (kick) +virtqueue_notify (vq->vq); + + req->rq_next =NULL ;Did you ask this part?
Quoted 8 lines
drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 451
Quoted 4 lines
drivers/block/virtio_blk.c +4
-0+ *rqlist = next; + prev =NULL ; + }else + prev = req;What guarantees that req is still alive after we called
virtblk_add_req()? The device may have seen it and completed it already by the time we get here.Isn't request completed after the kick?
If you asked about "
req->rq_next= NULL",
I think it should be placed before
"kick =virtqueue_kick_prepare(vq->vq);"
Quoted 7 lines
-----------
req->rq_next = NULL; kick = virtqueue_kick_prepare(vq->vq); spin_unlock_irqrestore(&vq->lock, flags); if (kick) virtqueue_notify(vq->vq);-----------
No, virtqueue_add_sgs() exposes vring descriptors to the device. The device may process immediately. In other words, VIRTIO devices may poll the vring instead of waiting for virtqueue_notify(). There is no guarantee that the request is alive until virtqueue_notify() is called.
The code has to handle the case where the request is completed during virtqueue_add_sgs().
Stefan
Re: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()
Quoted 76 lines · Stefan Hajnoczi
Quoted 35 lines · Suwan Kim
Quoted 26 lines · Stefan Hajnoczi
Quoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10
-0+static void virtio_queue_rqs (struct request **rqlist) +{ +struct request *req, *next, *prev =NULL ; +struct request *requeue_list =NULL ; + +rq_list_for_each_safe (rqlist, req, next) { +struct virtio_blk_vq *vq = req->mq_hctx->driver_data; +unsigned long flags; +bool kick; +drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
Quoted 10 lines
drivers/block/virtio_blk.c +10
-0+if (!virtblk_prep_rq_batch (vq, req)) { +rq_list_move (rqlist, &requeue_list, req, prev); + req = prev; + +if (!req) +continue ; + } + +if (!next || req->mq_hctx != next->mq_hctx) { +spin_lock_irqsave (&vq->lock, flags);Did you try calling
virtblk_add_req()here to avoid acquiring and releasing the lock multiple times? In other words, dovirtblk_prep_rq()but wait until we get here to dovirtblk_add_req().Quoted 3 lines
I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.
I tried as you said. I called
virtlblk_add_req()and added requests of rqlist to virtqueue in this if statement with holding the lock only once.Quoted 5 lines
I attach the code at the end of this mail.
Please refer the code.But I didn't see improvement. It showed slightly worse performance than the current patch.
Okay, thanks for trying it!
Quoted 24 lines
Quoted 6 lines
drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 447
Quoted 6 lines
drivers/block/virtio_blk.c +6
-0+ kick =virtqueue_kick_prepare (vq->vq); +spin_unlock_irqrestore (&vq->lock, flags); +if (kick) +virtqueue_notify (vq->vq); + + req->rq_next =NULL ;Did you ask this part?
Quoted 8 lines
drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 451
Quoted 4 lines
drivers/block/virtio_blk.c +4
-0+ *rqlist = next; + prev =NULL ; + }else + prev = req;What guarantees that req is still alive after we called
virtblk_add_req()? The device may have seen it and completed it already by the time we get here.Isn't request completed after the kick?
If you asked about "
req->rq_next= NULL",
I think it should be placed before
"kick =virtqueue_kick_prepare(vq->vq);"Quoted 7 lines
-----------
req->rq_next = NULL; kick = virtqueue_kick_prepare(vq->vq); spin_unlock_irqrestore(&vq->lock, flags); if (kick) virtqueue_notify(vq->vq);-----------
Quoted 7 lines
No,
virtqueue_add_sgs()exposes vring descriptors to the device. The device may process immediately. In other words, VIRTIO devices may poll the vring instead of waiting forvirtqueue_notify(). There is no guarantee that the request is alive untilvirtqueue_notify()is called.The code has to handle the case where the request is completed during
virtqueue_add_sgs().
Thanks for the explanation.
We should not use req again after virtblk_add_req(). I understand...
Then, as you commented in previous mail, is it ok that we do virtblk_add_req() in "if (!next || req->mq_hctx != next->mq_hctx)" statement to avoid use req again after virtblk_add_req() as below code?
In this code, It adds reqs to virtqueue in batch just before virtqueue_notify(), and it doesn't use req again after calling virtblk_add_req().
If it is fine, I will try it again.
This code is slightly different from the code I sent in previous mail.
---
static void virtio_queue_rqs(struct request **rqlist)
...
rq_list_for_each_safe(rqlist, req, next) {
...
if (!next || req->mq_hctx != next->mq_hctx) {
// Cut the list at current req
req->rq_next = NULL;
// Add req list to virtqueue in batch with holding lock once
kick = virtblk_add_req_batch(vq, rqlist, &requeue_list);
if (kick)
virtqueue_notify(vq->vq);
// setup new req list. Don't use previous req again.
*rqlist = next;
prev = NULL;...
---
Regards,
Suwan Kim
Stefan Hajnoczi
reviewerRe: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()
Quoted 117 lines · Suwan Kim
Quoted 76 lines · Stefan Hajnoczi
Quoted 35 lines · Suwan Kim
Quoted 26 lines · Stefan Hajnoczi
Quoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10
-0+static void virtio_queue_rqs (struct request **rqlist) +{ +struct request *req, *next, *prev =NULL ; +struct request *requeue_list =NULL ; + +rq_list_for_each_safe (rqlist, req, next) { +struct virtio_blk_vq *vq = req->mq_hctx->driver_data; +unsigned long flags; +bool kick; +drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
Quoted 10 lines
drivers/block/virtio_blk.c +10
-0+if (!virtblk_prep_rq_batch (vq, req)) { +rq_list_move (rqlist, &requeue_list, req, prev); + req = prev; + +if (!req) +continue ; + } + +if (!next || req->mq_hctx != next->mq_hctx) { +spin_lock_irqsave (&vq->lock, flags);Did you try calling
virtblk_add_req()here to avoid acquiring and releasing the lock multiple times? In other words, dovirtblk_prep_rq()but wait until we get here to dovirtblk_add_req().Quoted 3 lines
I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.
I tried as you said. I called
virtlblk_add_req()and added requests of rqlist to virtqueue in this if statement with holding the lock only once.Quoted 5 lines
I attach the code at the end of this mail.
Please refer the code.But I didn't see improvement. It showed slightly worse performance than the current patch.
Okay, thanks for trying it!
Quoted 24 lines
Quoted 6 lines
drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 447
Quoted 6 lines
drivers/block/virtio_blk.c +6
-0+ kick =virtqueue_kick_prepare (vq->vq); +spin_unlock_irqrestore (&vq->lock, flags); +if (kick) +virtqueue_notify (vq->vq); + + req->rq_next =NULL ;Did you ask this part?
Quoted 8 lines
drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 451
Quoted 4 lines
drivers/block/virtio_blk.c +4
-0+ *rqlist = next; + prev =NULL ; + }else + prev = req;What guarantees that req is still alive after we called
virtblk_add_req()? The device may have seen it and completed it already by the time we get here.Isn't request completed after the kick?
If you asked about "
req->rq_next= NULL",
I think it should be placed before
"kick =virtqueue_kick_prepare(vq->vq);"Quoted 7 lines
-----------
req->rq_next = NULL; kick = virtqueue_kick_prepare(vq->vq); spin_unlock_irqrestore(&vq->lock, flags); if (kick) virtqueue_notify(vq->vq);-----------
Quoted 7 lines
No,
virtqueue_add_sgs()exposes vring descriptors to the device. The device may process immediately. In other words, VIRTIO devices may poll the vring instead of waiting forvirtqueue_notify(). There is no guarantee that the request is alive untilvirtqueue_notify()is called.The code has to handle the case where the request is completed during
virtqueue_add_sgs().Thanks for the explanation.
We should not use req again after
virtblk_add_req(). I understand...Then, as you commented in previous mail, is it ok that we do
virtblk_add_req()in "if (!next ||req->mq_hctx!=next->mq_hctx)" statement to avoid use req again aftervirtblk_add_req()as below code?In this code, It adds reqs to virtqueue in batch just before
virtqueue_notify(), and it doesn't use req again after callingvirtblk_add_req().If it is fine, I will try it again.
This code is slightly different from the code I sent in previous mail.---
static voidvirtio_queue_rqs(struct request **rqlist)
...rq_list_for_each_safe(rqlist, req, next) { ... if (!next || req->mq_hctx != next->mq_hctx) { // Cut the list at current req req->rq_next = NULL; // Add req list to virtqueue in batch with holding lock once kick = virtblk_add_req_batch(vq, rqlist, &requeue_list); if (kick) virtqueue_notify(vq->vq); // setup new req list. Don't use previous req again. *rqlist = next; prev = NULL;
Quoted 1 line
...
Yes, that sounds good.
(I noticed struct request has a reference count so that might be a way to keep requests alive, if necessary, but I haven't investigated. See req_ref_put_and_test() though it's not used by block drivers and maybe virtio-blk shouldn't mess with it either.)
Stefan
Re: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()
Quoted 120 lines · Stefan Hajnoczi
Quoted 117 lines · Suwan Kim
Quoted 76 lines · Stefan Hajnoczi
Quoted 35 lines · Suwan Kim
Quoted 26 lines · Stefan Hajnoczi
Quoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10
-0+static void virtio_queue_rqs (struct request **rqlist) +{ +struct request *req, *next, *prev =NULL ; +struct request *requeue_list =NULL ; + +rq_list_for_each_safe (rqlist, req, next) { +struct virtio_blk_vq *vq = req->mq_hctx->driver_data; +unsigned long flags; +bool kick; +drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
Quoted 10 lines
drivers/block/virtio_blk.c +10
-0+if (!virtblk_prep_rq_batch (vq, req)) { +rq_list_move (rqlist, &requeue_list, req, prev); + req = prev; + +if (!req) +continue ; + } + +if (!next || req->mq_hctx != next->mq_hctx) { +spin_lock_irqsave (&vq->lock, flags);Did you try calling
virtblk_add_req()here to avoid acquiring and releasing the lock multiple times? In other words, dovirtblk_prep_rq()but wait until we get here to dovirtblk_add_req().Quoted 3 lines
I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.
I tried as you said. I called
virtlblk_add_req()and added requests of rqlist to virtqueue in this if statement with holding the lock only once.Quoted 5 lines
I attach the code at the end of this mail.
Please refer the code.But I didn't see improvement. It showed slightly worse performance than the current patch.
Okay, thanks for trying it!
Quoted 24 lines
Quoted 6 lines
drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 447
Quoted 6 lines
drivers/block/virtio_blk.c +6
-0+ kick =virtqueue_kick_prepare (vq->vq); +spin_unlock_irqrestore (&vq->lock, flags); +if (kick) +virtqueue_notify (vq->vq); + + req->rq_next =NULL ;Did you ask this part?
Quoted 8 lines
drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 451
Quoted 4 lines
drivers/block/virtio_blk.c +4
-0+ *rqlist = next; + prev =NULL ; + }else + prev = req;What guarantees that req is still alive after we called
virtblk_add_req()? The device may have seen it and completed it already by the time we get here.Isn't request completed after the kick?
If you asked about "
req->rq_next= NULL",
I think it should be placed before
"kick =virtqueue_kick_prepare(vq->vq);"Quoted 7 lines
-----------
req->rq_next = NULL; kick = virtqueue_kick_prepare(vq->vq); spin_unlock_irqrestore(&vq->lock, flags); if (kick) virtqueue_notify(vq->vq);-----------
Quoted 7 lines
No,
virtqueue_add_sgs()exposes vring descriptors to the device. The device may process immediately. In other words, VIRTIO devices may poll the vring instead of waiting forvirtqueue_notify(). There is no guarantee that the request is alive untilvirtqueue_notify()is called.The code has to handle the case where the request is completed during
virtqueue_add_sgs().Thanks for the explanation.
We should not use req again after
virtblk_add_req(). I understand...Then, as you commented in previous mail, is it ok that we do
virtblk_add_req()in "if (!next ||req->mq_hctx!=next->mq_hctx)" statement to avoid use req again aftervirtblk_add_req()as below code?In this code, It adds reqs to virtqueue in batch just before
virtqueue_notify(), and it doesn't use req again after callingvirtblk_add_req().If it is fine, I will try it again.
This code is slightly different from the code I sent in previous mail.---
static voidvirtio_queue_rqs(struct request **rqlist)
...rq_list_for_each_safe(rqlist, req, next) { ... if (!next || req->mq_hctx != next->mq_hctx) { // Cut the list at current req req->rq_next = NULL; // Add req list to virtqueue in batch with holding lock once kick = virtblk_add_req_batch(vq, rqlist, &requeue_list); if (kick) virtqueue_notify(vq->vq); // setup new req list. Don't use previous req again. *rqlist = next; prev = NULL;Quoted 1 line
...
Quoted 6 lines
Yes, that sounds good.
(I noticed struct request has a reference count so that might be a way to keep requests alive, if necessary, but I haven't investigated. See
req_ref_put_and_test()though it's not used by block drivers and maybe virtio-blk shouldn't mess with it either.)
I also think that using ref count is not a good idea. I will send the next version soon.
Regards,
Suwan Kim