Skip to contentSkip to conversation

virtio-blk: support polling I/O and mq_ops->queue_rqs()

PATCH Superseded Suwan Kim 4 participants 19 messages Mar 24, 2022 – Mar 29, 2022

Subsystems (3)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Suwan Kim

author
#0 lore raw

v3 cover virtio-blk: support polling I/O and mq_ops->queue_rqs()

This patch serise adds support for polling I/O and mq_ops->queue_rqs() to virtio-blk driver.

Changes

v2 -> v3
- Fix warning by kernel test robot

static int virtblk_poll()
...
if (!blk_mq_add_to_batch(req, iob, virtblk_result(vbr),
-> vbr->status,

v1 -> v2
- To receive the number of poll queues from user, use module parameter instead of QEMU uapi change.

- Add the comment about virtblk_map_queues().

- Add support for mq_ops->queue_rqs() to implement submit side batch.

Suwan Kim (2):
virtio-blk: support polling I/O
virtio-blk: support mq_ops->queue_rqs()

 drivers/block/virtio_blk.c | 194 ++++++++++++++++++++++++++++++++++---
 1 file changed, 181 insertions(+), 13 deletions(-)
Signature · 1 line
2.26.3
author
#1 lore raw

PATCH v3 1/2 virtio-blk: support polling I/O

This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.

The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.

virtio-blk polling is implemented upon 'batched completion' of block layer. virtblk_poll() queues completed request to io_comp_batch->req_list and later, virtblk_complete_batch() calls unmap function and ends the requests in batch.

virtio-blk reads the number of poll queues from module parameter "num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.

Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.

For verifying the improvement, I did Fio polling I/O performance test with io_uring engine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.

As a result, IOPS and average latency improved about 10%.

Test result:

- Fio io_uring poll without virtio-blk poll support

	-- numjobs=1 : IOPS = 339K, avg latency = 188.33us
	-- numjobs=2 : IOPS = 367K, avg latency = 347.33us
	-- numjobs=4 : IOPS = 383K, avg latency = 682.06us

- Fio io_uring poll with virtio-blk poll support

	-- numjobs=1 : IOPS = 380K, avg latency = 167.87us
	-- numjobs=2 : IOPS = 409K, avg latency = 312.6us
	-- numjobs=4 : IOPS = 413K, avg latency = 619.72us
---
 drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++--
 1 file changed, 97 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +97 -4
index 8c415be86732..3d16f8b753e7 100644
@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,
 		 "0 for no limit. "
 		 "Values > nr_cpu_ids truncated to nr_cpu_ids.");
 
+static unsigned int num_poll_queues;
+module_param(num_poll_queues, uint, 0644);
+MODULE_PARM_DESC(num_poll_queues, "The number of dedicated virtqueues for polling I/O");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 
@@ -81,6 +85,7 @@ struct virtio_blk {
 
 	/* num of vqs */
 	int num_vqs;
+	int io_queues[HCTX_MAX_TYPES];
 	struct virtio_blk_vq *vqs;
 };
 
@@ -548,6 +553,7 @@ static int init_vq(struct virtio_blk *vblk)
 	const char **names;
 	struct virtqueue **vqs;
 	unsigned short num_vqs;
+	unsigned int num_poll_vqs;
 	struct virtio_device *vdev = vblk->vdev;
 	struct irq_affinity desc = { 0, };
 
@@ -556,6 +562,7 @@ static int init_vq(struct virtio_blk *vblk)
 				   &num_vqs);
 	if (err)
 		num_vqs = 1;
+
 	if (!err && !num_vqs) {
 		dev_err(&vdev->dev, "MQ advertised but zero queues reported\n");
 		return -EINVAL;
@@ -565,6 +572,13 @@ static int init_vq(struct virtio_blk *vblk)
 			min_not_zero(num_request_queues, nr_cpu_ids),
 			num_vqs);
 
+	num_poll_vqs = min_t(unsigned int, num_poll_queues, num_vqs - 1);
+
+	memset(vblk->io_queues, 0, sizeof(int) * HCTX_MAX_TYPES);
+	vblk->io_queues[HCTX_TYPE_DEFAULT] = num_vqs - num_poll_vqs;
+	vblk->io_queues[HCTX_TYPE_READ] = 0;
+	vblk->io_queues[HCTX_TYPE_POLL] = num_poll_vqs;
+
 	vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
 	if (!vblk->vqs)
 		return -ENOMEM;
@@ -578,8 +592,13 @@ static int init_vq(struct virtio_blk *vblk)
 	}
 
 	for (i = 0; i < num_vqs; i++) {
-		callbacks[i] = virtblk_done;
-		snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req.%d", i);
+		if (i < num_vqs - num_poll_vqs) {
+			callbacks[i] = virtblk_done;
+			snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req.%d", i);
+		} else {
+			callbacks[i] = NULL;
+			snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req_poll.%d", i);
+		}
 		names[i] = vblk->vqs[i].name;
 	}
 
@@ -728,16 +747,87 @@ static const struct attribute_group *virtblk_attr_groups[] = {
 static int virtblk_map_queues(struct blk_mq_tag_set *set)
 {
 	struct virtio_blk *vblk = set->driver_data;
+	int i, qoff;
+
+	for (i = 0, qoff = 0; i < set->nr_maps; i++) {
+		struct blk_mq_queue_map *map = &set->map[i];
+
+		map->nr_queues = vblk->io_queues[i];
+		map->queue_offset = qoff;
+		qoff += map->nr_queues;
+
+		if (map->nr_queues == 0)
+			continue;
+
+		/*
+		 * Regular queues have interrupts and hence CPU affinity is
+		 * defined by the core virtio code, but polling queues have
+		 * no interrupts so we let the block layer assign CPU affinity.
+		 */
+		if (i == HCTX_TYPE_DEFAULT)
+			blk_mq_virtio_map_queues(&set->map[i], vblk->vdev, 0);
+		else
+			blk_mq_map_queues(&set->map[i]);
+	}
+
+	return 0;
+}
+
+static void virtblk_complete_batch(struct io_comp_batch *iob)
+{
+	struct request *req;
+	struct virtblk_req *vbr;
 
-	return blk_mq_virtio_map_queues(&set->map[HCTX_TYPE_DEFAULT],
-					vblk->vdev, 0);
+	rq_list_for_each(&iob->req_list, req) {
+		vbr = blk_mq_rq_to_pdu(req);
+		virtblk_unmap_data(req, vbr);
+		virtblk_cleanup_cmd(req);
+	}
+	blk_mq_end_request_batch(iob);
+}
+
+static int virtblk_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
+{
+	struct virtio_blk_vq *vq = hctx->driver_data;
+	struct virtblk_req *vbr;
+	unsigned long flags;
+	unsigned int len;
+	int found = 0;
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	while ((vbr = virtqueue_get_buf(vq->vq, &len)) != NULL) {
+		struct request *req = blk_mq_rq_from_pdu(vbr);
+
+		found++;
+		if (!blk_mq_add_to_batch(req, iob, vbr->status,
+						virtblk_complete_batch))
+			blk_mq_complete_request(req);
+	}
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+
+	return found;
+}
+
+static int virtblk_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			  unsigned int hctx_idx)
+{
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq = &vblk->vqs[hctx_idx];
+
+	WARN_ON(vblk->tag_set.tags[hctx_idx] != hctx->tags);
+	hctx->driver_data = vq;
+	return 0;
 }
 
 static const struct blk_mq_ops virtio_mq_ops = {
 	.queue_rq	= virtio_queue_rq,
 	.commit_rqs	= virtio_commit_rqs,
+	.init_hctx	= virtblk_init_hctx,
 	.complete	= virtblk_request_done,
 	.map_queues	= virtblk_map_queues,
+	.poll		= virtblk_poll,
 };
 
 static unsigned int virtblk_queue_depth;
@@ -816,6 +906,9 @@ static int virtblk_probe(struct virtio_device *vdev)
 		sizeof(struct scatterlist) * VIRTIO_BLK_INLINE_SG_CNT;
 	vblk->tag_set.driver_data = vblk;
 	vblk->tag_set.nr_hw_queues = vblk->num_vqs;
+	vblk->tag_set.nr_maps = 1;
+	if (vblk->io_queues[HCTX_TYPE_POLL])
+		vblk->tag_set.nr_maps = 3;
 
 	err = blk_mq_alloc_tag_set(&vblk->tag_set);
 	if (err)
Signature · 1 line
2.26.3
#2 lore raw

Re: [PATCH v3 1/2] virtio-blk: support polling I/O

Quoted 56 lines · Suwan Kim

This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.

The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.

virtio-blk polling is implemented upon 'batched completion' of block layer. virtblk_poll() queues completed request to io_comp_batch->req_list and later, virtblk_complete_batch() calls unmap function and ends the requests in batch.

virtio-blk reads the number of poll queues from module parameter "num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.

Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.

For verifying the improvement, I did Fio polling I/O performance test with io_uring engine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.

As a result, IOPS and average latency improved about 10%.

Test result:

- Fio io_uring poll without virtio-blk poll support

	-- numjobs=1 : IOPS = 339K, avg latency = 188.33us
	-- numjobs=2 : IOPS = 367K, avg latency = 347.33us
	-- numjobs=4 : IOPS = 383K, avg latency = 682.06us

- Fio io_uring poll with virtio-blk poll support

	-- numjobs=1 : IOPS = 380K, avg latency = 167.87us
	-- numjobs=2 : IOPS = 409K, avg latency = 312.6us
	-- numjobs=4 : IOPS = 413K, avg latency = 619.72us
---
 drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++--
 1 file changed, 97 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +0 -0
index 8c415be86732..3d16f8b753e7 100644

drivers/block/virtio_blk.c MODULE_PARM_DESC(num_request_queues, · line 46

Quoted 10 lines
drivers/block/virtio_blk.c +4 -0
@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,
 		 "0 for no limit. "
 		 "Values > nr_cpu_ids truncated to nr_cpu_ids.");
 
+static unsigned int num_poll_queues;
+module_param(num_poll_queues, uint, 0644);
+MODULE_PARM_DESC(num_poll_queues, "The number of dedicated virtqueues for polling I/O");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 

Is there some way to make it work reasonably without need to set module parameters? I don't see any other devices with a num_poll_queues parameter - how do they handle this?

Quoted 156 lines
drivers/block/virtio_blk.c +93 -4
@@ -81,6 +85,7 @@ struct virtio_blk {
 
 	/* num of vqs */
 	int num_vqs;
+	int io_queues[HCTX_MAX_TYPES];
 	struct virtio_blk_vq *vqs;
 };
 
@@ -548,6 +553,7 @@ static int init_vq(struct virtio_blk *vblk)
 	const char **names;
 	struct virtqueue **vqs;
 	unsigned short num_vqs;
+	unsigned int num_poll_vqs;
 	struct virtio_device *vdev = vblk->vdev;
 	struct irq_affinity desc = { 0, };
 
@@ -556,6 +562,7 @@ static int init_vq(struct virtio_blk *vblk)
 				   &num_vqs);
 	if (err)
 		num_vqs = 1;
+
 	if (!err && !num_vqs) {
 		dev_err(&vdev->dev, "MQ advertised but zero queues reported\n");
 		return -EINVAL;
@@ -565,6 +572,13 @@ static int init_vq(struct virtio_blk *vblk)
 			min_not_zero(num_request_queues, nr_cpu_ids),
 			num_vqs);
 
+	num_poll_vqs = min_t(unsigned int, num_poll_queues, num_vqs - 1);
+
+	memset(vblk->io_queues, 0, sizeof(int) * HCTX_MAX_TYPES);
+	vblk->io_queues[HCTX_TYPE_DEFAULT] = num_vqs - num_poll_vqs;
+	vblk->io_queues[HCTX_TYPE_READ] = 0;
+	vblk->io_queues[HCTX_TYPE_POLL] = num_poll_vqs;
+
 	vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
 	if (!vblk->vqs)
 		return -ENOMEM;
@@ -578,8 +592,13 @@ static int init_vq(struct virtio_blk *vblk)
 	}
 
 	for (i = 0; i < num_vqs; i++) {
-		callbacks[i] = virtblk_done;
-		snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req.%d", i);
+		if (i < num_vqs - num_poll_vqs) {
+			callbacks[i] = virtblk_done;
+			snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req.%d", i);
+		} else {
+			callbacks[i] = NULL;
+			snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req_poll.%d", i);
+		}
 		names[i] = vblk->vqs[i].name;
 	}
 
@@ -728,16 +747,87 @@ static const struct attribute_group *virtblk_attr_groups[] = {
 static int virtblk_map_queues(struct blk_mq_tag_set *set)
 {
 	struct virtio_blk *vblk = set->driver_data;
+	int i, qoff;
+
+	for (i = 0, qoff = 0; i < set->nr_maps; i++) {
+		struct blk_mq_queue_map *map = &set->map[i];
+
+		map->nr_queues = vblk->io_queues[i];
+		map->queue_offset = qoff;
+		qoff += map->nr_queues;
+
+		if (map->nr_queues == 0)
+			continue;
+
+		/*
+		 * Regular queues have interrupts and hence CPU affinity is
+		 * defined by the core virtio code, but polling queues have
+		 * no interrupts so we let the block layer assign CPU affinity.
+		 */
+		if (i == HCTX_TYPE_DEFAULT)
+			blk_mq_virtio_map_queues(&set->map[i], vblk->vdev, 0);
+		else
+			blk_mq_map_queues(&set->map[i]);
+	}
+
+	return 0;
+}
+
+static void virtblk_complete_batch(struct io_comp_batch *iob)
+{
+	struct request *req;
+	struct virtblk_req *vbr;
 
-	return blk_mq_virtio_map_queues(&set->map[HCTX_TYPE_DEFAULT],
-					vblk->vdev, 0);
+	rq_list_for_each(&iob->req_list, req) {
+		vbr = blk_mq_rq_to_pdu(req);
+		virtblk_unmap_data(req, vbr);
+		virtblk_cleanup_cmd(req);
+	}
+	blk_mq_end_request_batch(iob);
+}
+
+static int virtblk_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
+{
+	struct virtio_blk_vq *vq = hctx->driver_data;
+	struct virtblk_req *vbr;
+	unsigned long flags;
+	unsigned int len;
+	int found = 0;
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	while ((vbr = virtqueue_get_buf(vq->vq, &len)) != NULL) {
+		struct request *req = blk_mq_rq_from_pdu(vbr);
+
+		found++;
+		if (!blk_mq_add_to_batch(req, iob, vbr->status,
+						virtblk_complete_batch))
+			blk_mq_complete_request(req);
+	}
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+
+	return found;
+}
+
+static int virtblk_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			  unsigned int hctx_idx)
+{
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq = &vblk->vqs[hctx_idx];
+
+	WARN_ON(vblk->tag_set.tags[hctx_idx] != hctx->tags);
+	hctx->driver_data = vq;
+	return 0;
 }
 
 static const struct blk_mq_ops virtio_mq_ops = {
 	.queue_rq	= virtio_queue_rq,
 	.commit_rqs	= virtio_commit_rqs,
+	.init_hctx	= virtblk_init_hctx,
 	.complete	= virtblk_request_done,
 	.map_queues	= virtblk_map_queues,
+	.poll		= virtblk_poll,
 };
 
 static unsigned int virtblk_queue_depth;
@@ -816,6 +906,9 @@ static int virtblk_probe(struct virtio_device *vdev)
 		sizeof(struct scatterlist) * VIRTIO_BLK_INLINE_SG_CNT;
 	vblk->tag_set.driver_data = vblk;
 	vblk->tag_set.nr_hw_queues = vblk->num_vqs;
+	vblk->tag_set.nr_maps = 1;
+	if (vblk->io_queues[HCTX_TYPE_POLL])
+		vblk->tag_set.nr_maps = 3;
 
 	err = blk_mq_alloc_tag_set(&vblk->tag_set);
 	if (err)
Signature · 1 line
2.26.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#3 lore raw

Re: [PATCH v3 1/2] virtio-blk: support polling I/O

Quoted 60 lines · Michael S. Tsirkin
Quoted 59 lines · Suwan Kim

This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.

The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.

virtio-blk polling is implemented upon 'batched completion' of block layer. virtblk_poll() queues completed request to io_comp_batch->req_list and later, virtblk_complete_batch() calls unmap function and ends the requests in batch.

virtio-blk reads the number of poll queues from module parameter "num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.

Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.

For verifying the improvement, I did Fio polling I/O performance test with io_uring engine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.

As a result, IOPS and average latency improved about 10%.

Test result:

- Fio io_uring poll without virtio-blk poll support

	-- numjobs=1 : IOPS = 339K, avg latency = 188.33us
	-- numjobs=2 : IOPS = 367K, avg latency = 347.33us
	-- numjobs=4 : IOPS = 383K, avg latency = 682.06us

- Fio io_uring poll with virtio-blk poll support

	-- numjobs=1 : IOPS = 380K, avg latency = 167.87us
	-- numjobs=2 : IOPS = 409K, avg latency = 312.6us
	-- numjobs=4 : IOPS = 413K, avg latency = 619.72us
---
 drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++--
 1 file changed, 97 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +0 -0
index 8c415be86732..3d16f8b753e7 100644
@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,
 		 "0 for no limit. "
 		 "Values > nr_cpu_ids truncated to nr_cpu_ids.");
 
Quoted 11 lines

drivers/block/virtio_blk.c MODULE_PARM_DESC(num_request_queues, · line 46

Quoted 7 lines
drivers/block/virtio_blk.c +4 -0
+static unsigned int num_poll_queues;
+module_param(num_poll_queues, uint, 0644);
+MODULE_PARM_DESC(num_poll_queues, "The number of dedicated virtqueues for polling I/O");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 

Is there some way to make it work reasonably without need to set module parameters? I don't see any other devices with a num_poll_queues parameter - how do they handle this?

Hi Michael,

NVMe driver uses module parameter.

Please refer to this.
-----
drivers/nvme/host/pci.c

static unsigned int poll_queues; module_param_cb(poll_queues, &io_queue_count_ops, &poll_queues, 0644); MODULE_PARM_DESC(poll_queues, "Number of queues to use for polled IO.");
-----

Regards,
Suwan Kim

#4 lore raw

Re: [PATCH v3 1/2] virtio-blk: support polling I/O

Quoted 81 lines · Suwan Kim
Quoted 60 lines · Michael S. Tsirkin
Quoted 59 lines · Suwan Kim

This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.

The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.

virtio-blk polling is implemented upon 'batched completion' of block layer. virtblk_poll() queues completed request to io_comp_batch->req_list and later, virtblk_complete_batch() calls unmap function and ends the requests in batch.

virtio-blk reads the number of poll queues from module parameter "num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.

Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.

For verifying the improvement, I did Fio polling I/O performance test with io_uring engine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.

As a result, IOPS and average latency improved about 10%.

Test result:

- Fio io_uring poll without virtio-blk poll support

	-- numjobs=1 : IOPS = 339K, avg latency = 188.33us
	-- numjobs=2 : IOPS = 367K, avg latency = 347.33us
	-- numjobs=4 : IOPS = 383K, avg latency = 682.06us

- Fio io_uring poll with virtio-blk poll support

	-- numjobs=1 : IOPS = 380K, avg latency = 167.87us
	-- numjobs=2 : IOPS = 409K, avg latency = 312.6us
	-- numjobs=4 : IOPS = 413K, avg latency = 619.72us
---
 drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++--
 1 file changed, 97 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +0 -0
index 8c415be86732..3d16f8b753e7 100644
@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,
 		 "0 for no limit. "
 		 "Values > nr_cpu_ids truncated to nr_cpu_ids.");
 
Quoted 11 lines

drivers/block/virtio_blk.c MODULE_PARM_DESC(num_request_queues, · line 46

Quoted 7 lines
drivers/block/virtio_blk.c +4 -0
+static unsigned int num_poll_queues;
+module_param(num_poll_queues, uint, 0644);
+MODULE_PARM_DESC(num_poll_queues, "The number of dedicated virtqueues for polling I/O");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 

Is there some way to make it work reasonably without need to set module parameters? I don't see any other devices with a num_poll_queues parameter - how do they handle this?

Hi Michael,

NVMe driver uses module parameter.

Please refer to this.
-----
drivers/nvme/host/pci.c

Quoted 7 lines

static unsigned int poll_queues; module_param_cb(poll_queues, &io_queue_count_ops, &poll_queues, 0644); MODULE_PARM_DESC(poll_queues, "Number of queues to use for polled IO.");
-----

Regards,
Suwan Kim

OK then. Let's maybe be consistent wrt parameter naming?

Signature · 1 line
MST
author
#5 lore raw

Re: [PATCH v3 1/2] virtio-blk: support polling I/O

Quoted 87 lines · Michael S. Tsirkin
Quoted 86 lines · Suwan Kim
Quoted 60 lines · Michael S. Tsirkin
Quoted 59 lines · Suwan Kim

This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.

The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.

virtio-blk polling is implemented upon 'batched completion' of block layer. virtblk_poll() queues completed request to io_comp_batch->req_list and later, virtblk_complete_batch() calls unmap function and ends the requests in batch.

virtio-blk reads the number of poll queues from module parameter "num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.

Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.

For verifying the improvement, I did Fio polling I/O performance test with io_uring engine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.

As a result, IOPS and average latency improved about 10%.

Test result:

- Fio io_uring poll without virtio-blk poll support

	-- numjobs=1 : IOPS = 339K, avg latency = 188.33us
	-- numjobs=2 : IOPS = 367K, avg latency = 347.33us
	-- numjobs=4 : IOPS = 383K, avg latency = 682.06us

- Fio io_uring poll with virtio-blk poll support

	-- numjobs=1 : IOPS = 380K, avg latency = 167.87us
	-- numjobs=2 : IOPS = 409K, avg latency = 312.6us
	-- numjobs=4 : IOPS = 413K, avg latency = 619.72us
---
 drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++--
 1 file changed, 97 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +0 -0
index 8c415be86732..3d16f8b753e7 100644
@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,
 		 "0 for no limit. "
 		 "Values > nr_cpu_ids truncated to nr_cpu_ids.");
 
Quoted 11 lines

drivers/block/virtio_blk.c MODULE_PARM_DESC(num_request_queues, · line 46

Quoted 7 lines
drivers/block/virtio_blk.c +4 -0
+static unsigned int num_poll_queues;
+module_param(num_poll_queues, uint, 0644);
+MODULE_PARM_DESC(num_poll_queues, "The number of dedicated virtqueues for polling I/O");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 

Is there some way to make it work reasonably without need to set module parameters? I don't see any other devices with a num_poll_queues parameter - how do they handle this?

Hi Michael,

NVMe driver uses module parameter.

Please refer to this.
-----
drivers/nvme/host/pci.c

static unsigned int poll_queues; module_param_cb(poll_queues, &io_queue_count_ops, &poll_queues, 0644); MODULE_PARM_DESC(poll_queues, "Number of queues to use for polled IO.");
-----

Quoted 4 lines
Quoted 2 lines

Regards,
Suwan Kim

OK then. Let's maybe be consistent wrt parameter naming?

Ok. Consistent naming scheme seems to be better for code readability. I will rename it to 'poll_queues' in next version.

Regards,
Suwan Kim

#6 lore raw

Re: [PATCH v3 1/2] virtio-blk: support polling I/O

In reply to Suwan Kim

Hi Suwan,

The NVMe prints something like below by nvme_setup_io_queues() to confirm if the configuration takes effect.

"[ 0.620458] nvme nvme0: 4/0/0 default/read/poll queues".

How about to print in virtio-blk as well?

Thank you very much!

Dongli Zhang

Quoted 211 lines · Suwan Kim

This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.

The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.

virtio-blk polling is implemented upon 'batched completion' of block layer. virtblk_poll() queues completed request to io_comp_batch->req_list and later, virtblk_complete_batch() calls unmap function and ends the requests in batch.

virtio-blk reads the number of poll queues from module parameter "num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.

Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.

For verifying the improvement, I did Fio polling I/O performance test with io_uring engine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.

As a result, IOPS and average latency improved about 10%.

Test result:

- Fio io_uring poll without virtio-blk poll support

	-- numjobs=1 : IOPS = 339K, avg latency = 188.33us
	-- numjobs=2 : IOPS = 367K, avg latency = 347.33us
	-- numjobs=4 : IOPS = 383K, avg latency = 682.06us

- Fio io_uring poll with virtio-blk poll support

	-- numjobs=1 : IOPS = 380K, avg latency = 167.87us
	-- numjobs=2 : IOPS = 409K, avg latency = 312.6us
	-- numjobs=4 : IOPS = 413K, avg latency = 619.72us
---
 drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++--
 1 file changed, 97 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +94 -4
index 8c415be86732..3d16f8b753e7 100644
@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,
 		 "0 for no limit. "
 		 "Values > nr_cpu_ids truncated to nr_cpu_ids.");
 
+static unsigned int num_poll_queues;
+module_param(num_poll_queues, uint, 0644);
+MODULE_PARM_DESC(num_poll_queues, "The number of dedicated virtqueues for polling I/O");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 
@@ -81,6 +85,7 @@ struct virtio_blk {
 
 	/* num of vqs */
 	int num_vqs;
+	int io_queues[HCTX_MAX_TYPES];
 	struct virtio_blk_vq *vqs;
 };
 
@@ -548,6 +553,7 @@ static int init_vq(struct virtio_blk *vblk)
 	const char **names;
 	struct virtqueue **vqs;
 	unsigned short num_vqs;
+	unsigned int num_poll_vqs;
 	struct virtio_device *vdev = vblk->vdev;
 	struct irq_affinity desc = { 0, };
 
@@ -556,6 +562,7 @@ static int init_vq(struct virtio_blk *vblk)
 				   &num_vqs);
 	if (err)
 		num_vqs = 1;
+
 	if (!err && !num_vqs) {
 		dev_err(&vdev->dev, "MQ advertised but zero queues reported\n");
 		return -EINVAL;
@@ -565,6 +572,13 @@ static int init_vq(struct virtio_blk *vblk)
 			min_not_zero(num_request_queues, nr_cpu_ids),
 			num_vqs);
 
+	num_poll_vqs = min_t(unsigned int, num_poll_queues, num_vqs - 1);
+
+	memset(vblk->io_queues, 0, sizeof(int) * HCTX_MAX_TYPES);
+	vblk->io_queues[HCTX_TYPE_DEFAULT] = num_vqs - num_poll_vqs;
+	vblk->io_queues[HCTX_TYPE_READ] = 0;
+	vblk->io_queues[HCTX_TYPE_POLL] = num_poll_vqs;
+
 	vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
 	if (!vblk->vqs)
 		return -ENOMEM;
@@ -578,8 +592,13 @@ static int init_vq(struct virtio_blk *vblk)
 	}
 
 	for (i = 0; i < num_vqs; i++) {
-		callbacks[i] = virtblk_done;
-		snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req.%d", i);
+		if (i < num_vqs - num_poll_vqs) {
+			callbacks[i] = virtblk_done;
+			snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req.%d", i);
+		} else {
+			callbacks[i] = NULL;
+			snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req_poll.%d", i);
+		}
 		names[i] = vblk->vqs[i].name;
 	}
 
@@ -728,16 +747,87 @@ static const struct attribute_group *virtblk_attr_groups[] = {
 static int virtblk_map_queues(struct blk_mq_tag_set *set)
 {
 	struct virtio_blk *vblk = set->driver_data;
+	int i, qoff;
+
+	for (i = 0, qoff = 0; i < set->nr_maps; i++) {
+		struct blk_mq_queue_map *map = &set->map[i];
+
+		map->nr_queues = vblk->io_queues[i];
+		map->queue_offset = qoff;
+		qoff += map->nr_queues;
+
+		if (map->nr_queues == 0)
+			continue;
+
+		/*
+		 * Regular queues have interrupts and hence CPU affinity is
+		 * defined by the core virtio code, but polling queues have
+		 * no interrupts so we let the block layer assign CPU affinity.
+		 */
+		if (i == HCTX_TYPE_DEFAULT)
+			blk_mq_virtio_map_queues(&set->map[i], vblk->vdev, 0);
+		else
+			blk_mq_map_queues(&set->map[i]);
+	}
+
+	return 0;
+}
+
+static void virtblk_complete_batch(struct io_comp_batch *iob)
+{
+	struct request *req;
+	struct virtblk_req *vbr;
 
-	return blk_mq_virtio_map_queues(&set->map[HCTX_TYPE_DEFAULT],
-					vblk->vdev, 0);
+	rq_list_for_each(&iob->req_list, req) {
+		vbr = blk_mq_rq_to_pdu(req);
+		virtblk_unmap_data(req, vbr);
+		virtblk_cleanup_cmd(req);
+	}
+	blk_mq_end_request_batch(iob);
+}
+
+static int virtblk_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
+{
+	struct virtio_blk_vq *vq = hctx->driver_data;
+	struct virtblk_req *vbr;
+	unsigned long flags;
+	unsigned int len;
+	int found = 0;
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	while ((vbr = virtqueue_get_buf(vq->vq, &len)) != NULL) {
+		struct request *req = blk_mq_rq_from_pdu(vbr);
+
+		found++;
+		if (!blk_mq_add_to_batch(req, iob, vbr->status,
+						virtblk_complete_batch))
+			blk_mq_complete_request(req);
+	}
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+
+	return found;
+}
+
+static int virtblk_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			  unsigned int hctx_idx)
+{
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq = &vblk->vqs[hctx_idx];
+
+	WARN_ON(vblk->tag_set.tags[hctx_idx] != hctx->tags);
+	hctx->driver_data = vq;
+	return 0;
 }
 
 static const struct blk_mq_ops virtio_mq_ops = {
 	.queue_rq	= virtio_queue_rq,
 	.commit_rqs	= virtio_commit_rqs,
+	.init_hctx	= virtblk_init_hctx,
 	.complete	= virtblk_request_done,
 	.map_queues	= virtblk_map_queues,
+	.poll		= virtblk_poll,
 };
 
 static unsigned int virtblk_queue_depth;

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 914

Quoted 9 lines
drivers/block/virtio_blk.c +3 -0
@@ -816,6 +906,9 @@ static int virtblk_probe(struct virtio_device *vdev)
 		sizeof(struct scatterlist) * VIRTIO_BLK_INLINE_SG_CNT;
 	vblk->tag_set.driver_data = vblk;
 	vblk->tag_set.nr_hw_queues = vblk->num_vqs;
+	vblk->tag_set.nr_maps = 1;
+	if (vblk->io_queues[HCTX_TYPE_POLL])
+		vblk->tag_set.nr_maps = 3;
 
 	err = blk_mq_alloc_tag_set(&vblk->tag_set);
 	if (err)

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#7 lore raw

Re: [PATCH v3 1/2] virtio-blk: support polling I/O

Quoted 8 lines · Dongli Zhang

Hi Suwan,

The NVMe prints something like below by nvme_setup_io_queues() to confirm if the configuration takes effect.

"[ 0.620458] nvme nvme0: 4/0/0 default/read/poll queues".

How about to print in virtio-blk as well?

Hi Dongli,

Thansk for your feedback. It is good idea.
I will add it in next version.

Regards,
Suwan Kim

#8 lore raw

Re: [PATCH v3 1/2] virtio-blk: support polling I/O

In reply to Suwan Kim
Quoted 211 lines · Suwan Kim

This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.

The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.

virtio-blk polling is implemented upon 'batched completion' of block layer. virtblk_poll() queues completed request to io_comp_batch->req_list and later, virtblk_complete_batch() calls unmap function and ends the requests in batch.

virtio-blk reads the number of poll queues from module parameter "num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.

Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.

For verifying the improvement, I did Fio polling I/O performance test with io_uring engine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.

As a result, IOPS and average latency improved about 10%.

Test result:

- Fio io_uring poll without virtio-blk poll support

	-- numjobs=1 : IOPS = 339K, avg latency = 188.33us
	-- numjobs=2 : IOPS = 367K, avg latency = 347.33us
	-- numjobs=4 : IOPS = 383K, avg latency = 682.06us

- Fio io_uring poll with virtio-blk poll support

	-- numjobs=1 : IOPS = 380K, avg latency = 167.87us
	-- numjobs=2 : IOPS = 409K, avg latency = 312.6us
	-- numjobs=4 : IOPS = 413K, avg latency = 619.72us
---
 drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++--
 1 file changed, 97 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +94 -4
index 8c415be86732..3d16f8b753e7 100644
@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,
 		 "0 for no limit. "
 		 "Values > nr_cpu_ids truncated to nr_cpu_ids.");
 
+static unsigned int num_poll_queues;
+module_param(num_poll_queues, uint, 0644);
+MODULE_PARM_DESC(num_poll_queues, "The number of dedicated virtqueues for polling I/O");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 
@@ -81,6 +85,7 @@ struct virtio_blk {
 
 	/* num of vqs */
 	int num_vqs;
+	int io_queues[HCTX_MAX_TYPES];
 	struct virtio_blk_vq *vqs;
 };
 
@@ -548,6 +553,7 @@ static int init_vq(struct virtio_blk *vblk)
 	const char **names;
 	struct virtqueue **vqs;
 	unsigned short num_vqs;
+	unsigned int num_poll_vqs;
 	struct virtio_device *vdev = vblk->vdev;
 	struct irq_affinity desc = { 0, };
 
@@ -556,6 +562,7 @@ static int init_vq(struct virtio_blk *vblk)
 				   &num_vqs);
 	if (err)
 		num_vqs = 1;
+
 	if (!err && !num_vqs) {
 		dev_err(&vdev->dev, "MQ advertised but zero queues reported\n");
 		return -EINVAL;
@@ -565,6 +572,13 @@ static int init_vq(struct virtio_blk *vblk)
 			min_not_zero(num_request_queues, nr_cpu_ids),
 			num_vqs);
 
+	num_poll_vqs = min_t(unsigned int, num_poll_queues, num_vqs - 1);
+
+	memset(vblk->io_queues, 0, sizeof(int) * HCTX_MAX_TYPES);
+	vblk->io_queues[HCTX_TYPE_DEFAULT] = num_vqs - num_poll_vqs;
+	vblk->io_queues[HCTX_TYPE_READ] = 0;
+	vblk->io_queues[HCTX_TYPE_POLL] = num_poll_vqs;
+
 	vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
 	if (!vblk->vqs)
 		return -ENOMEM;
@@ -578,8 +592,13 @@ static int init_vq(struct virtio_blk *vblk)
 	}
 
 	for (i = 0; i < num_vqs; i++) {
-		callbacks[i] = virtblk_done;
-		snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req.%d", i);
+		if (i < num_vqs - num_poll_vqs) {
+			callbacks[i] = virtblk_done;
+			snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req.%d", i);
+		} else {
+			callbacks[i] = NULL;
+			snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req_poll.%d", i);
+		}
 		names[i] = vblk->vqs[i].name;
 	}
 
@@ -728,16 +747,87 @@ static const struct attribute_group *virtblk_attr_groups[] = {
 static int virtblk_map_queues(struct blk_mq_tag_set *set)
 {
 	struct virtio_blk *vblk = set->driver_data;
+	int i, qoff;
+
+	for (i = 0, qoff = 0; i < set->nr_maps; i++) {
+		struct blk_mq_queue_map *map = &set->map[i];
+
+		map->nr_queues = vblk->io_queues[i];
+		map->queue_offset = qoff;
+		qoff += map->nr_queues;
+
+		if (map->nr_queues == 0)
+			continue;
+
+		/*
+		 * Regular queues have interrupts and hence CPU affinity is
+		 * defined by the core virtio code, but polling queues have
+		 * no interrupts so we let the block layer assign CPU affinity.
+		 */
+		if (i == HCTX_TYPE_DEFAULT)
+			blk_mq_virtio_map_queues(&set->map[i], vblk->vdev, 0);
+		else
+			blk_mq_map_queues(&set->map[i]);
+	}
+
+	return 0;
+}
+
+static void virtblk_complete_batch(struct io_comp_batch *iob)
+{
+	struct request *req;
+	struct virtblk_req *vbr;
 
-	return blk_mq_virtio_map_queues(&set->map[HCTX_TYPE_DEFAULT],
-					vblk->vdev, 0);
+	rq_list_for_each(&iob->req_list, req) {
+		vbr = blk_mq_rq_to_pdu(req);
+		virtblk_unmap_data(req, vbr);
+		virtblk_cleanup_cmd(req);
+	}
+	blk_mq_end_request_batch(iob);
+}
+
+static int virtblk_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
+{
+	struct virtio_blk_vq *vq = hctx->driver_data;
+	struct virtblk_req *vbr;
+	unsigned long flags;
+	unsigned int len;
+	int found = 0;
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	while ((vbr = virtqueue_get_buf(vq->vq, &len)) != NULL) {
+		struct request *req = blk_mq_rq_from_pdu(vbr);
+
+		found++;
+		if (!blk_mq_add_to_batch(req, iob, vbr->status,
+						virtblk_complete_batch))
+			blk_mq_complete_request(req);
+	}
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+
+	return found;
+}
+
+static int virtblk_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			  unsigned int hctx_idx)
+{
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq = &vblk->vqs[hctx_idx];
+
+	WARN_ON(vblk->tag_set.tags[hctx_idx] != hctx->tags);
+	hctx->driver_data = vq;
+	return 0;
 }
 
 static const struct blk_mq_ops virtio_mq_ops = {
 	.queue_rq	= virtio_queue_rq,
 	.commit_rqs	= virtio_commit_rqs,
+	.init_hctx	= virtblk_init_hctx,
 	.complete	= virtblk_request_done,
 	.map_queues	= virtblk_map_queues,
+	.poll		= virtblk_poll,
 };
 
 static unsigned int virtblk_queue_depth;

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 914

Quoted 9 lines
drivers/block/virtio_blk.c +3 -0
@@ -816,6 +906,9 @@ static int virtblk_probe(struct virtio_device *vdev)
 		sizeof(struct scatterlist) * VIRTIO_BLK_INLINE_SG_CNT;
 	vblk->tag_set.driver_data = vblk;
 	vblk->tag_set.nr_hw_queues = vblk->num_vqs;
+	vblk->tag_set.nr_maps = 1;
+	if (vblk->io_queues[HCTX_TYPE_POLL])
+		vblk->tag_set.nr_maps = 3;
 
 	err = blk_mq_alloc_tag_set(&vblk->tag_set);
 	if (err)

So wrt cleanup, does something poll for all buffers to be used when device is removed?

Quoted 2 lines
Signature · 1 line
2.26.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#9 lore raw

Re: [PATCH v3 1/2] virtio-blk: support polling I/O

Quoted 219 lines · Michael S. Tsirkin
Quoted 218 lines · Suwan Kim

This patch supports polling I/O via virtio-blk driver. Polling feature is enabled by module parameter "num_poll_queues" and it sets dedicated polling queues for virtio-blk. This patch improves the polling I/O throughput and latency.

The virtio-blk driver doesn't not have a poll function and a poll queue and it has been operating in interrupt driven method even if the polling function is called in the upper layer.

virtio-blk polling is implemented upon 'batched completion' of block layer. virtblk_poll() queues completed request to io_comp_batch->req_list and later, virtblk_complete_batch() calls unmap function and ends the requests in batch.

virtio-blk reads the number of poll queues from module parameter "num_poll_queues". If VM sets queue parameter as below, ("num-queues=N" [QEMU property], "num_poll_queues=M" [module parameter]) It allocates N virtqueues to virtio_blk->vqs[N] and it uses [0..(N-M-1)] as default queues and [(N-M)..(N-1)] as poll queues. Unlike the default queues, the poll queues have no callback function.

Regarding HW-SW queue mapping, the default queue mapping uses the existing method that condsiders MSI irq vector. But the poll queue doesn't have an irq, so it uses the regular blk-mq cpu mapping.

For verifying the improvement, I did Fio polling I/O performance test with io_uring engine with the options below.
(io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=N) I set 4 vcpu and 4 virtio-blk queues - 2 default queues and 2 poll queues for VM.

As a result, IOPS and average latency improved about 10%.

Test result:

- Fio io_uring poll without virtio-blk poll support

	-- numjobs=1 : IOPS = 339K, avg latency = 188.33us
	-- numjobs=2 : IOPS = 367K, avg latency = 347.33us
	-- numjobs=4 : IOPS = 383K, avg latency = 682.06us

- Fio io_uring poll with virtio-blk poll support

	-- numjobs=1 : IOPS = 380K, avg latency = 167.87us
	-- numjobs=2 : IOPS = 409K, avg latency = 312.6us
	-- numjobs=4 : IOPS = 413K, avg latency = 619.72us
---
 drivers/block/virtio_blk.c | 101 +++++++++++++++++++++++++++++++++++--
 1 file changed, 97 insertions(+), 4 deletions(-)
drivers/block/virtio_blk.c +97 -4
index 8c415be86732..3d16f8b753e7 100644
@@ -37,6 +37,10 @@ MODULE_PARM_DESC(num_request_queues,
 		 "0 for no limit. "
 		 "Values > nr_cpu_ids truncated to nr_cpu_ids.");
 
+static unsigned int num_poll_queues;
+module_param(num_poll_queues, uint, 0644);
+MODULE_PARM_DESC(num_poll_queues, "The number of dedicated virtqueues for polling I/O");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 
@@ -81,6 +85,7 @@ struct virtio_blk {
 
 	/* num of vqs */
 	int num_vqs;
+	int io_queues[HCTX_MAX_TYPES];
 	struct virtio_blk_vq *vqs;
 };
 
@@ -548,6 +553,7 @@ static int init_vq(struct virtio_blk *vblk)
 	const char **names;
 	struct virtqueue **vqs;
 	unsigned short num_vqs;
+	unsigned int num_poll_vqs;
 	struct virtio_device *vdev = vblk->vdev;
 	struct irq_affinity desc = { 0, };
 
@@ -556,6 +562,7 @@ static int init_vq(struct virtio_blk *vblk)
 				   &num_vqs);
 	if (err)
 		num_vqs = 1;
+
 	if (!err && !num_vqs) {
 		dev_err(&vdev->dev, "MQ advertised but zero queues reported\n");
 		return -EINVAL;
@@ -565,6 +572,13 @@ static int init_vq(struct virtio_blk *vblk)
 			min_not_zero(num_request_queues, nr_cpu_ids),
 			num_vqs);
 
+	num_poll_vqs = min_t(unsigned int, num_poll_queues, num_vqs - 1);
+
+	memset(vblk->io_queues, 0, sizeof(int) * HCTX_MAX_TYPES);
+	vblk->io_queues[HCTX_TYPE_DEFAULT] = num_vqs - num_poll_vqs;
+	vblk->io_queues[HCTX_TYPE_READ] = 0;
+	vblk->io_queues[HCTX_TYPE_POLL] = num_poll_vqs;
+
 	vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
 	if (!vblk->vqs)
 		return -ENOMEM;
@@ -578,8 +592,13 @@ static int init_vq(struct virtio_blk *vblk)
 	}
 
 	for (i = 0; i < num_vqs; i++) {
-		callbacks[i] = virtblk_done;
-		snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req.%d", i);
+		if (i < num_vqs - num_poll_vqs) {
+			callbacks[i] = virtblk_done;
+			snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req.%d", i);
+		} else {
+			callbacks[i] = NULL;
+			snprintf(vblk->vqs[i].name, VQ_NAME_LEN, "req_poll.%d", i);
+		}
 		names[i] = vblk->vqs[i].name;
 	}
 
@@ -728,16 +747,87 @@ static const struct attribute_group *virtblk_attr_groups[] = {
 static int virtblk_map_queues(struct blk_mq_tag_set *set)
 {
 	struct virtio_blk *vblk = set->driver_data;
+	int i, qoff;
+
+	for (i = 0, qoff = 0; i < set->nr_maps; i++) {
+		struct blk_mq_queue_map *map = &set->map[i];
+
+		map->nr_queues = vblk->io_queues[i];
+		map->queue_offset = qoff;
+		qoff += map->nr_queues;
+
+		if (map->nr_queues == 0)
+			continue;
+
+		/*
+		 * Regular queues have interrupts and hence CPU affinity is
+		 * defined by the core virtio code, but polling queues have
+		 * no interrupts so we let the block layer assign CPU affinity.
+		 */
+		if (i == HCTX_TYPE_DEFAULT)
+			blk_mq_virtio_map_queues(&set->map[i], vblk->vdev, 0);
+		else
+			blk_mq_map_queues(&set->map[i]);
+	}
+
+	return 0;
+}
+
+static void virtblk_complete_batch(struct io_comp_batch *iob)
+{
+	struct request *req;
+	struct virtblk_req *vbr;
 
-	return blk_mq_virtio_map_queues(&set->map[HCTX_TYPE_DEFAULT],
-					vblk->vdev, 0);
+	rq_list_for_each(&iob->req_list, req) {
+		vbr = blk_mq_rq_to_pdu(req);
+		virtblk_unmap_data(req, vbr);
+		virtblk_cleanup_cmd(req);
+	}
+	blk_mq_end_request_batch(iob);
+}
+
+static int virtblk_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
+{
+	struct virtio_blk_vq *vq = hctx->driver_data;
+	struct virtblk_req *vbr;
+	unsigned long flags;
+	unsigned int len;
+	int found = 0;
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	while ((vbr = virtqueue_get_buf(vq->vq, &len)) != NULL) {
+		struct request *req = blk_mq_rq_from_pdu(vbr);
+
+		found++;
+		if (!blk_mq_add_to_batch(req, iob, vbr->status,
+						virtblk_complete_batch))
+			blk_mq_complete_request(req);
+	}
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+
+	return found;
+}
+
+static int virtblk_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			  unsigned int hctx_idx)
+{
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq = &vblk->vqs[hctx_idx];
+
+	WARN_ON(vblk->tag_set.tags[hctx_idx] != hctx->tags);
+	hctx->driver_data = vq;
+	return 0;
 }
 
 static const struct blk_mq_ops virtio_mq_ops = {
 	.queue_rq	= virtio_queue_rq,
 	.commit_rqs	= virtio_commit_rqs,
+	.init_hctx	= virtblk_init_hctx,
 	.complete	= virtblk_request_done,
 	.map_queues	= virtblk_map_queues,
+	.poll		= virtblk_poll,
 };
 
 static unsigned int virtblk_queue_depth;
@@ -816,6 +906,9 @@ static int virtblk_probe(struct virtio_device *vdev)
 		sizeof(struct scatterlist) * VIRTIO_BLK_INLINE_SG_CNT;
 	vblk->tag_set.driver_data = vblk;
 	vblk->tag_set.nr_hw_queues = vblk->num_vqs;
+	vblk->tag_set.nr_maps = 1;
+	if (vblk->io_queues[HCTX_TYPE_POLL])
+		vblk->tag_set.nr_maps = 3;
 
Quoted 7 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 914

Quoted 2 lines
drivers/block/virtio_blk.c +0 -0
 	err = blk_mq_alloc_tag_set(&vblk->tag_set);
 	if (err)

So wrt cleanup, does something poll for all buffers to be used when device is removed?

Sorry for late reply.

Maybe below function calls iterate each HW queue and flush requests before device is removed?

-----
virtblk_remove() -> blk_cleanup_disk()/blk_cleanup_queue() -> blk_queue_start_drain()/blk_freeze_queue()
-----

Regards,
Suwan Kim

#10 lore raw

Re: [PATCH v3 1/2] virtio-blk: support polling I/O

In reply to Suwan Kim
Quoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10 -0
+static int virtblk_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
+{
+	struct virtio_blk_vq *vq = hctx->driver_data;
+	struct virtblk_req *vbr;
+	unsigned long flags;
+	unsigned int len;
+	int found = 0;
+
+	spin_lock_irqsave(&vq->lock, flags);
+

drivers/block/virtio_blk.c static const struct attribute_group *virtblk_attr_groups[] = { · line 808

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+	while ((vbr = virtqueue_get_buf(vq->vq, &len)) != NULL) {
+		struct request *req = blk_mq_rq_from_pdu(vbr);
+
+		found++;
+		if (!blk_mq_add_to_batch(req, iob, vbr->status,
+						virtblk_complete_batch))
+			blk_mq_complete_request(req);
+	}
+
+	spin_unlock_irqrestore(&vq->lock, flags);

virtblk_done() does:

/* In case queue is stopped waiting for more buffers. */ if (req_done)

          blk_mq_start_stopped_hw_queues(vblk->disk->queue, true);

Is the same thing needed here in virtblk_poll() so that stopped queues are restarted when requests complete?

author
#11 lore raw

Re: [PATCH v3 1/2] virtio-blk: support polling I/O

Quoted 24 lines · Stefan Hajnoczi
Quoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10 -0
+static int virtblk_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
+{
+	struct virtio_blk_vq *vq = hctx->driver_data;
+	struct virtblk_req *vbr;
+	unsigned long flags;
+	unsigned int len;
+	int found = 0;
+
+	spin_lock_irqsave(&vq->lock, flags);
+

drivers/block/virtio_blk.c static const struct attribute_group *virtblk_attr_groups[] = { · line 808

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+	while ((vbr = virtqueue_get_buf(vq->vq, &len)) != NULL) {
+		struct request *req = blk_mq_rq_from_pdu(vbr);
+
+		found++;
+		if (!blk_mq_add_to_batch(req, iob, vbr->status,
+						virtblk_complete_batch))
+			blk_mq_complete_request(req);
+	}
+
+	spin_unlock_irqrestore(&vq->lock, flags);

virtblk_done() does:

Quoted 6 lines

/* In case queue is stopped waiting for more buffers. */ if (req_done)

          blk_mq_start_stopped_hw_queues(vblk->disk->queue, true);

Is the same thing needed here in virtblk_poll() so that stopped queues are restarted when requests complete?

I think you are right. I missed that.

I just added blk_mq_start_stopped_hw_queues() to virtblk_poll as you commented and did performance test again.

It showed higher peak performance than virtblk_poll without blk_mq_start_stopped_hw_queues().

I will add it in next version.
Thanks for the comment!

Regards,
Suwan Kim

author
#12 lore raw

PATCH v3 2/2 virtio-blk: support mq_ops->queue_rqs()

In reply to Suwan Kim

This patch supports mq_ops->queue_rqs() hook. It has an advantage of batch submission to virtio-blk driver. It also helps polling I/O because polling uses batched completion of block layer. Batch submission in queue_rqs() can boost polling performance.

In queue_rqs(), it iterates plug->mq_list, collects requests that belong to same HW queue and adds requests into virtqueue until it encounters a request from other HW queue or sees the end of the list. Then, virtio-blk kicks virtqueue to submit requests.

If there is an error, it inserts error request to requeue_list and passes it to ordinary block layer path.

For verification, I did fio test.
(io_uring, randread, direct=1, bs=4K, iodepth=64 numjobs=N) I set 4 vcpu and 2 virtio-blk queues for VM and run fio test 5 times. It shows about 1-4% improvement.

| numjobs=2 | numjobs=4
-----------------------------------------------------------
fio without queue_rqs() | 282K IOPS | 245K IOPS
-----------------------------------------------------------
fio with queue_rqs() | 294K IOPS | 249K IOPS

For polling I/O performance, I also did fio test as below. (io_uring, hipri, randread, direct=1, bs=512, iodepth=64 numjobs=4) I set 4 vcpu and 2 poll queues for VM.
It shows upto 7% improvement in polling I/O.

| IOPS | avg latency
-----------------------------------------------------------
fio poll without queue_rqs() | 413K | 619.72 usec
-----------------------------------------------------------
fio poll with queue_rqs() | 445K | 581.2 usec

---
 drivers/block/virtio_blk.c | 93 ++++++++++++++++++++++++++++++++++----
 1 file changed, 84 insertions(+), 9 deletions(-)
drivers/block/virtio_blk.c +84 -9
index 3d16f8b753e7..4a0a3b2d9caf 100644
@@ -311,6 +311,28 @@ static void virtio_commit_rqs(struct blk_mq_hw_ctx *hctx)
 		virtqueue_notify(vq->vq);
 }
 
+static blk_status_t virtblk_prep_rq(struct blk_mq_hw_ctx *hctx,
+					struct virtio_blk *vblk,
+					struct request *req,
+					struct virtblk_req *vbr, int *num)
+{
+	blk_status_t status;
+
+	status = virtblk_setup_cmd(vblk->vdev, req, vbr);
+	if (unlikely(status))
+		return status;
+
+	blk_mq_start_request(req);
+
+	*num = virtblk_map_data(hctx, req, vbr);
+	if (unlikely(*num < 0)) {
+		virtblk_cleanup_cmd(req);
+		return BLK_STS_RESOURCE;
+	}
+
+	return BLK_STS_OK;
+}
+
 static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
 			   const struct blk_mq_queue_data *bd)
 {
@@ -324,18 +346,10 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
 	blk_status_t status;
 	int err;
 
-	status = virtblk_setup_cmd(vblk->vdev, req, vbr);
+	status = virtblk_prep_rq(hctx, vblk, req, vbr, &num);
 	if (unlikely(status))
 		return status;
 
-	blk_mq_start_request(req);
-
-	num = virtblk_map_data(hctx, req, vbr);
-	if (unlikely(num < 0)) {
-		virtblk_cleanup_cmd(req);
-		return BLK_STS_RESOURCE;
-	}
-
 	spin_lock_irqsave(&vblk->vqs[qid].lock, flags);
 	err = virtblk_add_req(vblk->vqs[qid].vq, vbr, vbr->sg_table.sgl, num);
 	if (err) {
@@ -367,6 +381,66 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
 	return BLK_STS_OK;
 }
 
+static bool virtblk_prep_rq_batch(struct virtio_blk_vq *vq, struct request *req)
+{
+	struct virtio_blk *vblk = req->mq_hctx->queue->queuedata;
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(req);
+	unsigned long flags;
+	int num, err;
+
+	req->mq_hctx->tags->rqs[req->tag] = req;
+
+	if (virtblk_prep_rq(req->mq_hctx, vblk, req, vbr, &num) != BLK_STS_OK)
+		return false;
+
+	spin_lock_irqsave(&vq->lock, flags);
+	err = virtblk_add_req(vq->vq, vbr, vbr->sg_table.sgl, num);
+	if (err) {
+		spin_unlock_irqrestore(&vq->lock, flags);
+		virtblk_unmap_data(req, vbr);
+		virtblk_cleanup_cmd(req);
+		return false;
+	}
+	spin_unlock_irqrestore(&vq->lock, flags);
+
+	return true;
+}
+
+static void virtio_queue_rqs(struct request **rqlist)
+{
+	struct request *req, *next, *prev = NULL;
+	struct request *requeue_list = NULL;
+
+	rq_list_for_each_safe(rqlist, req, next) {
+		struct virtio_blk_vq *vq = req->mq_hctx->driver_data;
+		unsigned long flags;
+		bool kick;
+
+		if (!virtblk_prep_rq_batch(vq, req)) {
+			rq_list_move(rqlist, &requeue_list, req, prev);
+			req = prev;
+
+			if (!req)
+				continue;
+		}
+
+		if (!next || req->mq_hctx != next->mq_hctx) {
+			spin_lock_irqsave(&vq->lock, flags);
+			kick = virtqueue_kick_prepare(vq->vq);
+			spin_unlock_irqrestore(&vq->lock, flags);
+			if (kick)
+				virtqueue_notify(vq->vq);
+
+			req->rq_next = NULL;
+			*rqlist = next;
+			prev = NULL;
+		} else
+			prev = req;
+	}
+
+	*rqlist = requeue_list;
+}
+
 /* return id (s/n) string for *disk to *id_str
  */
 static int virtblk_get_id(struct gendisk *disk, char *id_str)
@@ -823,6 +897,7 @@ static int virtblk_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
 
 static const struct blk_mq_ops virtio_mq_ops = {
 	.queue_rq	= virtio_queue_rq,
+	.queue_rqs	= virtio_queue_rqs,
 	.commit_rqs	= virtio_commit_rqs,
 	.init_hctx	= virtblk_init_hctx,
 	.complete	= virtblk_request_done,
Signature · 1 line
2.26.3
#13 lore raw

Re: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()

Quoted 15 lines · Suwan Kim
drivers/block/virtio_blk.c +11 -0
@@ -367,6 +381,66 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
 	return BLK_STS_OK;
 }
 
+static bool virtblk_prep_rq_batch(struct virtio_blk_vq *vq, struct request *req)
+{
+	struct virtio_blk *vblk = req->mq_hctx->queue->queuedata;
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(req);
+	unsigned long flags;
+	int num, err;
+
+	req->mq_hctx->tags->rqs[req->tag] = req;
+
+	if (virtblk_prep_rq(req->mq_hctx, vblk, req, vbr, &num) != BLK_STS_OK)
+		return false;

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 404

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+
+	spin_lock_irqsave(&vq->lock, flags);
+	err = virtblk_add_req(vq->vq, vbr, vbr->sg_table.sgl, num);
+	if (err) {
+		spin_unlock_irqrestore(&vq->lock, flags);
+		virtblk_unmap_data(req, vbr);
+		virtblk_cleanup_cmd(req);
+		return false;
+	}
+	spin_unlock_irqrestore(&vq->lock, flags);

Simplification:

  spin_lock_irqsave(&vq->lock, flags);
  err = virtblk_add_req(vq->vq, vbr, vbr->sg_table.sgl, num);
  spin_unlock_irqrestore(&vq->lock, flags);
  if (err) {
      virtblk_unmap_data(req, vbr);
      virtblk_cleanup_cmd(req);
      return false;
  }
Quoted 14 lines
drivers/block/virtio_blk.c +14 -0
+
+	return true;
+}
+
+static void virtio_queue_rqs(struct request **rqlist)
+{
+	struct request *req, *next, *prev = NULL;
+	struct request *requeue_list = NULL;
+
+	rq_list_for_each_safe(rqlist, req, next) {
+		struct virtio_blk_vq *vq = req->mq_hctx->driver_data;
+		unsigned long flags;
+		bool kick;
+

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 428

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+		if (!virtblk_prep_rq_batch(vq, req)) {
+			rq_list_move(rqlist, &requeue_list, req, prev);
+			req = prev;
+
+			if (!req)
+				continue;
+		}
+
+		if (!next || req->mq_hctx != next->mq_hctx) {
+			spin_lock_irqsave(&vq->lock, flags);

Did you try calling virtblk_add_req() here to avoid acquiring and releasing the lock multiple times? In other words, do virtblk_prep_rq() but wait until we get here to do virtblk_add_req().

I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 438

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+			kick = virtqueue_kick_prepare(vq->vq);
+			spin_unlock_irqrestore(&vq->lock, flags);
+			if (kick)
+				virtqueue_notify(vq->vq);
+
+			req->rq_next = NULL;
+			*rqlist = next;
+			prev = NULL;
+		} else
+			prev = req;

What guarantees that req is still alive after we called virtblk_add_req()? The device may have seen it and completed it already by the time we get here.

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#14 lore raw

Re: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()

Quoted 29 lines · Stefan Hajnoczi
Quoted 15 lines · Suwan Kim
(quoted diff) +11 -0
@@ -367,6 +381,66 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
 	return BLK_STS_OK;
 }
 
+static bool virtblk_prep_rq_batch(struct virtio_blk_vq *vq, struct request *req)
+{
+	struct virtio_blk *vblk = req->mq_hctx->queue->queuedata;
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(req);
+	unsigned long flags;
+	int num, err;
+
+	req->mq_hctx->tags->rqs[req->tag] = req;
+
+	if (virtblk_prep_rq(req->mq_hctx, vblk, req, vbr, &num) != BLK_STS_OK)
+		return false;

Review static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 404

Quoted 10 lines
(quoted diff) +10 -0
+
+	spin_lock_irqsave(&vq->lock, flags);
+	err = virtblk_add_req(vq->vq, vbr, vbr->sg_table.sgl, num);
+	if (err) {
+		spin_unlock_irqrestore(&vq->lock, flags);
+		virtblk_unmap_data(req, vbr);
+		virtblk_cleanup_cmd(req);
+		return false;
+	}
+	spin_unlock_irqrestore(&vq->lock, flags);

Simplification:

Quoted 9 lines
  spin_lock_irqsave(&vq->lock, flags);
  err = virtblk_add_req(vq->vq, vbr, vbr->sg_table.sgl, num);
  spin_unlock_irqrestore(&vq->lock, flags);
  if (err) {
      virtblk_unmap_data(req, vbr);
      virtblk_cleanup_cmd(req);
      return false;
  }

Thanks! I will fix it.

Quoted 29 lines
Quoted 14 lines
(quoted diff) +14 -0
+
+	return true;
+}
+
+static void virtio_queue_rqs(struct request **rqlist)
+{
+	struct request *req, *next, *prev = NULL;
+	struct request *requeue_list = NULL;
+
+	rq_list_for_each_safe(rqlist, req, next) {
+		struct virtio_blk_vq *vq = req->mq_hctx->driver_data;
+		unsigned long flags;
+		bool kick;
+

Review static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 428

Quoted 10 lines
(quoted diff) +10 -0
+		if (!virtblk_prep_rq_batch(vq, req)) {
+			rq_list_move(rqlist, &requeue_list, req, prev);
+			req = prev;
+
+			if (!req)
+				continue;
+		}
+
+		if (!next || req->mq_hctx != next->mq_hctx) {
+			spin_lock_irqsave(&vq->lock, flags);

Did you try calling virtblk_add_req() here to avoid acquiring and releasing the lock multiple times? In other words, do virtblk_prep_rq() but wait until we get here to do virtblk_add_req().

Quoted 3 lines

I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.

I tried as you said. I called virtlblk_add_req() and added requests of rqlist to virtqueue in this if statement with holding the lock only once.

I attach the code at the end of this mail.
Please refer the code.

But I didn't see improvement. It showed slightly worse performance than the current patch.

Quoted 6 lines

Review static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 434

Quoted 6 lines
(quoted diff) +6 -0
+			kick = virtqueue_kick_prepare(vq->vq);
+			spin_unlock_irqrestore(&vq->lock, flags);
+			if (kick)
+				virtqueue_notify(vq->vq);
+
+			req->rq_next = NULL;

Did you ask this part?

Quoted 8 lines

Review static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 438

Quoted 4 lines
(quoted diff) +4 -0
+			*rqlist = next;
+			prev = NULL;
+		} else
+			prev = req;

What guarantees that req is still alive after we called virtblk_add_req()? The device may have seen it and completed it already by the time we get here.

Isn't request completed after the kick?

If you asked about "req->rq_next = NULL",
I think it should be placed before
"kick = virtqueue_kick_prepare(vq->vq);"

-----------

	req->rq_next = NULL;
	kick = virtqueue_kick_prepare(vq->vq);
	spin_unlock_irqrestore(&vq->lock, flags);
	if (kick)
		virtqueue_notify(vq->vq);

-----------

Regards,
Suwan Kim

---

drivers/block/virtio_blk.c +99 -11
index 2218cab39c72..d972d3042068 100644
@@ -92,6 +92,7 @@ struct virtio_blk {
 struct virtblk_req {
 	struct virtio_blk_outhdr out_hdr;
 	u8 status;
+	int sg_num;
 	struct sg_table sg_table;
 	struct scatterlist sg[];
 };
@@ -311,18 +312,13 @@ static void virtio_commit_rqs(struct blk_mq_hw_ctx *hctx)
 		virtqueue_notify(vq->vq);
 }
 
-static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
-			   const struct blk_mq_queue_data *bd)
+static blk_status_t virtblk_prep_rq(struct blk_mq_hw_ctx *hctx,
+					struct virtio_blk *vblk,
+					struct request *req,
+					struct virtblk_req *vbr)
 {
-	struct virtio_blk *vblk = hctx->queue->queuedata;
-	struct request *req = bd->rq;
-	struct virtblk_req *vbr = blk_mq_rq_to_pdu(req);
-	unsigned long flags;
-	int num;
-	int qid = hctx->queue_num;
-	bool notify = false;
 	blk_status_t status;
-	int err;
+	int num;
 
 	status = virtblk_setup_cmd(vblk->vdev, req, vbr);
 	if (unlikely(status))
@@ -335,9 +331,30 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
 		virtblk_cleanup_cmd(req);
 		return BLK_STS_RESOURCE;
 	}
+	vbr->sg_num = num;
+
+	return BLK_STS_OK;
+}
+
+static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
+			   const struct blk_mq_queue_data *bd)
+{
+	struct virtio_blk *vblk = hctx->queue->queuedata;
+	struct request *req = bd->rq;
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(req);
+	unsigned long flags;
+	int qid = hctx->queue_num;
+	bool notify = false;
+	blk_status_t status;
+	int err;
+
+	status = virtblk_prep_rq(hctx, vblk, req, vbr);
+	if (unlikely(status))
+		return status;
 
 	spin_lock_irqsave(&vblk->vqs[qid].lock, flags);
-	err = virtblk_add_req(vblk->vqs[qid].vq, vbr, vbr->sg_table.sgl, num);
+	err = virtblk_add_req(vblk->vqs[qid].vq, vbr,
+				vbr->sg_table.sgl, vbr->sg_num);
 	if (err) {
 		virtqueue_kick(vblk->vqs[qid].vq);
 		/* Don't stop the queue if -ENOMEM: we may have failed to
@@ -367,6 +384,76 @@ static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,
 	return BLK_STS_OK;
 }
 
+static bool virtblk_prep_rq_batch(struct virtio_blk_vq *vq, struct request *req)
+{
+	struct virtio_blk *vblk = req->mq_hctx->queue->queuedata;
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(req);
+
+	req->mq_hctx->tags->rqs[req->tag] = req;
+
+	return virtblk_prep_rq(req->mq_hctx, vblk, req, vbr) == BLK_STS_OK;
+}
+
+static bool virtblk_add_req_batch(struct virtio_blk_vq *vq,
+					struct request **rqlist,
+					struct request **requeue_list)
+{
+	unsigned long flags;
+	int err;
+	bool kick;
+
+	spin_lock_irqsave(&vq->lock, flags);
+	while (!rq_list_empty(*rqlist)) {
+		struct request *req = rq_list_pop(rqlist);
+		struct virtblk_req *vbr = blk_mq_rq_to_pdu(req);
+
+		err = virtblk_add_req(vq->vq, vbr,
+					vbr->sg_table.sgl, vbr->sg_num);
+		if (err) {
+			virtblk_unmap_data(req, vbr);
+			virtblk_cleanup_cmd(req);
+			rq_list_add(requeue_list, req);
+		}
+	}
+
+	kick = virtqueue_kick_prepare(vq->vq);
+	spin_unlock_irqrestore(&vq->lock, flags);
+
+	return kick;
+}
+
+static void virtio_queue_rqs(struct request **rqlist)
+{
+	struct request *req, *next, *prev = NULL;
+	struct request *requeue_list = NULL;
+
+	rq_list_for_each_safe(rqlist, req, next) {
+		struct virtio_blk_vq *vq = req->mq_hctx->driver_data;
+		bool kick;
+
+		if (!virtblk_prep_rq_batch(vq, req)) {
+			rq_list_move(rqlist, &requeue_list, req, prev);
+			req = prev;
+
+			if (!req)
+				continue;
+		}
+
+		if (!next || req->mq_hctx != next->mq_hctx) {
+			kick = virtblk_add_req_batch(vq, rqlist, &requeue_list);
+			if (kick)
+				virtqueue_notify(vq->vq);
+
+			req->rq_next = NULL;
+			*rqlist = next;
+			prev = NULL;
+		} else
+			prev = req;
+	}
+
+	*rqlist = requeue_list;
+}
+
 /* return id (s/n) string for *disk to *id_str
  */
 static int virtblk_get_id(struct gendisk *disk, char *id_str)
@@ -818,6 +905,7 @@ static int virtblk_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
 
 static const struct blk_mq_ops virtio_mq_ops = {
 	.queue_rq	= virtio_queue_rq,
+	.queue_rqs	= virtio_queue_rqs,
 	.commit_rqs	= virtio_commit_rqs,
 	.init_hctx	= virtblk_init_hctx,
 	.complete	= virtblk_request_done,
#15 lore raw

Re: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()

Quoted 35 lines · Suwan Kim
Quoted 26 lines · Stefan Hajnoczi
Quoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10 -0
+static void virtio_queue_rqs(struct request **rqlist)
+{
+	struct request *req, *next, *prev = NULL;
+	struct request *requeue_list = NULL;
+
+	rq_list_for_each_safe(rqlist, req, next) {
+		struct virtio_blk_vq *vq = req->mq_hctx->driver_data;
+		unsigned long flags;
+		bool kick;
+

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+		if (!virtblk_prep_rq_batch(vq, req)) {
+			rq_list_move(rqlist, &requeue_list, req, prev);
+			req = prev;
+
+			if (!req)
+				continue;
+		}
+
+		if (!next || req->mq_hctx != next->mq_hctx) {
+			spin_lock_irqsave(&vq->lock, flags);

Did you try calling virtblk_add_req() here to avoid acquiring and releasing the lock multiple times? In other words, do virtblk_prep_rq() but wait until we get here to do virtblk_add_req().

Quoted 3 lines

I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.

I tried as you said. I called virtlblk_add_req() and added requests of rqlist to virtqueue in this if statement with holding the lock only once.

Quoted 5 lines

I attach the code at the end of this mail.
Please refer the code.

But I didn't see improvement. It showed slightly worse performance than the current patch.

Okay, thanks for trying it!

Quoted 24 lines
Quoted 6 lines

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 447

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+			kick = virtqueue_kick_prepare(vq->vq);
+			spin_unlock_irqrestore(&vq->lock, flags);
+			if (kick)
+				virtqueue_notify(vq->vq);
+
+			req->rq_next = NULL;

Did you ask this part?

Quoted 8 lines

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 451

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+			*rqlist = next;
+			prev = NULL;
+		} else
+			prev = req;

What guarantees that req is still alive after we called virtblk_add_req()? The device may have seen it and completed it already by the time we get here.

Isn't request completed after the kick?

If you asked about "req->rq_next = NULL",
I think it should be placed before
"kick = virtqueue_kick_prepare(vq->vq);"

Quoted 7 lines

-----------

	req->rq_next = NULL;
	kick = virtqueue_kick_prepare(vq->vq);
	spin_unlock_irqrestore(&vq->lock, flags);
	if (kick)
		virtqueue_notify(vq->vq);

-----------

No, virtqueue_add_sgs() exposes vring descriptors to the device. The device may process immediately. In other words, VIRTIO devices may poll the vring instead of waiting for virtqueue_notify(). There is no guarantee that the request is alive until virtqueue_notify() is called.

The code has to handle the case where the request is completed during virtqueue_add_sgs().

Stefan

author
#16 lore raw

Re: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()

Quoted 76 lines · Stefan Hajnoczi
Quoted 35 lines · Suwan Kim
Quoted 26 lines · Stefan Hajnoczi
Quoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10 -0
+static void virtio_queue_rqs(struct request **rqlist)
+{
+	struct request *req, *next, *prev = NULL;
+	struct request *requeue_list = NULL;
+
+	rq_list_for_each_safe(rqlist, req, next) {
+		struct virtio_blk_vq *vq = req->mq_hctx->driver_data;
+		unsigned long flags;
+		bool kick;
+

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+		if (!virtblk_prep_rq_batch(vq, req)) {
+			rq_list_move(rqlist, &requeue_list, req, prev);
+			req = prev;
+
+			if (!req)
+				continue;
+		}
+
+		if (!next || req->mq_hctx != next->mq_hctx) {
+			spin_lock_irqsave(&vq->lock, flags);

Did you try calling virtblk_add_req() here to avoid acquiring and releasing the lock multiple times? In other words, do virtblk_prep_rq() but wait until we get here to do virtblk_add_req().

Quoted 3 lines

I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.

I tried as you said. I called virtlblk_add_req() and added requests of rqlist to virtqueue in this if statement with holding the lock only once.

Quoted 5 lines

I attach the code at the end of this mail.
Please refer the code.

But I didn't see improvement. It showed slightly worse performance than the current patch.

Okay, thanks for trying it!

Quoted 24 lines
Quoted 6 lines

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 447

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+			kick = virtqueue_kick_prepare(vq->vq);
+			spin_unlock_irqrestore(&vq->lock, flags);
+			if (kick)
+				virtqueue_notify(vq->vq);
+
+			req->rq_next = NULL;

Did you ask this part?

Quoted 8 lines

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 451

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+			*rqlist = next;
+			prev = NULL;
+		} else
+			prev = req;

What guarantees that req is still alive after we called virtblk_add_req()? The device may have seen it and completed it already by the time we get here.

Isn't request completed after the kick?

If you asked about "req->rq_next = NULL",
I think it should be placed before
"kick = virtqueue_kick_prepare(vq->vq);"

Quoted 7 lines

-----------

	req->rq_next = NULL;
	kick = virtqueue_kick_prepare(vq->vq);
	spin_unlock_irqrestore(&vq->lock, flags);
	if (kick)
		virtqueue_notify(vq->vq);

-----------

Quoted 7 lines

No, virtqueue_add_sgs() exposes vring descriptors to the device. The device may process immediately. In other words, VIRTIO devices may poll the vring instead of waiting for virtqueue_notify(). There is no guarantee that the request is alive until virtqueue_notify() is called.

The code has to handle the case where the request is completed during virtqueue_add_sgs().

Thanks for the explanation.

We should not use req again after virtblk_add_req(). I understand...

Then, as you commented in previous mail, is it ok that we do virtblk_add_req() in "if (!next || req->mq_hctx != next->mq_hctx)" statement to avoid use req again after virtblk_add_req() as below code?

In this code, It adds reqs to virtqueue in batch just before virtqueue_notify(), and it doesn't use req again after calling virtblk_add_req().

If it is fine, I will try it again.
This code is slightly different from the code I sent in previous mail.

---
static void virtio_queue_rqs(struct request **rqlist)
...

	rq_list_for_each_safe(rqlist, req, next) {
...
		if (!next || req->mq_hctx != next->mq_hctx) {
			// Cut the list at current req
			req->rq_next = NULL;
			// Add req list to virtqueue in batch with holding lock once
			kick = virtblk_add_req_batch(vq, rqlist, &requeue_list);
			if (kick)
				virtqueue_notify(vq->vq);

			// setup new req list. Don't use previous req again.
			*rqlist = next;
			prev = NULL;

...
---

Regards,
Suwan Kim

#17 lore raw

Re: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()

Quoted 117 lines · Suwan Kim
Quoted 76 lines · Stefan Hajnoczi
Quoted 35 lines · Suwan Kim
Quoted 26 lines · Stefan Hajnoczi
Quoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10 -0
+static void virtio_queue_rqs(struct request **rqlist)
+{
+	struct request *req, *next, *prev = NULL;
+	struct request *requeue_list = NULL;
+
+	rq_list_for_each_safe(rqlist, req, next) {
+		struct virtio_blk_vq *vq = req->mq_hctx->driver_data;
+		unsigned long flags;
+		bool kick;
+

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+		if (!virtblk_prep_rq_batch(vq, req)) {
+			rq_list_move(rqlist, &requeue_list, req, prev);
+			req = prev;
+
+			if (!req)
+				continue;
+		}
+
+		if (!next || req->mq_hctx != next->mq_hctx) {
+			spin_lock_irqsave(&vq->lock, flags);

Did you try calling virtblk_add_req() here to avoid acquiring and releasing the lock multiple times? In other words, do virtblk_prep_rq() but wait until we get here to do virtblk_add_req().

Quoted 3 lines

I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.

I tried as you said. I called virtlblk_add_req() and added requests of rqlist to virtqueue in this if statement with holding the lock only once.

Quoted 5 lines

I attach the code at the end of this mail.
Please refer the code.

But I didn't see improvement. It showed slightly worse performance than the current patch.

Okay, thanks for trying it!

Quoted 24 lines
Quoted 6 lines

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 447

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+			kick = virtqueue_kick_prepare(vq->vq);
+			spin_unlock_irqrestore(&vq->lock, flags);
+			if (kick)
+				virtqueue_notify(vq->vq);
+
+			req->rq_next = NULL;

Did you ask this part?

Quoted 8 lines

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 451

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+			*rqlist = next;
+			prev = NULL;
+		} else
+			prev = req;

What guarantees that req is still alive after we called virtblk_add_req()? The device may have seen it and completed it already by the time we get here.

Isn't request completed after the kick?

If you asked about "req->rq_next = NULL",
I think it should be placed before
"kick = virtqueue_kick_prepare(vq->vq);"

Quoted 7 lines

-----------

	req->rq_next = NULL;
	kick = virtqueue_kick_prepare(vq->vq);
	spin_unlock_irqrestore(&vq->lock, flags);
	if (kick)
		virtqueue_notify(vq->vq);

-----------

Quoted 7 lines

No, virtqueue_add_sgs() exposes vring descriptors to the device. The device may process immediately. In other words, VIRTIO devices may poll the vring instead of waiting for virtqueue_notify(). There is no guarantee that the request is alive until virtqueue_notify() is called.

The code has to handle the case where the request is completed during virtqueue_add_sgs().

Thanks for the explanation.

We should not use req again after virtblk_add_req(). I understand...

Then, as you commented in previous mail, is it ok that we do virtblk_add_req() in "if (!next || req->mq_hctx != next->mq_hctx)" statement to avoid use req again after virtblk_add_req() as below code?

In this code, It adds reqs to virtqueue in batch just before virtqueue_notify(), and it doesn't use req again after calling virtblk_add_req().

If it is fine, I will try it again.
This code is slightly different from the code I sent in previous mail.

---
static void virtio_queue_rqs(struct request **rqlist)
...

	rq_list_for_each_safe(rqlist, req, next) {
...
		if (!next || req->mq_hctx != next->mq_hctx) {
			// Cut the list at current req
			req->rq_next = NULL;
			// Add req list to virtqueue in batch with holding lock once
			kick = virtblk_add_req_batch(vq, rqlist, &requeue_list);
			if (kick)
				virtqueue_notify(vq->vq);

			// setup new req list. Don't use previous req again.
			*rqlist = next;
			prev = NULL;
Quoted 1 line

...

Yes, that sounds good.

(I noticed struct request has a reference count so that might be a way to keep requests alive, if necessary, but I haven't investigated. See req_ref_put_and_test() though it's not used by block drivers and maybe virtio-blk shouldn't mess with it either.)

Stefan

author
#18 lore raw

Re: [PATCH v3 2/2] virtio-blk: support mq_ops->queue_rqs()

Quoted 120 lines · Stefan Hajnoczi
Quoted 117 lines · Suwan Kim
Quoted 76 lines · Stefan Hajnoczi
Quoted 35 lines · Suwan Kim
Quoted 26 lines · Stefan Hajnoczi
Quoted 10 lines · Suwan Kim
drivers/block/virtio_blk.c +10 -0
+static void virtio_queue_rqs(struct request **rqlist)
+{
+	struct request *req, *next, *prev = NULL;
+	struct request *requeue_list = NULL;
+
+	rq_list_for_each_safe(rqlist, req, next) {
+		struct virtio_blk_vq *vq = req->mq_hctx->driver_data;
+		unsigned long flags;
+		bool kick;
+

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx,

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+		if (!virtblk_prep_rq_batch(vq, req)) {
+			rq_list_move(rqlist, &requeue_list, req, prev);
+			req = prev;
+
+			if (!req)
+				continue;
+		}
+
+		if (!next || req->mq_hctx != next->mq_hctx) {
+			spin_lock_irqsave(&vq->lock, flags);

Did you try calling virtblk_add_req() here to avoid acquiring and releasing the lock multiple times? In other words, do virtblk_prep_rq() but wait until we get here to do virtblk_add_req().

Quoted 3 lines

I don't know if it has any measurable effect on performance or maybe the code would become too complex, but I noticed that we're not fully exploiting batching.

I tried as you said. I called virtlblk_add_req() and added requests of rqlist to virtqueue in this if statement with holding the lock only once.

Quoted 5 lines

I attach the code at the end of this mail.
Please refer the code.

But I didn't see improvement. It showed slightly worse performance than the current patch.

Okay, thanks for trying it!

Quoted 24 lines
Quoted 6 lines

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 447

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+			kick = virtqueue_kick_prepare(vq->vq);
+			spin_unlock_irqrestore(&vq->lock, flags);
+			if (kick)
+				virtqueue_notify(vq->vq);
+
+			req->rq_next = NULL;

Did you ask this part?

Quoted 8 lines

drivers/block/virtio_blk.c static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, · line 451

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+			*rqlist = next;
+			prev = NULL;
+		} else
+			prev = req;

What guarantees that req is still alive after we called virtblk_add_req()? The device may have seen it and completed it already by the time we get here.

Isn't request completed after the kick?

If you asked about "req->rq_next = NULL",
I think it should be placed before
"kick = virtqueue_kick_prepare(vq->vq);"

Quoted 7 lines

-----------

	req->rq_next = NULL;
	kick = virtqueue_kick_prepare(vq->vq);
	spin_unlock_irqrestore(&vq->lock, flags);
	if (kick)
		virtqueue_notify(vq->vq);

-----------

Quoted 7 lines

No, virtqueue_add_sgs() exposes vring descriptors to the device. The device may process immediately. In other words, VIRTIO devices may poll the vring instead of waiting for virtqueue_notify(). There is no guarantee that the request is alive until virtqueue_notify() is called.

The code has to handle the case where the request is completed during virtqueue_add_sgs().

Thanks for the explanation.

We should not use req again after virtblk_add_req(). I understand...

Then, as you commented in previous mail, is it ok that we do virtblk_add_req() in "if (!next || req->mq_hctx != next->mq_hctx)" statement to avoid use req again after virtblk_add_req() as below code?

In this code, It adds reqs to virtqueue in batch just before virtqueue_notify(), and it doesn't use req again after calling virtblk_add_req().

If it is fine, I will try it again.
This code is slightly different from the code I sent in previous mail.

---
static void virtio_queue_rqs(struct request **rqlist)
...

	rq_list_for_each_safe(rqlist, req, next) {
...
		if (!next || req->mq_hctx != next->mq_hctx) {
			// Cut the list at current req
			req->rq_next = NULL;
			// Add req list to virtqueue in batch with holding lock once
			kick = virtblk_add_req_batch(vq, rqlist, &requeue_list);
			if (kick)
				virtqueue_notify(vq->vq);

			// setup new req list. Don't use previous req again.
			*rqlist = next;
			prev = NULL;
Quoted 1 line

...

Quoted 6 lines

Yes, that sounds good.

(I noticed struct request has a reference count so that might be a way to keep requests alive, if necessary, but I haven't investigated. See req_ref_put_and_test() though it's not used by block drivers and maybe virtio-blk shouldn't mess with it either.)

I also think that using ref count is not a good idea. I will send the next version soon.

Regards,
Suwan Kim