Skip to contentSkip to conversation

add virtio_blk device support to vdpa/ifc

PATCH Superseded Andy Pei 5 participants 263 messages Jan 25, 2022 – Jun 1, 2022 Page 2 of 2

Subsystems (2)
lore
Share & tools
lore
#132 lore raw

RE: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc driver

In reply to Xia, Chenbo
Quoted 34 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:22 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc driver

Quoted 7 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc

driver

Review

Quoted 9 lines

Add get device type ops to ifc driver.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c

Quoted 7 lines

index 8a260b7..99a6ab0 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
@@ -1300,6 +1300,15 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
Quoted 10 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1309

Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7 -0
+static int
+ifcvf_blk_get_device_type(struct rte_vdpa_device *vdev,
+	uint32_t *type)
+{
+	RTE_SET_USED(vdev);
+	*type = VDPA_DEVICE_TYPE_BLK;
+	return 0;

This is not right. Remember net and blk are both using the driver? This will lead to using net also returns BLK.

Correct a bit for above: will lead to net not working. So implement callback for both (although in the API, NULL callback can return NET type)

Thanks,
Chenbo

Quoted 23 lines

And I suggest the patch-set validated with both blk and net...

Besides, ifcvf_blk_get_device_type should be ifcvf_get_device_type

Thanks,
Chenbo

Quoted 15 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +3 -1
+}
+
 static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
 	.get_queue_num = ifcvf_get_queue_num,
 	.get_features = ifcvf_get_vdpa_features,
@@ -1313,6 +1322,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
 	.get_config = ifcvf_blk_get_config,
+	.get_dev_type = ifcvf_blk_get_device_type,
 };
 
 struct rte_vdpa_dev_info dev_info[] = {
--

1.8.3.1

author
#133 lore raw

RE: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc driver

HI Chenbo,

Thanks for your reply.
I will implement a function "ifcvf_get_device_type" for both ifc net and blk ops. "ifcvf_get_device_type" will return device type according to internal->device_type.

Quoted 85 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:41 PM
To: Xia, Chenbo <[email protected]>; Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc driver

Quoted 35 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:22 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc driver

Quoted 8 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc

driver

Review

Quoted 9 lines

Add get device type ops to ifc driver.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c

Quoted 7 lines

index 8a260b7..99a6ab0 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
@@ -1300,6 +1300,15 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
Quoted 10 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1309

Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7 -0
+static int
+ifcvf_blk_get_device_type(struct rte_vdpa_device *vdev,
+	uint32_t *type)
+{
+	RTE_SET_USED(vdev);
+	*type = VDPA_DEVICE_TYPE_BLK;
+	return 0;

This is not right. Remember net and blk are both using the driver? This will lead to using net also returns BLK.

Correct a bit for above: will lead to net not working. So implement callback for both (although in the API, NULL callback can return NET type)

Thanks,
Chenbo

Quoted 23 lines

And I suggest the patch-set validated with both blk and net...

Besides, ifcvf_blk_get_device_type should be ifcvf_get_device_type

Thanks,
Chenbo

Quoted 15 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +2 -0
+}
+
 static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
 	.get_queue_num = ifcvf_get_queue_num,
 	.get_features = ifcvf_get_vdpa_features, @@ -1313,6 +1322,7 @@

struct rte_vdpa_dev_info {

drivers/vdpa/ifc/ifcvf_vdpa.c +1 -1
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
 	.get_config = ifcvf_blk_get_config,
+	.get_dev_type = ifcvf_blk_get_device_type,
 };
 
 struct rte_vdpa_dev_info dev_info[] = {
--

1.8.3.1

author
#134 lore raw

PATCH v7 9/18 examples/vdpa: add vDPA blk support in example

In reply to Andy Pei

Add virtio blk device support to vDPA example.

---
 examples/vdpa/main.c             | 57 +++++++++++++++++++++++++++++++++++
 examples/vdpa/vdpa_blk_compact.h | 65 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 122 insertions(+)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c +57 -0
index 5ab0765..2544141 100644
@@ -20,6 +20,7 @@
 #include <cmdline_parse_string.h>
 #include <cmdline_parse_num.h>
 #include <cmdline.h>
+#include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
 #define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,54 @@ struct vdpa_port {
 };
 
 static int
+vdpa_blk_device_set_features_and_protocol(const char *path)
+{
+	uint64_t protocol_features = 0;
+	int ret;
+
+	ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_disable_features(path,
+		VHOST_BLK_DISABLED_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_disable_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_get_protocol_features(path, &protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_get_protocol_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	protocol_features |= VHOST_BLK_PROTOCOL_FEATURES;
+
+	ret = rte_vhost_driver_set_protocol_features(path, protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_protocol_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+out:
+	return ret;
+}
+
+static int
 start_vdpa(struct vdpa_port *vport)
 {
+	uint32_t device_type = 0;
 	int ret;
 	char *socket_path = vport->ifname;
 
@@ -192,6 +239,16 @@ struct vdpa_port {
 			"attach vdpa device failed: %s\n",
 			socket_path);
 
+	ret = rte_vhost_driver_get_vdpa_dev_type(socket_path, &device_type);
+	if (ret == 0 && device_type == VDPA_DEVICE_TYPE_BLK) {
+		RTE_LOG(NOTICE, VDPA, "is a blk device\n");
+		ret = vdpa_blk_device_set_features_and_protocol(socket_path);
+		if (ret != 0)
+			rte_exit(EXIT_FAILURE,
+				"set vhost blk driver features and protocol features failed: %s\n",
+				socket_path);
+	}
+
 	if (rte_vhost_driver_start(socket_path) < 0)
 		rte_exit(EXIT_FAILURE,
 			"start vhost driver failed: %s\n",
examples/vdpa/vdpa_blk_compact.h new file +65 -0

65 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#135 lore raw

RE: [PATCH v7 09/18] examples/vdpa: add vDPA blk support in example

Quoted 7 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 09/18] examples/vdpa: add vDPA blk support in example

Better be: examples/vdpa: add virtio blk support

Quoted 59 lines

Add virtio blk device support to vDPA example.

---
 examples/vdpa/main.c             | 57 +++++++++++++++++++++++++++++++++++
 examples/vdpa/vdpa_blk_compact.h | 65
++++++++++++++++++++++++++++++++++++++++
 2 files changed, 122 insertions(+)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c +31 -0
index 5ab0765..2544141 100644
@@ -20,6 +20,7 @@
 #include <cmdline_parse_string.h>
 #include <cmdline_parse_num.h>
 #include <cmdline.h>
+#include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
 #define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,54 @@ struct vdpa_port {
 };
 
 static int
+vdpa_blk_device_set_features_and_protocol(const char *path)
+{
+	uint64_t protocol_features = 0;
+	int ret;
+
+	ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_disable_features(path,
+		VHOST_BLK_DISABLED_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_disable_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_get_protocol_features(path,
 &protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_get_protocol_features for %s
 failed.\n",
+			path);
+		goto out;
+	}
+

examples/vdpa/main.c struct vdpa_port { · line 204

Quoted 10 lines
examples/vdpa/main.c +8 -0
+	protocol_features |= VHOST_BLK_PROTOCOL_FEATURES;
+
+	ret = rte_vhost_driver_set_protocol_features(path,
 protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_protocol_features for %s
 failed.\n",
+			path);
+		goto out;

This goto is not needed.

Quoted 14 lines
examples/vdpa/main.c +8 -0
+	}
+
+out:
+	return ret;
+}
+
+static int
 start_vdpa(struct vdpa_port *vport)
 {
+	uint32_t device_type = 0;
 	int ret;
 	char *socket_path = vport->ifname;

examples/vdpa/main.c struct vdpa_port { · line 244

Quoted 6 lines
examples/vdpa/main.c +3 -0
@@ -192,6 +239,16 @@ struct vdpa_port {
 			"attach vdpa device failed: %s\n",
 			socket_path);
 
+	ret = rte_vhost_driver_get_vdpa_dev_type(socket_path, &device_type);
+	if (ret == 0 && device_type == VDPA_DEVICE_TYPE_BLK) {
+		RTE_LOG(NOTICE, VDPA, "is a blk device\n");

Should add info of socket path

Quoted 24 lines
examples/vdpa/main.c +7 -0
+		ret = vdpa_blk_device_set_features_and_protocol(socket_path);
+		if (ret != 0)
+			rte_exit(EXIT_FAILURE,
+				"set vhost blk driver features and protocol
 features failed: %s\n",
+				socket_path);
+	}
+
 	if (rte_vhost_driver_start(socket_path) < 0)
 		rte_exit(EXIT_FAILURE,
examples/vdpa/main.c +0 -0
 			"start vhost driver failed: %s\n",
examples/vdpa/vdpa_blk_compact.h new file +6 -0
new file mode 100644
index 0000000..136c3f6
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#ifndef _VDPA_BLK_COMPACT_H_
+#define _VDPA_BLK_COMPACT_H_

examples/vdpa/vdpa_blk_compact.h line 16

Quoted 10 lines
examples/vdpa/vdpa_blk_compact.h new file +10 -0
+
+/**
+ * @file
+ *
+ * Device specific vhost lib
+ */
+
+#include <stdbool.h>
+
+#include <rte_pci.h>

above two headers are not used in this file?

Quoted 21 lines
examples/vdpa/vdpa_blk_compact.h new file +4 -0
+#include <rte_vhost.h>
+
+/* Feature bits */
+#define VIRTIO_BLK_F_SIZE_MAX     1    /* Indicates maximum segment size

*/
+#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of segments
*/

examples/vdpa/vdpa_blk_compact.h new file +2 -0
+#define VIRTIO_BLK_F_GEOMETRY     4    /* Legacy geometry available  */
+#define VIRTIO_BLK_F_BLK_SIZE     6    /* Block size of disk is available

*/
+#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is
available */

examples/vdpa/vdpa_blk_compact.h new file +6 -0
+#define VIRTIO_BLK_F_MQ           12   /* support more than one vq */
+
+/* Legacy feature bits */
+#ifndef VIRTIO_BLK_NO_LEGACY
+#define VIRTIO_BLK_F_BARRIER      0    /* Does host support barriers? */
+#define VIRTIO_BLK_F_SCSI         7    /* Supports scsi command passthru

*/
+#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in
config */

examples/vdpa/vdpa_blk_compact.h line 36

Quoted 5 lines
examples/vdpa/vdpa_blk_compact.h new file +5 -0
+#endif /* !VIRTIO_BLK_NO_LEGACY */
+
+#ifndef VHOST_USER_F_PROTOCOL_FEATURES
+#define VHOST_USER_F_PROTOCOL_FEATURES 30
+#endif

It's already in rte_vhost.h, so no need to re-define.

Thanks,
Chenbo

Quoted 35 lines
examples/vdpa/vdpa_blk_compact.h new file +9 -0
+
+#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \
+	(1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \
+	(1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \
+	(1ULL << VIRTIO_RING_F_EVENT_IDX) | \
+	(1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \
+	(1ULL << VIRTIO_F_VERSION_1))
+
+#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL <<

VIRTIO_F_NOTIFY_ON_EMPTY) | \

examples/vdpa/vdpa_blk_compact.h new file +5 -0
+	(1ULL << VIRTIO_RING_F_EVENT_IDX))
+
+#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \
+	(1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \
+	(1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) |

\

examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER)  |

\

examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_SCSI)     | (1ULL << VIRTIO_BLK_F_CONFIG_WCE)

| \

examples/vdpa/vdpa_blk_compact.h new file +13 -1
+	(1ULL << VIRTIO_BLK_F_MQ))
+
+/* Not supported features */
+#define VHOST_BLK_DISABLED_FEATURES (VHOST_BLK_DISABLED_FEATURES_BASE | \
+	(1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \
+	(1ULL << VIRTIO_BLK_F_SCSI)  | (1ULL << VIRTIO_BLK_F_CONFIG_WCE))
+
+/* Vhost-blk support protocol features */
+#define VHOST_BLK_PROTOCOL_FEATURES \
+	((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \
+	(1ULL << VHOST_USER_PROTOCOL_F_CONFIG))
+
+#endif /* _VDPA_BLK_COMPACT_H_ */
--

1.8.3.1

author
#136 lore raw

RE: [PATCH v7 09/18] examples/vdpa: add vDPA blk support in example

HI Chenbo,

Thanks for your reply.
My reply is inline.

Quoted 19 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:34 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 09/18] examples/vdpa: add vDPA blk support in example

Quoted 9 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 09/18] examples/vdpa: add vDPA blk support in example

Quoted 2 lines

Better be: examples/vdpa: add virtio blk support

Sure.

Quoted 62 lines
Quoted 6 lines

Add virtio blk device support to vDPA example.

---
 examples/vdpa/main.c             | 57

+++++++++++++++++++++++++++++++++++

Quoted 55 lines
 examples/vdpa/vdpa_blk_compact.h | 65
++++++++++++++++++++++++++++++++++++++++
 2 files changed, 122 insertions(+)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c index renamed from examples/vdpa/main.c +0 -0

5ab0765..2544141 100644

examples/vdpa/main.c +33 -0
@@ -20,6 +20,7 @@
 #include <cmdline_parse_string.h>
 #include <cmdline_parse_num.h>
 #include <cmdline.h>
+#include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
 #define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,54 @@ struct vdpa_port {  };
 
 static int
+vdpa_blk_device_set_features_and_protocol(const char *path) {
+	uint64_t protocol_features = 0;
+	int ret;
+
+	ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_disable_features(path,
+		VHOST_BLK_DISABLED_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_disable_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_get_protocol_features(path,
 &protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_get_protocol_features for %s
 failed.\n",
+			path);
+		goto out;
+	}
+
+	protocol_features |= VHOST_BLK_PROTOCOL_FEATURES;
+
+	ret = rte_vhost_driver_set_protocol_features(path,
 protocol_features);
Quoted 9 lines

examples/vdpa/main.c struct vdpa_port { · line 173

Quoted 6 lines
examples/vdpa/main.c +5 -0
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_protocol_features for %s
 failed.\n",
+			path);
+		goto out;

This goto is not needed.

Yes, you are right. I will fix it.

Quoted 17 lines
Quoted 17 lines
examples/vdpa/main.c +8 -0
+	}
+
+out:
+	return ret;
+}
+
+static int
 start_vdpa(struct vdpa_port *vport)
 {
+	uint32_t device_type = 0;
 	int ret;
 	char *socket_path = vport->ifname;
@@ -192,6 +239,16 @@ struct vdpa_port {
 			"attach vdpa device failed: %s\n",
 			socket_path);
 
Quoted 7 lines

examples/vdpa/main.c struct vdpa_port { · line 242

Quoted 1 line
examples/vdpa/main.c +1 -0
+	ret = rte_vhost_driver_get_vdpa_dev_type(socket_path,

&device_type);

examples/vdpa/main.c struct vdpa_port { · line 244

Quoted 2 lines
examples/vdpa/main.c +2 -0
+	if (ret == 0 && device_type == VDPA_DEVICE_TYPE_BLK) {
+		RTE_LOG(NOTICE, VDPA, "is a blk device\n");

Should add info of socket path

Yes, you are right. I will fix it.

Quoted 28 lines

examples/vdpa/main.c struct vdpa_port { · line 245

Quoted 1 line
examples/vdpa/main.c +1 -0
+		ret =

vdpa_blk_device_set_features_and_protocol(socket_path);

Quoted 26 lines
examples/vdpa/main.c +6 -0
+		if (ret != 0)
+			rte_exit(EXIT_FAILURE,
+				"set vhost blk driver features and protocol
 features failed: %s\n",
+				socket_path);
+	}
+
 	if (rte_vhost_driver_start(socket_path) < 0)
 		rte_exit(EXIT_FAILURE,

"start vhost driver failed: %s\n", diff --git
a/examples/vdpa/vdpa_blk_compact.h b/examples/vdpa/vdpa_blk_compact.h
new file mode 100644
index 0000000..136c3f6

examples/vdpa/vdpa_blk_compact.h new file +9 -0
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation  */
+
+#ifndef _VDPA_BLK_COMPACT_H_
+#define _VDPA_BLK_COMPACT_H_
+
+/**
+ * @file
+ *
Quoted 9 lines

examples/vdpa/vdpa_blk_compact.h line 16

Quoted 6 lines
examples/vdpa/vdpa_blk_compact.h new file +6 -0
+ * Device specific vhost lib
+ */
+
+#include <stdbool.h>
+
+#include <rte_pci.h>

above two headers are not used in this file?

I will remove these two include.

Quoted 30 lines

examples/vdpa/vdpa_blk_compact.h

Quoted 4 lines
examples/vdpa/vdpa_blk_compact.h new file +4 -0
+#include <rte_vhost.h>
+
+/* Feature bits */
+#define VIRTIO_BLK_F_SIZE_MAX     1    /* Indicates maximum segment

size

Quoted 2 lines

*/
+#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of

segments

Quoted 15 lines

*/

examples/vdpa/vdpa_blk_compact.h new file +2 -0
+#define VIRTIO_BLK_F_GEOMETRY     4    /* Legacy geometry available  */
+#define VIRTIO_BLK_F_BLK_SIZE     6    /* Block size of disk is available

*/
+#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is
available */

examples/vdpa/vdpa_blk_compact.h new file +6 -0
+#define VIRTIO_BLK_F_MQ           12   /* support more than one vq */
+
+/* Legacy feature bits */
+#ifndef VIRTIO_BLK_NO_LEGACY
+#define VIRTIO_BLK_F_BARRIER      0    /* Does host support barriers? */
+#define VIRTIO_BLK_F_SCSI         7    /* Supports scsi command passthru

*/
+#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in
config */

examples/vdpa/vdpa_blk_compact.h

Quoted 4 lines
examples/vdpa/vdpa_blk_compact.h new file +4 -0
+#endif /* !VIRTIO_BLK_NO_LEGACY */
+
+#ifndef VHOST_USER_F_PROTOCOL_FEATURES #define
+VHOST_USER_F_PROTOCOL_FEATURES 30 #endif

It's already in rte_vhost.h, so no need to re-define.

Quoted 3 lines

Thanks,
Chenbo

Yes, you are right. I will fix it.

Quoted 43 lines
Quoted 10 lines
examples/vdpa/vdpa_blk_compact.h new file +9 -0
+
+#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \
+	(1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \
+	(1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \
+	(1ULL << VIRTIO_RING_F_EVENT_IDX) | \
+	(1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \
+	(1ULL << VIRTIO_F_VERSION_1))
+
+#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL <<

VIRTIO_F_NOTIFY_ON_EMPTY) | \

examples/vdpa/vdpa_blk_compact.h

Quoted 4 lines
examples/vdpa/vdpa_blk_compact.h new file +4 -0
+	(1ULL << VIRTIO_RING_F_EVENT_IDX))
+
+#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \
+	(1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL <<

VIRTIO_BLK_F_SEG_MAX) | \

examples/vdpa/vdpa_blk_compact.h

Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL <<

VIRTIO_BLK_F_BLK_SIZE) |

examples/vdpa/vdpa_blk_compact.h

Quoted 2 lines

\

examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL <<

VIRTIO_BLK_F_BARRIER) |

examples/vdpa/vdpa_blk_compact.h

Quoted 2 lines

\

examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_SCSI)     | (1ULL <<

VIRTIO_BLK_F_CONFIG_WCE)

examples/vdpa/vdpa_blk_compact.h

Quoted 5 lines

| \

examples/vdpa/vdpa_blk_compact.h new file +4 -0
+	(1ULL << VIRTIO_BLK_F_MQ))
+
+/* Not supported features */
+#define VHOST_BLK_DISABLED_FEATURES

(VHOST_BLK_DISABLED_FEATURES_BASE | \

examples/vdpa/vdpa_blk_compact.h

Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL <<

VIRTIO_BLK_F_BARRIER) | \

examples/vdpa/vdpa_blk_compact.h

Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_SCSI)  | (1ULL <<

VIRTIO_BLK_F_CONFIG_WCE))

Quoted 9 lines
examples/vdpa/vdpa_blk_compact.h new file +7 -1
+
+/* Vhost-blk support protocol features */ #define
+VHOST_BLK_PROTOCOL_FEATURES \
+	((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \
+	(1ULL << VHOST_USER_PROTOCOL_F_CONFIG))
+
+#endif /* _VDPA_BLK_COMPACT_H_ */
--

1.8.3.1

author
#137 lore raw

PATCH v7 10/18 usertools: add support for virtio blk device

In reply to Andy Pei

Add virtio blk device support to devbind.

---
 usertools/dpdk-devbind.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-devbind.py +4 -1
index ace4627..7231be4 100755
@@ -72,6 +72,9 @@
 cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4',
                  'SVendor': None, 'SDevice': None}
 
+virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001',
+                    'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 baseband_devices = [acceleration_class]
 crypto_devices = [encryption_class, intel_processor_class]
@@ -82,7 +85,7 @@
 compress_devices = [cavium_zip]
 regex_devices = [cn9k_ree]
 misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev,
-                intel_ntb_skx, intel_ntb_icx]
+                intel_ntb_skx, intel_ntb_icx, virtio_blk]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
Signature · 1 line
1.8.3.1
author
#138 lore raw

PATCH v7 11/18 vdpa/ifc: add set vring state for blk device

In reply to Andy Pei

Set_vring_state op is mandatory, add set_vring_state for blk device. Currently set_vring_state for blk device is not implemented.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +11 -1
index 99a6ab0..ca49bc3 100644
@@ -1290,6 +1290,16 @@ struct rte_vdpa_dev_info {
 }
 
 static int
+ifcvf_blk_set_vring_state(int vid, int vring, int state)
+{
+	RTE_SET_USED(vid);
+	RTE_SET_USED(vring);
+	RTE_SET_USED(state);
+
+	return 0;
+}
+
+static int
 ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev,
 	uint64_t *features)
 {
@@ -1316,7 +1326,7 @@ struct rte_vdpa_dev_info {
 	.get_protocol_features = ifcvf_blk_get_protocol_features,
 	.dev_conf = ifcvf_dev_config,
 	.dev_close = ifcvf_dev_close,
-	.set_vring_state = NULL,
+	.set_vring_state = ifcvf_blk_set_vring_state,
 	.migration_done = NULL,
 	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
Signature · 1 line
1.8.3.1
#139 lore raw

RE: [PATCH v7 11/18] vdpa/ifc: add set vring state for blk device

Quoted 38 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 11/18] vdpa/ifc: add set vring state for blk device

Set_vring_state op is mandatory, add set_vring_state for blk device. Currently set_vring_state for blk device is not implemented.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +10 -0
index 99a6ab0..ca49bc3 100644
@@ -1290,6 +1290,16 @@ struct rte_vdpa_dev_info {
 }
 
 static int
+ifcvf_blk_set_vring_state(int vid, int vring, int state)
+{
+	RTE_SET_USED(vid);
+	RTE_SET_USED(vring);
+	RTE_SET_USED(state);
+
+	return 0;
+}
+
+static int
 ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev,
 	uint64_t *features)
 {

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1329

Quoted 5 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -1
@@ -1316,7 +1326,7 @@ struct rte_vdpa_dev_info {
 	.get_protocol_features = ifcvf_blk_get_protocol_features,
 	.dev_conf = ifcvf_dev_config,
 	.dev_close = ifcvf_dev_close,
-	.set_vring_state = NULL,
+	.set_vring_state = ifcvf_blk_set_vring_state,

About naming in this driver, if you use two functions for net/blk and name the one for blk as XXX_blk_XXX. Better rename the net one as XXX_net_XXX.

In this case, better to rename ifcvf_set_vring_state

Thanks,
Chenbo

Quoted 5 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -1
 	.migration_done = NULL,
 	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
--

1.8.3.1

author
#140 lore raw

PATCH v7 12/18 vdpa/ifc: add some log at vDPA launch before qemu connect

In reply to Andy Pei

Add some log of virtio blk device config space information at VDPA launch before qemu connects.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +28 -0
index ca49bc3..4060a44 100644
@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
 	int16_t device_id;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+	uint32_t i;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = {
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_BLK].features;
+
+		/* cannot read 64-bit register in one attempt,
+		 * so read byte by byte.
+		 */
+		for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+			byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+			capacity |= (uint64_t)*byte << (i * 8);
+		}
+		DRV_LOG(INFO, "capacity  : %"PRIu64"G", capacity >> 21);
+
+		DRV_LOG(INFO, "size_max  : 0x%08x",
+			internal->hw.blk_cfg->size_max);
+		DRV_LOG(INFO, "seg_max   : 0x%08x",
+			internal->hw.blk_cfg->seg_max);
+		DRV_LOG(INFO, "blk_size  : 0x%08x",
+			internal->hw.blk_cfg->blk_size);
+		DRV_LOG(INFO, "geometry");
+		DRV_LOG(INFO, "    cylinders: %u",
+			internal->hw.blk_cfg->geometry.cylinders);
+		DRV_LOG(INFO, "    heads    : %u",
+			internal->hw.blk_cfg->geometry.heads);
+		DRV_LOG(INFO, "    sectors  : %u",
+			internal->hw.blk_cfg->geometry.sectors);
+		DRV_LOG(INFO, "num_queues: 0x%08x",
+			internal->hw.blk_cfg->num_queues);
 	}
 
 	list->internal = internal;
Signature · 1 line
1.8.3.1
#141 lore raw

RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Quoted 8 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Title can be:

vdpa/ifc: add log for config space of virtio blk

Quoted 26 lines

Add some log of virtio blk device config space information at VDPA launch before qemu connects.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +3 -0
index ca49bc3..4060a44 100644
@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
 	int16_t device_id;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+	uint32_t i;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = {
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1443

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9 -0
 		internal->features |= dev_info[IFCVF_BLK].features;
+
+		/* cannot read 64-bit register in one attempt,
+		 * so read byte by byte.
+		 */
+		for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+			byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+			capacity |= (uint64_t)*byte << (i * 8);
+		}
+		DRV_LOG(INFO, "capacity  : %"PRIu64"G", capacity >> 21);

I believe this '21' should be calculated rather than hard-code.

And should these log all be debug log?

Thanks,
Chenbo

Quoted 21 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +16 -1
+
+		DRV_LOG(INFO, "size_max  : 0x%08x",
+			internal->hw.blk_cfg->size_max);
+		DRV_LOG(INFO, "seg_max   : 0x%08x",
+			internal->hw.blk_cfg->seg_max);
+		DRV_LOG(INFO, "blk_size  : 0x%08x",
+			internal->hw.blk_cfg->blk_size);
+		DRV_LOG(INFO, "geometry");
+		DRV_LOG(INFO, "    cylinders: %u",
+			internal->hw.blk_cfg->geometry.cylinders);
+		DRV_LOG(INFO, "    heads    : %u",
+			internal->hw.blk_cfg->geometry.heads);
+		DRV_LOG(INFO, "    sectors  : %u",
+			internal->hw.blk_cfg->geometry.sectors);
+		DRV_LOG(INFO, "num_queues: 0x%08x",
+			internal->hw.blk_cfg->num_queues);
 	}
 
 	list->internal = internal;
--

1.8.3.1

author
#142 lore raw

RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Hi Chenbo,

Thanks for your reply.
My reply is inline.

Quoted 19 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:53 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Quoted 9 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Quoted 4 lines

Title can be:

vdpa/ifc: add log for config space of virtio blk

Sure.

Quoted 27 lines
Quoted 24 lines

Add some log of virtio blk device config space information at VDPA launch before qemu connects.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index ca49bc3..4060a44 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +3 -0
@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
 	int16_t device_id;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+	uint32_t i;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1433

Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = {
 		internal->features = features &
 					~(1ULL <<

VIRTIO_F_IOMMU_PLATFORM);

Quoted 13 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1443

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9 -0
 		internal->features |= dev_info[IFCVF_BLK].features;
+
+		/* cannot read 64-bit register in one attempt,
+		 * so read byte by byte.
+		 */
+		for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+			byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+			capacity |= (uint64_t)*byte << (i * 8);
+		}
+		DRV_LOG(INFO, "capacity  : %"PRIu64"G", capacity >> 21);

I believe this '21' should be calculated rather than hard-code.

/* The capacity (in 512-byte sectors). */
So right shift 1 we gwt in K. Another right shift 10 we get in M, right shift 10 more bits, we get in G. 10 + 10 + 1 = 21.
I think add some comments in code is fine, what do you think?

Quoted 2 lines

And should these log all be debug log?

These information is hardware ability. I think INFO is fine.

Quoted 25 lines

Thanks,
Chenbo

Quoted 21 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +16 -1
+
+		DRV_LOG(INFO, "size_max  : 0x%08x",
+			internal->hw.blk_cfg->size_max);
+		DRV_LOG(INFO, "seg_max   : 0x%08x",
+			internal->hw.blk_cfg->seg_max);
+		DRV_LOG(INFO, "blk_size  : 0x%08x",
+			internal->hw.blk_cfg->blk_size);
+		DRV_LOG(INFO, "geometry");
+		DRV_LOG(INFO, "    cylinders: %u",
+			internal->hw.blk_cfg->geometry.cylinders);
+		DRV_LOG(INFO, "    heads    : %u",
+			internal->hw.blk_cfg->geometry.heads);
+		DRV_LOG(INFO, "    sectors  : %u",
+			internal->hw.blk_cfg->geometry.sectors);
+		DRV_LOG(INFO, "num_queues: 0x%08x",
+			internal->hw.blk_cfg->num_queues);
 	}
 
 	list->internal = internal;
--

1.8.3.1

#143 lore raw

RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Quoted 79 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Friday, May 13, 2022 4:35 PM
To: Xia, Chenbo <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Hi Chenbo,

Thanks for your reply.
My reply is inline.

Quoted 7 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:53 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch

before

Quoted 12 lines

qemu connect

Quoted 9 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Quoted 4 lines

Title can be:

vdpa/ifc: add log for config space of virtio blk

Sure.

Quoted 27 lines
Quoted 24 lines

Add some log of virtio blk device config space information at VDPA launch before qemu connects.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index ca49bc3..4060a44 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +3 -0
@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 struct rte_kvargs *kvlist = NULL;
 int ret = 0;
 int16_t device_id;
+uint64_t capacity = 0;
+uint8_t *byte;
+uint32_t i;
 
 if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 return 0;

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1433

Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = {
 internal->features = features &
 ~(1ULL <<

VIRTIO_F_IOMMU_PLATFORM);

Quoted 13 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1443

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9 -0
 internal->features |= dev_info[IFCVF_BLK].features;
+
+/* cannot read 64-bit register in one attempt,
+ * so read byte by byte.
+ */
+for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+capacity |= (uint64_t)*byte << (i * 8);
+}
+DRV_LOG(INFO, "capacity  : %"PRIu64"G", capacity >> 21);

I believe this '21' should be calculated rather than hard-code.

Quoted 5 lines

/* The capacity (in 512-byte sectors). */
So right shift 1 we gwt in K. Another right shift 10 we get in M, right shift 10 more bits, we get in G.
10 + 10 + 1 = 21.
I think add some comments in code is fine, what do you think?

Ok. Work for me.

Thanks,
Chenbo

Quoted 31 lines
Quoted 3 lines

And should these log all be debug log?

These information is hardware ability. I think INFO is fine.

Quoted 25 lines

Thanks,
Chenbo

Quoted 21 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +16 -1
+
+DRV_LOG(INFO, "size_max  : 0x%08x",
+internal->hw.blk_cfg->size_max);
+DRV_LOG(INFO, "seg_max   : 0x%08x",
+internal->hw.blk_cfg->seg_max);
+DRV_LOG(INFO, "blk_size  : 0x%08x",
+internal->hw.blk_cfg->blk_size);
+DRV_LOG(INFO, "geometry");
+DRV_LOG(INFO, "    cylinders: %u",
+internal->hw.blk_cfg->geometry.cylinders);
+DRV_LOG(INFO, "    heads    : %u",
+internal->hw.blk_cfg->geometry.heads);
+DRV_LOG(INFO, "    sectors  : %u",
+internal->hw.blk_cfg->geometry.sectors);
+DRV_LOG(INFO, "num_queues: 0x%08x",
+internal->hw.blk_cfg->num_queues);
 }
 
 list->internal = internal;
--

1.8.3.1

#144 lore raw

Re: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

In reply to Pei, Andy
Quoted 24 lines · Andy

Hi Chenbo,

Thanks for your reply.
My reply is inline.

Quoted 19 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:53 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Quoted 9 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Quoted 44 lines
Quoted 4 lines

Title can be:

vdpa/ifc: add log for config space of virtio blk

Sure.

Quoted 39 lines
Quoted 24 lines

Add some log of virtio blk device config space information at VDPA launch before qemu connects.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index ca49bc3..4060a44 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +3 -0
@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
 	int16_t device_id;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+	uint32_t i;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1433

Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = {
 		internal->features = features &
 					~(1ULL <<  

VIRTIO_F_IOMMU_PLATFORM);

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1443

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9 -0
 		internal->features |= dev_info[IFCVF_BLK].features;
+
+		/* cannot read 64-bit register in one attempt,
+		 * so read byte by byte.
+		 */
+		for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+			byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+			capacity |= (uint64_t)*byte << (i * 8);
+		}
+		DRV_LOG(INFO, "capacity  : %"PRIu64"G", capacity >> 21);  

I believe this '21' should be calculated rather than hard-code.

Sounds like a debug not info message.
It is meant for developer not end user.

author
#145 lore raw

RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Hi Stephen,

Thanks for your reply.
I will change to DEBUG level log in next version.

Quoted 84 lines

-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Friday, May 13, 2022 11:38 PM
To: Pei, Andy <[email protected]>
Cc: Xia, Chenbo <[email protected]>; [email protected]; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: Re: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Quoted 24 lines · Andy

Hi Chenbo,

Thanks for your reply.
My reply is inline.

Quoted 19 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:53 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Quoted 9 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect

Quoted 44 lines
Quoted 4 lines

Title can be:

vdpa/ifc: add log for config space of virtio blk

Sure.

Quoted 39 lines
Quoted 24 lines

Add some log of virtio blk device config space information at VDPA launch before qemu connects.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index ca49bc3..4060a44 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +3 -0
@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
 	int16_t device_id;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+	uint32_t i;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1433

Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = {
 		internal->features = features &
 					~(1ULL <<

VIRTIO_F_IOMMU_PLATFORM);

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1443

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9 -0
 		internal->features |= dev_info[IFCVF_BLK].features;
+
+		/* cannot read 64-bit register in one attempt,
+		 * so read byte by byte.
+		 */
+		for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+			byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+			capacity |= (uint64_t)*byte << (i * 8);
+		}
+		DRV_LOG(INFO, "capacity  : %"PRIu64"G", capacity >> 21);

I believe this '21' should be calculated rather than hard-code.

Sounds like a debug not info message.
It is meant for developer not end user.

author
#146 lore raw

PATCH v7 13/18 vdpa/ifc: read virtio max queues from hardware

In reply to Andy Pei

Original code max_queues is set to IFCVF_MAX_QUEUES.
New code max_queues is the min of IFCVF_MAX_QUEUES and hardware num_queues.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
index 4060a44..5a8cf1c 100644
@@ -1457,6 +1457,10 @@ struct rte_vdpa_dev_info dev_info[] = {
 			internal->hw.blk_cfg->geometry.sectors);
 		DRV_LOG(INFO, "num_queues: 0x%08x",
 			internal->hw.blk_cfg->num_queues);
+
+		/* reset max_queue here, to minimum modification */
+		internal->max_queues = RTE_MIN(IFCVF_MAX_QUEUES,
+			internal->hw.blk_cfg->num_queues);
 	}
 
 	list->internal = internal;
Signature · 1 line
1.8.3.1
#147 lore raw

RE: [PATCH v7 13/18] vdpa/ifc: read virtio max queues from hardware

Quoted 21 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 13/18] vdpa/ifc: read virtio max queues from hardware

Original code max_queues is set to IFCVF_MAX_QUEUES.
New code max_queues is the min of IFCVF_MAX_QUEUES and hardware num_queues.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
index 4060a44..5a8cf1c 100644

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1463

Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
@@ -1457,6 +1457,10 @@ struct rte_vdpa_dev_info dev_info[] = {
 			internal->hw.blk_cfg->geometry.sectors);
 		DRV_LOG(INFO, "num_queues: 0x%08x",
 			internal->hw.blk_cfg->num_queues);
+
+		/* reset max_queue here, to minimum modification */
+		internal->max_queues = RTE_MIN(IFCVF_MAX_QUEUES,
+			internal->hw.blk_cfg->num_queues);

MQ is not supported now in this driver, should we make this change later?

Thanks,
Chenbo

Quoted 5 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -1
 	}
 
 	list->internal = internal;
--

1.8.3.1

author
#148 lore raw

RE: [PATCH v7 13/18] vdpa/ifc: read virtio max queues from hardware

Hi Chenbo,

Thanks for your reply.
This patch will be omitted in next version.

Quoted 52 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:55 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 13/18] vdpa/ifc: read virtio max queues from hardware

Quoted 10 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 13/18] vdpa/ifc: read virtio max queues from hardware

Quoted 2 lines

Original code max_queues is set to IFCVF_MAX_QUEUES. New code max_queues is the min of IFCVF_MAX_QUEUES and hardware

num_queues.

Quoted 11 lines
---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++
 1 file changed, 4 insertions(+)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index 4060a44..5a8cf1c 100644

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1463

Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
@@ -1457,6 +1457,10 @@ struct rte_vdpa_dev_info dev_info[] = {
 			internal->hw.blk_cfg->geometry.sectors);
 		DRV_LOG(INFO, "num_queues: 0x%08x",
 			internal->hw.blk_cfg->num_queues);
+
+		/* reset max_queue here, to minimum modification */
+		internal->max_queues = RTE_MIN(IFCVF_MAX_QUEUES,
+			internal->hw.blk_cfg->num_queues);

MQ is not supported now in this driver, should we make this change later?

Thanks,
Chenbo

Quoted 5 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -1
 	}
 
 	list->internal = internal;
--

1.8.3.1

author
#149 lore raw

PATCH v7 14/18 vdpa/ifc: add interrupt and handle for virtio blk

In reply to Andy Pei

Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to info qemu.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 112 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 112 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +112 -0
index 5a8cf1c..0e94e1f 100644
@@ -53,7 +53,9 @@ struct ifcvf_internal {
 	int vfio_group_fd;
 	int vfio_dev_fd;
 	pthread_t tid;	/* thread for notify relay */
+	pthread_t intr_tid;	/* thread for intr relay */
 	int epfd;
+	int csc_fd;
 	int vid;
 	struct rte_vdpa_device *vdev;
 	uint16_t max_queues;
@@ -558,6 +560,107 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static void
+virtio_interrupt_handler(struct ifcvf_internal *internal)
+{
+	int vid = internal->vid;
+	int ret;
+
+	ret = rte_vhost_slave_config_change(vid, 1);
+	if (ret)
+		DRV_LOG(ERR, "failed to notify the guest about configuration space change.");
+}
+
+static void *
+intr_relay(void *arg)
+{
+	struct ifcvf_internal *internal = (struct ifcvf_internal *)arg;
+	struct epoll_event csc_event;
+	struct epoll_event ev;
+	uint64_t buf;
+	int nbytes;
+	int csc_fd, csc_val = 0;
+
+	csc_fd = epoll_create(1);
+	if (csc_fd < 0) {
+		DRV_LOG(ERR, "failed to create epoll for config space change.");
+		return NULL;
+	}
+
+	ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP;
+	ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle);
+	if (epoll_ctl(csc_fd, EPOLL_CTL_ADD,
+		rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) {
+		DRV_LOG(ERR, "epoll add error: %s", strerror(errno));
+		return NULL;
+	}
+
+	internal->csc_fd = csc_fd;
+
+	for (;;) {
+		csc_val = epoll_wait(csc_fd, &csc_event, 1, -1);
+		if (csc_val < 0) {
+			if (errno == EINTR)
+				continue;
+			DRV_LOG(ERR, "epoll_wait return fail\n");
+			return NULL;
+		} else if (csc_val == 0) {
+			continue;
+		} else {
+			/* csc_val > 0 */
+			nbytes = read(csc_event.data.fd, &buf, 8);
+			if (nbytes < 0) {
+				if (errno == EINTR || errno == EWOULDBLOCK)
+					continue;
+				DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n",
+					csc_event.data.fd,
+					strerror(errno));
+				return NULL;
+			} else if (nbytes == 0) {
+				DRV_LOG(ERR, "Read nothing from file descriptor %d\n",
+					csc_event.data.fd);
+				continue;
+			} else {
+				virtio_interrupt_handler(internal);
+			}
+		}
+	}
+
+	return NULL;
+}
+
+static int
+setup_intr_relay(struct ifcvf_internal *internal)
+{
+	int ret;
+
+	ret = pthread_create(&internal->intr_tid, NULL, intr_relay,
+			(void *)internal);
+	if (ret) {
+		DRV_LOG(ERR, "failed to create notify relay pthread.");
+		return -1;
+	}
+	return 0;
+}
+
+static int
+unset_intr_relay(struct ifcvf_internal *internal)
+{
+	void *status;
+
+	if (internal->intr_tid) {
+		pthread_cancel(internal->intr_tid);
+		pthread_join(internal->intr_tid, &status);
+	}
+	internal->intr_tid = 0;
+
+	if (internal->csc_fd >= 0)
+		close(internal->csc_fd);
+	internal->csc_fd = -1;
+
+	return 0;
+}
+
 static int
 update_datapath(struct ifcvf_internal *internal)
 {
@@ -584,10 +687,16 @@ struct rte_vdpa_dev_info {
 		if (ret)
 			goto err;
 
+		ret = setup_intr_relay(internal);
+		if (ret)
+			goto err;
+
 		rte_atomic32_set(&internal->running, 1);
 	} else if (rte_atomic32_read(&internal->running) &&
 		   (!rte_atomic32_read(&internal->started) ||
 		    !rte_atomic32_read(&internal->dev_attached))) {
+		ret = unset_intr_relay(internal);
+
 		ret = unset_notify_relay(internal);
 		if (ret)
 			goto err;
@@ -880,6 +989,9 @@ struct rte_vdpa_dev_info {
 	/* stop the direct IO data path */
 	unset_notify_relay(internal);
 	vdpa_ifcvf_stop(internal);
+
+	unset_intr_relay(internal);
+
 	vdpa_disable_vfio_intr(internal);
 
 	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
Signature · 1 line
1.8.3.1
#150 lore raw

RE: [PATCH v7 14/18] vdpa/ifc: add interrupt and handle for virtio blk

Quoted 8 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 14/18] vdpa/ifc: add interrupt and handle for virtio blk

Better be: vdpa/ifc: add interrupt handling for config space

Quoted 3 lines

Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to info qemu.

Inform QEMU. You don't need to save words in commit log. The commit log should be as detailed as possible to make readers understand quickly what the commit is doing :)

Quoted 12 lines
---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 112
++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 112 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
index 5a8cf1c..0e94e1f 100644

drivers/vdpa/ifc/ifcvf_vdpa.c struct ifcvf_internal { · line 56

Quoted 4 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
@@ -53,7 +53,9 @@ struct ifcvf_internal {
 	int vfio_group_fd;
 	int vfio_dev_fd;
 	pthread_t tid;	/* thread for notify relay */
+	pthread_t intr_tid;	/* thread for intr relay */

Thread for virtio-blk config space change interrupt relay

drivers/vdpa/ifc/ifcvf_vdpa.c struct ifcvf_internal { · line 58

Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
 	int epfd;
+	int csc_fd;

csc_epfd

Quoted 32 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +24 -0
 	int vid;
 	struct rte_vdpa_device *vdev;
 	uint16_t max_queues;
@@ -558,6 +560,107 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static void
+virtio_interrupt_handler(struct ifcvf_internal *internal)
+{
+	int vid = internal->vid;
+	int ret;
+
+	ret = rte_vhost_slave_config_change(vid, 1);
+	if (ret)
+		DRV_LOG(ERR, "failed to notify the guest about configuration
 space change.");
+}
+
+static void *
+intr_relay(void *arg)
+{
+	struct ifcvf_internal *internal = (struct ifcvf_internal *)arg;
+	struct epoll_event csc_event;
+	struct epoll_event ev;
+	uint64_t buf;
+	int nbytes;
+	int csc_fd, csc_val = 0;
+
+	csc_fd = epoll_create(1);
+	if (csc_fd < 0) {
+		DRV_LOG(ERR, "failed to create epoll for config space

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 597

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9 -0
 change.");
+		return NULL;
+	}
+
+	ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP;
+	ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle);
+	if (epoll_ctl(csc_fd, EPOLL_CTL_ADD,
+		rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) {
+		DRV_LOG(ERR, "epoll add error: %s", strerror(errno));
+		return NULL;

Close the epfd and set to -1 if err.

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 605

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10 -0
+	}
+
+	internal->csc_fd = csc_fd;
+
+	for (;;) {
+		csc_val = epoll_wait(csc_fd, &csc_event, 1, -1);
+		if (csc_val < 0) {
+			if (errno == EINTR)
+				continue;
+			DRV_LOG(ERR, "epoll_wait return fail\n");

Save '\n', it's not needed for DRV_LOG. Please check other DRV_LOGs

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 613

Quoted 8 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +8 -0
+			return NULL;
+		} else if (csc_val == 0) {
+			continue;
+		} else {
+			/* csc_val > 0 */
+			nbytes = read(csc_event.data.fd, &buf, 8);
+			if (nbytes < 0) {
+				if (errno == EINTR || errno == EWOULDBLOCK)

EAGAIN should also be this case?

Quoted 17 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7 -0
+					continue;
+				DRV_LOG(ERR, "Error reading from file
 descriptor %d: %s\n",
+					csc_event.data.fd,
+					strerror(errno));
+				return NULL;
+			} else if (nbytes == 0) {
+				DRV_LOG(ERR, "Read nothing from file

descriptor %d\n",

drivers/vdpa/ifc/ifcvf_vdpa.c +8 -0
+					csc_event.data.fd);
+				continue;
+			} else {
+				virtio_interrupt_handler(internal);
+			}
+		}
+	}
+

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 638

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10 -0
+	return NULL;
+}
+
+static int
+setup_intr_relay(struct ifcvf_internal *internal)
+{
+	int ret;
+
+	ret = pthread_create(&internal->intr_tid, NULL, intr_relay,
+			(void *)internal);

EAL API: rte_ctrl_thread_create, will be preferred.

Quoted 29 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +25 -0
+	if (ret) {
+		DRV_LOG(ERR, "failed to create notify relay pthread.");
+		return -1;
+	}
+	return 0;
+}
+
+static int
+unset_intr_relay(struct ifcvf_internal *internal)
+{
+	void *status;
+
+	if (internal->intr_tid) {
+		pthread_cancel(internal->intr_tid);
+		pthread_join(internal->intr_tid, &status);
+	}
+	internal->intr_tid = 0;
+
+	if (internal->csc_fd >= 0)
+		close(internal->csc_fd);
+	internal->csc_fd = -1;
+
+	return 0;
+}
+
 static int
 update_datapath(struct ifcvf_internal *internal)
 {

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 693

Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
@@ -584,10 +687,16 @@ struct rte_vdpa_dev_info {
 		if (ret)
 			goto err;
 
+		ret = setup_intr_relay(internal);
+		if (ret)
+			goto err;
+

But this is not needed for net, right? As I said, we should include validation for net also.

Thanks,
Chenbo

Quoted 21 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5 -1
 		rte_atomic32_set(&internal->running, 1);
 	} else if (rte_atomic32_read(&internal->running) &&
 		   (!rte_atomic32_read(&internal->started) ||
 		    !rte_atomic32_read(&internal->dev_attached))) {
+		ret = unset_intr_relay(internal);
+
 		ret = unset_notify_relay(internal);
 		if (ret)
 			goto err;
@@ -880,6 +989,9 @@ struct rte_vdpa_dev_info {
 	/* stop the direct IO data path */
 	unset_notify_relay(internal);
 	vdpa_ifcvf_stop(internal);
+
+	unset_intr_relay(internal);
+
 	vdpa_disable_vfio_intr(internal);
 
 	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
--

1.8.3.1

author
#151 lore raw

RE: [PATCH v7 14/18] vdpa/ifc: add interrupt and handle for virtio blk

HI Chenbo,

Thanks for your reply.
My reply is inline.

Quoted 19 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Friday, May 13, 2022 10:53 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 14/18] vdpa/ifc: add interrupt and handle for virtio blk

Quoted 9 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 14/18] vdpa/ifc: add interrupt and handle for virtio blk

Quoted 2 lines

Better be: vdpa/ifc: add interrupt handling for config space

Sure. I will fix it in next version.

Quoted 8 lines
Quoted 3 lines

Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to info qemu.

Inform QEMU. You don't need to save words in commit log. The commit log should be as detailed as possible to make readers understand quickly what the commit is doing :)

Sure. I will fix it in next version.

Quoted 7 lines
Quoted 7 lines
---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 112
++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 112 insertions(+)
Quoted 12 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct ifcvf_internal { · line 56

Quoted 9 lines
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index 5a8cf1c..0e94e1f 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
@@ -53,7 +53,9 @@ struct ifcvf_internal {
 	int vfio_group_fd;
 	int vfio_dev_fd;
 	pthread_t tid;	/* thread for notify relay */
+	pthread_t intr_tid;	/* thread for intr relay */

Thread for virtio-blk config space change interrupt relay

Sure.

Quoted 5 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct ifcvf_internal { · line 58

Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
 	int epfd;
+	int csc_fd;

csc_epfd

OK.

Quoted 35 lines
Quoted 35 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +26 -0
 	int vid;
 	struct rte_vdpa_device *vdev;
 	uint16_t max_queues;
@@ -558,6 +560,107 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static void
+virtio_interrupt_handler(struct ifcvf_internal *internal) {
+	int vid = internal->vid;
+	int ret;
+
+	ret = rte_vhost_slave_config_change(vid, 1);
+	if (ret)
+		DRV_LOG(ERR, "failed to notify the guest about configuration
 space change.");
+}
+
+static void *
+intr_relay(void *arg)
+{
+	struct ifcvf_internal *internal = (struct ifcvf_internal *)arg;
+	struct epoll_event csc_event;
+	struct epoll_event ev;
+	uint64_t buf;
+	int nbytes;
+	int csc_fd, csc_val = 0;
+
+	csc_fd = epoll_create(1);
+	if (csc_fd < 0) {
+		DRV_LOG(ERR, "failed to create epoll for config space
 change.");
+		return NULL;
+	}
+
Quoted 9 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 595

Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +6 -0
+	ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP;
+	ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle);
+	if (epoll_ctl(csc_fd, EPOLL_CTL_ADD,
+		rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) {
+		DRV_LOG(ERR, "epoll add error: %s", strerror(errno));
+		return NULL;

Close the epfd and set to -1 if err.

I check other epoll usage in DPDK, it seems most usage do not close the epfd and set to -1. I am not sure whether it is needed.

Quoted 4 lines
Quoted 4 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
+	}
+
+	internal->csc_fd = csc_fd;
+
Quoted 9 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 605

Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +6 -0
+	for (;;) {
+		csc_val = epoll_wait(csc_fd, &csc_event, 1, -1);
+		if (csc_val < 0) {
+			if (errno == EINTR)
+				continue;
+			DRV_LOG(ERR, "epoll_wait return fail\n");

Save '\n', it's not needed for DRV_LOG. Please check other DRV_LOGs

OK

Quoted 11 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 613

Quoted 8 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +8 -0
+			return NULL;
+		} else if (csc_val == 0) {
+			continue;
+		} else {
+			/* csc_val > 0 */
+			nbytes = read(csc_event.data.fd, &buf, 8);
+			if (nbytes < 0) {
+				if (errno == EINTR || errno == EWOULDBLOCK)

EAGAIN should also be this case?

Yes, it will be add in next version.

Quoted 20 lines
Quoted 20 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7 -0
+					continue;
+				DRV_LOG(ERR, "Error reading from file
 descriptor %d: %s\n",
+					csc_event.data.fd,
+					strerror(errno));
+				return NULL;
+			} else if (nbytes == 0) {
+				DRV_LOG(ERR, "Read nothing from file

descriptor %d\n",

drivers/vdpa/ifc/ifcvf_vdpa.c +11 -0
+					csc_event.data.fd);
+				continue;
+			} else {
+				virtio_interrupt_handler(internal);
+			}
+		}
+	}
+
+	return NULL;
+}
+
Quoted 9 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 638

Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +6 -0
+static int
+setup_intr_relay(struct ifcvf_internal *internal) {
+	int ret;
+
+	ret = pthread_create(&internal->intr_tid, NULL, intr_relay,
+			(void *)internal);

EAL API: rte_ctrl_thread_create, will be preferred.

Sure, I will use " rte_ctrl_thread_create " in next version.

Quoted 38 lines
Quoted 24 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +24 -0
+	if (ret) {
+		DRV_LOG(ERR, "failed to create notify relay pthread.");
+		return -1;
+	}
+	return 0;
+}
+
+static int
+unset_intr_relay(struct ifcvf_internal *internal) {
+	void *status;
+
+	if (internal->intr_tid) {
+		pthread_cancel(internal->intr_tid);
+		pthread_join(internal->intr_tid, &status);
+	}
+	internal->intr_tid = 0;
+
+	if (internal->csc_fd >= 0)
+		close(internal->csc_fd);
+	internal->csc_fd = -1;
+
+	return 0;
+}
+

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info {

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5 -0
 static int
 update_datapath(struct ifcvf_internal *internal)  { @@ -584,10
+687,16 @@ struct rte_vdpa_dev_info {
 		if (ret)
 			goto err;
 
+		ret = setup_intr_relay(internal);
+		if (ret)
+			goto err;
+

But this is not needed for net, right? As I said, we should include validation for net also.

Quoted 3 lines

Thanks,
Chenbo

For net device, especially the harden virtio device, fabric plug in or out will cause config change. I think net device may also need this interrupt, but I am not sure.

Quoted 23 lines
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +2 -0
 		rte_atomic32_set(&internal->running, 1);
 	} else if (rte_atomic32_read(&internal->running) &&
 		   (!rte_atomic32_read(&internal->started) ||
 		    !rte_atomic32_read(&internal->dev_attached))) {
+		ret = unset_intr_relay(internal);
+
 		ret = unset_notify_relay(internal);
 		if (ret)
 			goto err;

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 997

Quoted 9 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +3 -0
@@ -880,6 +989,9 @@ struct rte_vdpa_dev_info {
 	/* stop the direct IO data path */
 	unset_notify_relay(internal);
 	vdpa_ifcvf_stop(internal);
+
+	unset_intr_relay(internal);
+
 	vdpa_disable_vfio_intr(internal);
 
 	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL,

false);

Quoted 2 lines

--
1.8.3.1

author
#152 lore raw

PATCH v7 15/18 vdpa/ifc: add is blk flag to ifcvf HW struct

In reply to Andy Pei

Add is_blk flag to ifcvf_hw, and init is_blk during probe.

---
 drivers/vdpa/ifc/base/ifcvf.h | 1 +
 drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++
 2 files changed, 3 insertions(+)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
index 769c603..8591ef1 100644
@@ -149,6 +149,7 @@ struct ifcvf_hw {
 	u8     *lm_cfg;
 	struct vring_info vring[IFCVF_MAX_QUEUES * 2];
 	u8 nr_vring;
+	u8 is_blk;
 	struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE];
 };
 
drivers/vdpa/ifc/ifcvf_vdpa.c +2 -0
index 0e94e1f..4923bc1 100644
@@ -1536,11 +1536,13 @@ struct rte_vdpa_dev_info dev_info[] = {
 
 	if (device_id == VIRTIO_ID_NET) {
 		internal->device_type = IFCVF_NET;
+		internal->hw.is_blk = IFCVF_NET;
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_NET].features;
 	} else if (device_id == VIRTIO_ID_BLOCK) {
 		internal->device_type = IFCVF_BLK;
+		internal->hw.is_blk = IFCVF_BLK;
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_BLK].features;
Signature · 1 line
1.8.3.1
#153 lore raw

RE: [PATCH v7 15/18] vdpa/ifc: add is blk flag to ifcvf HW struct

Quoted 33 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 15/18] vdpa/ifc: add is blk flag to ifcvf HW struct

Add is_blk flag to ifcvf_hw, and init is_blk during probe.

---
 drivers/vdpa/ifc/base/ifcvf.h | 1 +
 drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++
 2 files changed, 3 insertions(+)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
index 769c603..8591ef1 100644
@@ -149,6 +149,7 @@ struct ifcvf_hw {
 	u8     *lm_cfg;
 	struct vring_info vring[IFCVF_MAX_QUEUES * 2];
 	u8 nr_vring;
+	u8 is_blk;
 	struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE];
 };
 
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
index 0e94e1f..4923bc1 100644

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1539

Quoted 4 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
@@ -1536,11 +1536,13 @@ struct rte_vdpa_dev_info dev_info[] = {
 
 	if (device_id == VIRTIO_ID_NET) {
 		internal->device_type = IFCVF_NET;
+		internal->hw.is_blk = IFCVF_NET;

I believe it's enough to keep only device_type. Device type can be defined in internal or internal->hw. Choose the way you prefer.

Thanks,
Chenbo

Quoted 11 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -1
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_NET].features;
 	} else if (device_id == VIRTIO_ID_BLOCK) {
 		internal->device_type = IFCVF_BLK;
+		internal->hw.is_blk = IFCVF_BLK;
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_BLK].features;
--

1.8.3.1

author
#154 lore raw

RE: [PATCH v7 15/18] vdpa/ifc: add is blk flag to ifcvf HW struct

Hi Chenbo,

Quoted 51 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Friday, May 13, 2022 10:55 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 15/18] vdpa/ifc: add is blk flag to ifcvf HW struct

Quoted 22 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 15/18] vdpa/ifc: add is blk flag to ifcvf HW struct

Add is_blk flag to ifcvf_hw, and init is_blk during probe.

---
 drivers/vdpa/ifc/base/ifcvf.h | 1 +
 drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++
 2 files changed, 3 insertions(+)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/base/ifcvf.h index 769c603..8591ef1 100644

drivers/vdpa/ifc/base/ifcvf.h struct ifcvf_hw { · line 153

Quoted 5 lines
drivers/vdpa/ifc/base/ifcvf.h +1 -0
@@ -149,6 +149,7 @@ struct ifcvf_hw {
 	u8     *lm_cfg;
 	struct vring_info vring[IFCVF_MAX_QUEUES * 2];
 	u8 nr_vring;
+	u8 is_blk;
 	struct ifcvf_pci_mem_resource

mem_resource[IFCVF_PCI_MAX_RESOURCE];

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1539

Quoted 11 lines
drivers/vdpa/ifc/base/ifcvf.h +0 -0
 };
 
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index 0e94e1f..4923bc1 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
@@ -1536,11 +1536,13 @@ struct rte_vdpa_dev_info dev_info[] = {
 
 	if (device_id == VIRTIO_ID_NET) {
 		internal->device_type = IFCVF_NET;
+		internal->hw.is_blk = IFCVF_NET;

I believe it's enough to keep only device_type. Device type can be defined in internal or internal->hw. Choose the way you prefer.

Quoted 3 lines

Thanks,
Chenbo

Yes, you are right. To keep API not changed, I think it is better to keep only device_type in internal->hw. So I have to make some change to other patch in this patch set.

Quoted 14 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1541

Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
 		internal->features = features &
 					~(1ULL <<

VIRTIO_F_IOMMU_PLATFORM);

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = {

Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
 		internal->features |= dev_info[IFCVF_NET].features;
 	} else if (device_id == VIRTIO_ID_BLOCK) {
 		internal->device_type = IFCVF_BLK;
+		internal->hw.is_blk = IFCVF_BLK;
 		internal->features = features &
 					~(1ULL <<

VIRTIO_F_IOMMU_PLATFORM);

Quoted 3 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0 -1
 		internal->features |= dev_info[IFCVF_BLK].features;
--

1.8.3.1

author
#155 lore raw

PATCH v7 16/18 vdpa/ifc/base: access correct register for blk device

In reply to Andy Pei

1.last_avail_idx is lower 16 bit of the register.
2.address of ring_state register is different between net and blk device.

---
 drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++-------
 drivers/vdpa/ifc/base/ifcvf.h |  1 +
 2 files changed, 30 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +29 -7
index d10c1fd..4d5881a 100644
@@ -218,10 +218,18 @@
 				&cfg->queue_used_hi);
 		IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size);
 
-		*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) =
-			(u32)hw->vring[i].last_avail_idx |
-			((u32)hw->vring[i].last_used_idx << 16);
+		if (hw->is_blk == IFCVF_BLK) {
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				i * IFCVF_LM_CFG_SIZE) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
+		} else if (hw->is_blk == IFCVF_NET) {
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				(i / 2) * IFCVF_LM_CFG_SIZE +
+				(i % 2) * 4) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
+		}
 
 		IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector);
 		if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -254,9 +262,23 @@
 		IFCVF_WRITE_REG16(i, &cfg->queue_select);
 		IFCVF_WRITE_REG16(0, &cfg->queue_enable);
 		IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector);
-		ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4);
-		hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
+
+		if (hw->is_blk) {
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					i * IFCVF_LM_CFG_SIZE);
+		} else if (hw->is_blk == IFCVF_NET) {
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					(i / 2) * IFCVF_LM_CFG_SIZE +
+					(i % 2) * 4);
+		}
+
+		if (hw->is_blk == IFCVF_BLK)
+			hw->vring[i].last_avail_idx =
+				(u16)(ring_state & IFCVF_16_BIT_MASK);
+		else if (hw->is_blk == IFCVF_NET)
+			hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
 		hw->vring[i].last_used_idx = (u16)(ring_state >> 16);
 	}
 }
drivers/vdpa/ifc/base/ifcvf.h +1 -0
index 8591ef1..ff11b12 100644
@@ -65,6 +65,7 @@
 #define IFCVF_MEDIATED_VRING		0x200000000000
 
 #define IFCVF_32_BIT_MASK		0xffffffff
+#define IFCVF_16_BIT_MASK		0xffff
 
 #ifndef VHOST_USER_PROTOCOL_F_CONFIG
 #define VHOST_USER_PROTOCOL_F_CONFIG	9
Signature · 1 line
1.8.3.1
#156 lore raw

RE: [PATCH v7 16/18] vdpa/ifc/base: access correct register for blk device

Quoted 11 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 16/18] vdpa/ifc/base: access correct register for blk device

1.last_avail_idx is lower 16 bit of the register.
2.address of ring_state register is different between net and blk device.

Not a good commit log. The commit log should illustrate more on what's the commit is doing.

Thanks,
Chenbo

Quoted 77 lines
---
 drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++-------
 drivers/vdpa/ifc/base/ifcvf.h |  1 +
 2 files changed, 30 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +30 -7
index d10c1fd..4d5881a 100644
@@ -218,10 +218,18 @@
 				&cfg->queue_used_hi);
 		IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size);
 
-		*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) =
-			(u32)hw->vring[i].last_avail_idx |
-			((u32)hw->vring[i].last_used_idx << 16);
+		if (hw->is_blk == IFCVF_BLK) {
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				i * IFCVF_LM_CFG_SIZE) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
+		} else if (hw->is_blk == IFCVF_NET) {
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				(i / 2) * IFCVF_LM_CFG_SIZE +
+				(i % 2) * 4) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
+		}
 
 		IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector);
 		if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -254,9 +262,23 @@
 		IFCVF_WRITE_REG16(i, &cfg->queue_select);
 		IFCVF_WRITE_REG16(0, &cfg->queue_enable);
 		IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg-
 >queue_msix_vector);
-		ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET
+
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4);
-		hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
+
+		if (hw->is_blk) {
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					i * IFCVF_LM_CFG_SIZE);
+		} else if (hw->is_blk == IFCVF_NET) {
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					(i / 2) * IFCVF_LM_CFG_SIZE +
+					(i % 2) * 4);
+		}
+
+		if (hw->is_blk == IFCVF_BLK)
+			hw->vring[i].last_avail_idx =
+				(u16)(ring_state & IFCVF_16_BIT_MASK);
+		else if (hw->is_blk == IFCVF_NET)
+			hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
 		hw->vring[i].last_used_idx = (u16)(ring_state >> 16);
 	}
 }
drivers/vdpa/ifc/base/ifcvf.h +1 -1
index 8591ef1..ff11b12 100644
@@ -65,6 +65,7 @@
 #define IFCVF_MEDIATED_VRING		0x200000000000
 
 #define IFCVF_32_BIT_MASK		0xffffffff
+#define IFCVF_16_BIT_MASK		0xffff
 
 #ifndef VHOST_USER_PROTOCOL_F_CONFIG
 #define VHOST_USER_PROTOCOL_F_CONFIG	9
--

1.8.3.1

author
#157 lore raw

RE: [PATCH v7 16/18] vdpa/ifc/base: access correct register for blk device

HI Chenbo,

Thanks for your reply.
I will send out a new version to fix this.

Quoted 109 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Friday, May 13, 2022 10:58 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 16/18] vdpa/ifc/base: access correct register for blk device

Quoted 10 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 16/18] vdpa/ifc/base: access correct register for blk device

Quoted 2 lines

1.last_avail_idx is lower 16 bit of the register.
2.address of ring_state register is different between net and blk device.

Not a good commit log. The commit log should illustrate more on what's the commit is doing.

Thanks,
Chenbo

Quoted 25 lines
---
 drivers/vdpa/ifc/base/ifcvf.c | 36
+++++++++++++++++++++++++++++-------
 drivers/vdpa/ifc/base/ifcvf.h |  1 +
 2 files changed, 30 insertions(+), 7 deletions(-)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/base/ifcvf.c index d10c1fd..4d5881a 100644

drivers/vdpa/ifc/base/ifcvf.c +5 -4
@@ -218,10 +218,18 @@
 				&cfg->queue_used_hi);
 		IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size);
 
-		*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) =
-			(u32)hw->vring[i].last_avail_idx |
-			((u32)hw->vring[i].last_used_idx << 16);
+		if (hw->is_blk == IFCVF_BLK) {
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				i * IFCVF_LM_CFG_SIZE) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);

drivers/vdpa/ifc/base/ifcvf.c line 235

Quoted 10 lines
drivers/vdpa/ifc/base/ifcvf.c +7 -0
+		} else if (hw->is_blk == IFCVF_NET) {
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				(i / 2) * IFCVF_LM_CFG_SIZE +
+				(i % 2) * 4) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
+		}
 
 		IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector);
 		if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -

254,9 +262,23

drivers/vdpa/ifc/base/ifcvf.c

Quoted 6 lines

@@

drivers/vdpa/ifc/base/ifcvf.c +0 -0
 		IFCVF_WRITE_REG16(i, &cfg->queue_select);
 		IFCVF_WRITE_REG16(0, &cfg->queue_enable);
 		IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg-
Quoted 1 line

queue_msix_vector);

drivers/vdpa/ifc/base/ifcvf.c +0 -1
-		ring_state = *(u32 *)(hw->lm_cfg +

IFCVF_LM_RING_STATE_OFFSET

Quoted 37 lines
drivers/vdpa/ifc/base/ifcvf.c +18 -2
+
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4);
-		hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
+
+		if (hw->is_blk) {
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					i * IFCVF_LM_CFG_SIZE);
+		} else if (hw->is_blk == IFCVF_NET) {
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					(i / 2) * IFCVF_LM_CFG_SIZE +
+					(i % 2) * 4);
+		}
+
+		if (hw->is_blk == IFCVF_BLK)
+			hw->vring[i].last_avail_idx =
+				(u16)(ring_state & IFCVF_16_BIT_MASK);
+		else if (hw->is_blk == IFCVF_NET)
+			hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
 		hw->vring[i].last_used_idx = (u16)(ring_state >> 16);
 	}
 }
(quoted diff) +0 -0

b/drivers/vdpa/ifc/base/ifcvf.h index 8591ef1..ff11b12 100644

drivers/vdpa/ifc/base/ifcvf.h +1 -1
@@ -65,6 +65,7 @@
 #define IFCVF_MEDIATED_VRING		0x200000000000
 
 #define IFCVF_32_BIT_MASK		0xffffffff
+#define IFCVF_16_BIT_MASK		0xffff
 
 #ifndef VHOST_USER_PROTOCOL_F_CONFIG
 #define VHOST_USER_PROTOCOL_F_CONFIG	9
--

1.8.3.1

author
#158 lore raw

PATCH v7 17/18 vdpa/ifc: blk device pause without no inflight IO

In reply to Andy Pei

When virtio blk device is pause, make sure hardware last_avail_idx and last_used_idx are the same.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +17 -8
index 4923bc1..def6adf 100644
@@ -314,12 +314,12 @@ struct rte_vdpa_dev_info {
 vdpa_ifcvf_stop(struct ifcvf_internal *internal)
 {
 	struct ifcvf_hw *hw = &internal->hw;
-	struct rte_vhost_vring vq;
 	uint32_t i;
 	int vid;
 	uint64_t features = 0;
 	uint64_t log_base = 0, log_size = 0;
 	uint64_t len;
+	u32 ring_state = 0;
 
 	vid = internal->vid;
 
@@ -328,13 +328,22 @@ struct rte_vdpa_dev_info {
 	 */
 	if (internal->device_type == IFCVF_BLK) {
 		for (i = 0; i < hw->nr_vring; i++) {
-			rte_vhost_get_vhost_vring(internal->vid, i, &vq);
-			while (vq.avail->idx != vq.used->idx) {
-				ifcvf_notify_queue(hw, i);
-				usleep(10);
-			}
-			hw->vring[i].last_avail_idx = vq.avail->idx;
-			hw->vring[i].last_used_idx = vq.used->idx;
+			do {
+				if (hw->lm_cfg != NULL)
+					ring_state = *(u32 *)(hw->lm_cfg +
+						IFCVF_LM_RING_STATE_OFFSET +
+						i * IFCVF_LM_CFG_SIZE);
+				hw->vring[i].last_avail_idx =
+					(u16)(ring_state & IFCVF_16_BIT_MASK);
+				hw->vring[i].last_used_idx =
+					(u16)(ring_state >> 16);
+				if (hw->vring[i].last_avail_idx !=
+					hw->vring[i].last_used_idx) {
+					ifcvf_notify_queue(hw, i);
+					usleep(10);
+				}
+			} while (hw->vring[i].last_avail_idx !=
+				hw->vring[i].last_used_idx);
 		}
 	}
 
Signature · 1 line
1.8.3.1
#159 lore raw

RE: [PATCH v7 17/18] vdpa/ifc: blk device pause without no inflight IO

Quoted 11 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 17/18] vdpa/ifc: blk device pause without no inflight IO

When virtio blk device is pause, make sure hardware last_avail_idx and last_used_idx are the same.

Patch is good. But illustrate more on avoiding in-flight packet in commit log

Thanks,
Chenbo

Quoted 56 lines
---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +17 -9
index 4923bc1..def6adf 100644
@@ -314,12 +314,12 @@ struct rte_vdpa_dev_info {
 vdpa_ifcvf_stop(struct ifcvf_internal *internal)
 {
 	struct ifcvf_hw *hw = &internal->hw;
-	struct rte_vhost_vring vq;
 	uint32_t i;
 	int vid;
 	uint64_t features = 0;
 	uint64_t log_base = 0, log_size = 0;
 	uint64_t len;
+	u32 ring_state = 0;
 
 	vid = internal->vid;
 
@@ -328,13 +328,22 @@ struct rte_vdpa_dev_info {
 	 */
 	if (internal->device_type == IFCVF_BLK) {
 		for (i = 0; i < hw->nr_vring; i++) {
-			rte_vhost_get_vhost_vring(internal->vid, i, &vq);
-			while (vq.avail->idx != vq.used->idx) {
-				ifcvf_notify_queue(hw, i);
-				usleep(10);
-			}
-			hw->vring[i].last_avail_idx = vq.avail->idx;
-			hw->vring[i].last_used_idx = vq.used->idx;
+			do {
+				if (hw->lm_cfg != NULL)
+					ring_state = *(u32 *)(hw->lm_cfg +
+						IFCVF_LM_RING_STATE_OFFSET +
+						i * IFCVF_LM_CFG_SIZE);
+				hw->vring[i].last_avail_idx =
+					(u16)(ring_state & IFCVF_16_BIT_MASK);
+				hw->vring[i].last_used_idx =
+					(u16)(ring_state >> 16);
+				if (hw->vring[i].last_avail_idx !=
+					hw->vring[i].last_used_idx) {
+					ifcvf_notify_queue(hw, i);
+					usleep(10);
+				}
+			} while (hw->vring[i].last_avail_idx !=
+				hw->vring[i].last_used_idx);
 		}
 	}
 
--

1.8.3.1

author
#160 lore raw

RE: [PATCH v7 17/18] vdpa/ifc: blk device pause without no inflight IO

Hi Chenbo,

Thanks for reply.
This will be addressed in next version.

Quoted 85 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Friday, May 13, 2022 10:59 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 17/18] vdpa/ifc: blk device pause without no inflight IO

Quoted 10 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 17/18] vdpa/ifc: blk device pause without no inflight IO

Quoted 2 lines

When virtio blk device is pause, make sure hardware last_avail_idx and last_used_idx are the same.

Patch is good. But illustrate more on avoiding in-flight packet in commit log

Thanks,
Chenbo

Quoted 28 lines
---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index 4923bc1..def6adf 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +1 -2
@@ -314,12 +314,12 @@ struct rte_vdpa_dev_info {
 vdpa_ifcvf_stop(struct ifcvf_internal *internal)  {
 	struct ifcvf_hw *hw = &internal->hw;
-	struct rte_vhost_vring vq;
 	uint32_t i;
 	int vid;
 	uint64_t features = 0;
 	uint64_t log_base = 0, log_size = 0;
 	uint64_t len;
+	u32 ring_state = 0;
 
 	vid = internal->vid;
 
@@ -328,13 +328,22 @@ struct rte_vdpa_dev_info {
 	 */
 	if (internal->device_type == IFCVF_BLK) {
 		for (i = 0; i < hw->nr_vring; i++) {
-			rte_vhost_get_vhost_vring(internal->vid, i, &vq);

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 334

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -6
-			while (vq.avail->idx != vq.used->idx) {
-				ifcvf_notify_queue(hw, i);
-				usleep(10);
-			}
-			hw->vring[i].last_avail_idx = vq.avail->idx;
-			hw->vring[i].last_used_idx = vq.used->idx;
+			do {
+				if (hw->lm_cfg != NULL)
+					ring_state = *(u32 *)(hw->lm_cfg +
+

IFCVF_LM_RING_STATE_OFFSET +

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info {

Quoted 3 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +3 -0
+						i * IFCVF_LM_CFG_SIZE);
+				hw->vring[i].last_avail_idx =
+					(u16)(ring_state &

IFCVF_16_BIT_MASK);

Quoted 14 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9 -1
+				hw->vring[i].last_used_idx =
+					(u16)(ring_state >> 16);
+				if (hw->vring[i].last_avail_idx !=
+					hw->vring[i].last_used_idx) {
+					ifcvf_notify_queue(hw, i);
+					usleep(10);
+				}
+			} while (hw->vring[i].last_avail_idx !=
+				hw->vring[i].last_used_idx);
 		}
 	}
 
--

1.8.3.1

author
#161 lore raw

PATCH v7 18/18 vhost: make sure each queue callfd is configured

In reply to Andy Pei

During the vhost data path building process, qemu will create a call fd at first, and create another call fd in the end. The final call fd will be used to relay notify.
In the original code, after kick fd is set, dev_conf will set the first call fd. Even though the actual call fd will set, the data path will not work correctly.

---
 lib/vhost/vhost_user.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
lib/vhost/vhost_user.c +14 -0
index e925428..82122b6 100644
@@ -3230,12 +3230,26 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev,
 	if (!vdpa_dev)
 		goto out;
 
+	if (request != VHOST_USER_SET_VRING_CALL)
+		goto out;
+
 	if (!(dev->flags & VIRTIO_DEV_VDPA_CONFIGURED)) {
 		if (vdpa_dev->ops->dev_conf(dev->vid))
 			VHOST_LOG_CONFIG(ERR, "(%s) failed to configure vDPA device\n",
 					dev->ifname);
 		else
 			dev->flags |= VIRTIO_DEV_VDPA_CONFIGURED;
+	} else {
+		/* when VIRTIO_DEV_VDPA_CONFIGURED already configured
+		 * close the device and config the device again,
+		 * make sure the call fd of each queue is configured correctly.
+		 */
+		if (vdpa_dev->ops->dev_close(dev->vid))
+			VHOST_LOG_CONFIG(ERR,
+					 "Failed to close vDPA device\n");
+		if (vdpa_dev->ops->dev_conf(dev->vid))
+			VHOST_LOG_CONFIG(ERR,
+					 "Failed to re-config vDPA device\n");
 	}
 
 out:
Signature · 1 line
1.8.3.1
#162 lore raw

RE: [PATCH v7 18/18] vhost: make sure each queue callfd is configured

Quoted 8 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 18/18] vhost: make sure each queue callfd is configured

Quoted 6 lines

During the vhost data path building process, qemu will create a call fd at first, and create another call fd in the end. The final call fd will be used to relay notify.
In the original code, after kick fd is set, dev_conf will set the first call fd. Even though the actual call fd will set, the data path will not work correctly.

I don't understand the problem. If callfd reset, we should re-configure in the set_vring_state callback, and issue can be handled there.

Thanks,
Chenbo

Quoted 41 lines
---
 lib/vhost/vhost_user.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
lib/vhost/vhost_user.c +14 -0
index e925428..82122b6 100644
@@ -3230,12 +3230,26 @@ typedef int (*vhost_message_handler_t)(struct
 virtio_net **pdev,
 	if (!vdpa_dev)
 		goto out;
 
+	if (request != VHOST_USER_SET_VRING_CALL)
+		goto out;
+
 	if (!(dev->flags & VIRTIO_DEV_VDPA_CONFIGURED)) {
 		if (vdpa_dev->ops->dev_conf(dev->vid))
 			VHOST_LOG_CONFIG(ERR, "(%s) failed to configure vDPA
 device\n",
 					dev->ifname);
 		else
 			dev->flags |= VIRTIO_DEV_VDPA_CONFIGURED;
+	} else {
+		/* when VIRTIO_DEV_VDPA_CONFIGURED already configured
+		 * close the device and config the device again,
+		 * make sure the call fd of each queue is configured correctly.
+		 */
+		if (vdpa_dev->ops->dev_close(dev->vid))
+			VHOST_LOG_CONFIG(ERR,
+					 "Failed to close vDPA device\n");
+		if (vdpa_dev->ops->dev_conf(dev->vid))
+			VHOST_LOG_CONFIG(ERR,
+					 "Failed to re-config vDPA device\n");
 	}

out:
--
1.8.3.1

author
#163 lore raw

v8 cover add virtio_blk device support to vdpa/ifc

In reply to Andy Pei

This patch set add virtio_blk device support to vdpa/ifc driver. With a lot of similarities, I re-use part of vdpa/ifc driver. Distinguish the virtio net and blk device by device id, and implement specific features and ops.
Add example to vdpa to support virtio_blk device.
To support blk device live migration, some modification to vhost lib. Perform dev_conf op only under VHOST_USER_SET_VRING_CALL msg.

v8:
delete some redundant code.
fix some commit log.

v7:
Check on expected fd num in new vhost msg handler. Sanity check on vhost msg size.
Fix typo.
Add commit log to help understand code. Remove duplicated code.
Add new API to get vDPA device type.

v6:
fix some commit log.
add vhost socket in log output to make it more user-friendly. when driver ops fail, just output some log, do not break message handler. check vhost msg size in msg handler.
v5:
fix some coding style issues.
v4:
add args "isblk" to vdpa example to specify a block device, fix some issue in example.
Make sure code specify for block device does not affect net device.
v3:
Fix some compile issues.
v2:
Fix some coding style issues.

Andy Pei (13):
vdpa/ifc: add support for virtio blk device
vhost: add vDPA ops for blk device
vhost: add vhost msg support for get/set config vdpa/ifc: add blk ops for ifc device vdpa/ifc: add vDPA interrupt relay for blk device vdpa/ifc: add block device SW live-migration
vhost: add API to get vDPA device type vdpa/ifc: add get device type ops to ifc driver examples/vdpa: add add virtio blk support
usertools: add support for virtio blk device vdpa/ifc: add log for config space of virtio blk vdpa/ifc: add interrupt handling for config space vdpa/ifc/base: access correct register for blk device

 doc/guides/prog_guide/vhost_lib.rst    |   5 +
 doc/guides/rel_notes/release_22_07.rst |   4 +
 drivers/vdpa/ifc/base/ifcvf.c          |  34 ++-
 drivers/vdpa/ifc/base/ifcvf.h          |  21 +-
 drivers/vdpa/ifc/ifcvf_vdpa.c          | 408 +++++++++++++++++++++++++++++++--
 examples/vdpa/main.c                   |  56 +++++
 examples/vdpa/vdpa_blk_compact.h       |  58 +++++
 lib/vhost/rte_vhost.h                  |  17 ++
 lib/vhost/socket.c                     |  44 ++++
 lib/vhost/vdpa_driver.h                |  11 +-
 lib/vhost/version.map                  |   1 +
 lib/vhost/vhost_user.c                 |  73 ++++++
 lib/vhost/vhost_user.h                 |  13 ++
 usertools/dpdk-devbind.py              |   5 +-
 14 files changed, 725 insertions(+), 25 deletions(-)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
Signature · 1 line
1.8.3.1
author
#164 lore raw

PATCH v8 1/13 vdpa/ifc: add support for virtio blk device

Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops.

---
 drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++-
 drivers/vdpa/ifc/ifcvf_vdpa.c | 91 +++++++++++++++++++++++++++++++++++++++----
 2 files changed, 98 insertions(+), 9 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +15 -1
index 573a35f..483d38b 100644
@@ -5,8 +5,17 @@
 #ifndef _IFCVF_H_
 #define _IFCVF_H_
 
+#include <linux/virtio_blk.h>
 #include "ifcvf_osdep.h"
 
+#define IFCVF_NET	0
+#define IFCVF_BLK	1
+
+/* for BLK */
+#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID    0x1001
+#define IFCVF_BLK_MODERN_DEVICE_ID          0x1042
+#define IFCVF_BLK_DEVICE_ID                 0x0002
+
 #define IFCVF_VENDOR_ID		0x1AF4
 #define IFCVF_DEVICE_ID		0x1041
 #define IFCVF_SUBSYS_VENDOR_ID	0x8086
@@ -126,13 +135,18 @@ struct ifcvf_hw {
 	u8     notify_region;
 	u32    notify_off_multiplier;
 	struct ifcvf_pci_common_cfg *common_cfg;
-	struct ifcvf_net_config *dev_cfg;
+	union {
+		struct ifcvf_net_config *net_cfg;
+		struct virtio_blk_config *blk_cfg;
+		void *dev_cfg;
+	};
 	u8     *isr;
 	u16    *notify_base;
 	u16    *notify_addr[IFCVF_MAX_QUEUES * 2];
 	u8     *lm_cfg;
 	struct vring_info vring[IFCVF_MAX_QUEUES * 2];
 	u8 nr_vring;
+	int device_type;
 	struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE];
 };
 
drivers/vdpa/ifc/ifcvf_vdpa.c +83 -8

127 lines not on this page · raw message

Signature · 1 line
1.8.3.1
Reviewed-by on v8 1/13
#165 lore raw

RE: [PATCH v8 01/13] vdpa/ifc: add support for virtio blk device

Hi Andy,

Quoted 31 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 01/13] vdpa/ifc: add support for virtio blk device

Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops.

---
 drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++-
 drivers/vdpa/ifc/ifcvf_vdpa.c | 91
+++++++++++++++++++++++++++++++++++++++----
 2 files changed, 98 insertions(+), 9 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +2 -0
index 573a35f..483d38b 100644
@@ -5,8 +5,17 @@
 #ifndef _IFCVF_H_
 #define _IFCVF_H_
 
+#include <linux/virtio_blk.h>
 #include "ifcvf_osdep.h"
 
+#define IFCVF_NET	0

drivers/vdpa/ifc/base/ifcvf.h line 21

Quoted 10 lines
drivers/vdpa/ifc/base/ifcvf.h +7 -0
+#define IFCVF_BLK	1
+
+/* for BLK */
+#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID    0x1001
+#define IFCVF_BLK_MODERN_DEVICE_ID          0x1042
+#define IFCVF_BLK_DEVICE_ID                 0x0002
+
 #define IFCVF_VENDOR_ID		0x1AF4
 #define IFCVF_DEVICE_ID		0x1041
 #define IFCVF_SUBSYS_VENDOR_ID	0x8086

Let's rename IFCVF_DEVICE_ID to IFCVF_NET_DEVICE_ID as it's only used for net now.

Quoted 65 lines
drivers/vdpa/ifc/base/ifcvf.h +6 -1
@@ -126,13 +135,18 @@ struct ifcvf_hw {
 	u8     notify_region;
 	u32    notify_off_multiplier;
 	struct ifcvf_pci_common_cfg *common_cfg;
-	struct ifcvf_net_config *dev_cfg;
+	union {
+		struct ifcvf_net_config *net_cfg;
+		struct virtio_blk_config *blk_cfg;
+		void *dev_cfg;
+	};
 	u8     *isr;
 	u16    *notify_base;
 	u16    *notify_addr[IFCVF_MAX_QUEUES * 2];
 	u8     *lm_cfg;
 	struct vring_info vring[IFCVF_MAX_QUEUES * 2];
 	u8 nr_vring;
+	int device_type;
 	struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE];
 };
 
drivers/vdpa/ifc/ifcvf_vdpa.c +30 -0
index 9f05595..be0efd3 100644
@@ -75,6 +75,12 @@ struct internal_list {
 	struct ifcvf_internal *internal;
 };
 
+/* vdpa device info includes device features and devcic operation. */
+struct rte_vdpa_dev_info {
+	uint64_t features;
+	struct rte_vdpa_dev_ops *ops;
+};
+
 TAILQ_HEAD(internal_list_head, internal_list);
 static struct internal_list_head internal_list =
 	TAILQ_HEAD_INITIALIZER(internal_list);
@@ -1167,6 +1173,48 @@ struct internal_list {
 	return 0;
 }
 
+static int16_t
+ifcvf_pci_get_device_type(struct rte_pci_device *pci_dev)
+{
+	uint16_t pci_device_id = pci_dev->id.device_id;
+	uint16_t device_id;
+
+	if (pci_device_id < 0x1000 || pci_device_id > 0x107f) {
+		DRV_LOG(ERR, "Probe device is not a virtio device\n");
+		return -1;
+	}
+
+	if (pci_device_id < 0x1040) {
+		/* Transitional devices: use the PCI subsystem device id as
+		 * virtio device id, same as legacy driver always did.
+		 */
+		device_id = pci_dev->id.subsystem_device_id;
+	} else {
+		/* Modern devices: simply use PCI device id,
+		 * but start from 0x1040.
+		 */
+		device_id = pci_device_id - 0x1040;
+	}
+
+	return device_id;

drivers/vdpa/ifc/ifcvf_vdpa.c struct internal_list { · line 1209

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10 -0
+}
+
+struct rte_vdpa_dev_info dev_info[] = {
+	{
+		.features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |
+			    (1ULL << VIRTIO_NET_F_CTRL_VQ) |
+			    (1ULL << VIRTIO_NET_F_STATUS) |
+			    (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) |
+			    (1ULL << VHOST_F_LOG_ALL),
+		.ops = &ifcvf_ops,

Rename ifcvf_ops -> ifcvf_net_ops

Overall the patch LGTM. With above fixed:

Quoted 85 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +29 -8
+	},
+	{
+		.features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) |
+			    (1ULL << VHOST_F_LOG_ALL),
+		.ops = NULL,
+	},
+};
+
 static int
 ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		struct rte_pci_device *pci_dev)
@@ -1178,6 +1226,7 @@ struct internal_list {
 	int sw_fallback_lm = 0;
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
+	int16_t device_id;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
@@ -1227,13 +1276,24 @@ struct internal_list {
 	internal->configured = 0;
 	internal->max_queues = IFCVF_MAX_QUEUES;
 	features = ifcvf_get_features(&internal->hw);
-	internal->features = (features &
-		~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) |
-		(1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |
-		(1ULL << VIRTIO_NET_F_CTRL_VQ) |
-		(1ULL << VIRTIO_NET_F_STATUS) |
-		(1ULL << VHOST_USER_F_PROTOCOL_FEATURES) |
-		(1ULL << VHOST_F_LOG_ALL);
+
+	device_id = ifcvf_pci_get_device_type(pci_dev);
+	if (device_id < 0) {
+		DRV_LOG(ERR, "failed to get device %s type", pci_dev->name);
+		goto error;
+	}
+
+	if (device_id == VIRTIO_ID_NET) {
+		internal->hw.device_type = IFCVF_NET;
+		internal->features = features &
+					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
+		internal->features |= dev_info[IFCVF_NET].features;
+	} else if (device_id == VIRTIO_ID_BLOCK) {
+		internal->hw.device_type = IFCVF_BLK;
+		internal->features = features &
+					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
+		internal->features |= dev_info[IFCVF_BLK].features;
+	}
 
 	list->internal = internal;
 
@@ -1245,7 +1305,8 @@ struct internal_list {
 	}
 	internal->sw_lm = sw_fallback_lm;
 
-	internal->vdev = rte_vdpa_register_device(&pci_dev->device,
 &ifcvf_ops);
+	internal->vdev = rte_vdpa_register_device(&pci_dev->device,
+				dev_info[internal->hw.device_type].ops);
 	if (internal->vdev == NULL) {
 		DRV_LOG(ERR, "failed to register device %s", pci_dev->name);
drivers/vdpa/ifc/ifcvf_vdpa.c +14 -1
 		goto error;
@@ -1313,6 +1374,20 @@ struct internal_list {
 	  .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID,
 	},
 
+	{ .class_id = RTE_CLASS_ANY_ID,
+	  .vendor_id = IFCVF_VENDOR_ID,
+	  .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID,
+	  .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID,
+	  .subsystem_device_id = IFCVF_BLK_DEVICE_ID,
+	},
+
+	{ .class_id = RTE_CLASS_ANY_ID,
+	  .vendor_id = IFCVF_VENDOR_ID,
+	  .device_id = IFCVF_BLK_MODERN_DEVICE_ID,
+	  .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID,
+	  .subsystem_device_id = IFCVF_BLK_DEVICE_ID,
+	},
+
 	{ .vendor_id = 0, /* sentinel */
 	},
 };
--

1.8.3.1

author
#166 lore raw

RE: [PATCH v8 01/13] vdpa/ifc: add support for virtio blk device

Hi Chenbo,

Thanks for your reply, my reply is inline.

Quoted 42 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 11:42 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 01/13] vdpa/ifc: add support for virtio blk device

Hi Andy,

Quoted 31 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 01/13] vdpa/ifc: add support for virtio blk device

Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops.

---
 drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++-
drivers/vdpa/ifc/ifcvf_vdpa.c | 91
+++++++++++++++++++++++++++++++++++++++----
 2 files changed, 98 insertions(+), 9 deletions(-)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/base/ifcvf.h index 573a35f..483d38b 100644

drivers/vdpa/ifc/base/ifcvf.h +1 -0
@@ -5,8 +5,17 @@
 #ifndef _IFCVF_H_
 #define _IFCVF_H_
 
+#include <linux/virtio_blk.h>
 #include "ifcvf_osdep.h"
 
Quoted 15 lines

drivers/vdpa/ifc/base/ifcvf.h line 21

Quoted 11 lines
drivers/vdpa/ifc/base/ifcvf.h +8 -0
+#define IFCVF_NET	0
+#define IFCVF_BLK	1
+
+/* for BLK */
+#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID    0x1001
+#define IFCVF_BLK_MODERN_DEVICE_ID          0x1042
+#define IFCVF_BLK_DEVICE_ID                 0x0002
+
 #define IFCVF_VENDOR_ID		0x1AF4
 #define IFCVF_DEVICE_ID		0x1041
 #define IFCVF_SUBSYS_VENDOR_ID	0x8086

Let's rename IFCVF_DEVICE_ID to IFCVF_NET_DEVICE_ID as it's only used for net now.

Sure, I will do it in next version.

Quoted 78 lines
Quoted 8 lines
drivers/vdpa/ifc/base/ifcvf.h +3 -1
@@ -126,13 +135,18 @@ struct ifcvf_hw {
 	u8     notify_region;
 	u32    notify_off_multiplier;
 	struct ifcvf_pci_common_cfg *common_cfg;
-	struct ifcvf_net_config *dev_cfg;
+	union {
+		struct ifcvf_net_config *net_cfg;
+		struct virtio_blk_config *blk_cfg;

drivers/vdpa/ifc/base/ifcvf.h struct ifcvf_hw { · line 150

Quoted 10 lines
drivers/vdpa/ifc/base/ifcvf.h +3 -0
+		void *dev_cfg;
+	};
 	u8     *isr;
 	u16    *notify_base;
 	u16    *notify_addr[IFCVF_MAX_QUEUES * 2];
 	u8     *lm_cfg;
 	struct vring_info vring[IFCVF_MAX_QUEUES * 2];
 	u8 nr_vring;
+	int device_type;
 	struct ifcvf_pci_mem_resource

mem_resource[IFCVF_PCI_MAX_RESOURCE];

Quoted 46 lines
drivers/vdpa/ifc/base/ifcvf.h +0 -0
 };
 
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index 9f05595..be0efd3 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +6 -0
@@ -75,6 +75,12 @@ struct internal_list {
 	struct ifcvf_internal *internal;
 };
 
+/* vdpa device info includes device features and devcic operation. */
+struct rte_vdpa_dev_info {
+	uint64_t features;
+	struct rte_vdpa_dev_ops *ops;
+};
+
 TAILQ_HEAD(internal_list_head, internal_list);  static struct

internal_list_head internal_list =

drivers/vdpa/ifc/ifcvf_vdpa.c +23 -0
 	TAILQ_HEAD_INITIALIZER(internal_list);
@@ -1167,6 +1173,48 @@ struct internal_list {
 	return 0;
 }
 
+static int16_t
+ifcvf_pci_get_device_type(struct rte_pci_device *pci_dev) {
+	uint16_t pci_device_id = pci_dev->id.device_id;
+	uint16_t device_id;
+
+	if (pci_device_id < 0x1000 || pci_device_id > 0x107f) {
+		DRV_LOG(ERR, "Probe device is not a virtio device\n");
+		return -1;
+	}
+
+	if (pci_device_id < 0x1040) {
+		/* Transitional devices: use the PCI subsystem device id as
+		 * virtio device id, same as legacy driver always did.
+		 */
+		device_id = pci_dev->id.subsystem_device_id;
+	} else {
+		/* Modern devices: simply use PCI device id,
+		 * but start from 0x1040.
+		 */
+		device_id = pci_device_id - 0x1040;
+	}
+
+	return device_id;

drivers/vdpa/ifc/ifcvf_vdpa.c struct internal_list { · line 1208

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10 -0
+}
+
+struct rte_vdpa_dev_info dev_info[] = {
+	{
+		.features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |
+			    (1ULL << VIRTIO_NET_F_CTRL_VQ) |
+			    (1ULL << VIRTIO_NET_F_STATUS) |
+			    (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) |
+			    (1ULL << VHOST_F_LOG_ALL),
+		.ops = &ifcvf_ops,

Rename ifcvf_ops -> ifcvf_net_ops

Quoted 4 lines

Overall the patch LGTM. With above fixed:

OK, I will do this in next version.

Quoted 89 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct internal_list {

Quoted 3 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +3 -0
+	},
+	{
+		.features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES)

|

Quoted 29 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +8 -7
+			    (1ULL << VHOST_F_LOG_ALL),
+		.ops = NULL,
+	},
+};
+
 static int
 ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		struct rte_pci_device *pci_dev)
@@ -1178,6 +1226,7 @@ struct internal_list {
 	int sw_fallback_lm = 0;
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
+	int16_t device_id;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
@@ -1227,13 +1276,24 @@ struct internal_list {
 	internal->configured = 0;
 	internal->max_queues = IFCVF_MAX_QUEUES;
 	features = ifcvf_get_features(&internal->hw);
-	internal->features = (features &
-		~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) |
-		(1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |
-		(1ULL << VIRTIO_NET_F_CTRL_VQ) |
-		(1ULL << VIRTIO_NET_F_STATUS) |
-		(1ULL << VHOST_USER_F_PROTOCOL_FEATURES) |
-		(1ULL << VHOST_F_LOG_ALL);
+
+	device_id = ifcvf_pci_get_device_type(pci_dev);

drivers/vdpa/ifc/ifcvf_vdpa.c struct internal_list { · line 1290

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9 -0
+	if (device_id < 0) {
+		DRV_LOG(ERR, "failed to get device %s type", pci_dev-
 name);
+		goto error;
+	}
+
+	if (device_id == VIRTIO_ID_NET) {
+		internal->hw.device_type = IFCVF_NET;
+		internal->features = features &
+					~(1ULL <<

VIRTIO_F_IOMMU_PLATFORM);

drivers/vdpa/ifc/ifcvf_vdpa.c struct internal_list {

Quoted 5 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5 -0
+		internal->features |= dev_info[IFCVF_NET].features;
+	} else if (device_id == VIRTIO_ID_BLOCK) {
+		internal->hw.device_type = IFCVF_BLK;
+		internal->features = features &
+					~(1ULL <<

VIRTIO_F_IOMMU_PLATFORM);

Quoted 39 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -1
+		internal->features |= dev_info[IFCVF_BLK].features;
+	}
 
 	list->internal = internal;
@@ -1245,7 +1305,8 @@ struct internal_list {
 	}
 	internal->sw_lm = sw_fallback_lm;
 
-	internal->vdev = rte_vdpa_register_device(&pci_dev->device,
 &ifcvf_ops);
+	internal->vdev = rte_vdpa_register_device(&pci_dev->device,
+				dev_info[internal->hw.device_type].ops);
 	if (internal->vdev == NULL) {
 		DRV_LOG(ERR, "failed to register device %s", pci_dev->name);
drivers/vdpa/ifc/ifcvf_vdpa.c +14 -1
 		goto error;
@@ -1313,6 +1374,20 @@ struct internal_list {
 	  .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID,
 	},
 
+	{ .class_id = RTE_CLASS_ANY_ID,
+	  .vendor_id = IFCVF_VENDOR_ID,
+	  .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID,
+	  .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID,
+	  .subsystem_device_id = IFCVF_BLK_DEVICE_ID,
+	},
+
+	{ .class_id = RTE_CLASS_ANY_ID,
+	  .vendor_id = IFCVF_VENDOR_ID,
+	  .device_id = IFCVF_BLK_MODERN_DEVICE_ID,
+	  .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID,
+	  .subsystem_device_id = IFCVF_BLK_DEVICE_ID,
+	},
+
 	{ .vendor_id = 0, /* sentinel */
 	},
 };
--

1.8.3.1

author
#167 lore raw

PATCH v8 2/13 vhost: add vDPA ops for blk device

In reply to Andy Pei

Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops.

---
 lib/vhost/vdpa_driver.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
lib/vhost/vdpa_driver.h +6 -2
index 88138be..e59a834 100644
@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {
 	/** Reset statistics of the queue */
 	int (*reset_stats)(struct rte_vdpa_device *dev, int qid);
 
-	/** Reserved for future extension */
-	void *reserved[2];
+	/** Get the device configuration space */
+	int (*get_config)(int vid, uint8_t *config, uint32_t len);
+
+	/** Set the device configuration space */
+	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
+		      uint32_t size, uint32_t flags);
 };
 
 /**
Signature · 1 line
1.8.3.1
Reviewed-by on v8 2/13
#168 lore raw

RE: [PATCH v8 02/13] vhost: add vDPA ops for blk device

Quoted 23 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 02/13] vhost: add vDPA ops for blk device

Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops.

---
 lib/vhost/vdpa_driver.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
lib/vhost/vdpa_driver.h +0 -0
index 88138be..e59a834 100644

lib/vhost/vdpa_driver.h struct rte_vdpa_dev_ops { · line 76

Quoted 7 lines
lib/vhost/vdpa_driver.h +2 -2
@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {
 	/** Reset statistics of the queue */
 	int (*reset_stats)(struct rte_vdpa_device *dev, int qid);
 
-	/** Reserved for future extension */
-	void *reserved[2];
+	/** Get the device configuration space */
+	int (*get_config)(int vid, uint8_t *config, uint32_t len);

One suggestion is renaming 'len' to 'size' to make it aligned to set_config.

With above fixed:

Quoted 9 lines
lib/vhost/vdpa_driver.h +4 -1
+
+	/** Set the device configuration space */
+	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
+		      uint32_t size, uint32_t flags);
 };
 
 /**
--

1.8.3.1

author
#169 lore raw

RE: [PATCH v8 02/13] vhost: add vDPA ops for blk device

Hi Chenbo,

Thanks for your reply, I will send out a new version to address this.

Quoted 56 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 11:46 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 02/13] vhost: add vDPA ops for blk device

Quoted 24 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 02/13] vhost: add vDPA ops for blk device

Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops.

---
 lib/vhost/vdpa_driver.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
lib/vhost/vdpa_driver.h index renamed from lib/vhost/vdpa_driver.h +0 -0

88138be..e59a834 100644

lib/vhost/vdpa_driver.h struct rte_vdpa_dev_ops { · line 76

Quoted 7 lines
lib/vhost/vdpa_driver.h +2 -2
@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {
 	/** Reset statistics of the queue */
 	int (*reset_stats)(struct rte_vdpa_device *dev, int qid);
 
-	/** Reserved for future extension */
-	void *reserved[2];
+	/** Get the device configuration space */
+	int (*get_config)(int vid, uint8_t *config, uint32_t len);

One suggestion is renaming 'len' to 'size' to make it aligned to set_config.

With above fixed:

Quoted 9 lines
lib/vhost/vdpa_driver.h +4 -1
+
+	/** Set the device configuration space */
+	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
+		      uint32_t size, uint32_t flags);
 };
 
 /**
--

1.8.3.1

author
#170 lore raw

PATCH v8 3/13 vhost: add vhost msg support for get/set config

In reply to Andy Pei

Add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device.

---
 lib/vhost/vhost_user.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/vhost/vhost_user.h | 13 +++++++++
 2 files changed, 86 insertions(+)
lib/vhost/vhost_user.c +73 -0
index 850848c..1dd1e25f 100644
@@ -2468,6 +2468,77 @@ static int is_vring_iotlb(struct virtio_net *dev,
 }
 
 static int
+vhost_user_get_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (vdpa_dev->ops->get_config) {
+		ret = vdpa_dev->ops->get_config(dev->vid,
+					   ctx->msg.payload.cfg.region,
+					   ctx->msg.payload.cfg.size);
+		if (ret != 0) {
+			ctx->msg.size = 0;
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) get_config() return error!\n",
+					 dev->ifname);
+		}
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) get_config() not supported!\n",
+				 dev->ifname);
+	}
+
+	return RTE_VHOST_MSG_RESULT_REPLY;
+}
+
+static int
+vhost_user_set_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) {
+		VHOST_LOG_CONFIG(ERR,
+			"(%s) vhost_user_config size: %"PRIu32", should not be larger than %d\n",
+			dev->ifname, ctx->msg.payload.cfg.size,
+			VHOST_USER_MAX_CONFIG_SIZE);
+		goto out;
+	}
+
+	if (vdpa_dev->ops->set_config) {
+		ret = vdpa_dev->ops->set_config(dev->vid,
+			ctx->msg.payload.cfg.region,
+			ctx->msg.payload.cfg.offset,
+			ctx->msg.payload.cfg.size,
+			ctx->msg.payload.cfg.flags);
+		if (ret)
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) set_config() return error!\n",
+					 dev->ifname);
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) set_config() not supported!\n",
+				 dev->ifname);
+	}
+
+	return RTE_VHOST_MSG_RESULT_OK;
+
+out:
+	return RTE_VHOST_MSG_RESULT_ERR;
+}
+
+static int
 vhost_user_iotlb_msg(struct virtio_net **pdev,
 			struct vhu_msg_context *ctx,
 			int main_fd __rte_unused)
@@ -2686,6 +2757,8 @@ static int is_vring_iotlb(struct virtio_net *dev,
 VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU, vhost_user_net_set_mtu, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD, vhost_user_set_req_fd, true) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG, vhost_user_get_config, false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG, vhost_user_set_config, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE, vhost_user_set_postcopy_advise, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END, vhost_user_postcopy_end, false) \
lib/vhost/vhost_user.h +13 -0
index ba1c5c7..c4d091e 100644
@@ -50,6 +50,8 @@
 	VHOST_USER_NET_SET_MTU = 20,
 	VHOST_USER_SET_SLAVE_REQ_FD = 21,
 	VHOST_USER_IOTLB_MSG = 22,
+	VHOST_USER_GET_CONFIG = 24,
+	VHOST_USER_SET_CONFIG = 25,
 	VHOST_USER_CRYPTO_CREATE_SESS = 26,
 	VHOST_USER_CRYPTO_CLOSE_SESS = 27,
 	VHOST_USER_POSTCOPY_ADVISE = 28,
@@ -123,6 +125,16 @@
 	uint16_t queue_size;
 } VhostUserInflight;
 
+#define VHOST_USER_MAX_CONFIG_SIZE		256
+
+/** Get/set config msg payload */
+struct vhost_user_config {
+	uint32_t offset;
+	uint32_t size;
+	uint32_t flags;
+	uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+};
+
 typedef struct VhostUserMsg {
 	union {
 		uint32_t master; /* a VhostUserRequest value */
@@ -146,6 +158,7 @@
 		VhostUserCryptoSessionParam crypto_session;
 		VhostUserVringArea area;
 		VhostUserInflight inflight;
+		struct vhost_user_config cfg;
 	} payload;
 	/* Nothing should be added after the payload */
 } __rte_packed VhostUserMsg;
Signature · 1 line
1.8.3.1
#171 lore raw

RE: [PATCH v8 03/13] vhost: add vhost msg support for get/set config

Quoted 88 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 03/13] vhost: add vhost msg support for get/set config

Add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device.

---
 lib/vhost/vhost_user.c | 73
++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/vhost/vhost_user.h | 13 +++++++++
 2 files changed, 86 insertions(+)
lib/vhost/vhost_user.c +59 -0
index 850848c..1dd1e25f 100644
@@ -2468,6 +2468,77 @@ static int is_vring_iotlb(struct virtio_net *dev,
 }
 
 static int
+vhost_user_get_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (vdpa_dev->ops->get_config) {
+		ret = vdpa_dev->ops->get_config(dev->vid,
+					   ctx->msg.payload.cfg.region,
+					   ctx->msg.payload.cfg.size);
+		if (ret != 0) {
+			ctx->msg.size = 0;
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) get_config() return error!\n",
+					 dev->ifname);
+		}
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) get_config() not supported!\n",
+				 dev->ifname);
+	}
+
+	return RTE_VHOST_MSG_RESULT_REPLY;
+}
+
+static int
+vhost_user_set_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) {
+		VHOST_LOG_CONFIG(ERR,
+			"(%s) vhost_user_config size: %"PRIu32", should not be
 larger than %d\n",
+			dev->ifname, ctx->msg.payload.cfg.size,
+			VHOST_USER_MAX_CONFIG_SIZE);
+		goto out;
+	}
+
+	if (vdpa_dev->ops->set_config) {
+		ret = vdpa_dev->ops->set_config(dev->vid,
+			ctx->msg.payload.cfg.region,
+			ctx->msg.payload.cfg.offset,
+			ctx->msg.payload.cfg.size,
+			ctx->msg.payload.cfg.flags);
+		if (ret)
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) set_config() return error!\n",
+					 dev->ifname);

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev, · line 2540

Quoted 10 lines
lib/vhost/vhost_user.c +10 -0
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) set_config() not supported!\n",
+				 dev->ifname);
+	}
+
+	return RTE_VHOST_MSG_RESULT_OK;
+
+out:
+	return RTE_VHOST_MSG_RESULT_ERR;
+}

I missed one thing in previous reviews: vdpa_dev pointer needed to be checked (Not Null) in both set/get config.

Thanks,
Chenbo

Quoted 62 lines
lib/vhost/vhost_user.c +2 -0
+
+static int
 vhost_user_iotlb_msg(struct virtio_net **pdev,
 			struct vhu_msg_context *ctx,
lib/vhost/vhost_user.c +1 -0
 			int main_fd __rte_unused)
@@ -2686,6 +2757,8 @@ static int is_vring_iotlb(struct virtio_net *dev,
 VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU, vhost_user_net_set_mtu,
 false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD, vhost_user_set_req_fd,
 true) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false)
 \
+VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG, vhost_user_get_config, false)

\
+VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG, vhost_user_set_config, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE,
vhost_user_set_postcopy_advise, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN,
vhost_user_set_postcopy_listen, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END, vhost_user_postcopy_end,
false) \

lib/vhost/vhost_user.h +13 -1
index ba1c5c7..c4d091e 100644
@@ -50,6 +50,8 @@
 	VHOST_USER_NET_SET_MTU = 20,
 	VHOST_USER_SET_SLAVE_REQ_FD = 21,
 	VHOST_USER_IOTLB_MSG = 22,
+	VHOST_USER_GET_CONFIG = 24,
+	VHOST_USER_SET_CONFIG = 25,
 	VHOST_USER_CRYPTO_CREATE_SESS = 26,
 	VHOST_USER_CRYPTO_CLOSE_SESS = 27,
 	VHOST_USER_POSTCOPY_ADVISE = 28,
@@ -123,6 +125,16 @@
 	uint16_t queue_size;
 } VhostUserInflight;
 
+#define VHOST_USER_MAX_CONFIG_SIZE		256
+
+/** Get/set config msg payload */
+struct vhost_user_config {
+	uint32_t offset;
+	uint32_t size;
+	uint32_t flags;
+	uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+};
+
 typedef struct VhostUserMsg {
 	union {
 		uint32_t master; /* a VhostUserRequest value */
@@ -146,6 +158,7 @@
 		VhostUserCryptoSessionParam crypto_session;
 		VhostUserVringArea area;
 		VhostUserInflight inflight;
+		struct vhost_user_config cfg;
 	} payload;
 	/* Nothing should be added after the payload */
 } __rte_packed VhostUserMsg;
--

1.8.3.1

author
#172 lore raw

RE: [PATCH v8 03/13] vhost: add vhost msg support for get/set config

Hi Chenbo,

Thanks for your reply.
I will check it in next version patch.

Quoted 191 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 11:55 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 03/13] vhost: add vhost msg support for get/set config

Quoted 11 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 03/13] vhost: add vhost msg support for get/set config

Quoted 1 line

Add support for VHOST_USER_GET_CONFIG and

VHOST_USER_SET_CONFIG.

Quoted 31 lines

VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device.

---
 lib/vhost/vhost_user.c | 73
++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/vhost/vhost_user.h | 13 +++++++++
 2 files changed, 86 insertions(+)
lib/vhost/vhost_user.c index renamed from lib/vhost/vhost_user.c +0 -0

850848c..1dd1e25f 100644

lib/vhost/vhost_user.c +13 -0
@@ -2468,6 +2468,77 @@ static int is_vring_iotlb(struct virtio_net
 *dev,  }
 
 static int
+vhost_user_get_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (vdpa_dev->ops->get_config) {
+		ret = vdpa_dev->ops->get_config(dev->vid,

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net · line 2493

Quoted 10 lines
lib/vhost/vhost_user.c +10 -0
+					   ctx->msg.payload.cfg.region,
+					   ctx->msg.payload.cfg.size);
+		if (ret != 0) {
+			ctx->msg.size = 0;
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) get_config() return error!\n",
+					 dev->ifname);
+		}
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) get_config() not

supported!\n",

Quoted 11 lines
lib/vhost/vhost_user.c +11 -0
+				 dev->ifname);
+	}
+
+	return RTE_VHOST_MSG_RESULT_REPLY;
+}
+
+static int
+vhost_user_set_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev,

Quoted 10 lines
lib/vhost/vhost_user.c +10 -0
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) {
+		VHOST_LOG_CONFIG(ERR,
+			"(%s) vhost_user_config size: %"PRIu32", should not

be

Quoted 8 lines
lib/vhost/vhost_user.c +7 -0
 larger than %d\n",
+			dev->ifname, ctx->msg.payload.cfg.size,
+			VHOST_USER_MAX_CONFIG_SIZE);
+		goto out;
+	}
+
+	if (vdpa_dev->ops->set_config) {
+		ret = vdpa_dev->ops->set_config(dev->vid,

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev,

Quoted 10 lines
lib/vhost/vhost_user.c +10 -0
+			ctx->msg.payload.cfg.region,
+			ctx->msg.payload.cfg.offset,
+			ctx->msg.payload.cfg.size,
+			ctx->msg.payload.cfg.flags);
+		if (ret)
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) set_config() return error!\n",
+					 dev->ifname);
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) set_config() not

supported!\n",

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev, · line 2539

Quoted 8 lines
lib/vhost/vhost_user.c +8 -0
+				 dev->ifname);
+	}
+
+	return RTE_VHOST_MSG_RESULT_OK;
+
+out:
+	return RTE_VHOST_MSG_RESULT_ERR;
+}

I missed one thing in previous reviews: vdpa_dev pointer needed to be checked (Not Null) in both set/get config.

Thanks,
Chenbo

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net

Quoted 13 lines
lib/vhost/vhost_user.c +2 -0
+
+static int
 vhost_user_iotlb_msg(struct virtio_net **pdev,
 			struct vhu_msg_context *ctx,
lib/vhost/vhost_user.c +0 -0
 			int main_fd __rte_unused)
@@ -2686,6 +2757,8 @@ static int is_vring_iotlb(struct virtio_net
 *dev,  VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU,
 vhost_user_net_set_mtu,
 false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD,
 vhost_user_set_req_fd,
 true) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG,

vhost_user_iotlb_msg,

Quoted 2 lines

false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG,

vhost_user_get_config,

Quoted 3 lines

+false)
\
+VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG,

vhost_user_set_config,

Quoted 29 lines

+false)
\
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE,
vhost_user_set_postcopy_advise, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END, vhost_user_postcopy_end,
false) \

lib/vhost/vhost_user.h index renamed from lib/vhost/vhost_user.h +0 -0

ba1c5c7..c4d091e 100644

lib/vhost/vhost_user.h +5 -0
@@ -50,6 +50,8 @@
 	VHOST_USER_NET_SET_MTU = 20,
 	VHOST_USER_SET_SLAVE_REQ_FD = 21,
 	VHOST_USER_IOTLB_MSG = 22,
+	VHOST_USER_GET_CONFIG = 24,
+	VHOST_USER_SET_CONFIG = 25,
 	VHOST_USER_CRYPTO_CREATE_SESS = 26,
 	VHOST_USER_CRYPTO_CLOSE_SESS = 27,
 	VHOST_USER_POSTCOPY_ADVISE = 28,
@@ -123,6 +125,16 @@
 	uint16_t queue_size;
 } VhostUserInflight;
 
+#define VHOST_USER_MAX_CONFIG_SIZE		256
+
+/** Get/set config msg payload */

lib/vhost/vhost_user.h line 140

Quoted 10 lines
lib/vhost/vhost_user.h +7 -0
+struct vhost_user_config {
+	uint32_t offset;
+	uint32_t size;
+	uint32_t flags;
+	uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+};
+
 typedef struct VhostUserMsg {
 	union {
 		uint32_t master; /* a VhostUserRequest value */ @@ -146,6

+158,7 @@

Quoted 9 lines
lib/vhost/vhost_user.h +1 -0
 		VhostUserCryptoSessionParam crypto_session;
 		VhostUserVringArea area;
 		VhostUserInflight inflight;
+		struct vhost_user_config cfg;
 	} payload;
 	/* Nothing should be added after the payload */  } __rte_packed

VhostUserMsg;
--
1.8.3.1

author
#173 lore raw

PATCH v8 4/13 vdpa/ifc: add blk ops for ifc device

In reply to Andy Pei

For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.

---
 drivers/vdpa/ifc/base/ifcvf.h |  4 ++
 drivers/vdpa/ifc/ifcvf_vdpa.c | 91 ++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 94 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/base/ifcvf.h +4 -0
index 483d38b..244de46 100644
@@ -67,6 +67,10 @@
 #define IFCVF_32_BIT_MASK		0xffffffff
 
 
+#ifndef VHOST_USER_PROTOCOL_F_CONFIG
+#define VHOST_USER_PROTOCOL_F_CONFIG	9
+#endif
+
 struct ifcvf_pci_cap {
 	u8 cap_vndr;            /* Generic PCI field: PCI_CAP_ID_VNDR */
 	u8 cap_next;            /* Generic PCI field: next ptr. */
drivers/vdpa/ifc/ifcvf_vdpa.c +90 -1

109 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#174 lore raw

RE: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc device

Quoted 62 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc device

For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.

---
 drivers/vdpa/ifc/base/ifcvf.h |  4 ++
 drivers/vdpa/ifc/ifcvf_vdpa.c | 91
++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 94 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/base/ifcvf.h +4 -0
index 483d38b..244de46 100644
@@ -67,6 +67,10 @@
 #define IFCVF_32_BIT_MASK		0xffffffff
 
 
+#ifndef VHOST_USER_PROTOCOL_F_CONFIG
+#define VHOST_USER_PROTOCOL_F_CONFIG	9
+#endif
+
 struct ifcvf_pci_cap {
 	u8 cap_vndr;            /* Generic PCI field: PCI_CAP_ID_VNDR */
 	u8 cap_next;            /* Generic PCI field: next ptr. */
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
index be0efd3..350214a 100644
@@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info {
 		 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \
 		 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \
 		 1ULL << VHOST_USER_PROTOCOL_F_STATUS)
+
+#define VDPA_BLK_PROTOCOL_FEATURES \
+		(1ULL << VHOST_USER_PROTOCOL_F_CONFIG)
+
 static int
 ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t
 *features)

{

drivers/vdpa/ifc/ifcvf_vdpa.c +6 -0
@@ -1199,6 +1203,91 @@ struct rte_vdpa_dev_info {
 	return device_id;
 }
 
+static int
+ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t len)
+{
+	struct virtio_blk_config *dev_cfg;
+	struct ifcvf_internal *internal;
+	struct rte_vdpa_device *vdev;

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1221

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10 -0
+	struct internal_list *list;
+	uint32_t i;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+
+	if (len < sizeof(struct virtio_blk_config)) {
+		DRV_LOG(ERR, "Invalid len: %u, required: %u",
+			len, (uint32_t)sizeof(struct virtio_blk_config));
+		return -1;
+	}

I believe it should not be > sizeof(struct virtio_blk_config) too?

And one question is should we limit only reading the whole blk config. I guess we are having this check because of current usage of QEMU? Although this is fine as it's vendor-specific logic, just wondering if we should have this limit for blk device.

Thanks,
Chenbo

Quoted 84 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +69 -0
+
+	vdev = rte_vhost_get_vdpa_device(vid);
+	list = find_internal_resource_by_vdev(vdev);
+	if (list == NULL) {
+		DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -1;
+	}
+
+	internal = list->internal;
+
+	for (i = 0; i < sizeof(struct virtio_blk_config); i++)
+		config[i] = *((u8 *)internal->hw.blk_cfg + i);
+
+	dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg;
+
+	/* cannot read 64-bit register in one attempt, so read byte by byte.
 */
+	for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+		byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+		capacity |= (uint64_t)*byte << (i * 8);
+	}
+	/* The capacity is number of sectors in 512-byte.
+	 * So right shift 1 bit  we get in K,
+	 * another right shift 10 bits we get in M,
+	 * right shift 10 more bits, we get in G.
+	 * To show capacity in G, we right shift 21 bits in total.
+	 */
+	DRV_LOG(DEBUG, "capacity  : %"PRIu64"G", capacity >> 21);
+
+	DRV_LOG(DEBUG, "size_max  : 0x%08x", dev_cfg->size_max);
+	DRV_LOG(DEBUG, "seg_max   : 0x%08x", dev_cfg->seg_max);
+	DRV_LOG(DEBUG, "blk_size  : 0x%08x", dev_cfg->blk_size);
+	DRV_LOG(DEBUG, "geometry");
+	DRV_LOG(DEBUG, "      cylinders: %u", dev_cfg->geometry.cylinders);
+	DRV_LOG(DEBUG, "      heads    : %u", dev_cfg->geometry.heads);
+	DRV_LOG(DEBUG, "      sectors  : %u", dev_cfg->geometry.sectors);
+	DRV_LOG(DEBUG, "num_queues: 0x%08x", dev_cfg->num_queues);
+
+	DRV_LOG(DEBUG, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n",
+		config[0], config[1], config[2], config[3], config[4],
+		config[5], config[6], config[7]);
+	return 0;
+}
+
+static int
+ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev,
+	uint64_t *features)
+{
+	RTE_SET_USED(vdev);
+
+	*features = VDPA_SUPPORTED_PROTOCOL_FEATURES;
+	*features |= VDPA_BLK_PROTOCOL_FEATURES;
+	return 0;
+}
+
+static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
+	.get_queue_num = ifcvf_get_queue_num,
+	.get_features = ifcvf_get_vdpa_features,
+	.set_features = ifcvf_set_features,
+	.get_protocol_features = ifcvf_blk_get_protocol_features,
+	.dev_conf = ifcvf_dev_config,
+	.dev_close = ifcvf_dev_close,
+	.set_vring_state = ifcvf_set_vring_state,
+	.migration_done = NULL,
+	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
+	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
+	.get_notify_area = ifcvf_get_notify_area,
+	.get_config = ifcvf_blk_get_config,
+};
+
 struct rte_vdpa_dev_info dev_info[] = {
 	{
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -2
 		.features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |
@@ -1211,7 +1300,7 @@ struct rte_vdpa_dev_info dev_info[] = {
 	{
 		.features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) |
 			    (1ULL << VHOST_F_LOG_ALL),
-		.ops = NULL,
+		.ops = &ifcvf_blk_ops,
 	},
 };
 
--

1.8.3.1

author
#175 lore raw

RE: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc device

HI Chenbo,

Thanks for your reply, my reply is inline.

Quoted 74 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 12:08 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc device

Quoted 65 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc device

For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.

---
drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c
| 91
++++++++++++++++++++++++++++++++++++++++++-

 2 files changed, 94 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/base/ifcvf.h index 483d38b..244de46 100644

drivers/vdpa/ifc/base/ifcvf.h +4 -0
@@ -67,6 +67,10 @@
 #define IFCVF_32_BIT_MASK		0xffffffff
 
 
+#ifndef VHOST_USER_PROTOCOL_F_CONFIG
+#define VHOST_USER_PROTOCOL_F_CONFIG	9
+#endif
+
 struct ifcvf_pci_cap {
 	u8 cap_vndr;            /* Generic PCI field: PCI_CAP_ID_VNDR */
 	u8 cap_next;            /* Generic PCI field: next ptr. */
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index be0efd3..350214a 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
@@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info {
 		 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \
 		 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \
 		 1ULL << VHOST_USER_PROTOCOL_F_STATUS)
+
+#define VDPA_BLK_PROTOCOL_FEATURES \
+		(1ULL << VHOST_USER_PROTOCOL_F_CONFIG)
+
 static int
 ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t
 *features)

{

drivers/vdpa/ifc/ifcvf_vdpa.c +9 -0
@@ -1199,6 +1203,91 @@ struct rte_vdpa_dev_info {
 	return device_id;
 }
 
+static int
+ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t len) {
+	struct virtio_blk_config *dev_cfg;
+	struct ifcvf_internal *internal;
+	struct rte_vdpa_device *vdev;
+	struct internal_list *list;
+	uint32_t i;
+	uint64_t capacity = 0;
+	uint8_t *byte;
Quoted 9 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1221

Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +6 -0
+
+	if (len < sizeof(struct virtio_blk_config)) {
+		DRV_LOG(ERR, "Invalid len: %u, required: %u",
+			len, (uint32_t)sizeof(struct virtio_blk_config));
+		return -1;
+	}

I believe it should not be > sizeof(struct virtio_blk_config) too?

I think I can make it if (len != sizeof(struct virtio_blk_config))

Quoted 5 lines

And one question is should we limit only reading the whole blk config. I guess we are having this check because of current usage of QEMU? Although this is fine as it's vendor-specific logic, just wondering if we should have this limit for blk device.

I think if user want to read any part of the config space,
The args should be (int vid, uint8_t *config, uint32_t offset, uint32_t len). offset mean the offset of struct virtio_blk_config to be the start of user needed data, the len means the data size user need to read.
That will change the message between vhost and QEMU( or other software). I thinks that is another story.

Quoted 89 lines

Thanks,
Chenbo

Quoted 63 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +62 -0
+
+	vdev = rte_vhost_get_vdpa_device(vid);
+	list = find_internal_resource_by_vdev(vdev);
+	if (list == NULL) {
+		DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -1;
+	}
+
+	internal = list->internal;
+
+	for (i = 0; i < sizeof(struct virtio_blk_config); i++)
+		config[i] = *((u8 *)internal->hw.blk_cfg + i);
+
+	dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg;
+
+	/* cannot read 64-bit register in one attempt, so read byte by byte.
 */
+	for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+		byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+		capacity |= (uint64_t)*byte << (i * 8);
+	}
+	/* The capacity is number of sectors in 512-byte.
+	 * So right shift 1 bit  we get in K,
+	 * another right shift 10 bits we get in M,
+	 * right shift 10 more bits, we get in G.
+	 * To show capacity in G, we right shift 21 bits in total.
+	 */
+	DRV_LOG(DEBUG, "capacity  : %"PRIu64"G", capacity >> 21);
+
+	DRV_LOG(DEBUG, "size_max  : 0x%08x", dev_cfg->size_max);
+	DRV_LOG(DEBUG, "seg_max   : 0x%08x", dev_cfg->seg_max);
+	DRV_LOG(DEBUG, "blk_size  : 0x%08x", dev_cfg->blk_size);
+	DRV_LOG(DEBUG, "geometry");
+	DRV_LOG(DEBUG, "      cylinders: %u", dev_cfg->geometry.cylinders);
+	DRV_LOG(DEBUG, "      heads    : %u", dev_cfg->geometry.heads);
+	DRV_LOG(DEBUG, "      sectors  : %u", dev_cfg->geometry.sectors);
+	DRV_LOG(DEBUG, "num_queues: 0x%08x", dev_cfg->num_queues);
+
+	DRV_LOG(DEBUG, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n",
+		config[0], config[1], config[2], config[3], config[4],
+		config[5], config[6], config[7]);
+	return 0;
+}
+
+static int
+ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev,
+	uint64_t *features)
+{
+	RTE_SET_USED(vdev);
+
+	*features = VDPA_SUPPORTED_PROTOCOL_FEATURES;
+	*features |= VDPA_BLK_PROTOCOL_FEATURES;
+	return 0;
+}
+
+static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
+	.get_queue_num = ifcvf_get_queue_num,
+	.get_features = ifcvf_get_vdpa_features,
+	.set_features = ifcvf_set_features,
+	.get_protocol_features = ifcvf_blk_get_protocol_features,
+	.dev_conf = ifcvf_dev_config,
+	.dev_close = ifcvf_dev_close,
+	.set_vring_state = ifcvf_set_vring_state,

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1293

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7 -0
+	.migration_done = NULL,
+	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
+	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
+	.get_notify_area = ifcvf_get_notify_area,
+	.get_config = ifcvf_blk_get_config,
+};
+
 struct rte_vdpa_dev_info dev_info[] = {
 	{
 		.features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |

@@ -1211,7

drivers/vdpa/ifc/ifcvf_vdpa.c

Quoted 3 lines

+1300,7 @@ struct rte_vdpa_dev_info dev_info[] = {

drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
 	{
 		.features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES)

|

Quoted 8 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -2
 			    (1ULL << VHOST_F_LOG_ALL),
-		.ops = NULL,
+		.ops = &ifcvf_blk_ops,
 	},
 };
 
--

1.8.3.1

author
#176 lore raw

PATCH v8 5/13 vdpa/ifc: add vDPA interrupt relay for blk device

In reply to Andy Pei

For the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +8 -1
index 350214a..509a1ed 100644
@@ -369,6 +369,7 @@ struct rte_vdpa_dev_info {
 	irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;
 	irq_set->start = 0;
 	fd_ptr = (int *)&irq_set->data;
+	/* The first interrupt is for the configure space change notification */
 	fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] =
 		rte_intr_fd_get(internal->pdev->intr_handle);
 
@@ -378,7 +379,13 @@ struct rte_vdpa_dev_info {
 	for (i = 0; i < nr_vring; i++) {
 		rte_vhost_get_vhost_vring(internal->vid, i, &vring);
 		fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
-		if ((i & 1) == 0 && m_rx == true) {
+		if (m_rx == true &&
+			((i & 1) == 0 || internal->hw.device_type == IFCVF_BLK)) {
+			/* For the net we only need to relay rx queue,
+			 * which will change the mem of VM.
+			 * For the blk we need to relay all the read cmd
+			 * of each queue
+			 */
 			fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
 			if (fd < 0) {
 				DRV_LOG(ERR, "can't setup eventfd: %s",
Signature · 1 line
1.8.3.1
Reviewed-by on v8 5/13
#177 lore raw

RE: [PATCH v8 05/13] vdpa/ifc: add vDPA interrupt relay for blk device

Quoted 47 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 05/13] vdpa/ifc: add vDPA interrupt relay for blk device

For the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
index 350214a..509a1ed 100644
@@ -369,6 +369,7 @@ struct rte_vdpa_dev_info {
 	irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;
 	irq_set->start = 0;
 	fd_ptr = (int *)&irq_set->data;
+	/* The first interrupt is for the configure space change

notification */

drivers/vdpa/ifc/ifcvf_vdpa.c +7 -1
 	fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] =
 		rte_intr_fd_get(internal->pdev->intr_handle);
@@ -378,7 +379,13 @@ struct rte_vdpa_dev_info {
 	for (i = 0; i < nr_vring; i++) {
 		rte_vhost_get_vhost_vring(internal->vid, i, &vring);
 		fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
-		if ((i & 1) == 0 && m_rx == true) {
+		if (m_rx == true &&
+			((i & 1) == 0 || internal->hw.device_type == IFCVF_BLK))
 {
+			/* For the net we only need to relay rx queue,
+			 * which will change the mem of VM.
+			 * For the blk we need to relay all the read cmd
+			 * of each queue
+			 */
 			fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
 			if (fd < 0) {
Quoted 3 lines

DRV_LOG(ERR, "can't setup eventfd: %s",
--
1.8.3.1

author
#178 lore raw

PATCH v8 6/13 vdpa/ifc: add block device SW live-migration

In reply to Andy Pei

Add SW live-migration support to block device. For block device, it is critical that no packet should be dropped. So when virtio blk device is pause, make sure hardware last_avail_idx and last_used_idx are the same. This indicates all requests have recieved acks, and no inflight IO.

---
 drivers/vdpa/ifc/base/ifcvf.h |  1 +
 drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 39 insertions(+), 4 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
index 244de46..4fb1736 100644
@@ -65,6 +65,7 @@
 #define IFCVF_MEDIATED_VRING		0x200000000000
 
 #define IFCVF_32_BIT_MASK		0xffffffff
+#define IFCVF_16_BIT_MASK		0xffff
 
 
 #ifndef VHOST_USER_PROTOCOL_F_CONFIG
drivers/vdpa/ifc/ifcvf_vdpa.c +38 -4

69 lines not on this page · raw message

Signature · 1 line
1.8.3.1
Reviewed-by on v8 6/13
#179 lore raw

RE: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration

Quoted 9 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration

Quoted 4 lines

Add SW live-migration support to block device.
For block device, it is critical that no packet
should be dropped. So when virtio blk device is
pause, make sure hardware last_avail_idx and

paused

Quoted 2 lines

last_used_idx are the same. This indicates all
requests have recieved acks, and no inflight IO.

received

With above fixed:

Quoted 99 lines
---
 drivers/vdpa/ifc/base/ifcvf.h |  1 +
 drivers/vdpa/ifc/ifcvf_vdpa.c | 42
++++++++++++++++++++++++++++++++++++++----
 2 files changed, 39 insertions(+), 4 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
index 244de46..4fb1736 100644
@@ -65,6 +65,7 @@
 #define IFCVF_MEDIATED_VRING		0x200000000000
 
 #define IFCVF_32_BIT_MASK		0xffffffff
+#define IFCVF_16_BIT_MASK		0xffff
 
 
 #ifndef VHOST_USER_PROTOCOL_F_CONFIG
drivers/vdpa/ifc/ifcvf_vdpa.c +38 -5
index 509a1ed..3e78c7d 100644
@@ -316,8 +316,34 @@ struct rte_vdpa_dev_info {
 	uint64_t features = 0;
 	uint64_t log_base = 0, log_size = 0;
 	uint64_t len;
+	u32 ring_state = 0;
 
 	vid = internal->vid;
+
+	/* to make sure no packet is lost for blk device
+	 * do not stop until last_avail_idx == last_used_idx
+	 */
+	if (internal->hw.device_type == IFCVF_BLK) {
+		for (i = 0; i < hw->nr_vring; i++) {
+			do {
+				if (hw->lm_cfg != NULL)
+					ring_state = *(u32 *)(hw->lm_cfg +
+						IFCVF_LM_RING_STATE_OFFSET +
+						i * IFCVF_LM_CFG_SIZE);
+				hw->vring[i].last_avail_idx =
+					(u16)(ring_state & IFCVF_16_BIT_MASK);
+				hw->vring[i].last_used_idx =
+					(u16)(ring_state >> 16);
+				if (hw->vring[i].last_avail_idx !=
+					hw->vring[i].last_used_idx) {
+					ifcvf_notify_queue(hw, i);
+					usleep(10);
+				}
+			} while (hw->vring[i].last_avail_idx !=
+				hw->vring[i].last_used_idx);
+		}
+	}
+
 	ifcvf_stop_hw(hw);
 
 	for (i = 0; i < hw->nr_vring; i++)
@@ -641,8 +667,10 @@ struct rte_vdpa_dev_info {
 		}
 		hw->vring[i].avail = gpa;
 
-		/* Direct I/O for Tx queue, relay for Rx queue */
-		if (i & 1) {
+		/* NET: Direct I/O for Tx queue, relay for Rx queue
+		 * BLK: relay every queue
+		 */
+		if ((internal->hw.device_type == IFCVF_NET) && (i & 1)) {
 			gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used);
 			if (gpa == 0) {
 				DRV_LOG(ERR, "Fail to get GPA for used ring.");
@@ -692,8 +720,12 @@ struct rte_vdpa_dev_info {
 
 	for (i = 0; i < hw->nr_vring; i++) {
 		/* synchronize remaining new used entries if any */
-		if ((i & 1) == 0)
+		if (internal->hw.device_type == IFCVF_NET) {
+			if ((i & 1) == 0)
+				update_used_ring(internal, i);
+		} else if (internal->hw.device_type == IFCVF_BLK) {
 			update_used_ring(internal, i);
+		}
 
 		rte_vhost_get_vhost_vring(vid, i, &vq);
 		len = IFCVF_USED_RING_LEN(vq.size);
@@ -755,7 +787,9 @@ struct rte_vdpa_dev_info {
 		}
 	}
 
-	for (qid = 0; qid < q_num; qid += 2) {
+	for (qid = 0; qid < q_num; qid += 1) {
+		if ((internal->hw.device_type == IFCVF_NET) && (qid & 1))
+			continue;
 		ev.events = EPOLLIN | EPOLLPRI;
 		/* leave a flag to mark it's for interrupt */
 		ev.data.u64 = 1 | qid << 1 |
--

1.8.3.1

author
#180 lore raw

RE: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration

Hi Chenbo,

Thanks for your reply, I will fix them in next version.

Quoted 137 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 1:25 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration

Quoted 10 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration

Quoted 3 lines

Add SW live-migration support to block device.
For block device, it is critical that no packet should be dropped. So when virtio blk device is pause, make sure hardware last_avail_idx and

paused

Quoted 2 lines

last_used_idx are the same. This indicates all requests have recieved acks, and no inflight IO.

received

With above fixed:

Quoted 31 lines
---
 drivers/vdpa/ifc/base/ifcvf.h |  1 +
 drivers/vdpa/ifc/ifcvf_vdpa.c | 42
++++++++++++++++++++++++++++++++++++++----
 2 files changed, 39 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/base/ifcvf.h index 244de46..4fb1736 100644

drivers/vdpa/ifc/base/ifcvf.h +1 -0
@@ -65,6 +65,7 @@
 #define IFCVF_MEDIATED_VRING		0x200000000000
 
 #define IFCVF_32_BIT_MASK		0xffffffff
+#define IFCVF_16_BIT_MASK		0xffff
 
 
 #ifndef VHOST_USER_PROTOCOL_F_CONFIG
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index 509a1ed..3e78c7d 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
@@ -316,8 +316,34 @@ struct rte_vdpa_dev_info {
 	uint64_t features = 0;
 	uint64_t log_base = 0, log_size = 0;
 	uint64_t len;
+	u32 ring_state = 0;
 
 	vid = internal->vid;

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 331

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10 -0
+
+	/* to make sure no packet is lost for blk device
+	 * do not stop until last_avail_idx == last_used_idx
+	 */
+	if (internal->hw.device_type == IFCVF_BLK) {
+		for (i = 0; i < hw->nr_vring; i++) {
+			do {
+				if (hw->lm_cfg != NULL)
+					ring_state = *(u32 *)(hw->lm_cfg +
+

IFCVF_LM_RING_STATE_OFFSET +

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info {

Quoted 3 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +3 -0
+						i * IFCVF_LM_CFG_SIZE);
+				hw->vring[i].last_avail_idx =
+					(u16)(ring_state &

IFCVF_16_BIT_MASK);

Quoted 18 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +12 -0
+				hw->vring[i].last_used_idx =
+					(u16)(ring_state >> 16);
+				if (hw->vring[i].last_avail_idx !=
+					hw->vring[i].last_used_idx) {
+					ifcvf_notify_queue(hw, i);
+					usleep(10);
+				}
+			} while (hw->vring[i].last_avail_idx !=
+				hw->vring[i].last_used_idx);
+		}
+	}
+
 	ifcvf_stop_hw(hw);
 
 	for (i = 0; i < hw->nr_vring; i++)
@@ -641,8 +667,10 @@ struct rte_vdpa_dev_info {
 		}
 		hw->vring[i].avail = gpa;

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 676

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -2
 
-		/* Direct I/O for Tx queue, relay for Rx queue */
-		if (i & 1) {
+		/* NET: Direct I/O for Tx queue, relay for Rx queue
+		 * BLK: relay every queue
+		 */
+		if ((internal->hw.device_type == IFCVF_NET) && (i & 1)) {
 			gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used);
 			if (gpa == 0) {
 				DRV_LOG(ERR, "Fail to get GPA for used

ring."); @@ -692,8 +720,12

Quoted 4 lines

@@ struct rte_vdpa_dev_info {

drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
 	for (i = 0; i < hw->nr_vring; i++) {
 		/* synchronize remaining new used entries if any */

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info {

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5 -1
-		if ((i & 1) == 0)
+		if (internal->hw.device_type == IFCVF_NET) {
+			if ((i & 1) == 0)
+				update_used_ring(internal, i);
+		} else if (internal->hw.device_type == IFCVF_BLK) {
 			update_used_ring(internal, i);
+		}
 
 		rte_vhost_get_vhost_vring(vid, i, &vq);
 		len = IFCVF_USED_RING_LEN(vq.size); @@ -755,7 +787,9

@@ struct

Quoted 13 lines

rte_vdpa_dev_info {

drivers/vdpa/ifc/ifcvf_vdpa.c +3 -2
 		}
 	}
 
-	for (qid = 0; qid < q_num; qid += 2) {
+	for (qid = 0; qid < q_num; qid += 1) {
+		if ((internal->hw.device_type == IFCVF_NET) && (qid & 1))
+			continue;
 		ev.events = EPOLLIN | EPOLLPRI;
 		/* leave a flag to mark it's for interrupt */
 		ev.data.u64 = 1 | qid << 1 |
--

1.8.3.1

author
#181 lore raw

PATCH v8 7/13 vhost: add API to get vDPA device type

In reply to Andy Pei

Vhost backend of different devices have different features. Add a API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++++
 doc/guides/rel_notes/release_22_07.rst |  4 ++++
 lib/vhost/rte_vhost.h                  | 17 +++++++++++++
 lib/vhost/socket.c                     | 44 ++++++++++++++++++++++++++++++++++
 lib/vhost/vdpa_driver.h                |  3 +++
 lib/vhost/version.map                  |  1 +
 6 files changed, 74 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +5 -0
index f287b76..0337b38 100644
@@ -282,6 +282,11 @@ The following is an overview of some key Vhost API functions:
   Clear inflight packets which are submitted to DMA engine in vhost async data
   path. Completed packets are returned to applications through ``pkts``.
 
+* ``rte_vhost_driver_get_vdpa_dev_type(path, type)``
+
+  Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
+  VDPA_DEVICE_TYPE_BLK.
+
 Vhost-user Implementations
 --------------------------
 
doc/guides/rel_notes/release_22_07.rst +4 -0
index e49cace..9550977 100644
@@ -104,6 +104,10 @@ New Features
   * ``RTE_EVENT_QUEUE_ATTR_WEIGHT``
   * ``RTE_EVENT_QUEUE_ATTR_AFFINITY``
 
+* **Added vhost API to get the device type of a vDPA device.**
+
+  Added an API which can get the device type of vDPA device.
+
 
 Removed Items
 -------------
lib/vhost/rte_vhost.h +17 -0
index c733f85..c977a24 100644
@@ -117,6 +117,9 @@
 
 #define RTE_MAX_VHOST_DEVICE	1024
 
+#define VDPA_DEVICE_TYPE_NET 0
+#define VDPA_DEVICE_TYPE_BLK 1
+
 struct rte_vdpa_device;
 
 /**
@@ -486,6 +489,20 @@ struct rte_vdpa_device *
 rte_vhost_driver_get_vdpa_device(const char *path);
 
 /**
+ * Get the device type of the vdpa device.
+ *
+ * @param path
+ *  The vhost-user socket file path
+ * @param type
+ *  the device type of the vdpa device
+ * @return
+ *  0 on success, -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type);
+
+/**
  * Set the feature bits the vhost-user driver supports.
  *
  * @param path
lib/vhost/socket.c +44 -0

50 lines not on this page · raw message

lib/vhost/vdpa_driver.h +3 -0
index e59a834..9cbd7cd 100644
@@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops {
 	/** Set the device configuration space */
 	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
 		      uint32_t size, uint32_t flags);
+
+	/** get device type: net device, blk device... */
+	int (*get_dev_type)(struct rte_vdpa_device *dev, uint32_t *type);
 };
 
 /**
lib/vhost/version.map +1 -0
index 5841315..583b4f3 100644
@@ -90,6 +90,7 @@ EXPERIMENTAL {
 
 	# added in 22.07
 	rte_vhost_async_get_inflight_thread_unsafe;
+	rte_vhost_driver_get_vdpa_dev_type;
 
 };
 
Signature · 1 line
1.8.3.1
#182 lore raw

RE: [PATCH v8 07/13] vhost: add API to get vDPA device type

Quoted 11 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 07/13] vhost: add API to get vDPA device type

Vhost backend of different devices have different features. Add a API to get vDPA device type, net device or blk device

an API

Quoted 39 lines

currently, so users can set different features for different kinds of devices.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++++
 doc/guides/rel_notes/release_22_07.rst |  4 ++++
 lib/vhost/rte_vhost.h                  | 17 +++++++++++++
 lib/vhost/socket.c                     | 44
++++++++++++++++++++++++++++++++++
 lib/vhost/vdpa_driver.h                |  3 +++
 lib/vhost/version.map                  |  1 +
 6 files changed, 74 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +0 -0
index f287b76..0337b38 100644
@@ -282,6 +282,11 @@ The following is an overview of some key Vhost API

functions:
Clear inflight packets which are submitted to DMA engine in vhost async
data
path. Completed packets are returned to applications through ``pkts``.

doc/guides/prog_guide/vhost_lib.rst +5 -0
+* ``rte_vhost_driver_get_vdpa_dev_type(path, type)``
+
+  Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
+  VDPA_DEVICE_TYPE_BLK.
+
 Vhost-user Implementations
 --------------------------
doc/guides/rel_notes/release_22_07.rst +0 -0
index e49cace..9550977 100644

doc/guides/rel_notes/release_22_07.rst New Features · line 110

Quoted 7 lines
doc/guides/rel_notes/release_22_07.rst +4 -0
@@ -104,6 +104,10 @@ New Features
   * ``RTE_EVENT_QUEUE_ATTR_WEIGHT``
   * ``RTE_EVENT_QUEUE_ATTR_AFFINITY``
 
+* **Added vhost API to get the device type of a vDPA device.**
+
+  Added an API which can get the device type of vDPA device.
+

Release note has a requirement of order, check comment under 'New features' in release notes.

Based on these comments, you should put this after 'Added vhost API to get the number of in-flight packets'

lib/vhost/rte_vhost.h line 121

Quoted 13 lines
doc/guides/rel_notes/release_22_07.rst +0 -0
 Removed Items
 -------------
lib/vhost/rte_vhost.h +2 -0
index c733f85..c977a24 100644
@@ -117,6 +117,9 @@
 
 #define RTE_MAX_VHOST_DEVICE	1024
 
+#define VDPA_DEVICE_TYPE_NET 0
+#define VDPA_DEVICE_TYPE_BLK 1

These will be new APIs of vhost lib. I suggest to rename them by adding prefix 'RTE_VHOST_'

Thanks,
Chenbo

Quoted 109 lines
lib/vhost/rte_vhost.h +15 -0
+
 struct rte_vdpa_device;
 
 /**
@@ -486,6 +489,20 @@ struct rte_vdpa_device *
 rte_vhost_driver_get_vdpa_device(const char *path);
 
 /**
+ * Get the device type of the vdpa device.
+ *
+ * @param path
+ *  The vhost-user socket file path
+ * @param type
+ *  the device type of the vdpa device
+ * @return
+ *  0 on success, -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type);
+
+/**
  * Set the feature bits the vhost-user driver supports.
  *
  * @param path
lib/vhost/socket.c +44 -0
index b304339..ef0f401 100644
@@ -619,6 +619,50 @@ struct rte_vdpa_device *
 }
 
 int
+rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type)
+{
+	struct vhost_user_socket *vsocket;
+	struct rte_vdpa_device *vdpa_dev;
+	uint32_t vdpa_type = 0;
+	int ret = 0;
+
+	pthread_mutex_lock(&vhost_user.mutex);
+	vsocket = find_vhost_user_socket(path);
+	if (!vsocket) {
+		VHOST_LOG_CONFIG(ERR,
+				 "(%s) socket file is not registered yet.\n",
+				 path);
+		ret = -1;
+		goto unlock_exit;
+	}
+
+	vdpa_dev = vsocket->vdpa_dev;
+	if (!vdpa_dev) {
+		ret = -1;
+		goto unlock_exit;
+	}
+
+	if (vdpa_dev->ops->get_dev_type) {
+		ret = vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type);
+		if (ret) {
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) failed to get vdpa dev type for
 socket file.\n",
+					 path);
+			ret = -1;
+			goto unlock_exit;
+		}
+	} else {
+		vdpa_type = VDPA_DEVICE_TYPE_NET;
+	}
+
+	*type = vdpa_type;
+
+unlock_exit:
+	pthread_mutex_unlock(&vhost_user.mutex);
+	return ret;
+}
+
+int
 rte_vhost_driver_disable_features(const char *path, uint64_t features)
 {
lib/vhost/socket.c +0 -0
 	struct vhost_user_socket *vsocket;
lib/vhost/vdpa_driver.h +3 -0
index e59a834..9cbd7cd 100644
@@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops {
 	/** Set the device configuration space */
 	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
 		      uint32_t size, uint32_t flags);
+
+	/** get device type: net device, blk device... */
+	int (*get_dev_type)(struct rte_vdpa_device *dev, uint32_t *type);
 };
 
 /**
lib/vhost/version.map +1 -1
index 5841315..583b4f3 100644
@@ -90,6 +90,7 @@ EXPERIMENTAL {
 
 	# added in 22.07
 	rte_vhost_async_get_inflight_thread_unsafe;
+	rte_vhost_driver_get_vdpa_dev_type;
 
 };
 
--

1.8.3.1

author
#183 lore raw

RE: [PATCH v8 07/13] vhost: add API to get vDPA device type

Hi Chenbo,

Thanks for your reply.
I will send a new version to address these.

Quoted 207 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 3:26 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 07/13] vhost: add API to get vDPA device type

Quoted 10 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 07/13] vhost: add API to get vDPA device type

Quoted 2 lines

Vhost backend of different devices have different features. Add a API to get vDPA device type, net device or blk device

an API

Quoted 40 lines

currently, so users can set different features for different kinds of devices.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++++
 doc/guides/rel_notes/release_22_07.rst |  4 ++++
 lib/vhost/rte_vhost.h                  | 17 +++++++++++++
 lib/vhost/socket.c                     | 44
++++++++++++++++++++++++++++++++++
 lib/vhost/vdpa_driver.h                |  3 +++
 lib/vhost/version.map                  |  1 +
 6 files changed, 74 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +0 -0
index f287b76..0337b38 100644
@@ -282,6 +282,11 @@ The following is an overview of some key Vhost

API
functions:
Clear inflight packets which are submitted to DMA engine in vhost
async data
path. Completed packets are returned to applications through ``pkts``.

doc/guides/prog_guide/vhost_lib.rst +5 -0
+* ``rte_vhost_driver_get_vdpa_dev_type(path, type)``
+
+  Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
+ VDPA_DEVICE_TYPE_BLK.
+
 Vhost-user Implementations
 --------------------------
doc/guides/rel_notes/release_22_07.rst +0 -0
index e49cace..9550977 100644

doc/guides/rel_notes/release_22_07.rst New Features · line 110

Quoted 7 lines
doc/guides/rel_notes/release_22_07.rst +4 -0
@@ -104,6 +104,10 @@ New Features
   * ``RTE_EVENT_QUEUE_ATTR_WEIGHT``
   * ``RTE_EVENT_QUEUE_ATTR_AFFINITY``
 
+* **Added vhost API to get the device type of a vDPA device.**
+
+  Added an API which can get the device type of vDPA device.
+

Release note has a requirement of order, check comment under 'New features'
in release notes.

Based on these comments, you should put this after 'Added vhost API to get the number of in-flight packets'

lib/vhost/rte_vhost.h line 121

Quoted 13 lines
doc/guides/rel_notes/release_22_07.rst +0 -0
 Removed Items
 -------------
lib/vhost/rte_vhost.h index renamed from lib/vhost/rte_vhost.h +0 -0

c733f85..c977a24 100644

lib/vhost/rte_vhost.h +2 -0
@@ -117,6 +117,9 @@
 
 #define RTE_MAX_VHOST_DEVICE	1024
 
+#define VDPA_DEVICE_TYPE_NET 0
+#define VDPA_DEVICE_TYPE_BLK 1

These will be new APIs of vhost lib. I suggest to rename them by adding prefix 'RTE_VHOST_'

Thanks,
Chenbo

Quoted 108 lines
lib/vhost/rte_vhost.h +15 -0
+
 struct rte_vdpa_device;
 
 /**
@@ -486,6 +489,20 @@ struct rte_vdpa_device *
 rte_vhost_driver_get_vdpa_device(const char *path);
 
 /**
+ * Get the device type of the vdpa device.
+ *
+ * @param path
+ *  The vhost-user socket file path
+ * @param type
+ *  the device type of the vdpa device
+ * @return
+ *  0 on success, -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type);
+
+/**
  * Set the feature bits the vhost-user driver supports.
  *
  * @param path
lib/vhost/socket.c index renamed from lib/vhost/socket.c +0 -0

b304339..ef0f401 100644

lib/vhost/socket.c +44 -0
@@ -619,6 +619,50 @@ struct rte_vdpa_device *  }
 
 int
+rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type)
+{
+	struct vhost_user_socket *vsocket;
+	struct rte_vdpa_device *vdpa_dev;
+	uint32_t vdpa_type = 0;
+	int ret = 0;
+
+	pthread_mutex_lock(&vhost_user.mutex);
+	vsocket = find_vhost_user_socket(path);
+	if (!vsocket) {
+		VHOST_LOG_CONFIG(ERR,
+				 "(%s) socket file is not registered yet.\n",
+				 path);
+		ret = -1;
+		goto unlock_exit;
+	}
+
+	vdpa_dev = vsocket->vdpa_dev;
+	if (!vdpa_dev) {
+		ret = -1;
+		goto unlock_exit;
+	}
+
+	if (vdpa_dev->ops->get_dev_type) {
+		ret = vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type);
+		if (ret) {
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) failed to get vdpa dev type for
 socket file.\n",
+					 path);
+			ret = -1;
+			goto unlock_exit;
+		}
+	} else {
+		vdpa_type = VDPA_DEVICE_TYPE_NET;
+	}
+
+	*type = vdpa_type;
+
+unlock_exit:
+	pthread_mutex_unlock(&vhost_user.mutex);
+	return ret;
+}
+
+int
 rte_vhost_driver_disable_features(const char *path, uint64_t
 features)  {
 	struct vhost_user_socket *vsocket;
lib/vhost/vdpa_driver.h index renamed from lib/vhost/vdpa_driver.h +0 -0

e59a834..9cbd7cd 100644

lib/vhost/vdpa_driver.h +3 -0
@@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops {
 	/** Set the device configuration space */
 	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
 		      uint32_t size, uint32_t flags);
+
+	/** get device type: net device, blk device... */
+	int (*get_dev_type)(struct rte_vdpa_device *dev, uint32_t *type);
 };
 
 /**
lib/vhost/version.map index renamed from lib/vhost/version.map +0 -0

5841315..583b4f3 100644

lib/vhost/version.map +1 -1
@@ -90,6 +90,7 @@ EXPERIMENTAL {
 
 	# added in 22.07
 	rte_vhost_async_get_inflight_thread_unsafe;
+	rte_vhost_driver_get_vdpa_dev_type;
 
 };
 
--

1.8.3.1

author
#184 lore raw

PATCH v8 8/13 vdpa/ifc: add get device type ops to ifc driver

In reply to Andy Pei

Add get device type ops to ifc driver.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +25 -0
index 3e78c7d..0acfa8c 100644
@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static int
+ifcvf_get_device_type(struct rte_vdpa_device *vdev,
+	uint32_t *type)
+{
+	struct ifcvf_internal *internal;
+	struct internal_list *list;
+
+	list = find_internal_resource_by_vdev(vdev);
+	if (list == NULL) {
+		DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -1;
+	}
+
+	internal = list->internal;
+
+	if (internal->hw.device_type == IFCVF_BLK)
+		*type = VDPA_DEVICE_TYPE_BLK;
+	else
+		*type = VDPA_DEVICE_TYPE_NET;
+
+	return 0;
+}
+
 static struct rte_vdpa_dev_ops ifcvf_ops = {
 	.get_queue_num = ifcvf_get_queue_num,
 	.get_features = ifcvf_get_vdpa_features,
@@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 static inline int
@@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
 	.get_config = ifcvf_blk_get_config,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 struct rte_vdpa_dev_info dev_info[] = {
Signature · 1 line
1.8.3.1
Reviewed-by on v8 8/13
#185 lore raw

RE: [PATCH v8 08/13] vdpa/ifc: add get device type ops to ifc driver

Quoted 67 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 08/13] vdpa/ifc: add get device type ops to ifc driver

Add get device type ops to ifc driver.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +25 -1
index 3e78c7d..0acfa8c 100644
@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static int
+ifcvf_get_device_type(struct rte_vdpa_device *vdev,
+	uint32_t *type)
+{
+	struct ifcvf_internal *internal;
+	struct internal_list *list;
+
+	list = find_internal_resource_by_vdev(vdev);
+	if (list == NULL) {
+		DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -1;
+	}
+
+	internal = list->internal;
+
+	if (internal->hw.device_type == IFCVF_BLK)
+		*type = VDPA_DEVICE_TYPE_BLK;
+	else
+		*type = VDPA_DEVICE_TYPE_NET;
+
+	return 0;
+}
+
 static struct rte_vdpa_dev_ops ifcvf_ops = {
 	.get_queue_num = ifcvf_get_queue_num,
 	.get_features = ifcvf_get_vdpa_features,
@@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 static inline int
@@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
 	.get_config = ifcvf_blk_get_config,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 struct rte_vdpa_dev_info dev_info[] = {
--
Quoted 1 line

1.8.3.1

With renaming fixed mentioned in patch 7:

author
#186 lore raw

RE: [PATCH v8 08/13] vdpa/ifc: add get device type ops to ifc driver

Sure

Quoted 83 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 3:30 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 08/13] vdpa/ifc: add get device type ops to ifc driver

Quoted 69 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 08/13] vdpa/ifc: add get device type ops to ifc driver

Add get device type ops to ifc driver.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index 3e78c7d..0acfa8c 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +23 -0
@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static int
+ifcvf_get_device_type(struct rte_vdpa_device *vdev,
+	uint32_t *type)
+{
+	struct ifcvf_internal *internal;
+	struct internal_list *list;
+
+	list = find_internal_resource_by_vdev(vdev);
+	if (list == NULL) {
+		DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -1;
+	}
+
+	internal = list->internal;
+
+	if (internal->hw.device_type == IFCVF_BLK)
+		*type = VDPA_DEVICE_TYPE_BLK;
+	else
+		*type = VDPA_DEVICE_TYPE_NET;
+
+	return 0;
+}
+
 static struct rte_vdpa_dev_ops ifcvf_ops = {
 	.get_queue_num = ifcvf_get_queue_num,
 	.get_features = ifcvf_get_vdpa_features, @@ -1201,6 +1224,7 @@

struct rte_vdpa_dev_info {

drivers/vdpa/ifc/ifcvf_vdpa.c +2 -1
 	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 static inline int
@@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
 	.get_config = ifcvf_blk_get_config,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 struct rte_vdpa_dev_info dev_info[] = {
--
Quoted 1 line

1.8.3.1

With renaming fixed mentioned in patch 7:

author
#187 lore raw

PATCH v8 9/13 examples/vdpa: add add virtio blk support

In reply to Andy Pei

Add virtio blk device support to vDPA example.

---
 examples/vdpa/main.c             | 56 ++++++++++++++++++++++++++++++++++++++
 examples/vdpa/vdpa_blk_compact.h | 58 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 114 insertions(+)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c +56 -0
index 5ab0765..62b6660 100644
@@ -20,6 +20,7 @@
 #include <cmdline_parse_string.h>
 #include <cmdline_parse_num.h>
 #include <cmdline.h>
+#include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
 #define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,53 @@ struct vdpa_port {
 };
 
 static int
+vdpa_blk_device_set_features_and_protocol(const char *path)
+{
+	uint64_t protocol_features = 0;
+	int ret;
+
+	ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_disable_features(path,
+		VHOST_BLK_DISABLED_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_disable_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_get_protocol_features(path, &protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_get_protocol_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	protocol_features |= VHOST_BLK_PROTOCOL_FEATURES;
+
+	ret = rte_vhost_driver_set_protocol_features(path, protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_protocol_features for %s failed.\n",
+			path);
+	}
+
+out:
+	return ret;
+}
+
+static int
 start_vdpa(struct vdpa_port *vport)
 {
+	uint32_t device_type = 0;
 	int ret;
 	char *socket_path = vport->ifname;
 
@@ -192,6 +238,16 @@ struct vdpa_port {
 			"attach vdpa device failed: %s\n",
 			socket_path);
 
+	ret = rte_vhost_driver_get_vdpa_dev_type(socket_path, &device_type);
+	if (ret == 0 && device_type == VDPA_DEVICE_TYPE_BLK) {
+		RTE_LOG(NOTICE, VDPA, "%s is a blk device\n", socket_path);
+		ret = vdpa_blk_device_set_features_and_protocol(socket_path);
+		if (ret != 0)
+			rte_exit(EXIT_FAILURE,
+				"set vhost blk driver features and protocol features failed: %s\n",
+				socket_path);
+	}
+
 	if (rte_vhost_driver_start(socket_path) < 0)
 		rte_exit(EXIT_FAILURE,
 			"start vhost driver failed: %s\n",
examples/vdpa/vdpa_blk_compact.h new file +58 -0

58 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#188 lore raw

RE: [PATCH v8 09/13] examples/vdpa: add add virtio blk support

Quoted 8 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 09/13] examples/vdpa: add add virtio blk support

Double add in the title

Quoted 109 lines

Add virtio blk device support to vDPA example.

---
 examples/vdpa/main.c             | 56
++++++++++++++++++++++++++++++++++++++
 examples/vdpa/vdpa_blk_compact.h | 58
++++++++++++++++++++++++++++++++++++++++
 2 files changed, 114 insertions(+)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c +56 -0
index 5ab0765..62b6660 100644
@@ -20,6 +20,7 @@
 #include <cmdline_parse_string.h>
 #include <cmdline_parse_num.h>
 #include <cmdline.h>
+#include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
 #define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,53 @@ struct vdpa_port {
 };
 
 static int
+vdpa_blk_device_set_features_and_protocol(const char *path)
+{
+	uint64_t protocol_features = 0;
+	int ret;
+
+	ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_disable_features(path,
+		VHOST_BLK_DISABLED_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_disable_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_get_protocol_features(path,
 &protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_get_protocol_features for %s
 failed.\n",
+			path);
+		goto out;
+	}
+
+	protocol_features |= VHOST_BLK_PROTOCOL_FEATURES;
+
+	ret = rte_vhost_driver_set_protocol_features(path,
 protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_protocol_features for %s
 failed.\n",
+			path);
+	}
+
+out:
+	return ret;
+}
+
+static int
 start_vdpa(struct vdpa_port *vport)
 {
+	uint32_t device_type = 0;
 	int ret;
 	char *socket_path = vport->ifname;
@@ -192,6 +238,16 @@ struct vdpa_port {
 			"attach vdpa device failed: %s\n",
 			socket_path);
 
+	ret = rte_vhost_driver_get_vdpa_dev_type(socket_path, &device_type);
+	if (ret == 0 && device_type == VDPA_DEVICE_TYPE_BLK) {
+		RTE_LOG(NOTICE, VDPA, "%s is a blk device\n", socket_path);
+		ret = vdpa_blk_device_set_features_and_protocol(socket_path);
+		if (ret != 0)
+			rte_exit(EXIT_FAILURE,
+				"set vhost blk driver features and protocol
 features failed: %s\n",
+				socket_path);
+	}
+
 	if (rte_vhost_driver_start(socket_path) < 0)
 		rte_exit(EXIT_FAILURE,
examples/vdpa/main.c +0 -0
 			"start vhost driver failed: %s\n",
examples/vdpa/vdpa_blk_compact.h new file +2 -0
new file mode 100644
index 0000000..4193561
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation

examples/vdpa/vdpa_blk_compact.h line 12

Quoted 10 lines
examples/vdpa/vdpa_blk_compact.h new file +10 -0
+ */
+
+#ifndef _VDPA_BLK_COMPACT_H_
+#define _VDPA_BLK_COMPACT_H_
+
+/**
+ * @file
+ *
+ * Device specific vhost lib
+ */

You describe the file with more details, or maybe just delete it.

Quoted 13 lines
examples/vdpa/vdpa_blk_compact.h new file +5 -0
+
+#include <rte_vhost.h>
+
+/* Feature bits */
+#define VIRTIO_BLK_F_SIZE_MAX     1    /* Indicates maximum segment size

*/
+#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of segments
*/

examples/vdpa/vdpa_blk_compact.h new file +2 -0
+#define VIRTIO_BLK_F_GEOMETRY     4    /* Legacy geometry available  */
+#define VIRTIO_BLK_F_BLK_SIZE     6    /* Block size of disk is available

*/
+#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is
available */

examples/vdpa/vdpa_blk_compact.h line 25

Quoted 4 lines
examples/vdpa/vdpa_blk_compact.h new file +4 -0
+#define VIRTIO_BLK_F_MQ           12   /* support more than one vq */
+
+/* Legacy feature bits */
+#ifndef VIRTIO_BLK_NO_LEGACY

This is always true? Is it possible VIRTIO_BLK_NO_LEGACY get defined?

Thanks,
Chenbo

Quoted 41 lines
examples/vdpa/vdpa_blk_compact.h new file +2 -0
+#define VIRTIO_BLK_F_BARRIER      0    /* Does host support barriers? */
+#define VIRTIO_BLK_F_SCSI         7    /* Supports scsi command passthru

*/
+#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in
config */

examples/vdpa/vdpa_blk_compact.h new file +10 -0
+#endif /* !VIRTIO_BLK_NO_LEGACY */
+
+#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \
+	(1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \
+	(1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \
+	(1ULL << VIRTIO_RING_F_EVENT_IDX) | \
+	(1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \
+	(1ULL << VIRTIO_F_VERSION_1))
+
+#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL <<

VIRTIO_F_NOTIFY_ON_EMPTY) | \

examples/vdpa/vdpa_blk_compact.h new file +5 -0
+	(1ULL << VIRTIO_RING_F_EVENT_IDX))
+
+#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \
+	(1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \
+	(1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) |

\

examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER)  |

\

examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_SCSI)     | (1ULL << VIRTIO_BLK_F_CONFIG_WCE)

| \

examples/vdpa/vdpa_blk_compact.h new file +13 -1
+	(1ULL << VIRTIO_BLK_F_MQ))
+
+/* Not supported features */
+#define VHOST_BLK_DISABLED_FEATURES (VHOST_BLK_DISABLED_FEATURES_BASE | \
+	(1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \
+	(1ULL << VIRTIO_BLK_F_SCSI)  | (1ULL << VIRTIO_BLK_F_CONFIG_WCE))
+
+/* Vhost-blk support protocol features */
+#define VHOST_BLK_PROTOCOL_FEATURES \
+	((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \
+	(1ULL << VHOST_USER_PROTOCOL_F_CONFIG))
+
+#endif /* _VDPA_BLK_COMPACT_H_ */
--

1.8.3.1

author
#189 lore raw

RE: [PATCH v8 09/13] examples/vdpa: add add virtio blk support

Hi Chenbo,
Thanks for your reply, my reply is inline.

Quoted 19 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 3:40 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 09/13] examples/vdpa: add add virtio blk support

Quoted 9 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 09/13] examples/vdpa: add add virtio blk support

Quoted 2 lines

Double add in the title

I will fix this in next version.

Quoted 112 lines
Quoted 81 lines

Add virtio blk device support to vDPA example.

---
 examples/vdpa/main.c             | 56
++++++++++++++++++++++++++++++++++++++
 examples/vdpa/vdpa_blk_compact.h | 58
++++++++++++++++++++++++++++++++++++++++
 2 files changed, 114 insertions(+)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c index renamed from examples/vdpa/main.c +0 -0

5ab0765..62b6660 100644

examples/vdpa/main.c +45 -0
@@ -20,6 +20,7 @@
 #include <cmdline_parse_string.h>
 #include <cmdline_parse_num.h>
 #include <cmdline.h>
+#include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
 #define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,53 @@ struct vdpa_port {  };
 
 static int
+vdpa_blk_device_set_features_and_protocol(const char *path) {
+	uint64_t protocol_features = 0;
+	int ret;
+
+	ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_disable_features(path,
+		VHOST_BLK_DISABLED_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_disable_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_get_protocol_features(path,
 &protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_get_protocol_features for %s
 failed.\n",
+			path);
+		goto out;
+	}
+
+	protocol_features |= VHOST_BLK_PROTOCOL_FEATURES;
+
+	ret = rte_vhost_driver_set_protocol_features(path,
 protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_protocol_features for %s
 failed.\n",
+			path);
+	}
+
+out:
+	return ret;
+}
+
+static int
 start_vdpa(struct vdpa_port *vport)
 {
+	uint32_t device_type = 0;
 	int ret;
 	char *socket_path = vport->ifname;

examples/vdpa/main.c struct vdpa_port { · line 241

Quoted 4 lines
examples/vdpa/main.c +1 -0
@@ -192,6 +238,16 @@ struct vdpa_port {
 			"attach vdpa device failed: %s\n",
 			socket_path);
 
+	ret = rte_vhost_driver_get_vdpa_dev_type(socket_path,

&device_type);

examples/vdpa/main.c struct vdpa_port {

Quoted 3 lines
examples/vdpa/main.c +3 -0
+	if (ret == 0 && device_type == VDPA_DEVICE_TYPE_BLK) {
+		RTE_LOG(NOTICE, VDPA, "%s is a blk device\n", socket_path);
+		ret =

vdpa_blk_device_set_features_and_protocol(socket_path);

Quoted 22 lines
examples/vdpa/main.c +6 -0
+		if (ret != 0)
+			rte_exit(EXIT_FAILURE,
+				"set vhost blk driver features and protocol
 features failed: %s\n",
+				socket_path);
+	}
+
 	if (rte_vhost_driver_start(socket_path) < 0)
 		rte_exit(EXIT_FAILURE,

"start vhost driver failed: %s\n", diff --git
a/examples/vdpa/vdpa_blk_compact.h b/examples/vdpa/vdpa_blk_compact.h
new file mode 100644
index 0000000..4193561

examples/vdpa/vdpa_blk_compact.h new file +5 -0
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation  */
+
+#ifndef _VDPA_BLK_COMPACT_H_
+#define _VDPA_BLK_COMPACT_H_
Quoted 9 lines

examples/vdpa/vdpa_blk_compact.h line 12

Quoted 6 lines
examples/vdpa/vdpa_blk_compact.h new file +6 -0
+
+/**
+ * @file
+ *
+ * Device specific vhost lib
+ */

You describe the file with more details, or maybe just delete it.

I will delete it in next version.

Quoted 9 lines

examples/vdpa/vdpa_blk_compact.h

Quoted 5 lines
examples/vdpa/vdpa_blk_compact.h new file +5 -0
+
+#include <rte_vhost.h>
+
+/* Feature bits */
+#define VIRTIO_BLK_F_SIZE_MAX     1    /* Indicates maximum segment

size

Quoted 2 lines

*/
+#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of

segments

Quoted 13 lines

examples/vdpa/vdpa_blk_compact.h line 25

Quoted 10 lines

*/

examples/vdpa/vdpa_blk_compact.h new file +2 -0
+#define VIRTIO_BLK_F_GEOMETRY     4    /* Legacy geometry available  */
+#define VIRTIO_BLK_F_BLK_SIZE     6    /* Block size of disk is available

*/
+#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is
available */

examples/vdpa/vdpa_blk_compact.h new file +4 -0
+#define VIRTIO_BLK_F_MQ           12   /* support more than one vq */
+
+/* Legacy feature bits */
+#ifndef VIRTIO_BLK_NO_LEGACY

This is always true? Is it possible VIRTIO_BLK_NO_LEGACY get defined?

The I will delete this condition.

Quoted 51 lines

Thanks,
Chenbo

Quoted 16 lines
examples/vdpa/vdpa_blk_compact.h new file +2 -0
+#define VIRTIO_BLK_F_BARRIER      0    /* Does host support barriers? */
+#define VIRTIO_BLK_F_SCSI         7    /* Supports scsi command passthru

*/
+#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in
config */

examples/vdpa/vdpa_blk_compact.h new file +10 -0
+#endif /* !VIRTIO_BLK_NO_LEGACY */
+
+#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \
+	(1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \
+	(1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \
+	(1ULL << VIRTIO_RING_F_EVENT_IDX) | \
+	(1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \
+	(1ULL << VIRTIO_F_VERSION_1))
+
+#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL <<

VIRTIO_F_NOTIFY_ON_EMPTY) | \

examples/vdpa/vdpa_blk_compact.h

Quoted 4 lines
examples/vdpa/vdpa_blk_compact.h new file +4 -0
+	(1ULL << VIRTIO_RING_F_EVENT_IDX))
+
+#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \
+	(1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL <<

VIRTIO_BLK_F_SEG_MAX) | \

examples/vdpa/vdpa_blk_compact.h

Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL <<

VIRTIO_BLK_F_BLK_SIZE) |

examples/vdpa/vdpa_blk_compact.h

Quoted 2 lines

\

examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL <<

VIRTIO_BLK_F_BARRIER) |

examples/vdpa/vdpa_blk_compact.h

Quoted 2 lines

\

examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_SCSI)     | (1ULL <<

VIRTIO_BLK_F_CONFIG_WCE)

examples/vdpa/vdpa_blk_compact.h

Quoted 5 lines

| \

examples/vdpa/vdpa_blk_compact.h new file +4 -0
+	(1ULL << VIRTIO_BLK_F_MQ))
+
+/* Not supported features */
+#define VHOST_BLK_DISABLED_FEATURES

(VHOST_BLK_DISABLED_FEATURES_BASE | \

examples/vdpa/vdpa_blk_compact.h

Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL <<

VIRTIO_BLK_F_BARRIER) | \

examples/vdpa/vdpa_blk_compact.h

Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1 -0
+	(1ULL << VIRTIO_BLK_F_SCSI)  | (1ULL <<

VIRTIO_BLK_F_CONFIG_WCE))

Quoted 9 lines
examples/vdpa/vdpa_blk_compact.h new file +7 -1
+
+/* Vhost-blk support protocol features */ #define
+VHOST_BLK_PROTOCOL_FEATURES \
+	((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \
+	(1ULL << VHOST_USER_PROTOCOL_F_CONFIG))
+
+#endif /* _VDPA_BLK_COMPACT_H_ */
--

1.8.3.1

author
#190 lore raw

PATCH v8 10/13 usertools: add support for virtio blk device

In reply to Andy Pei

Add virtio blk device support to devbind.

---
 usertools/dpdk-devbind.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-devbind.py +4 -1
index ace4627..7231be4 100755
@@ -72,6 +72,9 @@
 cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4',
                  'SVendor': None, 'SDevice': None}
 
+virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001',
+                    'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 baseband_devices = [acceleration_class]
 crypto_devices = [encryption_class, intel_processor_class]
@@ -82,7 +85,7 @@
 compress_devices = [cavium_zip]
 regex_devices = [cn9k_ree]
 misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev,
-                intel_ntb_skx, intel_ntb_icx]
+                intel_ntb_skx, intel_ntb_icx, virtio_blk]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
Signature · 1 line
1.8.3.1
#191 lore raw

RE: [PATCH v8 10/13] usertools: add support for virtio blk device

Quoted 21 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 10/13] usertools: add support for virtio blk device

Add virtio blk device support to devbind.

---
 usertools/dpdk-devbind.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-devbind.py +0 -0
index ace4627..7231be4 100755

usertools/dpdk-devbind.py line 75

Quoted 4 lines
usertools/dpdk-devbind.py +1 -0
@@ -72,6 +72,9 @@
 cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4',
                  'SVendor': None, 'SDevice': None}
 
+virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001',

No modern device support?

Thanks,
Chenbo

Quoted 16 lines
usertools/dpdk-devbind.py +3 -2
+                    'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 baseband_devices = [acceleration_class]
 crypto_devices = [encryption_class, intel_processor_class]
@@ -82,7 +85,7 @@
 compress_devices = [cavium_zip]
 regex_devices = [cn9k_ree]
 misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev,
-                intel_ntb_skx, intel_ntb_icx]
+                intel_ntb_skx, intel_ntb_icx, virtio_blk]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
--

1.8.3.1

author
#192 lore raw

RE: [PATCH v8 10/13] usertools: add support for virtio blk device

Hi Chenbo,

Quoted 34 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 3:43 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 10/13] usertools: add support for virtio blk device

Quoted 25 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 10/13] usertools: add support for virtio blk device

Add virtio blk device support to devbind.

---
 usertools/dpdk-devbind.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-devbind.py +0 -0
index ace4627..7231be4 100755
@@ -72,6 +72,9 @@
 cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4',
                  'SVendor': None, 'SDevice': None}
 
Quoted 4 lines

usertools/dpdk-devbind.py line 75

Quoted 1 line
usertools/dpdk-devbind.py +1 -0
+virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001',

No modern device support?

It should be supported. I will add modern device in next version.

Quoted 17 lines

Thanks,
Chenbo

Quoted 14 lines
usertools/dpdk-devbind.py +2 -0
+                    'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]

baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] @@ -82,7 +85,7 @@ compress_devices = [cavium_zip] regex_devices = [cn9k_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev,

usertools/dpdk-devbind.py +1 -2
-                intel_ntb_skx, intel_ntb_icx]
+                intel_ntb_skx, intel_ntb_icx, virtio_blk]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
--

1.8.3.1

author
#193 lore raw

PATCH v8 11/13 vdpa/ifc: add log for config space of virtio blk

In reply to Andy Pei

Add some log of virtio blk device config space information at VDPA launch before qemu connects.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +34 -0
index 0acfa8c..376a1af 100644
@@ -1382,6 +1382,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
 	int16_t device_id;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+	uint32_t i;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
@@ -1448,6 +1451,37 @@ struct rte_vdpa_dev_info dev_info[] = {
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_BLK].features;
+
+		/* cannot read 64-bit register in one attempt,
+		 * so read byte by byte.
+		 */
+		for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+			byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+			capacity |= (uint64_t)*byte << (i * 8);
+		}
+		/* The capacity is number of sectors in 512-byte.
+		 * So right shift 1 bit  we get in K,
+		 * another right shift 10 bits we get in M,
+		 * right shift 10 more bits, we get in G.
+		 * To show capacity in G, we right shift 21 bits in total.
+		 */
+		DRV_LOG(DEBUG, "capacity  : %"PRIu64"G", capacity >> 21);
+
+		DRV_LOG(DEBUG, "size_max  : 0x%08x",
+			internal->hw.blk_cfg->size_max);
+		DRV_LOG(DEBUG, "seg_max   : 0x%08x",
+			internal->hw.blk_cfg->seg_max);
+		DRV_LOG(DEBUG, "blk_size  : 0x%08x",
+			internal->hw.blk_cfg->blk_size);
+		DRV_LOG(DEBUG, "geometry");
+		DRV_LOG(DEBUG, "    cylinders: %u",
+			internal->hw.blk_cfg->geometry.cylinders);
+		DRV_LOG(DEBUG, "    heads    : %u",
+			internal->hw.blk_cfg->geometry.heads);
+		DRV_LOG(DEBUG, "    sectors  : %u",
+			internal->hw.blk_cfg->geometry.sectors);
+		DRV_LOG(DEBUG, "num_queues: 0x%08x",
+			internal->hw.blk_cfg->num_queues);
 	}
 
 	list->internal = internal;
Signature · 1 line
1.8.3.1
Reviewed-by on v8 11/13
#194 lore raw

RE: [PATCH v8 11/13] vdpa/ifc: add log for config space of virtio blk

Quoted 70 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 11/13] vdpa/ifc: add log for config space of virtio blk

Add some log of virtio blk device config space information at VDPA launch before qemu connects.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +34 -1
index 0acfa8c..376a1af 100644
@@ -1382,6 +1382,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
 	int16_t device_id;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+	uint32_t i;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
@@ -1448,6 +1451,37 @@ struct rte_vdpa_dev_info dev_info[] = {
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_BLK].features;
+
+		/* cannot read 64-bit register in one attempt,
+		 * so read byte by byte.
+		 */
+		for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+			byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+			capacity |= (uint64_t)*byte << (i * 8);
+		}
+		/* The capacity is number of sectors in 512-byte.
+		 * So right shift 1 bit  we get in K,
+		 * another right shift 10 bits we get in M,
+		 * right shift 10 more bits, we get in G.
+		 * To show capacity in G, we right shift 21 bits in total.
+		 */
+		DRV_LOG(DEBUG, "capacity  : %"PRIu64"G", capacity >> 21);
+
+		DRV_LOG(DEBUG, "size_max  : 0x%08x",
+			internal->hw.blk_cfg->size_max);
+		DRV_LOG(DEBUG, "seg_max   : 0x%08x",
+			internal->hw.blk_cfg->seg_max);
+		DRV_LOG(DEBUG, "blk_size  : 0x%08x",
+			internal->hw.blk_cfg->blk_size);
+		DRV_LOG(DEBUG, "geometry");
+		DRV_LOG(DEBUG, "    cylinders: %u",
+			internal->hw.blk_cfg->geometry.cylinders);
+		DRV_LOG(DEBUG, "    heads    : %u",
+			internal->hw.blk_cfg->geometry.heads);
+		DRV_LOG(DEBUG, "    sectors  : %u",
+			internal->hw.blk_cfg->geometry.sectors);
+		DRV_LOG(DEBUG, "num_queues: 0x%08x",
+			internal->hw.blk_cfg->num_queues);
 	}
 
 	list->internal = internal;
--
Quoted 1 line

1.8.3.1

author
#195 lore raw

PATCH v8 12/13 vdpa/ifc: add interrupt handling for config space

In reply to Andy Pei

Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 118 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 117 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +117 -1
index 376a1af..8a49622 100644
@@ -53,7 +53,9 @@ struct ifcvf_internal {
 	int vfio_group_fd;
 	int vfio_dev_fd;
 	pthread_t tid;	/* thread for notify relay */
+	pthread_t intr_tid; /* thread for config space change interrupt relay */
 	int epfd;
+	int csc_epfd;
 	int vid;
 	struct rte_vdpa_device *vdev;
 	uint16_t max_queues;
@@ -566,6 +568,111 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static void
+virtio_interrupt_handler(struct ifcvf_internal *internal)
+{
+	int vid = internal->vid;
+	int ret;
+
+	ret = rte_vhost_slave_config_change(vid, 1);
+	if (ret)
+		DRV_LOG(ERR, "failed to notify the guest about configuration space change.");
+}
+
+static void *
+intr_relay(void *arg)
+{
+	struct ifcvf_internal *internal = (struct ifcvf_internal *)arg;
+	struct epoll_event csc_event;
+	struct epoll_event ev;
+	uint64_t buf;
+	int nbytes;
+	int csc_epfd, csc_val = 0;
+
+	csc_epfd = epoll_create(1);
+	if (csc_epfd < 0) {
+		DRV_LOG(ERR, "failed to create epoll for config space change.");
+		return NULL;
+	}
+
+	ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP;
+	ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle);
+	if (epoll_ctl(csc_epfd, EPOLL_CTL_ADD,
+		rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) {
+		DRV_LOG(ERR, "epoll add error: %s", strerror(errno));
+		return NULL;
+	}
+
+	internal->csc_epfd = csc_epfd;
+
+	for (;;) {
+		csc_val = epoll_wait(csc_epfd, &csc_event, 1, -1);
+		if (csc_val < 0) {
+			if (errno == EINTR)
+				continue;
+			DRV_LOG(ERR, "epoll_wait return fail.");
+			return NULL;
+		} else if (csc_val == 0) {
+			continue;
+		} else {
+			/* csc_val > 0 */
+			nbytes = read(csc_event.data.fd, &buf, 8);
+			if (nbytes < 0) {
+				if (errno == EINTR ||
+				    errno == EWOULDBLOCK ||
+				    errno == EAGAIN)
+					continue;
+				DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n",
+					csc_event.data.fd,
+					strerror(errno));
+				return NULL;
+			} else if (nbytes == 0) {
+				DRV_LOG(ERR, "Read nothing from file descriptor %d\n",
+					csc_event.data.fd);
+				continue;
+			} else {
+				virtio_interrupt_handler(internal);
+			}
+		}
+	}
+
+	return NULL;
+}
+
+static int
+setup_intr_relay(struct ifcvf_internal *internal)
+{
+	char name[THREAD_NAME_LEN];
+	int ret;
+
+	snprintf(name, sizeof(name), "ifc-intr-%d", internal->vid);
+	ret = rte_ctrl_thread_create(&internal->intr_tid, name, NULL,
+				     intr_relay, (void *)internal);
+	if (ret) {
+		DRV_LOG(ERR, "failed to create notify relay pthread.");
+		return -1;
+	}
+	return 0;
+}
+
+static int
+unset_intr_relay(struct ifcvf_internal *internal)
+{
+	void *status;
+
+	if (internal->intr_tid) {
+		pthread_cancel(internal->intr_tid);
+		pthread_join(internal->intr_tid, &status);
+	}
+	internal->intr_tid = 0;
+
+	if (internal->csc_epfd >= 0)
+		close(internal->csc_epfd);
+	internal->csc_epfd = -1;
+
+	return 0;
+}
+
 static int
 update_datapath(struct ifcvf_internal *internal)
 {
@@ -592,10 +699,16 @@ struct rte_vdpa_dev_info {
 		if (ret)
 			goto err;
 
+		ret = setup_intr_relay(internal);
+		if (ret)
+			goto err;
+
 		rte_atomic32_set(&internal->running, 1);
 	} else if (rte_atomic32_read(&internal->running) &&
 		   (!rte_atomic32_read(&internal->started) ||
 		    !rte_atomic32_read(&internal->dev_attached))) {
+		ret = unset_intr_relay(internal);
+
 		ret = unset_notify_relay(internal);
 		if (ret)
 			goto err;
@@ -812,7 +925,7 @@ struct rte_vdpa_dev_info {
 		if (nfds < 0) {
 			if (errno == EINTR)
 				continue;
-			DRV_LOG(ERR, "epoll_wait return fail\n");
+			DRV_LOG(ERR, "epoll_wait return fail.");
 			return NULL;
 		}
 
@@ -888,6 +1001,9 @@ struct rte_vdpa_dev_info {
 	/* stop the direct IO data path */
 	unset_notify_relay(internal);
 	vdpa_ifcvf_stop(internal);
+
+	unset_intr_relay(internal);
+
 	vdpa_disable_vfio_intr(internal);
 
 	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
Signature · 1 line
1.8.3.1
#196 lore raw

RE: [PATCH v8 12/13] vdpa/ifc: add interrupt handling for config space

Quoted 102 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 12/13] vdpa/ifc: add interrupt handling for config space

Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 118
+++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 117 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
index 376a1af..8a49622 100644
@@ -53,7 +53,9 @@ struct ifcvf_internal {
 	int vfio_group_fd;
 	int vfio_dev_fd;
 	pthread_t tid;	/* thread for notify relay */
+	pthread_t intr_tid; /* thread for config space change interrupt

relay */

drivers/vdpa/ifc/ifcvf_vdpa.c +61 -0
 	int epfd;
+	int csc_epfd;
 	int vid;
 	struct rte_vdpa_device *vdev;
 	uint16_t max_queues;
@@ -566,6 +568,111 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static void
+virtio_interrupt_handler(struct ifcvf_internal *internal)
+{
+	int vid = internal->vid;
+	int ret;
+
+	ret = rte_vhost_slave_config_change(vid, 1);
+	if (ret)
+		DRV_LOG(ERR, "failed to notify the guest about configuration
 space change.");
+}
+
+static void *
+intr_relay(void *arg)
+{
+	struct ifcvf_internal *internal = (struct ifcvf_internal *)arg;
+	struct epoll_event csc_event;
+	struct epoll_event ev;
+	uint64_t buf;
+	int nbytes;
+	int csc_epfd, csc_val = 0;
+
+	csc_epfd = epoll_create(1);
+	if (csc_epfd < 0) {
+		DRV_LOG(ERR, "failed to create epoll for config space
 change.");
+		return NULL;
+	}
+
+	ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP;
+	ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle);
+	if (epoll_ctl(csc_epfd, EPOLL_CTL_ADD,
+		rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) {
+		DRV_LOG(ERR, "epoll add error: %s", strerror(errno));
+		return NULL;
+	}
+
+	internal->csc_epfd = csc_epfd;
+
+	for (;;) {
+		csc_val = epoll_wait(csc_epfd, &csc_event, 1, -1);
+		if (csc_val < 0) {
+			if (errno == EINTR)
+				continue;
+			DRV_LOG(ERR, "epoll_wait return fail.");
+			return NULL;
+		} else if (csc_val == 0) {
+			continue;
+		} else {
+			/* csc_val > 0 */
+			nbytes = read(csc_event.data.fd, &buf, 8);
+			if (nbytes < 0) {
+				if (errno == EINTR ||
+				    errno == EWOULDBLOCK ||
+				    errno == EAGAIN)
+					continue;
+				DRV_LOG(ERR, "Error reading from file
 descriptor %d: %s\n",
+					csc_event.data.fd,
+					strerror(errno));
+				return NULL;
+			} else if (nbytes == 0) {
+				DRV_LOG(ERR, "Read nothing from file

descriptor %d\n",

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 640

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10 -0
+					csc_event.data.fd);
+				continue;
+			} else {
+				virtio_interrupt_handler(internal);
+			}
+		}
+	}
+
+	return NULL;
+}

I think we should not assume unset_intr_relay will help us close epfd when Error happens, so just close it when there's some error.

Quoted 24 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +24 -0
+
+static int
+setup_intr_relay(struct ifcvf_internal *internal)
+{
+	char name[THREAD_NAME_LEN];
+	int ret;
+
+	snprintf(name, sizeof(name), "ifc-intr-%d", internal->vid);
+	ret = rte_ctrl_thread_create(&internal->intr_tid, name, NULL,
+				     intr_relay, (void *)internal);
+	if (ret) {
+		DRV_LOG(ERR, "failed to create notify relay pthread.");
+		return -1;
+	}
+	return 0;
+}
+
+static int
+unset_intr_relay(struct ifcvf_internal *internal)
+{
+	void *status;
+
+	if (internal->intr_tid) {
+		pthread_cancel(internal->intr_tid);

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 674

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10 -0
+		pthread_join(internal->intr_tid, &status);
+	}
+	internal->intr_tid = 0;
+
+	if (internal->csc_epfd >= 0)
+		close(internal->csc_epfd);
+	internal->csc_epfd = -1;
+
+	return 0;
+}

It will always return 0, so return type should be void

Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
+
 static int
 update_datapath(struct ifcvf_internal *internal)
 {
@@ -592,10 +699,16 @@ struct rte_vdpa_dev_info {
 		if (ret)
 			goto err;

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 710

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5 -0
 
+		ret = setup_intr_relay(internal);
+		if (ret)
+			goto err;
+
 		rte_atomic32_set(&internal->running, 1);
 	} else if (rte_atomic32_read(&internal->running) &&
 		   (!rte_atomic32_read(&internal->started) ||
 		    !rte_atomic32_read(&internal->dev_attached))) {
+		ret = unset_intr_relay(internal);

This will be changed accordingly.

Thanks,
Chenbo

Quoted 25 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5 -2
+
 		ret = unset_notify_relay(internal);
 		if (ret)
 			goto err;
@@ -812,7 +925,7 @@ struct rte_vdpa_dev_info {
 		if (nfds < 0) {
 			if (errno == EINTR)
 				continue;
-			DRV_LOG(ERR, "epoll_wait return fail\n");
+			DRV_LOG(ERR, "epoll_wait return fail.");
 			return NULL;
 		}
 
@@ -888,6 +1001,9 @@ struct rte_vdpa_dev_info {
 	/* stop the direct IO data path */
 	unset_notify_relay(internal);
 	vdpa_ifcvf_stop(internal);
+
+	unset_intr_relay(internal);
+
 	vdpa_disable_vfio_intr(internal);
 
 	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
--

1.8.3.1

author
#197 lore raw

PATCH v8 13/13 vdpa/ifc/base: access correct register for blk device

In reply to Andy Pei

Register address is different between net and blk device. We are re-use most of the code, when register address is different, we have to check net and blk device go through different code.

---
 drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +27 -7
index d10c1fd..dd475a7 100644
@@ -218,10 +218,17 @@
 				&cfg->queue_used_hi);
 		IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size);
 
-		*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) =
-			(u32)hw->vring[i].last_avail_idx |
-			((u32)hw->vring[i].last_used_idx << 16);
+		if (hw->device_type == IFCVF_BLK)
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				i * IFCVF_LM_CFG_SIZE) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
+		else
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				(i / 2) * IFCVF_LM_CFG_SIZE +
+				(i % 2) * 4) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
 
 		IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector);
 		if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -254,9 +261,22 @@
 		IFCVF_WRITE_REG16(i, &cfg->queue_select);
 		IFCVF_WRITE_REG16(0, &cfg->queue_enable);
 		IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector);
-		ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4);
-		hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
+
+		if (hw->device_type == IFCVF_BLK)
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					i * IFCVF_LM_CFG_SIZE);
+		else
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					(i / 2) * IFCVF_LM_CFG_SIZE +
+					(i % 2) * 4);
+
+		if (hw->device_type == IFCVF_BLK)
+			hw->vring[i].last_avail_idx =
+				(u16)(ring_state & IFCVF_16_BIT_MASK);
+		else
+			hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
 		hw->vring[i].last_used_idx = (u16)(ring_state >> 16);
 	}
 }
Signature · 1 line
1.8.3.1
Reviewed-by on v8 13/13
#198 lore raw

RE: [PATCH v8 13/13] vdpa/ifc/base: access correct register for blk device

Quoted 10 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 13/13] vdpa/ifc/base: access correct register for blk device

Quoted 2 lines

Register address is different between net and blk device. We are re-use most of the code, when register address is

Re-using

With this fixed:

Quoted 64 lines

different, we have to check net and blk device go through different code.

---
 drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +28 -8
index d10c1fd..dd475a7 100644
@@ -218,10 +218,17 @@
 				&cfg->queue_used_hi);
 		IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size);
 
-		*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) =
-			(u32)hw->vring[i].last_avail_idx |
-			((u32)hw->vring[i].last_used_idx << 16);
+		if (hw->device_type == IFCVF_BLK)
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				i * IFCVF_LM_CFG_SIZE) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
+		else
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				(i / 2) * IFCVF_LM_CFG_SIZE +
+				(i % 2) * 4) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
 
 		IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector);
 		if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -254,9 +261,22 @@
 		IFCVF_WRITE_REG16(i, &cfg->queue_select);
 		IFCVF_WRITE_REG16(0, &cfg->queue_enable);
 		IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg-
 >queue_msix_vector);
-		ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET
+
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4);
-		hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
+
+		if (hw->device_type == IFCVF_BLK)
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					i * IFCVF_LM_CFG_SIZE);
+		else
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					(i / 2) * IFCVF_LM_CFG_SIZE +
+					(i % 2) * 4);
+
+		if (hw->device_type == IFCVF_BLK)
+			hw->vring[i].last_avail_idx =
+				(u16)(ring_state & IFCVF_16_BIT_MASK);
+		else
+			hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
 		hw->vring[i].last_used_idx = (u16)(ring_state >> 16);
 	}
 }
--

1.8.3.1

author
#199 lore raw

RE: [PATCH v8 13/13] vdpa/ifc/base: access correct register for blk device

Hi Chenbo,

Thanks for your reply.
I will address in next version.

Quoted 96 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 3:56 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 13/13] vdpa/ifc/base: access correct register for blk device

Quoted 11 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 13/13] vdpa/ifc/base: access correct register for blk device

Quoted 2 lines

Register address is different between net and blk device. We are re-use most of the code, when register address is

Re-using

With this fixed:

Quoted 24 lines

different, we have to check net and blk device go through different code.

---
 drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/base/ifcvf.c index d10c1fd..dd475a7 100644

drivers/vdpa/ifc/base/ifcvf.c +4 -4
@@ -218,10 +218,17 @@
 				&cfg->queue_used_hi);
 		IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size);
 
-		*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) =
-			(u32)hw->vring[i].last_avail_idx |
-			((u32)hw->vring[i].last_used_idx << 16);
+		if (hw->device_type == IFCVF_BLK)
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				i * IFCVF_LM_CFG_SIZE) =
+				(u32)hw->vring[i].last_avail_idx |

drivers/vdpa/ifc/base/ifcvf.c line 234

Quoted 10 lines
drivers/vdpa/ifc/base/ifcvf.c +7 -0
+				((u32)hw->vring[i].last_used_idx << 16);
+		else
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				(i / 2) * IFCVF_LM_CFG_SIZE +
+				(i % 2) * 4) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
 
 		IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector);
 		if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -

254,9 +261,22

drivers/vdpa/ifc/base/ifcvf.c

Quoted 6 lines

@@

drivers/vdpa/ifc/base/ifcvf.c +0 -0
 		IFCVF_WRITE_REG16(i, &cfg->queue_select);
 		IFCVF_WRITE_REG16(0, &cfg->queue_enable);
 		IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg-
Quoted 1 line

queue_msix_vector);

drivers/vdpa/ifc/base/ifcvf.c +0 -1
-		ring_state = *(u32 *)(hw->lm_cfg +

IFCVF_LM_RING_STATE_OFFSET

Quoted 24 lines
drivers/vdpa/ifc/base/ifcvf.c +17 -3
+
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4);
-		hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
+
+		if (hw->device_type == IFCVF_BLK)
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					i * IFCVF_LM_CFG_SIZE);
+		else
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					(i / 2) * IFCVF_LM_CFG_SIZE +
+					(i % 2) * 4);
+
+		if (hw->device_type == IFCVF_BLK)
+			hw->vring[i].last_avail_idx =
+				(u16)(ring_state & IFCVF_16_BIT_MASK);
+		else
+			hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
 		hw->vring[i].last_used_idx = (u16)(ring_state >> 16);
 	}
 }
--

1.8.3.1

author
#200 lore raw

v9 cover add virtio_blk device support to vdpa/ifc

In reply to Andy Pei

This patch set add virtio_blk device support to vdpa/ifc driver. With a lot of similarities, I re-use part of vdpa/ifc driver. Distinguish the virtio net and blk device by device id, and implement specific features and ops.
Add example to vdpa to support virtio_blk device.
To support blk device live migration, some modification to vhost lib. Perform dev_conf op only under VHOST_USER_SET_VRING_CALL msg.

v9:
fix some naming issue.
add more NULL pointer check.
close open file when errors occur.

v8:
delete some redundant code.
fix some commit log.

v7:
Check on expected fd num in new vhost msg handler. Sanity check on vhost msg size.
Fix typo.
Add commit log to help understand code. Remove duplicated code.
Add new API to get vDPA device type.

v6:
fix some commit log.
add vhost socket in log output to make it more user-friendly. when driver ops fail, just output some log, do not break message handler. check vhost msg size in msg handler.
v5:
fix some coding style issues.
v4:
add args "isblk" to vdpa example to specify a block device, fix some issue in example.
Make sure code specify for block device does not affect net device.
v3:
Fix some compile issues.
v2:
Fix some coding style issues.

Andy Pei (13):
vdpa/ifc: add support for virtio blk device
vhost: add vDPA ops for blk device
vhost: add vhost msg support for get/set config vdpa/ifc: add blk ops for ifc device vdpa/ifc: add vDPA interrupt relay for blk device vdpa/ifc: add block device SW live-migration
vhost: add API to get vDPA device type vdpa/ifc: add get device type ops to ifc driver examples/vdpa: add virtio blk support
usertools: add support for virtio blk device vdpa/ifc: add log for config space of virtio blk vdpa/ifc: add interrupt handling for config space vdpa/ifc/base: access correct register for blk device

 doc/guides/prog_guide/vhost_lib.rst    |   5 +
 doc/guides/rel_notes/release_22_07.rst |   4 +
 drivers/vdpa/ifc/base/ifcvf.c          |  34 ++-
 drivers/vdpa/ifc/base/ifcvf.h          |  27 ++-
 drivers/vdpa/ifc/ifcvf_vdpa.c          | 415 +++++++++++++++++++++++++++++++--
 examples/vdpa/main.c                   |  56 +++++
 examples/vdpa/vdpa_blk_compact.h       |  50 ++++
 lib/vhost/rte_vhost.h                  |  17 ++
 lib/vhost/socket.c                     |  44 ++++
 lib/vhost/vdpa_driver.h                |  11 +-
 lib/vhost/version.map                  |   1 +
 lib/vhost/vhost_user.c                 |  88 +++++++
 lib/vhost/vhost_user.h                 |  13 ++
 usertools/dpdk-devbind.py              |   5 +-
 14 files changed, 739 insertions(+), 31 deletions(-)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
Signature · 1 line
1.8.3.1
author
#201 lore raw

PATCH v9 1/13 vdpa/ifc: add support for virtio blk device

Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops.

---
 drivers/vdpa/ifc/base/ifcvf.h | 22 +++++++---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 95 ++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 102 insertions(+), 15 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +17 -5
index 573a35f..a761d49 100644
@@ -5,12 +5,19 @@
 #ifndef _IFCVF_H_
 #define _IFCVF_H_
 
+#include <linux/virtio_blk.h>
 #include "ifcvf_osdep.h"
 
-#define IFCVF_VENDOR_ID		0x1AF4
-#define IFCVF_DEVICE_ID		0x1041
-#define IFCVF_SUBSYS_VENDOR_ID	0x8086
-#define IFCVF_SUBSYS_DEVICE_ID	0x001A
+#define IFCVF_NET	0
+#define IFCVF_BLK	1
+
+#define IFCVF_VENDOR_ID                     0x1AF4
+#define IFCVF_NET_DEVICE_ID                 0x1041
+#define IFCVF_BLK_MODERN_DEVICE_ID          0x1042
+#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID    0x1001
+#define IFCVF_SUBSYS_VENDOR_ID              0x8086
+#define IFCVF_SUBSYS_DEVICE_ID              0x001A
+#define IFCVF_BLK_DEVICE_ID                 0x0002
 
 #define IFCVF_MAX_QUEUES		1
 
@@ -126,13 +133,18 @@ struct ifcvf_hw {
 	u8     notify_region;
 	u32    notify_off_multiplier;
 	struct ifcvf_pci_common_cfg *common_cfg;
-	struct ifcvf_net_config *dev_cfg;
+	union {
+		struct ifcvf_net_config *net_cfg;
+		struct virtio_blk_config *blk_cfg;
+		void *dev_cfg;
+	};
 	u8     *isr;
 	u16    *notify_base;
 	u16    *notify_addr[IFCVF_MAX_QUEUES * 2];
 	u8     *lm_cfg;
 	struct vring_info vring[IFCVF_MAX_QUEUES * 2];
 	u8 nr_vring;
+	int device_type;
 	struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE];
 };
 
drivers/vdpa/ifc/ifcvf_vdpa.c +85 -10

141 lines not on this page · raw message

Signature · 1 line
1.8.3.1
Reviewed-by on v9 1/13
#202 lore raw

RE: [PATCH v9 01/13] vdpa/ifc: add support for virtio blk device

Quoted 12 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 01/13] vdpa/ifc: add support for virtio blk device

Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops.

Quoted 6 lines
---
 drivers/vdpa/ifc/base/ifcvf.h | 22 +++++++---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 95
++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 102 insertions(+), 15 deletions(-)
author
#203 lore raw

PATCH v9 2/13 vhost: add vDPA ops for blk device

In reply to Andy Pei

Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops.

---
 lib/vhost/vdpa_driver.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
lib/vhost/vdpa_driver.h +6 -2
index 88138be..c4233a6 100644
@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {
 	/** Reset statistics of the queue */
 	int (*reset_stats)(struct rte_vdpa_device *dev, int qid);
 
-	/** Reserved for future extension */
-	void *reserved[2];
+	/** Get the device configuration space */
+	int (*get_config)(int vid, uint8_t *config, uint32_t size);
+
+	/** Set the device configuration space */
+	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
+		      uint32_t size, uint32_t flags);
 };
 
 /**
Signature · 1 line
1.8.3.1
Reviewed-by on v9 2/13
#204 lore raw

RE: [PATCH v9 02/13] vhost: add vDPA ops for blk device

Quoted 38 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 02/13] vhost: add vDPA ops for blk device

Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops.

---
 lib/vhost/vdpa_driver.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
lib/vhost/vdpa_driver.h +6 -3
index 88138be..c4233a6 100644
@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {
 	/** Reset statistics of the queue */
 	int (*reset_stats)(struct rte_vdpa_device *dev, int qid);
 
-	/** Reserved for future extension */
-	void *reserved[2];
+	/** Get the device configuration space */
+	int (*get_config)(int vid, uint8_t *config, uint32_t size);
+
+	/** Set the device configuration space */
+	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
+		      uint32_t size, uint32_t flags);
 };
 
 /**
--
Quoted 1 line

1.8.3.1

author
#205 lore raw

PATCH v9 3/13 vhost: add vhost msg support for get/set config

In reply to Andy Pei

Add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device.

---
 lib/vhost/vhost_user.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/vhost/vhost_user.h | 13 ++++++++
 2 files changed, 101 insertions(+)
lib/vhost/vhost_user.c +88 -0
index 850848c..b37d814 100644
@@ -2468,6 +2468,92 @@ static int is_vring_iotlb(struct virtio_net *dev,
 }
 
 static int
+vhost_user_get_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (!vdpa_dev) {
+		VHOST_LOG_CONFIG(ERR,
+				 "(%s) is not vDPA device!\n",
+				 dev->ifname);
+		goto out;
+	}
+
+	if (vdpa_dev->ops->get_config) {
+		ret = vdpa_dev->ops->get_config(dev->vid,
+					   ctx->msg.payload.cfg.region,
+					   ctx->msg.payload.cfg.size);
+		if (ret != 0) {
+			ctx->msg.size = 0;
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) get_config() return error!\n",
+					 dev->ifname);
+		}
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) get_config() not supported!\n",
+				 dev->ifname);
+	}
+
+out:
+	return RTE_VHOST_MSG_RESULT_REPLY;
+}
+
+static int
+vhost_user_set_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) {
+		VHOST_LOG_CONFIG(ERR,
+			"(%s) vhost_user_config size: %"PRIu32", should not be larger than %d\n",
+			dev->ifname, ctx->msg.payload.cfg.size,
+			VHOST_USER_MAX_CONFIG_SIZE);
+		goto out;
+	}
+
+	if (!vdpa_dev) {
+		VHOST_LOG_CONFIG(ERR,
+				 "(%s) is not vDPA device!\n",
+				 dev->ifname);
+		goto out;
+	}
+
+	if (vdpa_dev->ops->set_config) {
+		ret = vdpa_dev->ops->set_config(dev->vid,
+			ctx->msg.payload.cfg.region,
+			ctx->msg.payload.cfg.offset,
+			ctx->msg.payload.cfg.size,
+			ctx->msg.payload.cfg.flags);
+		if (ret)
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) set_config() return error!\n",
+					 dev->ifname);
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) set_config() not supported!\n",
+				 dev->ifname);
+	}
+
+	return RTE_VHOST_MSG_RESULT_OK;
+
+out:
+	return RTE_VHOST_MSG_RESULT_ERR;
+}
+
+static int
 vhost_user_iotlb_msg(struct virtio_net **pdev,
 			struct vhu_msg_context *ctx,
 			int main_fd __rte_unused)
@@ -2686,6 +2772,8 @@ static int is_vring_iotlb(struct virtio_net *dev,
 VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU, vhost_user_net_set_mtu, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD, vhost_user_set_req_fd, true) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG, vhost_user_get_config, false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG, vhost_user_set_config, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE, vhost_user_set_postcopy_advise, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END, vhost_user_postcopy_end, false) \
lib/vhost/vhost_user.h +13 -0
index ba1c5c7..c4d091e 100644
@@ -50,6 +50,8 @@
 	VHOST_USER_NET_SET_MTU = 20,
 	VHOST_USER_SET_SLAVE_REQ_FD = 21,
 	VHOST_USER_IOTLB_MSG = 22,
+	VHOST_USER_GET_CONFIG = 24,
+	VHOST_USER_SET_CONFIG = 25,
 	VHOST_USER_CRYPTO_CREATE_SESS = 26,
 	VHOST_USER_CRYPTO_CLOSE_SESS = 27,
 	VHOST_USER_POSTCOPY_ADVISE = 28,
@@ -123,6 +125,16 @@
 	uint16_t queue_size;
 } VhostUserInflight;
 
+#define VHOST_USER_MAX_CONFIG_SIZE		256
+
+/** Get/set config msg payload */
+struct vhost_user_config {
+	uint32_t offset;
+	uint32_t size;
+	uint32_t flags;
+	uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+};
+
 typedef struct VhostUserMsg {
 	union {
 		uint32_t master; /* a VhostUserRequest value */
@@ -146,6 +158,7 @@
 		VhostUserCryptoSessionParam crypto_session;
 		VhostUserVringArea area;
 		VhostUserInflight inflight;
+		struct vhost_user_config cfg;
 	} payload;
 	/* Nothing should be added after the payload */
 } __rte_packed VhostUserMsg;
Signature · 1 line
1.8.3.1
#206 lore raw

RE: [PATCH v9 03/13] vhost: add vhost msg support for get/set config

Quoted 35 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 03/13] vhost: add vhost msg support for get/set config

Add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device.

---
 lib/vhost/vhost_user.c | 88
++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/vhost/vhost_user.h | 13 ++++++++
 2 files changed, 101 insertions(+)
lib/vhost/vhost_user.c +7 -0
index 850848c..b37d814 100644
@@ -2468,6 +2468,92 @@ static int is_vring_iotlb(struct virtio_net *dev,
 }
 
 static int
+vhost_user_get_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev, · line 2487

Quoted 10 lines
lib/vhost/vhost_user.c +10 -0
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (!vdpa_dev) {
+		VHOST_LOG_CONFIG(ERR,
+				 "(%s) is not vDPA device!\n",
+				 dev->ifname);
+		goto out;
+	}

In my understanding, this message is only targeted for vDPA device. Traditional SW back-end will not use it. So if vdpa_dev is not attached, you can just return RTE_VHOST_MSG_RESULT_ERR here. Another way is setting msg size to 0 and reply, but I think in this case. We should just break the msg handling.

Besides, two lines are enough for the log above:

+ VHOST_LOG_CONFIG(ERR, "(%s) is not vDPA device!\n",
+ dev->ifname);

Thanks,
Chenbo

Quoted 130 lines
lib/vhost/vhost_user.c +69 -0
+
+	if (vdpa_dev->ops->get_config) {
+		ret = vdpa_dev->ops->get_config(dev->vid,
+					   ctx->msg.payload.cfg.region,
+					   ctx->msg.payload.cfg.size);
+		if (ret != 0) {
+			ctx->msg.size = 0;
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) get_config() return error!\n",
+					 dev->ifname);
+		}
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) get_config() not supported!\n",
+				 dev->ifname);
+	}
+
+out:
+	return RTE_VHOST_MSG_RESULT_REPLY;
+}
+
+static int
+vhost_user_set_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) {
+		VHOST_LOG_CONFIG(ERR,
+			"(%s) vhost_user_config size: %"PRIu32", should not be
 larger than %d\n",
+			dev->ifname, ctx->msg.payload.cfg.size,
+			VHOST_USER_MAX_CONFIG_SIZE);
+		goto out;
+	}
+
+	if (!vdpa_dev) {
+		VHOST_LOG_CONFIG(ERR,
+				 "(%s) is not vDPA device!\n",
+				 dev->ifname);
+		goto out;
+	}
+
+	if (vdpa_dev->ops->set_config) {
+		ret = vdpa_dev->ops->set_config(dev->vid,
+			ctx->msg.payload.cfg.region,
+			ctx->msg.payload.cfg.offset,
+			ctx->msg.payload.cfg.size,
+			ctx->msg.payload.cfg.flags);
+		if (ret)
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) set_config() return error!\n",
+					 dev->ifname);
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) set_config() not supported!\n",
+				 dev->ifname);
+	}
+
+	return RTE_VHOST_MSG_RESULT_OK;
+
+out:
+	return RTE_VHOST_MSG_RESULT_ERR;
+}
+
+static int
 vhost_user_iotlb_msg(struct virtio_net **pdev,
 			struct vhu_msg_context *ctx,
lib/vhost/vhost_user.c +1 -0
 			int main_fd __rte_unused)
@@ -2686,6 +2772,8 @@ static int is_vring_iotlb(struct virtio_net *dev,
 VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU, vhost_user_net_set_mtu,
 false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD, vhost_user_set_req_fd,
 true) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false)
 \
+VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG, vhost_user_get_config, false)

\
+VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG, vhost_user_set_config, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE,
vhost_user_set_postcopy_advise, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN,
vhost_user_set_postcopy_listen, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END, vhost_user_postcopy_end,
false) \

lib/vhost/vhost_user.h +13 -1
index ba1c5c7..c4d091e 100644
@@ -50,6 +50,8 @@
 	VHOST_USER_NET_SET_MTU = 20,
 	VHOST_USER_SET_SLAVE_REQ_FD = 21,
 	VHOST_USER_IOTLB_MSG = 22,
+	VHOST_USER_GET_CONFIG = 24,
+	VHOST_USER_SET_CONFIG = 25,
 	VHOST_USER_CRYPTO_CREATE_SESS = 26,
 	VHOST_USER_CRYPTO_CLOSE_SESS = 27,
 	VHOST_USER_POSTCOPY_ADVISE = 28,
@@ -123,6 +125,16 @@
 	uint16_t queue_size;
 } VhostUserInflight;
 
+#define VHOST_USER_MAX_CONFIG_SIZE		256
+
+/** Get/set config msg payload */
+struct vhost_user_config {
+	uint32_t offset;
+	uint32_t size;
+	uint32_t flags;
+	uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+};
+
 typedef struct VhostUserMsg {
 	union {
 		uint32_t master; /* a VhostUserRequest value */
@@ -146,6 +158,7 @@
 		VhostUserCryptoSessionParam crypto_session;
 		VhostUserVringArea area;
 		VhostUserInflight inflight;
+		struct vhost_user_config cfg;
 	} payload;
 	/* Nothing should be added after the payload */
 } __rte_packed VhostUserMsg;
--

1.8.3.1

author
#207 lore raw

RE: [PATCH v9 03/13] vhost: add vhost msg support for get/set config

Quoted 58 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, May 24, 2022 10:52 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v9 03/13] vhost: add vhost msg support for get/set config

Quoted 11 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 03/13] vhost: add vhost msg support for get/set config

Quoted 1 line

Add support for VHOST_USER_GET_CONFIG and

VHOST_USER_SET_CONFIG.

Quoted 33 lines

VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device.

---
 lib/vhost/vhost_user.c | 88
++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/vhost/vhost_user.h | 13 ++++++++
 2 files changed, 101 insertions(+)
lib/vhost/vhost_user.c index renamed from lib/vhost/vhost_user.c +0 -0

850848c..b37d814 100644

lib/vhost/vhost_user.c +15 -0
@@ -2468,6 +2468,92 @@ static int is_vring_iotlb(struct virtio_net
 *dev,  }
 
 static int
+vhost_user_get_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (!vdpa_dev) {
+		VHOST_LOG_CONFIG(ERR,
+				 "(%s) is not vDPA device!\n",
+				 dev->ifname);
Quoted 9 lines

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net · line 2487

Quoted 2 lines
lib/vhost/vhost_user.c +2 -0
+		goto out;
+	}

In my understanding, this message is only targeted for vDPA device. Traditional SW back-end will not use it. So if vdpa_dev is not attached, you can just return RTE_VHOST_MSG_RESULT_ERR here. Another way is setting msg size to 0 and reply, but I think in this case. We should just break the msg handling.

OK. I will just return RTE_VHOST_MSG_RESULT_ERR here.

Review

Quoted 5 lines

Besides, two lines are enough for the log above:

(quoted diff) +2 -0
+		VHOST_LOG_CONFIG(ERR, "(%s) is not vDPA device!\n",
+				 dev->ifname);

Sure.

Quoted 144 lines

Thanks,
Chenbo

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev,

Quoted 13 lines
lib/vhost/vhost_user.c +13 -0
+
+	if (vdpa_dev->ops->get_config) {
+		ret = vdpa_dev->ops->get_config(dev->vid,
+					   ctx->msg.payload.cfg.region,
+					   ctx->msg.payload.cfg.size);
+		if (ret != 0) {
+			ctx->msg.size = 0;
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) get_config() return error!\n",
+					 dev->ifname);
+		}
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) get_config() not

supported!\n",

Quoted 12 lines
lib/vhost/vhost_user.c +12 -0
+				 dev->ifname);
+	}
+
+out:
+	return RTE_VHOST_MSG_RESULT_REPLY;
+}
+
+static int
+vhost_user_set_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev,

Quoted 10 lines
lib/vhost/vhost_user.c +10 -0
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) {
+		VHOST_LOG_CONFIG(ERR,
+			"(%s) vhost_user_config size: %"PRIu32", should not

be

Quoted 15 lines
lib/vhost/vhost_user.c +14 -0
 larger than %d\n",
+			dev->ifname, ctx->msg.payload.cfg.size,
+			VHOST_USER_MAX_CONFIG_SIZE);
+		goto out;
+	}
+
+	if (!vdpa_dev) {
+		VHOST_LOG_CONFIG(ERR,
+				 "(%s) is not vDPA device!\n",
+				 dev->ifname);
+		goto out;
+	}
+
+	if (vdpa_dev->ops->set_config) {
+		ret = vdpa_dev->ops->set_config(dev->vid,

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev,

Quoted 10 lines
lib/vhost/vhost_user.c +10 -0
+			ctx->msg.payload.cfg.region,
+			ctx->msg.payload.cfg.offset,
+			ctx->msg.payload.cfg.size,
+			ctx->msg.payload.cfg.flags);
+		if (ret)
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) set_config() return error!\n",
+					 dev->ifname);
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) set_config() not

supported!\n",

Quoted 14 lines
lib/vhost/vhost_user.c +10 -0
+				 dev->ifname);
+	}
+
+	return RTE_VHOST_MSG_RESULT_OK;
+
+out:
+	return RTE_VHOST_MSG_RESULT_ERR;
+}
+
+static int
 vhost_user_iotlb_msg(struct virtio_net **pdev,
 			struct vhu_msg_context *ctx,
lib/vhost/vhost_user.c +0 -0
 			int main_fd __rte_unused)

lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net

Quoted 7 lines
lib/vhost/vhost_user.c +0 -0
@@ -2686,6 +2772,8 @@ static int is_vring_iotlb(struct virtio_net
 *dev,  VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU,
 vhost_user_net_set_mtu,
 false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD,
 vhost_user_set_req_fd,
 true) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG,

vhost_user_iotlb_msg,

Quoted 2 lines

false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG,

vhost_user_get_config,

Quoted 3 lines

+false)
\
+VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG,

vhost_user_set_config,

Quoted 29 lines

+false)
\
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE,
vhost_user_set_postcopy_advise, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END, vhost_user_postcopy_end,
false) \

lib/vhost/vhost_user.h index renamed from lib/vhost/vhost_user.h +0 -0

ba1c5c7..c4d091e 100644

lib/vhost/vhost_user.h +5 -0
@@ -50,6 +50,8 @@
 	VHOST_USER_NET_SET_MTU = 20,
 	VHOST_USER_SET_SLAVE_REQ_FD = 21,
 	VHOST_USER_IOTLB_MSG = 22,
+	VHOST_USER_GET_CONFIG = 24,
+	VHOST_USER_SET_CONFIG = 25,
 	VHOST_USER_CRYPTO_CREATE_SESS = 26,
 	VHOST_USER_CRYPTO_CLOSE_SESS = 27,
 	VHOST_USER_POSTCOPY_ADVISE = 28,
@@ -123,6 +125,16 @@
 	uint16_t queue_size;
 } VhostUserInflight;
 
+#define VHOST_USER_MAX_CONFIG_SIZE		256
+
+/** Get/set config msg payload */

lib/vhost/vhost_user.h line 140

Quoted 10 lines
lib/vhost/vhost_user.h +7 -0
+struct vhost_user_config {
+	uint32_t offset;
+	uint32_t size;
+	uint32_t flags;
+	uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+};
+
 typedef struct VhostUserMsg {
 	union {
 		uint32_t master; /* a VhostUserRequest value */ @@ -146,6

+158,7 @@

Quoted 9 lines
lib/vhost/vhost_user.h +1 -0
 		VhostUserCryptoSessionParam crypto_session;
 		VhostUserVringArea area;
 		VhostUserInflight inflight;
+		struct vhost_user_config cfg;
 	} payload;
 	/* Nothing should be added after the payload */  } __rte_packed

VhostUserMsg;
--
1.8.3.1

author
#208 lore raw

PATCH v9 4/13 vdpa/ifc: add blk ops for ifc device

In reply to Andy Pei

For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.

---
 drivers/vdpa/ifc/base/ifcvf.h |  4 ++
 drivers/vdpa/ifc/ifcvf_vdpa.c | 91 ++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 94 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/base/ifcvf.h +4 -0
index a761d49..b6fdfdb 100644
@@ -65,6 +65,10 @@
 #define IFCVF_32_BIT_MASK		0xffffffff
 
 
+#ifndef VHOST_USER_PROTOCOL_F_CONFIG
+#define VHOST_USER_PROTOCOL_F_CONFIG	9
+#endif
+
 struct ifcvf_pci_cap {
 	u8 cap_vndr;            /* Generic PCI field: PCI_CAP_ID_VNDR */
 	u8 cap_next;            /* Generic PCI field: next ptr. */
drivers/vdpa/ifc/ifcvf_vdpa.c +90 -1

109 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#209 lore raw

RE: [PATCH v9 04/13] vdpa/ifc: add blk ops for ifc device

Quoted 64 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 04/13] vdpa/ifc: add blk ops for ifc device

For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.

---
 drivers/vdpa/ifc/base/ifcvf.h |  4 ++
 drivers/vdpa/ifc/ifcvf_vdpa.c | 91
++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 94 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/base/ifcvf.h +4 -0
index a761d49..b6fdfdb 100644
@@ -65,6 +65,10 @@
 #define IFCVF_32_BIT_MASK		0xffffffff
 
 
+#ifndef VHOST_USER_PROTOCOL_F_CONFIG
+#define VHOST_USER_PROTOCOL_F_CONFIG	9
+#endif
+
 struct ifcvf_pci_cap {
 	u8 cap_vndr;            /* Generic PCI field: PCI_CAP_ID_VNDR */
 	u8 cap_next;            /* Generic PCI field: next ptr. */
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
index 1eed90b..c1fc1d7 100644
@@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info {
 		 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \
 		 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \
 		 1ULL << VHOST_USER_PROTOCOL_F_STATUS)
+
+#define VDPA_BLK_PROTOCOL_FEATURES \
+		(1ULL << VHOST_USER_PROTOCOL_F_CONFIG)
+
 static int
 ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t
 *features)

{

drivers/vdpa/ifc/ifcvf_vdpa.c +8 -0
@@ -1199,6 +1203,91 @@ struct rte_vdpa_dev_info {
 	return device_id;
 }
 
+static int
+ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t size)
+{
+	struct virtio_blk_config *dev_cfg;
+	struct ifcvf_internal *internal;
+	struct rte_vdpa_device *vdev;
+	struct internal_list *list;
+	uint32_t i;

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1223

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10 -0
+	uint64_t capacity = 0;
+	uint8_t *byte;
+
+	if (size != sizeof(struct virtio_blk_config)) {
+		DRV_LOG(ERR, "Invalid len: %u, required: %u",
+			size, (uint32_t)sizeof(struct virtio_blk_config));
+		return -1;
+	}
+
+	vdev = rte_vhost_get_vdpa_device(vid);

Check vdev is not NULL here.

Thanks
Chenbo

Quoted 82 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +67 -0
+	list = find_internal_resource_by_vdev(vdev);
+	if (list == NULL) {
+		DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -1;
+	}
+
+	internal = list->internal;
+
+	for (i = 0; i < sizeof(struct virtio_blk_config); i++)
+		config[i] = *((u8 *)internal->hw.blk_cfg + i);
+
+	dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg;
+
+	/* cannot read 64-bit register in one attempt, so read byte by byte.
 */
+	for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+		byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+		capacity |= (uint64_t)*byte << (i * 8);
+	}
+	/* The capacity is number of sectors in 512-byte.
+	 * So right shift 1 bit  we get in K,
+	 * another right shift 10 bits we get in M,
+	 * right shift 10 more bits, we get in G.
+	 * To show capacity in G, we right shift 21 bits in total.
+	 */
+	DRV_LOG(DEBUG, "capacity  : %"PRIu64"G", capacity >> 21);
+
+	DRV_LOG(DEBUG, "size_max  : 0x%08x", dev_cfg->size_max);
+	DRV_LOG(DEBUG, "seg_max   : 0x%08x", dev_cfg->seg_max);
+	DRV_LOG(DEBUG, "blk_size  : 0x%08x", dev_cfg->blk_size);
+	DRV_LOG(DEBUG, "geometry");
+	DRV_LOG(DEBUG, "      cylinders: %u", dev_cfg->geometry.cylinders);
+	DRV_LOG(DEBUG, "      heads    : %u", dev_cfg->geometry.heads);
+	DRV_LOG(DEBUG, "      sectors  : %u", dev_cfg->geometry.sectors);
+	DRV_LOG(DEBUG, "num_queues: 0x%08x", dev_cfg->num_queues);
+
+	DRV_LOG(DEBUG, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n",
+		config[0], config[1], config[2], config[3], config[4],
+		config[5], config[6], config[7]);
+	return 0;
+}
+
+static int
+ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev,
+	uint64_t *features)
+{
+	RTE_SET_USED(vdev);
+
+	*features = VDPA_SUPPORTED_PROTOCOL_FEATURES;
+	*features |= VDPA_BLK_PROTOCOL_FEATURES;
+	return 0;
+}
+
+static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
+	.get_queue_num = ifcvf_get_queue_num,
+	.get_features = ifcvf_get_vdpa_features,
+	.set_features = ifcvf_set_features,
+	.get_protocol_features = ifcvf_blk_get_protocol_features,
+	.dev_conf = ifcvf_dev_config,
+	.dev_close = ifcvf_dev_close,
+	.set_vring_state = ifcvf_set_vring_state,
+	.migration_done = NULL,
+	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
+	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
+	.get_notify_area = ifcvf_get_notify_area,
+	.get_config = ifcvf_blk_get_config,
+};
+
 struct rte_vdpa_dev_info dev_info[] = {
 	{
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -2
 		.features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |
@@ -1211,7 +1300,7 @@ struct rte_vdpa_dev_info dev_info[] = {
 	{
 		.features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) |
 			    (1ULL << VHOST_F_LOG_ALL),
-		.ops = NULL,
+		.ops = &ifcvf_blk_ops,
 	},
 };
 
--

1.8.3.1

author
#210 lore raw

RE: [PATCH v9 04/13] vdpa/ifc: add blk ops for ifc device

Quoted 78 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, May 24, 2022 10:55 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v9 04/13] vdpa/ifc: add blk ops for ifc device

Quoted 67 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 04/13] vdpa/ifc: add blk ops for ifc device

For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.

---
drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c
| 91
++++++++++++++++++++++++++++++++++++++++++-

 2 files changed, 94 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/vdpa/ifc/base/ifcvf.h index a761d49..b6fdfdb 100644

drivers/vdpa/ifc/base/ifcvf.h +4 -0
@@ -65,6 +65,10 @@
 #define IFCVF_32_BIT_MASK		0xffffffff
 
 
+#ifndef VHOST_USER_PROTOCOL_F_CONFIG
+#define VHOST_USER_PROTOCOL_F_CONFIG	9
+#endif
+
 struct ifcvf_pci_cap {
 	u8 cap_vndr;            /* Generic PCI field: PCI_CAP_ID_VNDR */
 	u8 cap_next;            /* Generic PCI field: next ptr. */
(quoted diff) +0 -0

b/drivers/vdpa/ifc/ifcvf_vdpa.c index 1eed90b..c1fc1d7 100644

drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
@@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info {
 		 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \
 		 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \
 		 1ULL << VHOST_USER_PROTOCOL_F_STATUS)
+
+#define VDPA_BLK_PROTOCOL_FEATURES \
+		(1ULL << VHOST_USER_PROTOCOL_F_CONFIG)
+
 static int
 ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t
 *features)

{

drivers/vdpa/ifc/ifcvf_vdpa.c +11 -0
@@ -1199,6 +1203,91 @@ struct rte_vdpa_dev_info {
 	return device_id;
 }
 
+static int
+ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t size) {
+	struct virtio_blk_config *dev_cfg;
+	struct ifcvf_internal *internal;
+	struct rte_vdpa_device *vdev;
+	struct internal_list *list;
+	uint32_t i;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+
+	if (size != sizeof(struct virtio_blk_config)) {
Quoted 9 lines

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1223

Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +6 -0
+		DRV_LOG(ERR, "Invalid len: %u, required: %u",
+			size, (uint32_t)sizeof(struct virtio_blk_config));
+		return -1;
+	}
+
+	vdev = rte_vhost_get_vdpa_device(vid);

Check vdev is not NULL here.

Sure, I will fix it in next version.

Quoted 87 lines

Thanks
Chenbo

Quoted 61 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +60 -0
+	list = find_internal_resource_by_vdev(vdev);
+	if (list == NULL) {
+		DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -1;
+	}
+
+	internal = list->internal;
+
+	for (i = 0; i < sizeof(struct virtio_blk_config); i++)
+		config[i] = *((u8 *)internal->hw.blk_cfg + i);
+
+	dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg;
+
+	/* cannot read 64-bit register in one attempt, so read byte by byte.
 */
+	for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+		byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+		capacity |= (uint64_t)*byte << (i * 8);
+	}
+	/* The capacity is number of sectors in 512-byte.
+	 * So right shift 1 bit  we get in K,
+	 * another right shift 10 bits we get in M,
+	 * right shift 10 more bits, we get in G.
+	 * To show capacity in G, we right shift 21 bits in total.
+	 */
+	DRV_LOG(DEBUG, "capacity  : %"PRIu64"G", capacity >> 21);
+
+	DRV_LOG(DEBUG, "size_max  : 0x%08x", dev_cfg->size_max);
+	DRV_LOG(DEBUG, "seg_max   : 0x%08x", dev_cfg->seg_max);
+	DRV_LOG(DEBUG, "blk_size  : 0x%08x", dev_cfg->blk_size);
+	DRV_LOG(DEBUG, "geometry");
+	DRV_LOG(DEBUG, "      cylinders: %u", dev_cfg->geometry.cylinders);
+	DRV_LOG(DEBUG, "      heads    : %u", dev_cfg->geometry.heads);
+	DRV_LOG(DEBUG, "      sectors  : %u", dev_cfg->geometry.sectors);
+	DRV_LOG(DEBUG, "num_queues: 0x%08x", dev_cfg->num_queues);
+
+	DRV_LOG(DEBUG, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n",
+		config[0], config[1], config[2], config[3], config[4],
+		config[5], config[6], config[7]);
+	return 0;
+}
+
+static int
+ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev,
+	uint64_t *features)
+{
+	RTE_SET_USED(vdev);
+
+	*features = VDPA_SUPPORTED_PROTOCOL_FEATURES;
+	*features |= VDPA_BLK_PROTOCOL_FEATURES;
+	return 0;
+}
+
+static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
+	.get_queue_num = ifcvf_get_queue_num,
+	.get_features = ifcvf_get_vdpa_features,
+	.set_features = ifcvf_set_features,
+	.get_protocol_features = ifcvf_blk_get_protocol_features,
+	.dev_conf = ifcvf_dev_config,
+	.dev_close = ifcvf_dev_close,
+	.set_vring_state = ifcvf_set_vring_state,

drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1293

Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7 -0
+	.migration_done = NULL,
+	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
+	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
+	.get_notify_area = ifcvf_get_notify_area,
+	.get_config = ifcvf_blk_get_config,
+};
+
 struct rte_vdpa_dev_info dev_info[] = {
 	{
 		.features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |

@@ -1211,7

drivers/vdpa/ifc/ifcvf_vdpa.c

Quoted 3 lines

+1300,7 @@ struct rte_vdpa_dev_info dev_info[] = {

drivers/vdpa/ifc/ifcvf_vdpa.c +0 -0
 	{
 		.features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES)

|

Quoted 8 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -2
 			    (1ULL << VHOST_F_LOG_ALL),
-		.ops = NULL,
+		.ops = &ifcvf_blk_ops,
 	},
 };
 
--

1.8.3.1

author
#211 lore raw

PATCH v9 5/13 vdpa/ifc: add vDPA interrupt relay for blk device

In reply to Andy Pei

For the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +8 -1
index c1fc1d7..1d05529 100644
@@ -369,6 +369,7 @@ struct rte_vdpa_dev_info {
 	irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;
 	irq_set->start = 0;
 	fd_ptr = (int *)&irq_set->data;
+	/* The first interrupt is for the configure space change notification */
 	fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] =
 		rte_intr_fd_get(internal->pdev->intr_handle);
 
@@ -378,7 +379,13 @@ struct rte_vdpa_dev_info {
 	for (i = 0; i < nr_vring; i++) {
 		rte_vhost_get_vhost_vring(internal->vid, i, &vring);
 		fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
-		if ((i & 1) == 0 && m_rx == true) {
+		if (m_rx == true &&
+			((i & 1) == 0 || internal->hw.device_type == IFCVF_BLK)) {
+			/* For the net we only need to relay rx queue,
+			 * which will change the mem of VM.
+			 * For the blk we need to relay all the read cmd
+			 * of each queue
+			 */
 			fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
 			if (fd < 0) {
 				DRV_LOG(ERR, "can't setup eventfd: %s",
Signature · 1 line
1.8.3.1
Reviewed-by on v9 5/13
#212 lore raw

RE: [PATCH v9 05/13] vdpa/ifc: add vDPA interrupt relay for blk device

Quoted 47 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 05/13] vdpa/ifc: add vDPA interrupt relay for blk device

For the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +1 -0
index c1fc1d7..1d05529 100644
@@ -369,6 +369,7 @@ struct rte_vdpa_dev_info {
 	irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;
 	irq_set->start = 0;
 	fd_ptr = (int *)&irq_set->data;
+	/* The first interrupt is for the configure space change

notification */

drivers/vdpa/ifc/ifcvf_vdpa.c +7 -1
 	fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] =
 		rte_intr_fd_get(internal->pdev->intr_handle);
@@ -378,7 +379,13 @@ struct rte_vdpa_dev_info {
 	for (i = 0; i < nr_vring; i++) {
 		rte_vhost_get_vhost_vring(internal->vid, i, &vring);
 		fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
-		if ((i & 1) == 0 && m_rx == true) {
+		if (m_rx == true &&
+			((i & 1) == 0 || internal->hw.device_type == IFCVF_BLK))
 {
+			/* For the net we only need to relay rx queue,
+			 * which will change the mem of VM.
+			 * For the blk we need to relay all the read cmd
+			 * of each queue
+			 */
 			fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
 			if (fd < 0) {
Quoted 3 lines

DRV_LOG(ERR, "can't setup eventfd: %s",
--
1.8.3.1

Btw: add my R-by in next version as I see in this version, they are missed.

author
#213 lore raw

PATCH v9 6/13 vdpa/ifc: add block device SW live-migration

In reply to Andy Pei

Add SW live-migration support to block device. For block device, it is critical that no packet should be dropped. So when virtio blk device is paused, make sure hardware last_avail_idx and last_used_idx are the same. This indicates all requests have received acks, and no inflight IO.

---
 drivers/vdpa/ifc/base/ifcvf.h |  1 +
 drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 39 insertions(+), 4 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
index b6fdfdb..9d95aac 100644
@@ -63,6 +63,7 @@
 #define IFCVF_MEDIATED_VRING		0x200000000000
 
 #define IFCVF_32_BIT_MASK		0xffffffff
+#define IFCVF_16_BIT_MASK		0xffff
 
 
 #ifndef VHOST_USER_PROTOCOL_F_CONFIG
drivers/vdpa/ifc/ifcvf_vdpa.c +38 -4

69 lines not on this page · raw message

Signature · 1 line
1.8.3.1
Reviewed-by on v9 6/13
#214 lore raw

RE: [PATCH v9 06/13] vdpa/ifc: add block device SW live-migration

Quoted 16 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 06/13] vdpa/ifc: add block device SW live-migration

Add SW live-migration support to block device.
For block device, it is critical that no packet
should be dropped. So when virtio blk device is
paused, make sure hardware last_avail_idx and
last_used_idx are the same. This indicates all
requests have received acks, and no inflight IO.

Quoted 6 lines
---
 drivers/vdpa/ifc/base/ifcvf.h |  1 +
 drivers/vdpa/ifc/ifcvf_vdpa.c | 42
++++++++++++++++++++++++++++++++++++++----
 2 files changed, 39 insertions(+), 4 deletions(-)
author
#215 lore raw

PATCH v9 7/13 vhost: add API to get vDPA device type

In reply to Andy Pei

Vhost backend of different devices have different features. Add an API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++++
 doc/guides/rel_notes/release_22_07.rst |  4 ++++
 lib/vhost/rte_vhost.h                  | 17 +++++++++++++
 lib/vhost/socket.c                     | 44 ++++++++++++++++++++++++++++++++++
 lib/vhost/vdpa_driver.h                |  3 +++
 lib/vhost/version.map                  |  1 +
 6 files changed, 74 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +5 -0
index f287b76..0337b38 100644
@@ -282,6 +282,11 @@ The following is an overview of some key Vhost API functions:
   Clear inflight packets which are submitted to DMA engine in vhost async data
   path. Completed packets are returned to applications through ``pkts``.
 
+* ``rte_vhost_driver_get_vdpa_dev_type(path, type)``
+
+  Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
+  VDPA_DEVICE_TYPE_BLK.
+
 Vhost-user Implementations
 --------------------------
 
doc/guides/rel_notes/release_22_07.rst +4 -0
index e49cace..63875b7 100644
@@ -60,6 +60,10 @@ New Features
   Added an API which can get the number of in-flight packets in
   vhost async data path without using lock.
 
+* **Added vhost API to get the device type of a vDPA device.**
+
+  Added an API which can get the device type of vDPA device.
+
 * **Updated Intel iavf driver.**
 
   * Added Tx QoS queue rate limitation support.
lib/vhost/rte_vhost.h +17 -0
index c733f85..2f130ec 100644
@@ -117,6 +117,9 @@
 
 #define RTE_MAX_VHOST_DEVICE	1024
 
+#define RTE_VHOST_VDPA_DEVICE_TYPE_NET 0
+#define RTE_VHOST_VDPA_DEVICE_TYPE_BLK 1
+
 struct rte_vdpa_device;
 
 /**
@@ -486,6 +489,20 @@ struct rte_vdpa_device *
 rte_vhost_driver_get_vdpa_device(const char *path);
 
 /**
+ * Get the device type of the vdpa device.
+ *
+ * @param path
+ *  The vhost-user socket file path
+ * @param type
+ *  the device type of the vdpa device
+ * @return
+ *  0 on success, -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type);
+
+/**
  * Set the feature bits the vhost-user driver supports.
  *
  * @param path
lib/vhost/socket.c +44 -0

50 lines not on this page · raw message

lib/vhost/vdpa_driver.h +3 -0
index c4233a6..8b88a53 100644
@@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops {
 	/** Set the device configuration space */
 	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
 		      uint32_t size, uint32_t flags);
+
+	/** get device type: net device, blk device... */
+	int (*get_dev_type)(struct rte_vdpa_device *dev, uint32_t *type);
 };
 
 /**
lib/vhost/version.map +1 -0
index 5841315..583b4f3 100644
@@ -90,6 +90,7 @@ EXPERIMENTAL {
 
 	# added in 22.07
 	rte_vhost_async_get_inflight_thread_unsafe;
+	rte_vhost_driver_get_vdpa_dev_type;
 
 };
 
Signature · 1 line
1.8.3.1
Reviewed-by on v9 7/13
#216 lore raw

RE: [PATCH v9 07/13] vhost: add API to get vDPA device type

Quoted 15 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 07/13] vhost: add API to get vDPA device type

Vhost backend of different devices have different features. Add an API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices.

Quoted 9 lines
---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++++
 doc/guides/rel_notes/release_22_07.rst |  4 ++++
 lib/vhost/rte_vhost.h                  | 17 +++++++++++++
 lib/vhost/socket.c                     | 44
++++++++++++++++++++++++++++++++++
 lib/vhost/vdpa_driver.h                |  3 +++
 lib/vhost/version.map                  |  1 +
 6 files changed, 74 insertions(+)
author
#217 lore raw

PATCH v9 8/13 vdpa/ifc: add get device type ops to ifc driver

In reply to Andy Pei

Add get device type ops to ifc driver.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +25 -0
index 7a0bdb4..1c5746a 100644
@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static int
+ifcvf_get_device_type(struct rte_vdpa_device *vdev,
+	uint32_t *type)
+{
+	struct ifcvf_internal *internal;
+	struct internal_list *list;
+
+	list = find_internal_resource_by_vdev(vdev);
+	if (list == NULL) {
+		DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -1;
+	}
+
+	internal = list->internal;
+
+	if (internal->hw.device_type == IFCVF_BLK)
+		*type = RTE_VHOST_VDPA_DEVICE_TYPE_BLK;
+	else
+		*type = RTE_VHOST_VDPA_DEVICE_TYPE_NET;
+
+	return 0;
+}
+
 static struct rte_vdpa_dev_ops ifcvf_net_ops = {
 	.get_queue_num = ifcvf_get_queue_num,
 	.get_features = ifcvf_get_vdpa_features,
@@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 static inline int
@@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
 	.get_config = ifcvf_blk_get_config,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 struct rte_vdpa_dev_info dev_info[] = {
Signature · 1 line
1.8.3.1
Reviewed-by on v9 8/13
#218 lore raw

RE: [PATCH v9 08/13] vdpa/ifc: add get device type ops to ifc driver

Quoted 67 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 08/13] vdpa/ifc: add get device type ops to ifc driver

Add get device type ops to ifc driver.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +25 -1
index 7a0bdb4..1c5746a 100644
@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static int
+ifcvf_get_device_type(struct rte_vdpa_device *vdev,
+	uint32_t *type)
+{
+	struct ifcvf_internal *internal;
+	struct internal_list *list;
+
+	list = find_internal_resource_by_vdev(vdev);
+	if (list == NULL) {
+		DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -1;
+	}
+
+	internal = list->internal;
+
+	if (internal->hw.device_type == IFCVF_BLK)
+		*type = RTE_VHOST_VDPA_DEVICE_TYPE_BLK;
+	else
+		*type = RTE_VHOST_VDPA_DEVICE_TYPE_NET;
+
+	return 0;
+}
+
 static struct rte_vdpa_dev_ops ifcvf_net_ops = {
 	.get_queue_num = ifcvf_get_queue_num,
 	.get_features = ifcvf_get_vdpa_features,
@@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 static inline int
@@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
 	.get_config = ifcvf_blk_get_config,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 struct rte_vdpa_dev_info dev_info[] = {
--
Quoted 1 line

1.8.3.1

author
#219 lore raw

PATCH v9 9/13 examples/vdpa: add virtio blk support

In reply to Andy Pei

Add virtio blk device support to vDPA example.

---
 examples/vdpa/main.c             | 56 ++++++++++++++++++++++++++++++++++++++++
 examples/vdpa/vdpa_blk_compact.h | 50 +++++++++++++++++++++++++++++++++++
 2 files changed, 106 insertions(+)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c +56 -0
index 5ab0765..7e11ef4 100644
@@ -20,6 +20,7 @@
 #include <cmdline_parse_string.h>
 #include <cmdline_parse_num.h>
 #include <cmdline.h>
+#include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
 #define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,53 @@ struct vdpa_port {
 };
 
 static int
+vdpa_blk_device_set_features_and_protocol(const char *path)
+{
+	uint64_t protocol_features = 0;
+	int ret;
+
+	ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_disable_features(path,
+		VHOST_BLK_DISABLED_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_disable_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_get_protocol_features(path, &protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_get_protocol_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	protocol_features |= VHOST_BLK_PROTOCOL_FEATURES;
+
+	ret = rte_vhost_driver_set_protocol_features(path, protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_protocol_features for %s failed.\n",
+			path);
+	}
+
+out:
+	return ret;
+}
+
+static int
 start_vdpa(struct vdpa_port *vport)
 {
+	uint32_t device_type = 0;
 	int ret;
 	char *socket_path = vport->ifname;
 
@@ -192,6 +238,16 @@ struct vdpa_port {
 			"attach vdpa device failed: %s\n",
 			socket_path);
 
+	ret = rte_vhost_driver_get_vdpa_dev_type(socket_path, &device_type);
+	if (ret == 0 && device_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK) {
+		RTE_LOG(NOTICE, VDPA, "%s is a blk device\n", socket_path);
+		ret = vdpa_blk_device_set_features_and_protocol(socket_path);
+		if (ret != 0)
+			rte_exit(EXIT_FAILURE,
+				"set vhost blk driver features and protocol features failed: %s\n",
+				socket_path);
+	}
+
 	if (rte_vhost_driver_start(socket_path) < 0)
 		rte_exit(EXIT_FAILURE,
 			"start vhost driver failed: %s\n",
examples/vdpa/vdpa_blk_compact.h new file +50 -0

50 lines not on this page · raw message

Signature · 1 line
1.8.3.1
Reviewed-by on v9 9/13
#220 lore raw

RE: [PATCH v9 09/13] examples/vdpa: add virtio blk support

Quoted 11 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 09/13] examples/vdpa: add virtio blk support

Add virtio blk device support to vDPA example.

Quoted 7 lines
---
 examples/vdpa/main.c             | 56
++++++++++++++++++++++++++++++++++++++++
 examples/vdpa/vdpa_blk_compact.h | 50 +++++++++++++++++++++++++++++++++++
 2 files changed, 106 insertions(+)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
author
#221 lore raw

PATCH v9 10/13 usertools: add support for virtio blk device

In reply to Andy Pei

Add virtio blk device support to devbind.

---
 usertools/dpdk-devbind.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-devbind.py +4 -1
index ace4627..18c7d67 100755
@@ -72,6 +72,9 @@
 cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4',
                  'SVendor': None, 'SDevice': None}
 
+virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001,1042',
+                    'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 baseband_devices = [acceleration_class]
 crypto_devices = [encryption_class, intel_processor_class]
@@ -82,7 +85,7 @@
 compress_devices = [cavium_zip]
 regex_devices = [cn9k_ree]
 misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev,
-                intel_ntb_skx, intel_ntb_icx]
+                intel_ntb_skx, intel_ntb_icx, virtio_blk]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
Signature · 1 line
1.8.3.1
Acked-by on v9 10/13
#222 lore raw

RE: [PATCH v9 10/13] usertools: add support for virtio blk device

Quoted 40 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 10/13] usertools: add support for virtio blk device

Add virtio blk device support to devbind.

---
 usertools/dpdk-devbind.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-devbind.py +4 -2
index ace4627..18c7d67 100755
@@ -72,6 +72,9 @@
 cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4',
                  'SVendor': None, 'SDevice': None}
 
+virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001,1042',
+                    'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 baseband_devices = [acceleration_class]
 crypto_devices = [encryption_class, intel_processor_class]
@@ -82,7 +85,7 @@
 compress_devices = [cavium_zip]
 regex_devices = [cn9k_ree]
 misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev,
-                intel_ntb_skx, intel_ntb_icx]
+                intel_ntb_skx, intel_ntb_icx, virtio_blk]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
--
Quoted 1 line

1.8.3.1

author
#223 lore raw

PATCH v9 11/13 vdpa/ifc: add log for config space of virtio blk

In reply to Andy Pei

Add some log of virtio blk device config space information at VDPA launch before qemu connects.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +34 -0
index 1c5746a..80d6261 100644
@@ -1382,6 +1382,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
 	int16_t device_id;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+	uint32_t i;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
@@ -1448,6 +1451,37 @@ struct rte_vdpa_dev_info dev_info[] = {
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_BLK].features;
+
+		/* cannot read 64-bit register in one attempt,
+		 * so read byte by byte.
+		 */
+		for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+			byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+			capacity |= (uint64_t)*byte << (i * 8);
+		}
+		/* The capacity is number of sectors in 512-byte.
+		 * So right shift 1 bit  we get in K,
+		 * another right shift 10 bits we get in M,
+		 * right shift 10 more bits, we get in G.
+		 * To show capacity in G, we right shift 21 bits in total.
+		 */
+		DRV_LOG(DEBUG, "capacity  : %"PRIu64"G", capacity >> 21);
+
+		DRV_LOG(DEBUG, "size_max  : 0x%08x",
+			internal->hw.blk_cfg->size_max);
+		DRV_LOG(DEBUG, "seg_max   : 0x%08x",
+			internal->hw.blk_cfg->seg_max);
+		DRV_LOG(DEBUG, "blk_size  : 0x%08x",
+			internal->hw.blk_cfg->blk_size);
+		DRV_LOG(DEBUG, "geometry");
+		DRV_LOG(DEBUG, "    cylinders: %u",
+			internal->hw.blk_cfg->geometry.cylinders);
+		DRV_LOG(DEBUG, "    heads    : %u",
+			internal->hw.blk_cfg->geometry.heads);
+		DRV_LOG(DEBUG, "    sectors  : %u",
+			internal->hw.blk_cfg->geometry.sectors);
+		DRV_LOG(DEBUG, "num_queues: 0x%08x",
+			internal->hw.blk_cfg->num_queues);
 	}
 
 	list->internal = internal;
Signature · 1 line
1.8.3.1
Reviewed-by on v9 11/13
#224 lore raw

RE: [PATCH v9 11/13] vdpa/ifc: add log for config space of virtio blk

Quoted 70 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 11/13] vdpa/ifc: add log for config space of virtio blk

Add some log of virtio blk device config space information at VDPA launch before qemu connects.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +34 -1
index 1c5746a..80d6261 100644
@@ -1382,6 +1382,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
 	int16_t device_id;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+	uint32_t i;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
@@ -1448,6 +1451,37 @@ struct rte_vdpa_dev_info dev_info[] = {
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_BLK].features;
+
+		/* cannot read 64-bit register in one attempt,
+		 * so read byte by byte.
+		 */
+		for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+			byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+			capacity |= (uint64_t)*byte << (i * 8);
+		}
+		/* The capacity is number of sectors in 512-byte.
+		 * So right shift 1 bit  we get in K,
+		 * another right shift 10 bits we get in M,
+		 * right shift 10 more bits, we get in G.
+		 * To show capacity in G, we right shift 21 bits in total.
+		 */
+		DRV_LOG(DEBUG, "capacity  : %"PRIu64"G", capacity >> 21);
+
+		DRV_LOG(DEBUG, "size_max  : 0x%08x",
+			internal->hw.blk_cfg->size_max);
+		DRV_LOG(DEBUG, "seg_max   : 0x%08x",
+			internal->hw.blk_cfg->seg_max);
+		DRV_LOG(DEBUG, "blk_size  : 0x%08x",
+			internal->hw.blk_cfg->blk_size);
+		DRV_LOG(DEBUG, "geometry");
+		DRV_LOG(DEBUG, "    cylinders: %u",
+			internal->hw.blk_cfg->geometry.cylinders);
+		DRV_LOG(DEBUG, "    heads    : %u",
+			internal->hw.blk_cfg->geometry.heads);
+		DRV_LOG(DEBUG, "    sectors  : %u",
+			internal->hw.blk_cfg->geometry.sectors);
+		DRV_LOG(DEBUG, "num_queues: 0x%08x",
+			internal->hw.blk_cfg->num_queues);
 	}
 
 	list->internal = internal;
--
Quoted 1 line

1.8.3.1

author
#225 lore raw

PATCH v9 12/13 vdpa/ifc: add interrupt handling for config space

In reply to Andy Pei

Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 121 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 120 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +120 -1
index 80d6261..af1f192 100644
@@ -53,7 +53,9 @@ struct ifcvf_internal {
 	int vfio_group_fd;
 	int vfio_dev_fd;
 	pthread_t tid;	/* thread for notify relay */
+	pthread_t intr_tid; /* thread for config space change interrupt relay */
 	int epfd;
+	int csc_epfd;
 	int vid;
 	struct rte_vdpa_device *vdev;
 	uint16_t max_queues;
@@ -566,6 +568,114 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static void
+virtio_interrupt_handler(struct ifcvf_internal *internal)
+{
+	int vid = internal->vid;
+	int ret;
+
+	ret = rte_vhost_slave_config_change(vid, 1);
+	if (ret)
+		DRV_LOG(ERR, "failed to notify the guest about configuration space change.");
+}
+
+static void *
+intr_relay(void *arg)
+{
+	struct ifcvf_internal *internal = (struct ifcvf_internal *)arg;
+	struct epoll_event csc_event;
+	struct epoll_event ev;
+	uint64_t buf;
+	int nbytes;
+	int csc_epfd, csc_val = 0;
+
+	csc_epfd = epoll_create(1);
+	if (csc_epfd < 0) {
+		DRV_LOG(ERR, "failed to create epoll for config space change.");
+		return NULL;
+	}
+
+	ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP;
+	ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle);
+	if (epoll_ctl(csc_epfd, EPOLL_CTL_ADD,
+		rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) {
+		DRV_LOG(ERR, "epoll add error: %s", strerror(errno));
+		goto out;
+	}
+
+	internal->csc_epfd = csc_epfd;
+
+	for (;;) {
+		csc_val = epoll_wait(csc_epfd, &csc_event, 1, -1);
+		if (csc_val < 0) {
+			if (errno == EINTR)
+				continue;
+			DRV_LOG(ERR, "epoll_wait return fail.");
+			goto out;
+		} else if (csc_val == 0) {
+			continue;
+		} else {
+			/* csc_val > 0 */
+			nbytes = read(csc_event.data.fd, &buf, 8);
+			if (nbytes < 0) {
+				if (errno == EINTR ||
+				    errno == EWOULDBLOCK ||
+				    errno == EAGAIN)
+					continue;
+				DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n",
+					csc_event.data.fd,
+					strerror(errno));
+				goto out;
+			} else if (nbytes == 0) {
+				DRV_LOG(ERR, "Read nothing from file descriptor %d\n",
+					csc_event.data.fd);
+				continue;
+			} else {
+				virtio_interrupt_handler(internal);
+			}
+		}
+	}
+
+out:
+	if (csc_epfd >= 0)
+		close(csc_epfd);
+	internal->csc_epfd = -1;
+
+	return NULL;
+}
+
+static int
+setup_intr_relay(struct ifcvf_internal *internal)
+{
+	char name[THREAD_NAME_LEN];
+	int ret;
+
+	snprintf(name, sizeof(name), "ifc-intr-%d", internal->vid);
+	ret = rte_ctrl_thread_create(&internal->intr_tid, name, NULL,
+				     intr_relay, (void *)internal);
+	if (ret) {
+		DRV_LOG(ERR, "failed to create notify relay pthread.");
+		return -1;
+	}
+	return 0;
+}
+
+static void
+unset_intr_relay(struct ifcvf_internal *internal)
+{
+	void *status;
+
+	if (internal->intr_tid) {
+		pthread_cancel(internal->intr_tid);
+		pthread_join(internal->intr_tid, &status);
+	}
+	internal->intr_tid = 0;
+
+	if (internal->csc_epfd >= 0)
+		close(internal->csc_epfd);
+	internal->csc_epfd = -1;
+}
+
 static int
 update_datapath(struct ifcvf_internal *internal)
 {
@@ -592,10 +702,16 @@ struct rte_vdpa_dev_info {
 		if (ret)
 			goto err;
 
+		ret = setup_intr_relay(internal);
+		if (ret)
+			goto err;
+
 		rte_atomic32_set(&internal->running, 1);
 	} else if (rte_atomic32_read(&internal->running) &&
 		   (!rte_atomic32_read(&internal->started) ||
 		    !rte_atomic32_read(&internal->dev_attached))) {
+		unset_intr_relay(internal);
+
 		ret = unset_notify_relay(internal);
 		if (ret)
 			goto err;
@@ -812,7 +928,7 @@ struct rte_vdpa_dev_info {
 		if (nfds < 0) {
 			if (errno == EINTR)
 				continue;
-			DRV_LOG(ERR, "epoll_wait return fail\n");
+			DRV_LOG(ERR, "epoll_wait return fail.");
 			return NULL;
 		}
 
@@ -888,6 +1004,9 @@ struct rte_vdpa_dev_info {
 	/* stop the direct IO data path */
 	unset_notify_relay(internal);
 	vdpa_ifcvf_stop(internal);
+
+	unset_intr_relay(internal);
+
 	vdpa_disable_vfio_intr(internal);
 
 	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
Signature · 1 line
1.8.3.1
Reviewed-by on v9 12/13
#226 lore raw

RE: [PATCH v9 12/13] vdpa/ifc: add interrupt handling for config space

Quoted 10 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 12/13] vdpa/ifc: add interrupt handling for config space

Quoted 8 lines

Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 121
+++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 120 insertions(+), 1 deletion(-)
author
#227 lore raw

PATCH v9 13/13 vdpa/ifc/base: access correct register for blk device

In reply to Andy Pei

Register address is different between net and blk device. We are re-useing most of the code, when register address is different, we have to check net and blk device go through different code.

---
 drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +27 -7
index d10c1fd..dd475a7 100644
@@ -218,10 +218,17 @@
 				&cfg->queue_used_hi);
 		IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size);
 
-		*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) =
-			(u32)hw->vring[i].last_avail_idx |
-			((u32)hw->vring[i].last_used_idx << 16);
+		if (hw->device_type == IFCVF_BLK)
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				i * IFCVF_LM_CFG_SIZE) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
+		else
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				(i / 2) * IFCVF_LM_CFG_SIZE +
+				(i % 2) * 4) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
 
 		IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector);
 		if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -254,9 +261,22 @@
 		IFCVF_WRITE_REG16(i, &cfg->queue_select);
 		IFCVF_WRITE_REG16(0, &cfg->queue_enable);
 		IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector);
-		ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4);
-		hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
+
+		if (hw->device_type == IFCVF_BLK)
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					i * IFCVF_LM_CFG_SIZE);
+		else
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					(i / 2) * IFCVF_LM_CFG_SIZE +
+					(i % 2) * 4);
+
+		if (hw->device_type == IFCVF_BLK)
+			hw->vring[i].last_avail_idx =
+				(u16)(ring_state & IFCVF_16_BIT_MASK);
+		else
+			hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
 		hw->vring[i].last_used_idx = (u16)(ring_state >> 16);
 	}
 }
Signature · 1 line
1.8.3.1
Reviewed-by on v9 13/13
#228 lore raw

RE: [PATCH v9 13/13] vdpa/ifc/base: access correct register for blk device

Quoted 10 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 13/13] vdpa/ifc/base: access correct register for blk device

Quoted 2 lines

Register address is different between net and blk device. We are re-useing most of the code, when register address is

re-using

With this fixed:

Quoted 8 lines

different, we have to check net and blk device go through different code.

---
 drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)
author
#229 lore raw

RE: [PATCH v9 13/13] vdpa/ifc/base: access correct register for blk device

Hi Chenbo,

Sure.
Thanks for your review.
I will address in V10.

Quoted 38 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, May 24, 2022 11:10 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v9 13/13] vdpa/ifc/base: access correct register for blk device

Quoted 11 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 13/13] vdpa/ifc/base: access correct register for blk device

Quoted 2 lines

Register address is different between net and blk device. We are re-useing most of the code, when register address is

re-using

With this fixed:

Quoted 8 lines

different, we have to check net and blk device go through different code.

---
 drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)
author
#230 lore raw

v10 cover add virtio_blk device support to vdpa/ifc

In reply to Andy Pei

This patch set add virtio_blk device support to vdpa/ifc driver. With a lot of similarities, I re-use part of vdpa/ifc driver. Distinguish the virtio net and blk device by device id, and implement specific features and ops.
Add example to vdpa to support virtio_blk device.
To support blk device live migration, some modification to vhost lib. Perform dev_conf op only under VHOST_USER_SET_VRING_CALL msg.

v10:
fix typo.
add more NULL pointer check.
fix a error handler.

v9:
fix some naming issue.
add more NULL pointer check.
close open file when errors occur.

v8:
delete some redundant code.
fix some commit log.

v7:
Check on expected fd num in new vhost msg handler. Sanity check on vhost msg size.
Fix typo.
Add commit log to help understand code. Remove duplicated code.
Add new API to get vDPA device type.

v6:
fix some commit log.
add vhost socket in log output to make it more user-friendly. when driver ops fail, just output some log, do not break message handler. check vhost msg size in msg handler.
v5:
fix some coding style issues.
v4:
add args "isblk" to vdpa example to specify a block device, fix some issue in example.
Make sure code specify for block device does not affect net device.
v3:
Fix some compile issues.
v2:
Fix some coding style issues.

Andy Pei (13):
vdpa/ifc: add support for virtio blk device
vhost: add vDPA ops for blk device
vhost: add vhost msg support for get/set config vdpa/ifc: add blk ops for ifc device vdpa/ifc: add vDPA interrupt relay for blk device vdpa/ifc: add block device SW live-migration
vhost: add API to get vDPA device type vdpa/ifc: add get device type ops to ifc driver examples/vdpa: add virtio blk support
usertools: add support for virtio blk device vdpa/ifc: add log for config space of virtio blk vdpa/ifc: add interrupt handling for config space vdpa/ifc/base: access correct register for blk device

 doc/guides/prog_guide/vhost_lib.rst    |   5 +
 doc/guides/rel_notes/release_22_07.rst |   4 +
 drivers/vdpa/ifc/base/ifcvf.c          |  34 ++-
 drivers/vdpa/ifc/base/ifcvf.h          |  27 ++-
 drivers/vdpa/ifc/ifcvf_vdpa.c          | 420 +++++++++++++++++++++++++++++++--
 examples/vdpa/main.c                   |  56 +++++
 examples/vdpa/vdpa_blk_compact.h       |  50 ++++
 lib/vhost/rte_vhost.h                  |  17 ++
 lib/vhost/socket.c                     |  44 ++++
 lib/vhost/vdpa_driver.h                |  11 +-
 lib/vhost/version.map                  |   1 +
 lib/vhost/vhost_user.c                 |  85 +++++++
 lib/vhost/vhost_user.h                 |  13 +
 usertools/dpdk-devbind.py              |   5 +-
 14 files changed, 741 insertions(+), 31 deletions(-)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
Signature · 1 line
1.8.3.1
author
#231 lore raw

PATCH v10 1/13 vdpa/ifc: add support for virtio blk device

Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops.

---
 drivers/vdpa/ifc/base/ifcvf.h | 22 +++++++---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 95 ++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 102 insertions(+), 15 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +17 -5
index 573a35f..a761d49 100644
@@ -5,12 +5,19 @@
 #ifndef _IFCVF_H_
 #define _IFCVF_H_
 
+#include <linux/virtio_blk.h>
 #include "ifcvf_osdep.h"
 
-#define IFCVF_VENDOR_ID		0x1AF4
-#define IFCVF_DEVICE_ID		0x1041
-#define IFCVF_SUBSYS_VENDOR_ID	0x8086
-#define IFCVF_SUBSYS_DEVICE_ID	0x001A
+#define IFCVF_NET	0
+#define IFCVF_BLK	1
+
+#define IFCVF_VENDOR_ID                     0x1AF4
+#define IFCVF_NET_DEVICE_ID                 0x1041
+#define IFCVF_BLK_MODERN_DEVICE_ID          0x1042
+#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID    0x1001
+#define IFCVF_SUBSYS_VENDOR_ID              0x8086
+#define IFCVF_SUBSYS_DEVICE_ID              0x001A
+#define IFCVF_BLK_DEVICE_ID                 0x0002
 
 #define IFCVF_MAX_QUEUES		1
 
@@ -126,13 +133,18 @@ struct ifcvf_hw {
 	u8     notify_region;
 	u32    notify_off_multiplier;
 	struct ifcvf_pci_common_cfg *common_cfg;
-	struct ifcvf_net_config *dev_cfg;
+	union {
+		struct ifcvf_net_config *net_cfg;
+		struct virtio_blk_config *blk_cfg;
+		void *dev_cfg;
+	};
 	u8     *isr;
 	u16    *notify_base;
 	u16    *notify_addr[IFCVF_MAX_QUEUES * 2];
 	u8     *lm_cfg;
 	struct vring_info vring[IFCVF_MAX_QUEUES * 2];
 	u8 nr_vring;
+	int device_type;
 	struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE];
 };
 
drivers/vdpa/ifc/ifcvf_vdpa.c +85 -10

141 lines not on this page · raw message

Signature · 1 line
1.8.3.1
author
#232 lore raw

PATCH v10 2/13 vhost: add vDPA ops for blk device

In reply to Andy Pei

Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops.

---
 lib/vhost/vdpa_driver.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
lib/vhost/vdpa_driver.h +6 -2
index 88138be..c4233a6 100644
@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {
 	/** Reset statistics of the queue */
 	int (*reset_stats)(struct rte_vdpa_device *dev, int qid);
 
-	/** Reserved for future extension */
-	void *reserved[2];
+	/** Get the device configuration space */
+	int (*get_config)(int vid, uint8_t *config, uint32_t size);
+
+	/** Set the device configuration space */
+	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
+		      uint32_t size, uint32_t flags);
 };
 
 /**
Signature · 1 line
1.8.3.1
author
#233 lore raw

PATCH v10 3/13 vhost: add vhost msg support for get/set config

In reply to Andy Pei

Add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device.

---
 lib/vhost/vhost_user.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/vhost/vhost_user.h | 13 ++++++++
 2 files changed, 98 insertions(+)
lib/vhost/vhost_user.c +85 -0
index 850848c..4f2a777 100644
@@ -2468,6 +2468,89 @@ static int is_vring_iotlb(struct virtio_net *dev,
 }
 
 static int
+vhost_user_get_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (!vdpa_dev) {
+		VHOST_LOG_CONFIG(ERR, "(%s) is not vDPA device!\n",
+				 dev->ifname);
+		return RTE_VHOST_MSG_RESULT_ERR;
+	}
+
+	if (vdpa_dev->ops->get_config) {
+		ret = vdpa_dev->ops->get_config(dev->vid,
+					   ctx->msg.payload.cfg.region,
+					   ctx->msg.payload.cfg.size);
+		if (ret != 0) {
+			ctx->msg.size = 0;
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) get_config() return error!\n",
+					 dev->ifname);
+		}
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) get_config() not supported!\n",
+				 dev->ifname);
+	}
+
+	return RTE_VHOST_MSG_RESULT_REPLY;
+}
+
+static int
+vhost_user_set_config(struct virtio_net **pdev,
+			struct vhu_msg_context *ctx,
+			int main_fd __rte_unused)
+{
+	struct virtio_net *dev = *pdev;
+	struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev;
+	int ret = 0;
+
+	if (validate_msg_fds(dev, ctx, 0) != 0)
+		return RTE_VHOST_MSG_RESULT_ERR;
+
+	if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) {
+		VHOST_LOG_CONFIG(ERR,
+			"(%s) vhost_user_config size: %"PRIu32", should not be larger than %d\n",
+			dev->ifname, ctx->msg.payload.cfg.size,
+			VHOST_USER_MAX_CONFIG_SIZE);
+		goto out;
+	}
+
+	if (!vdpa_dev) {
+		VHOST_LOG_CONFIG(ERR, "(%s) is not vDPA device!\n",
+				 dev->ifname);
+		goto out;
+	}
+
+	if (vdpa_dev->ops->set_config) {
+		ret = vdpa_dev->ops->set_config(dev->vid,
+			ctx->msg.payload.cfg.region,
+			ctx->msg.payload.cfg.offset,
+			ctx->msg.payload.cfg.size,
+			ctx->msg.payload.cfg.flags);
+		if (ret)
+			VHOST_LOG_CONFIG(ERR,
+					 "(%s) set_config() return error!\n",
+					 dev->ifname);
+	} else {
+		VHOST_LOG_CONFIG(ERR, "(%s) set_config() not supported!\n",
+				 dev->ifname);
+	}
+
+	return RTE_VHOST_MSG_RESULT_OK;
+
+out:
+	return RTE_VHOST_MSG_RESULT_ERR;
+}
+
+static int
 vhost_user_iotlb_msg(struct virtio_net **pdev,
 			struct vhu_msg_context *ctx,
 			int main_fd __rte_unused)
@@ -2686,6 +2769,8 @@ static int is_vring_iotlb(struct virtio_net *dev,
 VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU, vhost_user_net_set_mtu, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD, vhost_user_set_req_fd, true) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG, vhost_user_get_config, false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG, vhost_user_set_config, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE, vhost_user_set_postcopy_advise, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false) \
 VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END, vhost_user_postcopy_end, false) \
lib/vhost/vhost_user.h +13 -0
index ba1c5c7..c4d091e 100644
@@ -50,6 +50,8 @@
 	VHOST_USER_NET_SET_MTU = 20,
 	VHOST_USER_SET_SLAVE_REQ_FD = 21,
 	VHOST_USER_IOTLB_MSG = 22,
+	VHOST_USER_GET_CONFIG = 24,
+	VHOST_USER_SET_CONFIG = 25,
 	VHOST_USER_CRYPTO_CREATE_SESS = 26,
 	VHOST_USER_CRYPTO_CLOSE_SESS = 27,
 	VHOST_USER_POSTCOPY_ADVISE = 28,
@@ -123,6 +125,16 @@
 	uint16_t queue_size;
 } VhostUserInflight;
 
+#define VHOST_USER_MAX_CONFIG_SIZE		256
+
+/** Get/set config msg payload */
+struct vhost_user_config {
+	uint32_t offset;
+	uint32_t size;
+	uint32_t flags;
+	uint8_t region[VHOST_USER_MAX_CONFIG_SIZE];
+};
+
 typedef struct VhostUserMsg {
 	union {
 		uint32_t master; /* a VhostUserRequest value */
@@ -146,6 +158,7 @@
 		VhostUserCryptoSessionParam crypto_session;
 		VhostUserVringArea area;
 		VhostUserInflight inflight;
+		struct vhost_user_config cfg;
 	} payload;
 	/* Nothing should be added after the payload */
 } __rte_packed VhostUserMsg;
Signature · 1 line
1.8.3.1
Reviewed-by on v10 3/13
#234 lore raw

RE: [PATCH v10 03/13] vhost: add vhost msg support for get/set config

Quoted 13 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Tuesday, May 24, 2022 10:48 AM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v10 03/13] vhost: add vhost msg support for get/set config

Add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device.

Quoted 6 lines
---
 lib/vhost/vhost_user.c | 85
++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/vhost/vhost_user.h | 13 ++++++++
 2 files changed, 98 insertions(+)
author
#235 lore raw

PATCH v10 4/13 vdpa/ifc: add blk ops for ifc device

In reply to Andy Pei

For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.

---
 drivers/vdpa/ifc/base/ifcvf.h |  4 ++
 drivers/vdpa/ifc/ifcvf_vdpa.c | 96 ++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 99 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/base/ifcvf.h +4 -0
index a761d49..b6fdfdb 100644
@@ -65,6 +65,10 @@
 #define IFCVF_32_BIT_MASK		0xffffffff
 
 
+#ifndef VHOST_USER_PROTOCOL_F_CONFIG
+#define VHOST_USER_PROTOCOL_F_CONFIG	9
+#endif
+
 struct ifcvf_pci_cap {
 	u8 cap_vndr;            /* Generic PCI field: PCI_CAP_ID_VNDR */
 	u8 cap_next;            /* Generic PCI field: next ptr. */
drivers/vdpa/ifc/ifcvf_vdpa.c +95 -1

114 lines not on this page · raw message

Signature · 1 line
1.8.3.1
Reviewed-by on v10 4/13
#236 lore raw

RE: [PATCH v10 04/13] vdpa/ifc: add blk ops for ifc device

Quoted 14 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Tuesday, May 24, 2022 10:48 AM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v10 04/13] vdpa/ifc: add blk ops for ifc device

For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.

Quoted 6 lines
---
 drivers/vdpa/ifc/base/ifcvf.h |  4 ++
 drivers/vdpa/ifc/ifcvf_vdpa.c | 96
++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 99 insertions(+), 1 deletion(-)
author
#237 lore raw

RE: [PATCH v10 04/13] vdpa/ifc: add blk ops for ifc device

Hi Chenbo,

Thanks for your effort.

Quoted 32 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, May 24, 2022 2:28 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v10 04/13] vdpa/ifc: add blk ops for ifc device

Quoted 14 lines

-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Tuesday, May 24, 2022 10:48 AM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v10 04/13] vdpa/ifc: add blk ops for ifc device

For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.

Quoted 6 lines

---
drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c
| 96
++++++++++++++++++++++++++++++++++++++++++-

 2 files changed, 99 insertions(+), 1 deletion(-)
author
#238 lore raw

PATCH v10 5/13 vdpa/ifc: add vDPA interrupt relay for blk device

In reply to Andy Pei

For the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +8 -1
index 10a4a25..61fb427 100644
@@ -369,6 +369,7 @@ struct rte_vdpa_dev_info {
 	irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;
 	irq_set->start = 0;
 	fd_ptr = (int *)&irq_set->data;
+	/* The first interrupt is for the configure space change notification */
 	fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] =
 		rte_intr_fd_get(internal->pdev->intr_handle);
 
@@ -378,7 +379,13 @@ struct rte_vdpa_dev_info {
 	for (i = 0; i < nr_vring; i++) {
 		rte_vhost_get_vhost_vring(internal->vid, i, &vring);
 		fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
-		if ((i & 1) == 0 && m_rx == true) {
+		if (m_rx == true &&
+			((i & 1) == 0 || internal->hw.device_type == IFCVF_BLK)) {
+			/* For the net we only need to relay rx queue,
+			 * which will change the mem of VM.
+			 * For the blk we need to relay all the read cmd
+			 * of each queue
+			 */
 			fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
 			if (fd < 0) {
 				DRV_LOG(ERR, "can't setup eventfd: %s",
Signature · 1 line
1.8.3.1
author
#239 lore raw

PATCH v10 6/13 vdpa/ifc: add block device SW live-migration

In reply to Andy Pei

Add SW live-migration support to block device. For block device, it is critical that no packet should be dropped. So when virtio blk device is paused, make sure hardware last_avail_idx and last_used_idx are the same. This indicates all requests have received acks, and no inflight IO.

---
 drivers/vdpa/ifc/base/ifcvf.h |  1 +
 drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 39 insertions(+), 4 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
index b6fdfdb..9d95aac 100644
@@ -63,6 +63,7 @@
 #define IFCVF_MEDIATED_VRING		0x200000000000
 
 #define IFCVF_32_BIT_MASK		0xffffffff
+#define IFCVF_16_BIT_MASK		0xffff
 
 
 #ifndef VHOST_USER_PROTOCOL_F_CONFIG
drivers/vdpa/ifc/ifcvf_vdpa.c +38 -4

69 lines not on this page · raw message

Signature · 1 line
1.8.3.1
author
#240 lore raw

PATCH v10 7/13 vhost: add API to get vDPA device type

In reply to Andy Pei

Vhost backend of different devices have different features. Add an API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices.

---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++++
 doc/guides/rel_notes/release_22_07.rst |  4 ++++
 lib/vhost/rte_vhost.h                  | 17 +++++++++++++
 lib/vhost/socket.c                     | 44 ++++++++++++++++++++++++++++++++++
 lib/vhost/vdpa_driver.h                |  3 +++
 lib/vhost/version.map                  |  1 +
 6 files changed, 74 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +5 -0
index f287b76..0337b38 100644
@@ -282,6 +282,11 @@ The following is an overview of some key Vhost API functions:
   Clear inflight packets which are submitted to DMA engine in vhost async data
   path. Completed packets are returned to applications through ``pkts``.
 
+* ``rte_vhost_driver_get_vdpa_dev_type(path, type)``
+
+  Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET,
+  VDPA_DEVICE_TYPE_BLK.
+
 Vhost-user Implementations
 --------------------------
 
doc/guides/rel_notes/release_22_07.rst +4 -0
index e49cace..63875b7 100644
@@ -60,6 +60,10 @@ New Features
   Added an API which can get the number of in-flight packets in
   vhost async data path without using lock.
 
+* **Added vhost API to get the device type of a vDPA device.**
+
+  Added an API which can get the device type of vDPA device.
+
 * **Updated Intel iavf driver.**
 
   * Added Tx QoS queue rate limitation support.
lib/vhost/rte_vhost.h +17 -0
index c733f85..2f130ec 100644
@@ -117,6 +117,9 @@
 
 #define RTE_MAX_VHOST_DEVICE	1024
 
+#define RTE_VHOST_VDPA_DEVICE_TYPE_NET 0
+#define RTE_VHOST_VDPA_DEVICE_TYPE_BLK 1
+
 struct rte_vdpa_device;
 
 /**
@@ -486,6 +489,20 @@ struct rte_vdpa_device *
 rte_vhost_driver_get_vdpa_device(const char *path);
 
 /**
+ * Get the device type of the vdpa device.
+ *
+ * @param path
+ *  The vhost-user socket file path
+ * @param type
+ *  the device type of the vdpa device
+ * @return
+ *  0 on success, -1 on failure
+ */
+__rte_experimental
+int
+rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type);
+
+/**
  * Set the feature bits the vhost-user driver supports.
  *
  * @param path
lib/vhost/socket.c +44 -0

50 lines not on this page · raw message

lib/vhost/vdpa_driver.h +3 -0
index c4233a6..8b88a53 100644
@@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops {
 	/** Set the device configuration space */
 	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
 		      uint32_t size, uint32_t flags);
+
+	/** get device type: net device, blk device... */
+	int (*get_dev_type)(struct rte_vdpa_device *dev, uint32_t *type);
 };
 
 /**
lib/vhost/version.map +1 -0
index 5841315..583b4f3 100644
@@ -90,6 +90,7 @@ EXPERIMENTAL {
 
 	# added in 22.07
 	rte_vhost_async_get_inflight_thread_unsafe;
+	rte_vhost_driver_get_vdpa_dev_type;
 
 };
 
Signature · 1 line
1.8.3.1
author
#241 lore raw

PATCH v10 8/13 vdpa/ifc: add get device type ops to ifc driver

In reply to Andy Pei

Add get device type ops to ifc driver.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +25 -0
index 0f9db8a..f4c6198 100644
@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static int
+ifcvf_get_device_type(struct rte_vdpa_device *vdev,
+	uint32_t *type)
+{
+	struct ifcvf_internal *internal;
+	struct internal_list *list;
+
+	list = find_internal_resource_by_vdev(vdev);
+	if (list == NULL) {
+		DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -1;
+	}
+
+	internal = list->internal;
+
+	if (internal->hw.device_type == IFCVF_BLK)
+		*type = RTE_VHOST_VDPA_DEVICE_TYPE_BLK;
+	else
+		*type = RTE_VHOST_VDPA_DEVICE_TYPE_NET;
+
+	return 0;
+}
+
 static struct rte_vdpa_dev_ops ifcvf_net_ops = {
 	.get_queue_num = ifcvf_get_queue_num,
 	.get_features = ifcvf_get_vdpa_features,
@@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 static inline int
@@ -1332,6 +1356,7 @@ struct rte_vdpa_dev_info {
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
 	.get_notify_area = ifcvf_get_notify_area,
 	.get_config = ifcvf_blk_get_config,
+	.get_dev_type = ifcvf_get_device_type,
 };
 
 struct rte_vdpa_dev_info dev_info[] = {
Signature · 1 line
1.8.3.1
author
#242 lore raw

PATCH v10 9/13 examples/vdpa: add virtio blk support

In reply to Andy Pei

Add virtio blk device support to vDPA example.

---
 examples/vdpa/main.c             | 56 ++++++++++++++++++++++++++++++++++++++++
 examples/vdpa/vdpa_blk_compact.h | 50 +++++++++++++++++++++++++++++++++++
 2 files changed, 106 insertions(+)
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c +56 -0
index 5ab0765..7e11ef4 100644
@@ -20,6 +20,7 @@
 #include <cmdline_parse_string.h>
 #include <cmdline_parse_num.h>
 #include <cmdline.h>
+#include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
 #define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,53 @@ struct vdpa_port {
 };
 
 static int
+vdpa_blk_device_set_features_and_protocol(const char *path)
+{
+	uint64_t protocol_features = 0;
+	int ret;
+
+	ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_disable_features(path,
+		VHOST_BLK_DISABLED_FEATURES);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_disable_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	ret = rte_vhost_driver_get_protocol_features(path, &protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_get_protocol_features for %s failed.\n",
+			path);
+		goto out;
+	}
+
+	protocol_features |= VHOST_BLK_PROTOCOL_FEATURES;
+
+	ret = rte_vhost_driver_set_protocol_features(path, protocol_features);
+	if (ret != 0) {
+		RTE_LOG(ERR, VDPA,
+			"rte_vhost_driver_set_protocol_features for %s failed.\n",
+			path);
+	}
+
+out:
+	return ret;
+}
+
+static int
 start_vdpa(struct vdpa_port *vport)
 {
+	uint32_t device_type = 0;
 	int ret;
 	char *socket_path = vport->ifname;
 
@@ -192,6 +238,16 @@ struct vdpa_port {
 			"attach vdpa device failed: %s\n",
 			socket_path);
 
+	ret = rte_vhost_driver_get_vdpa_dev_type(socket_path, &device_type);
+	if (ret == 0 && device_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK) {
+		RTE_LOG(NOTICE, VDPA, "%s is a blk device\n", socket_path);
+		ret = vdpa_blk_device_set_features_and_protocol(socket_path);
+		if (ret != 0)
+			rte_exit(EXIT_FAILURE,
+				"set vhost blk driver features and protocol features failed: %s\n",
+				socket_path);
+	}
+
 	if (rte_vhost_driver_start(socket_path) < 0)
 		rte_exit(EXIT_FAILURE,
 			"start vhost driver failed: %s\n",
examples/vdpa/vdpa_blk_compact.h new file +50 -0

50 lines not on this page · raw message

Signature · 1 line
1.8.3.1
author
#243 lore raw

PATCH v10 10/13 usertools: add support for virtio blk device

In reply to Andy Pei

Add virtio blk device support to devbind.

---
 usertools/dpdk-devbind.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-devbind.py +4 -1
index ace4627..18c7d67 100755
@@ -72,6 +72,9 @@
 cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4',
                  'SVendor': None, 'SDevice': None}
 
+virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001,1042',
+                    'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 baseband_devices = [acceleration_class]
 crypto_devices = [encryption_class, intel_processor_class]
@@ -82,7 +85,7 @@
 compress_devices = [cavium_zip]
 regex_devices = [cn9k_ree]
 misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev,
-                intel_ntb_skx, intel_ntb_icx]
+                intel_ntb_skx, intel_ntb_icx, virtio_blk]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
Signature · 1 line
1.8.3.1
author
#244 lore raw

PATCH v10 11/13 vdpa/ifc: add log for config space of virtio blk

In reply to Andy Pei

Add some log of virtio blk device config space information at VDPA launch before qemu connects.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +34 -0
index f4c6198..30850b0 100644
@@ -1387,6 +1387,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
 	int16_t device_id;
+	uint64_t capacity = 0;
+	uint8_t *byte;
+	uint32_t i;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
@@ -1453,6 +1456,37 @@ struct rte_vdpa_dev_info dev_info[] = {
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_BLK].features;
+
+		/* cannot read 64-bit register in one attempt,
+		 * so read byte by byte.
+		 */
+		for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+			byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+			capacity |= (uint64_t)*byte << (i * 8);
+		}
+		/* The capacity is number of sectors in 512-byte.
+		 * So right shift 1 bit  we get in K,
+		 * another right shift 10 bits we get in M,
+		 * right shift 10 more bits, we get in G.
+		 * To show capacity in G, we right shift 21 bits in total.
+		 */
+		DRV_LOG(DEBUG, "capacity  : %"PRIu64"G", capacity >> 21);
+
+		DRV_LOG(DEBUG, "size_max  : 0x%08x",
+			internal->hw.blk_cfg->size_max);
+		DRV_LOG(DEBUG, "seg_max   : 0x%08x",
+			internal->hw.blk_cfg->seg_max);
+		DRV_LOG(DEBUG, "blk_size  : 0x%08x",
+			internal->hw.blk_cfg->blk_size);
+		DRV_LOG(DEBUG, "geometry");
+		DRV_LOG(DEBUG, "    cylinders: %u",
+			internal->hw.blk_cfg->geometry.cylinders);
+		DRV_LOG(DEBUG, "    heads    : %u",
+			internal->hw.blk_cfg->geometry.heads);
+		DRV_LOG(DEBUG, "    sectors  : %u",
+			internal->hw.blk_cfg->geometry.sectors);
+		DRV_LOG(DEBUG, "num_queues: 0x%08x",
+			internal->hw.blk_cfg->num_queues);
 	}
 
 	list->internal = internal;
Signature · 1 line
1.8.3.1
author
#245 lore raw

PATCH v10 12/13 vdpa/ifc: add interrupt handling for config space

In reply to Andy Pei

Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 121 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 120 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +120 -1
index 30850b0..40a18b2 100644
@@ -53,7 +53,9 @@ struct ifcvf_internal {
 	int vfio_group_fd;
 	int vfio_dev_fd;
 	pthread_t tid;	/* thread for notify relay */
+	pthread_t intr_tid; /* thread for config space change interrupt relay */
 	int epfd;
+	int csc_epfd;
 	int vid;
 	struct rte_vdpa_device *vdev;
 	uint16_t max_queues;
@@ -566,6 +568,114 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static void
+virtio_interrupt_handler(struct ifcvf_internal *internal)
+{
+	int vid = internal->vid;
+	int ret;
+
+	ret = rte_vhost_slave_config_change(vid, 1);
+	if (ret)
+		DRV_LOG(ERR, "failed to notify the guest about configuration space change.");
+}
+
+static void *
+intr_relay(void *arg)
+{
+	struct ifcvf_internal *internal = (struct ifcvf_internal *)arg;
+	struct epoll_event csc_event;
+	struct epoll_event ev;
+	uint64_t buf;
+	int nbytes;
+	int csc_epfd, csc_val = 0;
+
+	csc_epfd = epoll_create(1);
+	if (csc_epfd < 0) {
+		DRV_LOG(ERR, "failed to create epoll for config space change.");
+		return NULL;
+	}
+
+	ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP;
+	ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle);
+	if (epoll_ctl(csc_epfd, EPOLL_CTL_ADD,
+		rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) {
+		DRV_LOG(ERR, "epoll add error: %s", strerror(errno));
+		goto out;
+	}
+
+	internal->csc_epfd = csc_epfd;
+
+	for (;;) {
+		csc_val = epoll_wait(csc_epfd, &csc_event, 1, -1);
+		if (csc_val < 0) {
+			if (errno == EINTR)
+				continue;
+			DRV_LOG(ERR, "epoll_wait return fail.");
+			goto out;
+		} else if (csc_val == 0) {
+			continue;
+		} else {
+			/* csc_val > 0 */
+			nbytes = read(csc_event.data.fd, &buf, 8);
+			if (nbytes < 0) {
+				if (errno == EINTR ||
+				    errno == EWOULDBLOCK ||
+				    errno == EAGAIN)
+					continue;
+				DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n",
+					csc_event.data.fd,
+					strerror(errno));
+				goto out;
+			} else if (nbytes == 0) {
+				DRV_LOG(ERR, "Read nothing from file descriptor %d\n",
+					csc_event.data.fd);
+				continue;
+			} else {
+				virtio_interrupt_handler(internal);
+			}
+		}
+	}
+
+out:
+	if (csc_epfd >= 0)
+		close(csc_epfd);
+	internal->csc_epfd = -1;
+
+	return NULL;
+}
+
+static int
+setup_intr_relay(struct ifcvf_internal *internal)
+{
+	char name[THREAD_NAME_LEN];
+	int ret;
+
+	snprintf(name, sizeof(name), "ifc-intr-%d", internal->vid);
+	ret = rte_ctrl_thread_create(&internal->intr_tid, name, NULL,
+				     intr_relay, (void *)internal);
+	if (ret) {
+		DRV_LOG(ERR, "failed to create notify relay pthread.");
+		return -1;
+	}
+	return 0;
+}
+
+static void
+unset_intr_relay(struct ifcvf_internal *internal)
+{
+	void *status;
+
+	if (internal->intr_tid) {
+		pthread_cancel(internal->intr_tid);
+		pthread_join(internal->intr_tid, &status);
+	}
+	internal->intr_tid = 0;
+
+	if (internal->csc_epfd >= 0)
+		close(internal->csc_epfd);
+	internal->csc_epfd = -1;
+}
+
 static int
 update_datapath(struct ifcvf_internal *internal)
 {
@@ -592,10 +702,16 @@ struct rte_vdpa_dev_info {
 		if (ret)
 			goto err;
 
+		ret = setup_intr_relay(internal);
+		if (ret)
+			goto err;
+
 		rte_atomic32_set(&internal->running, 1);
 	} else if (rte_atomic32_read(&internal->running) &&
 		   (!rte_atomic32_read(&internal->started) ||
 		    !rte_atomic32_read(&internal->dev_attached))) {
+		unset_intr_relay(internal);
+
 		ret = unset_notify_relay(internal);
 		if (ret)
 			goto err;
@@ -812,7 +928,7 @@ struct rte_vdpa_dev_info {
 		if (nfds < 0) {
 			if (errno == EINTR)
 				continue;
-			DRV_LOG(ERR, "epoll_wait return fail\n");
+			DRV_LOG(ERR, "epoll_wait return fail.");
 			return NULL;
 		}
 
@@ -888,6 +1004,9 @@ struct rte_vdpa_dev_info {
 	/* stop the direct IO data path */
 	unset_notify_relay(internal);
 	vdpa_ifcvf_stop(internal);
+
+	unset_intr_relay(internal);
+
 	vdpa_disable_vfio_intr(internal);
 
 	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
Signature · 1 line
1.8.3.1
author
#246 lore raw

PATCH v10 13/13 vdpa/ifc/base: access correct register for blk device

In reply to Andy Pei

Register address is different between net and blk device. We are re-using most of the code, when register address is different, we have to check net and blk device go through different code.

---
 drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +27 -7
index d10c1fd..dd475a7 100644
@@ -218,10 +218,17 @@
 				&cfg->queue_used_hi);
 		IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size);
 
-		*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) =
-			(u32)hw->vring[i].last_avail_idx |
-			((u32)hw->vring[i].last_used_idx << 16);
+		if (hw->device_type == IFCVF_BLK)
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				i * IFCVF_LM_CFG_SIZE) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
+		else
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				(i / 2) * IFCVF_LM_CFG_SIZE +
+				(i % 2) * 4) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
 
 		IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector);
 		if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -254,9 +261,22 @@
 		IFCVF_WRITE_REG16(i, &cfg->queue_select);
 		IFCVF_WRITE_REG16(0, &cfg->queue_enable);
 		IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector);
-		ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4);
-		hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
+
+		if (hw->device_type == IFCVF_BLK)
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					i * IFCVF_LM_CFG_SIZE);
+		else
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					(i / 2) * IFCVF_LM_CFG_SIZE +
+					(i % 2) * 4);
+
+		if (hw->device_type == IFCVF_BLK)
+			hw->vring[i].last_avail_idx =
+				(u16)(ring_state & IFCVF_16_BIT_MASK);
+		else
+			hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
 		hw->vring[i].last_used_idx = (u16)(ring_state >> 16);
 	}
 }
Signature · 1 line
1.8.3.1
#247 lore raw
In reply to Andy Pei
Quoted 77 lines · Andy Pei

This patch set add virtio_blk device support to vdpa/ifc driver. With a lot of similarities, I re-use part of vdpa/ifc driver. Distinguish the virtio net and blk device by device id, and implement specific features and ops.
Add example to vdpa to support virtio_blk device.
To support blk device live migration, some modification to vhost lib. Perform dev_conf op only under VHOST_USER_SET_VRING_CALL msg.

v10:
fix typo.
add more NULL pointer check.
fix a error handler.

v9:
fix some naming issue.
add more NULL pointer check.
close open file when errors occur.

v8:
delete some redundant code.
fix some commit log.

v7:
Check on expected fd num in new vhost msg handler. Sanity check on vhost msg size.
Fix typo.
Add commit log to help understand code. Remove duplicated code.
Add new API to get vDPA device type.

v6:
fix some commit log.
add vhost socket in log output to make it more user-friendly. when driver ops fail, just output some log, do not break message handler. check vhost msg size in msg handler.
v5:
fix some coding style issues.
v4:
add args "isblk" to vdpa example to specify a block device, fix some issue in example.
Make sure code specify for block device does not affect net device.
v3:
Fix some compile issues.
v2:
Fix some coding style issues.

Andy Pei (13):
vdpa/ifc: add support for virtio blk device
vhost: add vDPA ops for blk device
vhost: add vhost msg support for get/set config vdpa/ifc: add blk ops for ifc device vdpa/ifc: add vDPA interrupt relay for blk device vdpa/ifc: add block device SW live-migration
vhost: add API to get vDPA device type vdpa/ifc: add get device type ops to ifc driver examples/vdpa: add virtio blk support
usertools: add support for virtio blk device vdpa/ifc: add log for config space of virtio blk vdpa/ifc: add interrupt handling for config space vdpa/ifc/base: access correct register for blk device

  doc/guides/prog_guide/vhost_lib.rst    |   5 +
  doc/guides/rel_notes/release_22_07.rst |   4 +
  drivers/vdpa/ifc/base/ifcvf.c          |  34 ++-
  drivers/vdpa/ifc/base/ifcvf.h          |  27 ++-
  drivers/vdpa/ifc/ifcvf_vdpa.c          | 420 +++++++++++++++++++++++++++++++--
  examples/vdpa/main.c                   |  56 +++++
  examples/vdpa/vdpa_blk_compact.h       |  50 ++++
  lib/vhost/rte_vhost.h                  |  17 ++
  lib/vhost/socket.c                     |  44 ++++
  lib/vhost/vdpa_driver.h                |  11 +-
  lib/vhost/version.map                  |   1 +
  lib/vhost/vhost_user.c                 |  85 +++++++
  lib/vhost/vhost_user.h                 |  13 +
  usertools/dpdk-devbind.py              |   5 +-
  14 files changed, 741 insertions(+), 31 deletions(-)
Quoted 2 lines

create mode 100644 examples/vdpa/vdpa_blk_compact.h

Applied to dpdk-next-virtio/main.

Thanks,
Maxime

author
#248 lore raw

Thanks Maxime

Quoted 97 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Wednesday, June 1, 2022 6:11 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: Re: [PATCH v10 00/13] add virtio_blk device support to vdpa/ifc

Quoted 62 lines · Andy Pei

This patch set add virtio_blk device support to vdpa/ifc driver. With a lot of similarities, I re-use part of vdpa/ifc driver. Distinguish the virtio net and blk device by device id, and implement specific features and ops.
Add example to vdpa to support virtio_blk device.
To support blk device live migration, some modification to vhost lib. Perform dev_conf op only under VHOST_USER_SET_VRING_CALL msg.

v10:
fix typo.
add more NULL pointer check.
fix a error handler.

v9:
fix some naming issue.
add more NULL pointer check.
close open file when errors occur.

v8:
delete some redundant code.
fix some commit log.

v7:
Check on expected fd num in new vhost msg handler. Sanity check on vhost msg size.
Fix typo.
Add commit log to help understand code. Remove duplicated code.
Add new API to get vDPA device type.

v6:
fix some commit log.
add vhost socket in log output to make it more user-friendly. when driver ops fail, just output some log, do not break message handler. check vhost msg size in msg handler.
v5:
fix some coding style issues.
v4:
add args "isblk" to vdpa example to specify a block device, fix some issue in example.
Make sure code specify for block device does not affect net device.
v3:
Fix some compile issues.
v2:
Fix some coding style issues.

Andy Pei (13):
vdpa/ifc: add support for virtio blk device
vhost: add vDPA ops for blk device
vhost: add vhost msg support for get/set config vdpa/ifc: add blk ops for ifc device vdpa/ifc: add vDPA interrupt relay for blk device vdpa/ifc: add block device SW live-migration
vhost: add API to get vDPA device type vdpa/ifc: add get device type ops to ifc driver examples/vdpa: add virtio blk support
usertools: add support for virtio blk device vdpa/ifc: add log for config space of virtio blk vdpa/ifc: add interrupt handling for config space vdpa/ifc/base: access correct register for blk device

Quoted 5 lines
  doc/guides/prog_guide/vhost_lib.rst    |   5 +
  doc/guides/rel_notes/release_22_07.rst |   4 +
  drivers/vdpa/ifc/base/ifcvf.c          |  34 ++-
  drivers/vdpa/ifc/base/ifcvf.h          |  27 ++-
  drivers/vdpa/ifc/ifcvf_vdpa.c          | 420

+++++++++++++++++++++++++++++++--

Quoted 10 lines
  examples/vdpa/main.c                   |  56 +++++
  examples/vdpa/vdpa_blk_compact.h       |  50 ++++
  lib/vhost/rte_vhost.h                  |  17 ++
  lib/vhost/socket.c                     |  44 ++++
  lib/vhost/vdpa_driver.h                |  11 +-
  lib/vhost/version.map                  |   1 +
  lib/vhost/vhost_user.c                 |  85 +++++++
  lib/vhost/vhost_user.h                 |  13 +
  usertools/dpdk-devbind.py              |   5 +-
  14 files changed, 741 insertions(+), 31 deletions(-)
Quoted 2 lines

create mode 100644 examples/vdpa/vdpa_blk_compact.h

Applied to dpdk-next-virtio/main.

Thanks,
Maxime

author
#249 lore raw

PATCH 2/15 vhost: add vdpa ops for blk device

In reply to Andy Pei

Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vdpa ops.

---
 lib/vhost/vdpa_driver.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
lib/vhost/vdpa_driver.h +6 -2
index fc2d6ac..9a23db9 100644
@@ -65,8 +65,12 @@ struct rte_vdpa_dev_ops {
 	/** Reset statistics of the queue */
 	int (*reset_stats)(struct rte_vdpa_device *dev, int qid);
 
-	/** Reserved for future extension */
-	void *reserved[2];
+	/** Get the device configuration space */
+	int (*get_config)(int vid, uint8_t *config, uint32_t len);
+
+	/** Set the device configuration space */
+	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
+		      uint32_t size, uint32_t flags);
 };
 
 /**
Signature · 1 line
1.8.3.1
author
#250 lore raw

PATCH 3/15 vdpa/ifc: add blk ops for ifc device

In reply to Andy Pei

For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.

---
 drivers/vdpa/ifc/base/ifcvf.h |  4 ++
 drivers/vdpa/ifc/ifcvf_vdpa.c | 88 ++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 91 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/base/ifcvf.h +4 -0
index 01522c6..769c603 100644
@@ -66,6 +66,10 @@
 
 #define IFCVF_32_BIT_MASK		0xffffffff
 
+#ifndef VHOST_USER_PROTOCOL_F_CONFIG
+#define VHOST_USER_PROTOCOL_F_CONFIG	9
+#endif
+
 struct ifcvf_pci_cap {
 	u8 cap_vndr;            /* Generic PCI field: PCI_CAP_ID_VNDR */
 	u8 cap_next;            /* Generic PCI field: next ptr. */
drivers/vdpa/ifc/ifcvf_vdpa.c +87 -1

106 lines not on this page · raw message

Signature · 1 line
1.8.3.1
author
#251 lore raw

PATCH 4/15 vdpa/ifc: add vdpa interrupt for blk device

In reply to Andy Pei

For the blk we need to relay all the cmd of each queue.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 48 +++++++++++++++++++++++++++++++++----------
 1 file changed, 37 insertions(+), 11 deletions(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +37 -11
index 965baa2..9729490 100644
@@ -374,24 +374,50 @@ struct rte_vdpa_dev_info {
 	irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;
 	irq_set->start = 0;
 	fd_ptr = (int *)&irq_set->data;
+	/* The first interrupt is for the configure space change notification */
 	fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] =
 		rte_intr_fd_get(internal->pdev->intr_handle);
 
 	for (i = 0; i < nr_vring; i++)
 		internal->intr_fd[i] = -1;
 
-	for (i = 0; i < nr_vring; i++) {
-		rte_vhost_get_vhost_vring(internal->vid, i, &vring);
-		fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
-		if ((i & 1) == 0 && m_rx == true) {
-			fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
-			if (fd < 0) {
-				DRV_LOG(ERR, "can't setup eventfd: %s",
-					strerror(errno));
-				return -1;
+	if (internal->device_type == IFCVF_NET) {
+		for (i = 0; i < nr_vring; i++) {
+			rte_vhost_get_vhost_vring(internal->vid, i, &vring);
+			fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
+			if ((i & 1) == 0 && m_rx == true) {
+				/**
+				** For the net we only need to relay rx queue,
+				** which will change the mem of VM.
+				**/
+				fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+				if (fd < 0) {
+					DRV_LOG(ERR, "can't setup eventfd: %s",
+						strerror(errno));
+					return -1;
+				}
+				internal->intr_fd[i] = fd;
+				fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd;
+			}
+		}
+	} else if (internal->device_type == IFCVF_BLK) {
+		for (i = 0; i < nr_vring; i++) {
+			rte_vhost_get_vhost_vring(internal->vid, i, &vring);
+			fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
+			if (m_rx == true) {
+				/**
+				** For the blk we need to relay all the read cmd
+				** of each queue
+				**/
+				fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+				if (fd < 0) {
+					DRV_LOG(ERR, "can't setup eventfd: %s",
+						strerror(errno));
+					return -1;
+				}
+				internal->intr_fd[i] = fd;
+				fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd;
 			}
-			internal->intr_fd[i] = fd;
-			fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd;
 		}
 	}
 
Signature · 1 line
1.8.3.1
author
#252 lore raw

PATCH 5/15 vdpa/ifc: add blk dev sw live migration

In reply to Andy Pei

Enable virtio blk sw live migration relay callfd and log the dirty page.
In this version we ignore the write cmd and still mark it dirty. Maybe we can improve it later.

---
 drivers/vdpa/ifc/base/ifcvf.c |   4 +-
 drivers/vdpa/ifc/base/ifcvf.h |   6 ++
 drivers/vdpa/ifc/ifcvf_vdpa.c | 130 +++++++++++++++++++++++++++++++++++-------
 3 files changed, 118 insertions(+), 22 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +2 -2
index 721cb1d..3a69e53 100644
@@ -189,7 +189,7 @@
 	IFCVF_WRITE_REG32(val >> 32, hi);
 }
 
-STATIC int
+int
 ifcvf_hw_enable(struct ifcvf_hw *hw)
 {
 	struct ifcvf_pci_common_cfg *cfg;
@@ -238,7 +238,7 @@
 	return 0;
 }
 
-STATIC void
+void
 ifcvf_hw_disable(struct ifcvf_hw *hw)
 {
 	u32 i;
drivers/vdpa/ifc/base/ifcvf.h +6 -0
index 769c603..6dd7925 100644
@@ -179,4 +179,10 @@ struct ifcvf_hw {
 u64
 ifcvf_get_queue_notify_off(struct ifcvf_hw *hw, int qid);
 
+int
+ifcvf_hw_enable(struct ifcvf_hw *hw);
+
+void
+ifcvf_hw_disable(struct ifcvf_hw *hw);
+
 #endif /* _IFCVF_H_ */
drivers/vdpa/ifc/ifcvf_vdpa.c +110 -20

180 lines not on this page · raw message

Signature · 1 line
1.8.3.1
author
#253 lore raw

PATCH 6/15 example/vdpa:add vdpa blk support in example

In reply to Andy Pei
---
 examples/vdpa/Makefile           |   2 +-
 examples/vdpa/main.c             |   8 ++
 examples/vdpa/meson.build        |   1 +
 examples/vdpa/vdpa_blk_compact.c | 152 +++++++++++++++++++++++++++++++
 examples/vdpa/vdpa_blk_compact.h | 118 ++++++++++++++++++++++++
 examples/vdpa/vhost_user.h       | 190 +++++++++++++++++++++++++++++++++++++++
 6 files changed, 470 insertions(+), 1 deletion(-)
 create mode 100644 examples/vdpa/vdpa_blk_compact.c
 create mode 100644 examples/vdpa/vdpa_blk_compact.h
 create mode 100644 examples/vdpa/vhost_user.h
examples/vdpa/Makefile +1 -1
index d974db4..9d0479b 100644
@@ -5,7 +5,7 @@
 APP = vdpa
 
 # all source are stored in SRCS-y
-SRCS-y := main.c
+SRCS-y := main.c vdpa_blk_compact.c
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 PKGCONF ?= pkg-config
examples/vdpa/main.c +8 -0
index 5ab0765..3fa3d3a 100644
@@ -20,6 +20,7 @@
 #include <cmdline_parse_string.h>
 #include <cmdline_parse_num.h>
 #include <cmdline.h>
+#include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
 #define MAX_VDPA_SAMPLE_PORTS 1024
@@ -156,6 +157,7 @@ struct vdpa_port {
 static const struct rte_vhost_device_ops vdpa_sample_devops = {
 	.new_device = new_device,
 	.destroy_device = destroy_device,
+	.new_connection = rte_vhost_blk_session_install_rte_compat_hooks,
 };
 
 static int
@@ -192,6 +194,12 @@ struct vdpa_port {
 			"attach vdpa device failed: %s\n",
 			socket_path);
 
+	if (vdpa_blk_device_set_features_and_protocol(socket_path, vport->dev)
+		< 0)
+		rte_exit(EXIT_FAILURE,
+			"set vhost blk driver features and protocal features failed: %s\n",
+			socket_path);
+
 	if (rte_vhost_driver_start(socket_path) < 0)
 		rte_exit(EXIT_FAILURE,
 			"start vhost driver failed: %s\n",
examples/vdpa/meson.build +1 -0
index bd08605..f0d111c 100644
@@ -15,4 +15,5 @@ deps += 'vhost'
 allow_experimental_apis = true
 sources = files(
         'main.c',
+	'vdpa_blk_compact.c',
 )
examples/vdpa/vdpa_blk_compact.c new file +152 -0

152 lines not on this page · raw message

examples/vdpa/vdpa_blk_compact.h new file +118 -0

118 lines not on this page · raw message

examples/vdpa/vhost_user.h new file +190 -0

190 lines not on this page · raw message

Signature · 1 line
1.8.3.1
author
#254 lore raw

PATCH 7/15 usertools: add support for virtio blk device

In reply to Andy Pei
---
 usertools/dpdk-devbind.py | 8 ++++++++
 1 file changed, 8 insertions(+)
usertools/dpdk-devbind.py +8 -0
index ace4627..cbe336f 100755
@@ -14,6 +14,8 @@
 from os.path import join as path_join
 
 # The PCI base class for all devices
+virtio_blk_class = {'Class': '01', 'Vendor': "1af4", 'Device': '1001',
+                    'SVendor': '8086', 'SDevice': '0002'}
 network_class = {'Class': '02', 'Vendor': None, 'Device': None,
                  'SVendor': None, 'SDevice': None}
 acceleration_class = {'Class': '12', 'Vendor': None, 'Device': None,
@@ -72,6 +74,7 @@
 cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4',
                  'SVendor': None, 'SDevice': None}
 
+virtio_blk_devices = [virtio_blk_class]
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 baseband_devices = [acceleration_class]
 crypto_devices = [encryption_class, intel_processor_class]
@@ -587,6 +590,9 @@ def show_status():
     Displays to the user what devices are bound to the igb_uio driver, the
     kernel driver or to no driver'''
 
+    if status_dev in ["virtio_blk", "all"]:
+        show_device_status(virtio_blk_devices, "virtio_blk")
+
     if status_dev in ["net", "all"]:
         show_device_status(network_devices, "Network", if_field=True)
 
@@ -746,6 +752,7 @@ def do_arg_actions():
         if b_flag is not None:
             clear_data()
             # refresh if we have changed anything
+            get_device_details(virtio_blk_devices)
             get_device_details(network_devices)
             get_device_details(baseband_devices)
             get_device_details(crypto_devices)
@@ -769,6 +776,7 @@ def main():
     parse_args()
     check_modules()
     clear_data()
+    get_device_details(virtio_blk_devices)
     get_device_details(network_devices)
     get_device_details(baseband_devices)
     get_device_details(crypto_devices)
Signature · 1 line
1.8.3.1
author
#255 lore raw

PATCH 8/15 vdpa/ifc: set_vring_state op is mandatory, add set_vring_state for blk device

In reply to Andy Pei
---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +11 -1
index 1f832a3..eff6ff3 100644
@@ -1384,6 +1384,16 @@ struct rte_vdpa_dev_info {
 }
 
 static int
+ifcvf_blk_set_vring_state(int vid, int vring, int state)
+{
+	RTE_SET_USED(vid);
+	RTE_SET_USED(vring);
+	RTE_SET_USED(state);
+
+	return 0;
+}
+
+static int
 ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev,
 	uint64_t *features)
 {
@@ -1401,7 +1411,7 @@ struct rte_vdpa_dev_info {
 	.get_protocol_features = ifcvf_blk_get_protocol_features,
 	.dev_conf = ifcvf_dev_config,
 	.dev_close = ifcvf_dev_close,
-	.set_vring_state = NULL,
+	.set_vring_state = ifcvf_blk_set_vring_state,
 	.migration_done = NULL,
 	.get_vfio_group_fd = ifcvf_get_vfio_group_fd,
 	.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
Signature · 1 line
1.8.3.1
author
#256 lore raw

PATCH 9/15 vdpa/ifc: add some log at VDPA lauch before qemu connect

In reply to Andy Pei
---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +29 -0
index eff6ff3..0b4b77f 100644
@@ -1447,6 +1447,9 @@ struct rte_vdpa_dev_info dev_info[] = {
 	struct rte_kvargs *kvlist = NULL;
 	int ret = 0;
 	int16_t device_id;
+	__u64 capacity = 0;
+	uint8_t *byte;
+	uint32_t i;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
@@ -1513,6 +1516,32 @@ struct rte_vdpa_dev_info dev_info[] = {
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_BLK].features;
+
+		/**
+		** cannot read 64-bit register in one attempt,
+		** so read byte by byte.
+		**/
+		for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) {
+			byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i;
+			capacity |= (__u64)*byte << (i * 8);
+		}
+		DRV_LOG(INFO, "capacity  : %quG", capacity >> 21);
+
+		DRV_LOG(INFO, "size_max  : 0x%08x",
+			internal->hw.blk_cfg->size_max);
+		DRV_LOG(INFO, "seg_max   : 0x%08x",
+			internal->hw.blk_cfg->seg_max);
+		DRV_LOG(INFO, "blk_size  : 0x%08x",
+			internal->hw.blk_cfg->blk_size);
+		DRV_LOG(INFO, "geometry");
+		DRV_LOG(INFO, "    cylinders: %u",
+			internal->hw.blk_cfg->geometry.cylinders);
+		DRV_LOG(INFO, "    heads    : %u",
+			internal->hw.blk_cfg->geometry.heads);
+		DRV_LOG(INFO, "    sectors  : %u",
+			internal->hw.blk_cfg->geometry.sectors);
+		DRV_LOG(INFO, "num_queues: 0x%08x",
+			internal->hw.blk_cfg->num_queues);
 	}
 
 	list->internal = internal;
Signature · 1 line
1.8.3.1
author
#257 lore raw

PATCH 10/15 vdpa/ifc: read virtio max_queues from hardware

In reply to Andy Pei

original code max_queues is set to IFCVF_MAX_QUEUES.
New code max_queues is the min of IFCVF_MAX_QUEUES and hardware num_queues.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
index 0b4b77f..f092aca 100644
@@ -1542,6 +1542,10 @@ struct rte_vdpa_dev_info dev_info[] = {
 			internal->hw.blk_cfg->geometry.sectors);
 		DRV_LOG(INFO, "num_queues: 0x%08x",
 			internal->hw.blk_cfg->num_queues);
+
+		/* reset max_queue here, to minimum modification */
+		internal->max_queues = RTE_MIN(IFCVF_MAX_QUEUES,
+			internal->hw.blk_cfg->num_queues);
 	}
 
 	list->internal = internal;
Signature · 1 line
1.8.3.1
author
#258 lore raw

PATCH 11/15 vdpa: add config space change interrupt register and handle for virtio_blk

In reply to Andy Pei

Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to info qemu.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 113 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 113 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +113 -0
index f092aca..2552375 100644
@@ -53,7 +53,9 @@ struct ifcvf_internal {
 	int vfio_group_fd;
 	int vfio_dev_fd;
 	pthread_t tid;	/* thread for notify relay */
+	pthread_t intr_tid;	/* thread for intr relay */
 	int epfd;
+	int csc_fd;
 	int vid;
 	struct rte_vdpa_device *vdev;
 	uint16_t max_queues;
@@ -622,6 +624,108 @@ struct rte_vdpa_dev_info {
 	return 0;
 }
 
+static void
+virtio_interrupt_handler(struct ifcvf_internal *internal)
+{
+	int vid = internal->vid;
+	int ret;
+
+	ret = rte_vhost_slave_config_change(vid, 1);
+	if (ret)
+		DRV_LOG(ERR, "failed to notify the guest about configuration space change.");
+
+	return;
+}
+
+static void *
+intr_relay(void *arg)
+{
+	struct ifcvf_internal *internal = (struct ifcvf_internal *)arg;
+	struct epoll_event csc_event;
+	struct epoll_event ev;
+	uint64_t buf;
+	int nbytes;
+	int csc_fd, csc_val = 0;
+
+	csc_fd = epoll_create(1);
+	if (csc_fd < 0) {
+		DRV_LOG(ERR, "failed to create epoll for config space change.");
+		return NULL;
+	}
+
+	ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP;
+	ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle);
+	if (epoll_ctl(csc_fd, EPOLL_CTL_ADD,
+		rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) {
+		DRV_LOG(ERR, "epoll add error: %s", strerror(errno));
+		return NULL;
+	}
+
+	internal->csc_fd = csc_fd;
+
+	for (;;) {
+		csc_val = epoll_wait(csc_fd, &csc_event, 1, -1);
+		if (csc_val < 0) {
+			if (errno == EINTR)
+				continue;
+			DRV_LOG(ERR, "epoll_wait return fail\n");
+			return NULL;
+		} else if (csc_val == 0) {
+			continue;
+		} else {
+			/* csc_val > 0 */
+			nbytes = read(csc_event.data.fd, &buf, 8);
+			if (nbytes < 0) {
+				if (errno == EINTR || errno == EWOULDBLOCK)
+					continue;
+				DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n",
+					csc_event.data.fd,
+					strerror(errno));
+				return NULL;
+			} else if (nbytes == 0) {
+				DRV_LOG(ERR, "Read nothing from file descriptor %d\n",
+					csc_event.data.fd);
+				continue;
+			} else {
+				virtio_interrupt_handler(internal);
+			}
+		}
+	}
+	return NULL;
+}
+
+static int
+setup_intr_relay(struct ifcvf_internal *internal)
+{
+	int ret;
+
+	ret = pthread_create(&internal->intr_tid, NULL, intr_relay,
+			(void *)internal);
+	if (ret) {
+		DRV_LOG(ERR, "failed to create notify relay pthread.");
+		return -1;
+	}
+	return 0;
+}
+
+static int
+unset_intr_relay(struct ifcvf_internal *internal)
+{
+	void *status;
+
+	if (internal->intr_tid) {
+		pthread_cancel(internal->intr_tid);
+		pthread_join(internal->intr_tid, &status);
+	}
+	internal->intr_tid = 0;
+
+	if (internal->csc_fd >= 0)
+		close(internal->csc_fd);
+	internal->csc_fd = -1;
+
+	return 0;
+}
+
 static int
 update_datapath(struct ifcvf_internal *internal)
 {
@@ -648,10 +752,16 @@ struct rte_vdpa_dev_info {
 		if (ret)
 			goto err;
 
+		ret = setup_intr_relay(internal);
+		if (ret)
+			goto err;
+
 		rte_atomic32_set(&internal->running, 1);
 	} else if (rte_atomic32_read(&internal->running) &&
 		   (!rte_atomic32_read(&internal->started) ||
 		    !rte_atomic32_read(&internal->dev_attached))) {
+		ret = unset_intr_relay(internal);
+
 		ret = unset_notify_relay(internal);
 		if (ret)
 			goto err;
@@ -969,6 +1079,9 @@ struct rte_vdpa_dev_info {
 		vdpa_ifcvf_stop(internal);
 	else if (internal->device_type == IFCVF_BLK)
 		vdpa_ifcvf_blk_pause(internal);
+
+	unset_intr_relay(internal);
+
 	vdpa_disable_vfio_intr(internal);
 
 	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
Signature · 1 line
1.8.3.1
author
#259 lore raw

PATCH 12/15 vdpa/ifc: add is_blk flag to ifcvf_hw, and init is_blk during probe

In reply to Andy Pei
---
 drivers/vdpa/ifc/base/ifcvf.h | 1 +
 drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++
 2 files changed, 3 insertions(+)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
index 6dd7925..8e602af 100644
@@ -149,6 +149,7 @@ struct ifcvf_hw {
 	u8     *lm_cfg;
 	struct vring_info vring[IFCVF_MAX_QUEUES * 2];
 	u8 nr_vring;
+	u8 is_blk;
 	struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE];
 };
 
drivers/vdpa/ifc/ifcvf_vdpa.c +2 -0
index 2552375..546f9bd 100644
@@ -1621,11 +1621,13 @@ struct rte_vdpa_dev_info dev_info[] = {
 
 	if (device_id == VIRTIO_ID_NET) {
 		internal->device_type = IFCVF_NET;
+		internal->hw.is_blk = IFCVF_NET;
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_NET].features;
 	} else if (device_id == VIRTIO_ID_BLOCK) {
 		internal->device_type = IFCVF_BLK;
+		internal->hw.is_blk = IFCVF_BLK;
 		internal->features = features &
 					~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 		internal->features |= dev_info[IFCVF_BLK].features;
Signature · 1 line
1.8.3.1
author
#260 lore raw

PATCH 13/15 vdpa/ifc/base: for blk device, live migration register is different from net device

In reply to Andy Pei

1.last_avail_idx is lower 16 bit of the register.
2.address of ring_state register is different between net and blk device.

---
 drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++-------
 drivers/vdpa/ifc/base/ifcvf.h |  1 +
 2 files changed, 30 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +29 -7
index 3a69e53..a8a4728 100644
@@ -216,10 +216,18 @@
 				&cfg->queue_used_hi);
 		IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size);
 
-		*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) =
-			(u32)hw->vring[i].last_avail_idx |
-			((u32)hw->vring[i].last_used_idx << 16);
+		if (hw->is_blk == IFCVF_BLK) {
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				i * IFCVF_LM_CFG_SIZE) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
+		} else if (hw->is_blk == IFCVF_NET) {
+			*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
+				(i / 2) * IFCVF_LM_CFG_SIZE +
+				(i % 2) * 4) =
+				(u32)hw->vring[i].last_avail_idx |
+				((u32)hw->vring[i].last_used_idx << 16);
+		}
 
 		IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector);
 		if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -252,9 +260,23 @@
 		IFCVF_WRITE_REG16(i, &cfg->queue_select);
 		IFCVF_WRITE_REG16(0, &cfg->queue_enable);
 		IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector);
-		ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET +
-				(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4);
-		hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
+
+		if (hw->is_blk) {
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					i * IFCVF_LM_CFG_SIZE);
+		} else if (hw->is_blk == IFCVF_NET) {
+			ring_state = *(u32 *)(hw->lm_cfg +
+					IFCVF_LM_RING_STATE_OFFSET +
+					(i / 2) * IFCVF_LM_CFG_SIZE +
+					(i % 2) * 4);
+		}
+
+		if (hw->is_blk == IFCVF_BLK)
+			hw->vring[i].last_avail_idx =
+				(u16)(ring_state & IFCVF_16_BIT_MASK);
+		else if (hw->is_blk == IFCVF_NET)
+			hw->vring[i].last_avail_idx = (u16)(ring_state >> 16);
 		hw->vring[i].last_used_idx = (u16)(ring_state >> 16);
 	}
 }
drivers/vdpa/ifc/base/ifcvf.h +1 -0
index 8e602af..7367094 100644
@@ -65,6 +65,7 @@
 #define IFCVF_MEDIATED_VRING		0x200000000000
 
 #define IFCVF_32_BIT_MASK		0xffffffff
+#define IFCVF_16_BIT_MASK		0xffff
 
 #ifndef VHOST_USER_PROTOCOL_F_CONFIG
 #define VHOST_USER_PROTOCOL_F_CONFIG	9
Signature · 1 line
1.8.3.1
author
#261 lore raw

PATCH 14/15 vdpa/ifc: make sure hardware last_avail_idx and last_used_idx is the same when blk device pause

In reply to Andy Pei
---
 drivers/vdpa/ifc/base/ifcvf.c |  2 +-
 drivers/vdpa/ifc/base/ifcvf.h |  3 +++
 drivers/vdpa/ifc/ifcvf_vdpa.c | 32 +++++++++++++++++++++++---------
 3 files changed, 27 insertions(+), 10 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +1 -1
index a8a4728..7018048 100644
@@ -116,7 +116,7 @@
 	IFCVF_WRITE_REG8(status, &hw->common_cfg->device_status);
 }
 
-STATIC void
+void
 ifcvf_reset(struct ifcvf_hw *hw)
 {
 	ifcvf_set_status(hw, 0);
drivers/vdpa/ifc/base/ifcvf.h +3 -0
index 7367094..f22d18b 100644
@@ -157,6 +157,9 @@ struct ifcvf_hw {
 int
 ifcvf_init_hw(struct ifcvf_hw *hw, PCI_DEV *dev);
 
+void
+ifcvf_reset(struct ifcvf_hw *hw);
+
 u64
 ifcvf_get_features(struct ifcvf_hw *hw);
 
drivers/vdpa/ifc/ifcvf_vdpa.c +23 -9
index 546f9bd..ff233bc 100644
@@ -359,23 +359,32 @@ struct rte_vdpa_dev_info {
 vdpa_ifcvf_blk_pause(struct ifcvf_internal *internal)
 {
 	struct ifcvf_hw *hw = &internal->hw;
-	struct rte_vhost_vring vq;
 	int i, vid;
 	uint64_t features = 0;
 	uint64_t log_base = 0, log_size = 0;
 	uint64_t len;
+	u32 ring_state = 0;
 
 	vid = internal->vid;
 
 	if (internal->device_type == IFCVF_BLK) {
 		for (i = 0; i < hw->nr_vring; i++) {
-			rte_vhost_get_vhost_vring(internal->vid, i, &vq);
-			while (vq.avail->idx != vq.used->idx) {
-				ifcvf_notify_queue(hw, i);
-				usleep(10);
-			}
-			hw->vring[i].last_avail_idx = vq.avail->idx;
-			hw->vring[i].last_used_idx = vq.used->idx;
+			do {
+				if (hw->lm_cfg != NULL)
+					ring_state = *(u32 *)(hw->lm_cfg +
+						IFCVF_LM_RING_STATE_OFFSET +
+						i * IFCVF_LM_CFG_SIZE);
+				hw->vring[i].last_avail_idx =
+					(u16)(ring_state & IFCVF_16_BIT_MASK);
+				hw->vring[i].last_used_idx =
+					(u16)(ring_state >> 16);
+				if (hw->vring[i].last_avail_idx !=
+					hw->vring[i].last_used_idx) {
+					ifcvf_notify_queue(hw, i);
+					usleep(10);
+				}
+			} while (hw->vring[i].last_avail_idx !=
+				hw->vring[i].last_used_idx);
 		}
 	}
 
@@ -766,7 +775,12 @@ struct rte_vdpa_dev_info {
 		if (ret)
 			goto err;
 
-		vdpa_ifcvf_stop(internal);
+		if (internal->device_type == IFCVF_BLK) {
+			vdpa_ifcvf_blk_pause(internal);
+			ifcvf_reset(&internal->hw);
+		} else {
+			vdpa_ifcvf_stop(internal);
+		}
 
 		ret = vdpa_disable_vfio_intr(internal);
 		if (ret)
Signature · 1 line
1.8.3.1
author
#262 lore raw

PATCH 15/15 vhost: make sure each queue callfd is configured

In reply to Andy Pei

During the vhost data path building process, qemu will create a call fd at first, and create another call fd in the end. The final call fd will be used to relay notify. In the original code, after kick fd is set, dev_conf will set the first call fd. Even though the actual call fd will set, the data path will not work correctly.

---
 lib/vhost/vhost_user.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
lib/vhost/vhost_user.c +15 -0
index 5eb1dd6..0be879a 100644
@@ -3137,12 +3137,27 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev,
 	if (!vdpa_dev)
 		goto out;
 
+	if (request != VHOST_USER_SET_VRING_CALL)
+		goto out;
+
 	if (!(dev->flags & VIRTIO_DEV_VDPA_CONFIGURED)) {
 		if (vdpa_dev->ops->dev_conf(dev->vid))
 			VHOST_LOG_CONFIG(ERR,
 					 "Failed to configure vDPA device\n");
 		else
 			dev->flags |= VIRTIO_DEV_VDPA_CONFIGURED;
+	} else {
+		/**
+		** when VIRTIO_DEV_VDPA_CONFIGURED already configured
+		** close the device and config the device again,
+		** make sure the call fd of each queue is configed correctly.
+		**/
+		if (vdpa_dev->ops->dev_close(dev->vid))
+			VHOST_LOG_CONFIG(ERR,
+					 "Failed to close vDPA device\n");
+		if (vdpa_dev->ops->dev_conf(dev->vid))
+			VHOST_LOG_CONFIG(ERR,
+					 "Failed to re-config vDPA device\n");
 	}
 
 out:
Signature · 1 line
1.8.3.1