add virtio_blk device support to vdpa/ifc
Subsystems (2)
RE: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc driver
In reply to Xia, ChenboQuoted 34 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:22 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc driverQuoted 7 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifcdriver
Review
Quoted 9 lines
Add get device type ops to ifc driver.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 10 ++++++++++ 1 file changed, 10 insertions(+)(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cQuoted 7 lines
index 8a260b7..99a6ab0 100644
drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0@@ -1300,6 +1300,15 @@ struct rte_vdpa_dev_info {return 0 ; }
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1309
Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7
-0+static int +ifcvf_blk_get_device_type (struct rte_vdpa_device *vdev, +uint32_t *type) +{ +RTE_SET_USED (vdev); + *type =VDPA_DEVICE_TYPE_BLK ; +return 0 ;This is not right. Remember net and blk are both using the driver? This will lead to using net also returns BLK.
Correct a bit for above: will lead to net not working. So implement callback for both (although in the API, NULL callback can return NET type)
Thanks,
Chenbo
Quoted 23 lines
And I suggest the patch-set validated with both blk and net...
Besides,
ifcvf_blk_get_device_typeshould beifcvf_get_device_typeThanks,
ChenboQuoted 15 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +3
-1+} +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features,@@ -1313,6 +1322,7 @@ struct rte_vdpa_dev_info { .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_blk_get_device_type, };struct rte_vdpa_dev_info dev_info[] = { --1.8.3.1
RE: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc driver
HI Chenbo,
Thanks for your reply.
I will implement a function "ifcvf_get_device_type" for both ifc net and blk ops. "ifcvf_get_device_type" will return device type according to internal->device_type.
Quoted 85 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:41 PM
To: Xia, Chenbo <[email protected]>; Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc driverQuoted 35 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:22 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc driverQuoted 8 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifcdriver
Review
Quoted 9 lines
Add get device type ops to ifc driver.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 10 ++++++++++ 1 file changed, 10 insertions(+)(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cQuoted 7 lines
index 8a260b7..99a6ab0 100644
drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0@@ -1300,6 +1300,15 @@ struct rte_vdpa_dev_info {return 0 ; }Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1309
Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7
-0+static int +ifcvf_blk_get_device_type (struct rte_vdpa_device *vdev, +uint32_t *type) +{ +RTE_SET_USED (vdev); + *type =VDPA_DEVICE_TYPE_BLK ; +return 0 ;This is not right. Remember net and blk are both using the driver? This will lead to using net also returns BLK.
Correct a bit for above: will lead to net not working. So implement callback for both (although in the API, NULL callback can return NET type)
Thanks,
ChenboQuoted 23 lines
And I suggest the patch-set validated with both blk and net...
Besides,
ifcvf_blk_get_device_typeshould beifcvf_get_device_typeThanks,
ChenboQuoted 15 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +2
-0+} +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features, @@ -1313,6 +1322,7 @@struct
rte_vdpa_dev_info{drivers/vdpa/ifc/ifcvf_vdpa.c +1
-1.get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_blk_get_device_type, };struct rte_vdpa_dev_info dev_info[] = { --1.8.3.1
PATCH v7 9/18 examples/vdpa: add vDPA blk support in example
In reply to Andy PeiAdd virtio blk device support to vDPA example.
--- examples/vdpa/main.c | 57 +++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 65 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c +57 -0
@@ -20,6 +20,7 @@#include <cmdline_parse_string.h> #include <cmdline_parse_num.h> #include <cmdline.h> +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128#define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,54 @@ struct vdpa_port { };static int +vdpa_blk_device_set_features_and_protocol (const char *path) +{ +uint64_t protocol_features =0 ; +int ret; + + ret =rte_vhost_driver_set_features (path,VHOST_BLK_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_disable_features (path, +VHOST_BLK_DISABLED_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_disable_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_get_protocol_features (path, &protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_get_protocol_features for %s failed.\n" , + path); + goto out; + } + + protocol_features |=VHOST_BLK_PROTOCOL_FEATURES ; + + ret =rte_vhost_driver_set_protocol_features (path, protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_protocol_features for %s failed.\n" , + path); + goto out; + } + +out: +return ret; +} + +static int start_vdpa (struct vdpa_port *vport) { +uint32_t device_type =0 ;int ret;char *socket_path = vport->ifname;
@@ -192,6 +239,16 @@ struct vdpa_port {"attach vdpa device failed: %s\n" , socket_path); + ret =rte_vhost_driver_get_vdpa_dev_type (socket_path, &device_type); +if (ret ==0 && device_type ==VDPA_DEVICE_TYPE_BLK ) { +RTE_LOG (NOTICE ,VDPA ,"is a blk device\n" ); + ret =vdpa_blk_device_set_features_and_protocol (socket_path); +if (ret !=0 ) +rte_exit (EXIT_FAILURE , +"set vhost blk driver features and protocol features failed: %s\n" , + socket_path); + } +if (rte_vhost_driver_start (socket_path) <0 ) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n",
examples/vdpa/vdpa_blk_compact.h new file +65 -0
65 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RE: [PATCH v7 09/18] examples/vdpa: add vDPA blk support in example
Quoted 7 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 09/18] examples/vdpa: add vDPA blk support in example
Better be: examples/vdpa: add virtio blk support
Quoted 59 lines
Add virtio blk device support to vDPA example.
Signed-off-by Andy Pei <[email protected]>--- examples/vdpa/main.c | 57 +++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 65 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.hexamples/vdpa/main.c +31
-0@@ -20,6 +20,7 @@#include <cmdline_parse_string.h> #include <cmdline_parse_num.h> #include <cmdline.h> +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128#define MAX_VDPA_SAMPLE_PORTS 1024@@ -159,8 +160,54 @@ struct vdpa_port { };static int +vdpa_blk_device_set_features_and_protocol (const char *path) +{ +uint64_t protocol_features =0 ; +int ret; + + ret =rte_vhost_driver_set_features (path,VHOST_BLK_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_disable_features (path, +VHOST_BLK_DISABLED_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_disable_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_get_protocol_features (path, &protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , + "rte_vhost_driver_get_protocol_features for %s failed.\n", + path); + goto out; + } +
examples/vdpa/main.c struct vdpa_port { · line 204
Quoted 10 lines
examples/vdpa/main.c +8
-0+ protocol_features |=VHOST_BLK_PROTOCOL_FEATURES ; + + ret =rte_vhost_driver_set_protocol_features (path, protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , + "rte_vhost_driver_set_protocol_features for %s failed.\n", + path); + goto out;
This goto is not needed.
Quoted 14 lines
examples/vdpa/main.c +8
-0+ } + +out: +return ret; +} + +static int start_vdpa (struct vdpa_port *vport) { +uint32_t device_type =0 ;int ret;char *socket_path = vport->ifname;
examples/vdpa/main.c struct vdpa_port { · line 244
Quoted 6 lines
examples/vdpa/main.c +3
-0@@ -192,6 +239,16 @@ struct vdpa_port {"attach vdpa device failed: %s\n" , socket_path); + ret =rte_vhost_driver_get_vdpa_dev_type (socket_path, &device_type); +if (ret ==0 && device_type ==VDPA_DEVICE_TYPE_BLK ) { +RTE_LOG (NOTICE ,VDPA ,"is a blk device\n" );
Should add info of socket path
Quoted 24 lines
examples/vdpa/main.c +7
-0+ ret =vdpa_blk_device_set_features_and_protocol (socket_path); +if (ret !=0 ) +rte_exit (EXIT_FAILURE , + "set vhost blk driver features and protocol features failed: %s\n", + socket_path); + } +if (rte_vhost_driver_start (socket_path) <0 ) rte_exit(EXIT_FAILURE,examples/vdpa/main.c +0
-0"start vhost driver failed: %s\n",examples/vdpa/vdpa_blk_compact.h new file +6
-0@@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2022 Intel Corporation +*/ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_
examples/vdpa/vdpa_blk_compact.h line 16
Quoted 10 lines
examples/vdpa/vdpa_blk_compact.h new file +10
-0+ +/** +* @file +* +* Device specific vhost lib +*/ + +#include <stdbool.h> + +#include <rte_pci.h>
above two headers are not used in this file?
Quoted 21 lines
examples/vdpa/vdpa_blk_compact.h new file +4
-0+#include <rte_vhost.h> + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1/* Indicates maximum segment size */
+#defineVIRTIO_BLK_F_SEG_MAX2 /* Indicates maximum # of segments
*/examples/vdpa/vdpa_blk_compact.h new file +2
-0+#define VIRTIO_BLK_F_GEOMETRY 4/* Legacy geometry available */ +#define VIRTIO_BLK_F_BLK_SIZE 6/* Block size of disk is available */
+#defineVIRTIO_BLK_F_TOPOLOGY10 /* Topology information is
available */examples/vdpa/vdpa_blk_compact.h new file +6
-0+#define VIRTIO_BLK_F_MQ 12/* support more than one vq */ + +/* Legacy feature bits */ +#ifndef VIRTIO_BLK_NO_LEGACY +#define VIRTIO_BLK_F_BARRIER 0/* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7/* Supports scsi command passthru */
+#defineVIRTIO_BLK_F_CONFIG_WCE11 /* Writeback mode available in
config */
examples/vdpa/vdpa_blk_compact.h line 36
Quoted 5 lines
examples/vdpa/vdpa_blk_compact.h new file +5
-0+#endif /* !VIRTIO_BLK_NO_LEGACY */ + +#ifndef VHOST_USER_F_PROTOCOL_FEATURES +#define VHOST_USER_F_PROTOCOL_FEATURES 30 +#endif
It's already in rte_vhost.h, so no need to re-define.
Thanks,
Chenbo
Quoted 35 lines
examples/vdpa/vdpa_blk_compact.h new file +9
-0+ +#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1)) + +#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL <<
VIRTIO_F_NOTIFY_ON_EMPTY) | \examples/vdpa/vdpa_blk_compact.h new file +5
-0+ (1ULL << VIRTIO_RING_F_EVENT_IDX)) + +#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) |\
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER) |\
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL <<VIRTIO_BLK_F_SCSI ) | (1ULL <<VIRTIO_BLK_F_CONFIG_WCE )| \
examples/vdpa/vdpa_blk_compact.h new file +13
-1+ (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define VHOST_BLK_DISABLED_FEATURES (VHOST_BLK_DISABLED_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE)) + +/* Vhost-blk support protocol features */ +#define VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG)) + +#endif /* _VDPA_BLK_COMPACT_H_ */ --1.8.3.1
RE: [PATCH v7 09/18] examples/vdpa: add vDPA blk support in example
HI Chenbo,
Thanks for your reply.
My reply is inline.
Quoted 19 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:34 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 09/18] examples/vdpa: add vDPA blk support in exampleQuoted 9 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 09/18] examples/vdpa: add vDPA blk support in example
Quoted 2 lines
Better be: examples/vdpa: add virtio blk support
Sure.
Quoted 62 lines
Quoted 6 lines
Add virtio blk device support to vDPA example.
Signed-off-by Andy Pei <[email protected]>--- examples/vdpa/main.c | 57+++++++++++++++++++++++++++++++++++
Quoted 55 lines
examples/vdpa/vdpa_blk_compact.h | 65 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.hexamples/vdpa/main.c index renamed from examples/vdpa/main.c +0
-05ab0765..2544141 100644
examples/vdpa/main.c +33
-0@@ -20,6 +20,7 @@#include <cmdline_parse_string.h> #include <cmdline_parse_num.h> #include <cmdline.h> +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128#define MAX_VDPA_SAMPLE_PORTS 1024@@ -159,8 +160,54 @@ struct vdpa_port { };static int +vdpa_blk_device_set_features_and_protocol (const char *path) { +uint64_t protocol_features =0 ; +int ret; + + ret =rte_vhost_driver_set_features (path,VHOST_BLK_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_disable_features (path, +VHOST_BLK_DISABLED_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_disable_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_get_protocol_features (path, &protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , + "rte_vhost_driver_get_protocol_features for %s failed.\n", + path); + goto out; + } + + protocol_features |=VHOST_BLK_PROTOCOL_FEATURES ; + + ret =rte_vhost_driver_set_protocol_features (path, protocol_features);
Quoted 9 lines
examples/vdpa/main.c struct vdpa_port { · line 173
Quoted 6 lines
examples/vdpa/main.c +5
-0+if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , + "rte_vhost_driver_set_protocol_features for %s failed.\n", + path); + goto out;This goto is not needed.
Yes, you are right. I will fix it.
Quoted 17 lines
Quoted 17 lines
examples/vdpa/main.c +8
-0+ } + +out: +return ret; +} + +static int start_vdpa (struct vdpa_port *vport) { +uint32_t device_type =0 ;int ret;char *socket_path = vport->ifname;@@ -192,6 +239,16 @@ struct vdpa_port {"attach vdpa device failed: %s\n" , socket_path);
Quoted 7 lines
examples/vdpa/main.c struct vdpa_port { · line 242
Quoted 1 line
examples/vdpa/main.c +1
-0+ ret = rte_vhost_driver_get_vdpa_dev_type(socket_path,
&device_type);examples/vdpa/main.c struct vdpa_port { · line 244
Quoted 2 lines
examples/vdpa/main.c +2
-0+if (ret ==0 && device_type ==VDPA_DEVICE_TYPE_BLK ) { +RTE_LOG (NOTICE ,VDPA ,"is a blk device\n" );Should add info of socket path
Yes, you are right. I will fix it.
Quoted 28 lines
examples/vdpa/main.c struct vdpa_port { · line 245
Quoted 1 line
examples/vdpa/main.c +1
-0+ ret =
vdpa_blk_device_set_features_and_protocol(socket_path);Quoted 26 lines
examples/vdpa/main.c +6
-0+if (ret !=0 ) +rte_exit (EXIT_FAILURE , + "set vhost blk driver features and protocol features failed: %s\n", + socket_path); + } +if (rte_vhost_driver_start (socket_path) <0 ) rte_exit(EXIT_FAILURE,"start vhost driver failed: %s\n", diff --git
a/examples/vdpa/vdpa_blk_compact.hb/examples/vdpa/vdpa_blk_compact.h
new file mode 100644
index 0000000..136c3f6examples/vdpa/vdpa_blk_compact.h new file +9
-0@@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2022 Intel Corporation */ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_ + +/** +* @file +*
Quoted 9 lines
examples/vdpa/vdpa_blk_compact.h line 16
Quoted 6 lines
examples/vdpa/vdpa_blk_compact.h new file +6
-0+* Device specific vhost lib +*/ + +#include <stdbool.h> + +#include <rte_pci.h> above two headers are not used in this file?
I will remove these two include.
Quoted 30 lines
examples/vdpa/vdpa_blk_compact.h
Quoted 4 lines
examples/vdpa/vdpa_blk_compact.h new file +4
-0+#include <rte_vhost.h> + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1/* Indicates maximum segment size
Quoted 2 lines
*/
+#defineVIRTIO_BLK_F_SEG_MAX2 /* Indicates maximum # ofsegments
Quoted 15 lines
*/
examples/vdpa/vdpa_blk_compact.h new file +2
-0+#define VIRTIO_BLK_F_GEOMETRY 4/* Legacy geometry available */ +#define VIRTIO_BLK_F_BLK_SIZE 6/* Block size of disk is available */
+#defineVIRTIO_BLK_F_TOPOLOGY10 /* Topology information is
available */examples/vdpa/vdpa_blk_compact.h new file +6
-0+#define VIRTIO_BLK_F_MQ 12/* support more than one vq */ + +/* Legacy feature bits */ +#ifndef VIRTIO_BLK_NO_LEGACY +#define VIRTIO_BLK_F_BARRIER 0/* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7/* Supports scsi command passthru */
+#defineVIRTIO_BLK_F_CONFIG_WCE11 /* Writeback mode available in
config */examples/vdpa/vdpa_blk_compact.h
Quoted 4 lines
examples/vdpa/vdpa_blk_compact.h new file +4
-0+#endif /* !VIRTIO_BLK_NO_LEGACY */ + +#ifndef VHOST_USER_F_PROTOCOL_FEATURES #define +VHOST_USER_F_PROTOCOL_FEATURES 30 #endifIt's already in
rte_vhost.h, so no need to re-define.
Quoted 3 lines
Thanks,
Chenbo
Yes, you are right. I will fix it.
Quoted 43 lines
Quoted 10 lines
examples/vdpa/vdpa_blk_compact.h new file +9
-0+ +#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1)) + +#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL <<
VIRTIO_F_NOTIFY_ON_EMPTY) | \examples/vdpa/vdpa_blk_compact.h
Quoted 4 lines
examples/vdpa/vdpa_blk_compact.h new file +4
-0+ (1ULL << VIRTIO_RING_F_EVENT_IDX)) + +#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL <<
VIRTIO_BLK_F_SEG_MAX) | \examples/vdpa/vdpa_blk_compact.h
Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL <<
VIRTIO_BLK_F_BLK_SIZE) |examples/vdpa/vdpa_blk_compact.h
Quoted 2 lines
\
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL <<
VIRTIO_BLK_F_BARRIER) |examples/vdpa/vdpa_blk_compact.h
Quoted 2 lines
\
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL <<
VIRTIO_BLK_F_CONFIG_WCE)examples/vdpa/vdpa_blk_compact.h
Quoted 5 lines
| \
examples/vdpa/vdpa_blk_compact.h new file +4
-0+ (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define VHOST_BLK_DISABLED_FEATURES (
VHOST_BLK_DISABLED_FEATURES_BASE| \examples/vdpa/vdpa_blk_compact.h
Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL <<
VIRTIO_BLK_F_BARRIER) | \examples/vdpa/vdpa_blk_compact.h
Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL <<
VIRTIO_BLK_F_CONFIG_WCE))Quoted 9 lines
examples/vdpa/vdpa_blk_compact.h new file +7
-1+ +/* Vhost-blk support protocol features */ #define +VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG)) + +#endif /* _VDPA_BLK_COMPACT_H_ */ --1.8.3.1
PATCH v7 10/18 usertools: add support for virtio blk device
In reply to Andy PeiAdd virtio blk device support to devbind.
--- usertools/dpdk-devbind.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-devbind.py +4 -1
@@ -72,6 +72,9 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class]
@@ -82,7 +85,7 @@ compress_devices = [cavium_zip] regex_devices = [cn9k_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, - intel_ntb_skx, intel_ntb_icx] + intel_ntb_skx, intel_ntb_icx, virtio_blk] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
Signature · 1 line
1.8.3.1
PATCH v7 11/18 vdpa/ifc: add set vring state for blk device
In reply to Andy PeiSet_vring_state op is mandatory, add set_vring_state for blk device. Currently set_vring_state for blk device is not implemented.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +11 -1
@@ -1290,6 +1290,16 @@ struct rte_vdpa_dev_info { }static int +ifcvf_blk_set_vring_state (int vid,int vring,int state) +{ +RTE_SET_USED (vid); +RTE_SET_USED (vring); +RTE_SET_USED (state); + +return 0 ; +} + +static int ifcvf_blk_get_protocol_features (struct rte_vdpa_device *vdev,uint64_t *features) {
@@ -1316,7 +1326,7 @@ struct rte_vdpa_dev_info { .get_protocol_features = ifcvf_blk_get_protocol_features, .dev_conf = ifcvf_dev_config, .dev_close = ifcvf_dev_close, - .set_vring_state = NULL, + .set_vring_state = ifcvf_blk_set_vring_state, .migration_done = NULL, .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd,
Signature · 1 line
1.8.3.1
RE: [PATCH v7 11/18] vdpa/ifc: add set vring state for blk device
Quoted 38 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 11/18] vdpa/ifc: add set vring state for blk device
Set_vring_stateop is mandatory, addset_vring_statefor blk device. Currentlyset_vring_statefor blk device is not implemented.Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)drivers/vdpa/ifc/ifcvf_vdpa.c +10
-0@@ -1290,6 +1290,16 @@ struct rte_vdpa_dev_info { }static int +ifcvf_blk_set_vring_state (int vid,int vring,int state) +{ +RTE_SET_USED (vid); +RTE_SET_USED (vring); +RTE_SET_USED (state); + +return 0 ; +} + +static int ifcvf_blk_get_protocol_features (struct rte_vdpa_device *vdev,uint64_t *features) {
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1329
Quoted 5 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1
-1@@ -1316,7 +1326,7 @@ struct rte_vdpa_dev_info { .get_protocol_features = ifcvf_blk_get_protocol_features, .dev_conf = ifcvf_dev_config, .dev_close = ifcvf_dev_close, - .set_vring_state = NULL, + .set_vring_state = ifcvf_blk_set_vring_state,
About naming in this driver, if you use two functions for net/blk and name the one for blk as XXX_blk_XXX. Better rename the net one as XXX_net_XXX.
In this case, better to rename ifcvf_set_vring_state
Thanks,
Chenbo
Quoted 5 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0
-1.migration_done = NULL, .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, --1.8.3.1
PATCH v7 12/18 vdpa/ifc: add some log at vDPA launch before qemu connect
In reply to Andy PeiAdd some log of virtio blk device config space information at VDPA launch before qemu connects.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +28 -0
@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +uint64_t capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;
@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_BLK ].features; + +/* cannot read 64-bit register in one attempt, +* so read byte by byte. +*/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +DRV_LOG (INFO ,"capacity : %" PRIu64"G" , capacity >>21 ); + +DRV_LOG (INFO ,"size_max : 0x%08x" , + internal->hw.blk_cfg->size_max); +DRV_LOG (INFO ,"seg_max : 0x%08x" , + internal->hw.blk_cfg->seg_max); +DRV_LOG (INFO ,"blk_size : 0x%08x" , + internal->hw.blk_cfg->blk_size); +DRV_LOG (INFO ,"geometry" ); +DRV_LOG (INFO ," cylinders: %u" , + internal->hw.blk_cfg->geometry.cylinders); +DRV_LOG (INFO ," heads : %u" , + internal->hw.blk_cfg->geometry.heads); +DRV_LOG (INFO ," sectors : %u" , + internal->hw.blk_cfg->geometry.sectors); +DRV_LOG (INFO ,"num_queues: 0x%08x" , + internal->hw.blk_cfg->num_queues); } list->internal = internal;
Signature · 1 line
1.8.3.1
RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect
Quoted 8 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect
Title can be:
vdpa/ifc: add log for config space of virtio blk
Quoted 26 lines
Add some log of virtio blk device config space information at VDPA launch before qemu connects.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)drivers/vdpa/ifc/ifcvf_vdpa.c +3
-0@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +uint64_t capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM );
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1443
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9
-0internal->features |= dev_info[IFCVF_BLK ].features; + +/* cannot read 64-bit register in one attempt, +* so read byte by byte. +*/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +DRV_LOG (INFO ,"capacity : %" PRIu64"G" , capacity >>21 );
I believe this '21' should be calculated rather than hard-code.
And should these log all be debug log?
Thanks,
Chenbo
Quoted 21 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +16
-1+ +DRV_LOG (INFO ,"size_max : 0x%08x" , + internal->hw.blk_cfg->size_max); +DRV_LOG (INFO ,"seg_max : 0x%08x" , + internal->hw.blk_cfg->seg_max); +DRV_LOG (INFO ,"blk_size : 0x%08x" , + internal->hw.blk_cfg->blk_size); +DRV_LOG (INFO ,"geometry" ); +DRV_LOG (INFO ," cylinders: %u" , + internal->hw.blk_cfg->geometry.cylinders); +DRV_LOG (INFO ," heads : %u" , + internal->hw.blk_cfg->geometry.heads); +DRV_LOG (INFO ," sectors : %u" , + internal->hw.blk_cfg->geometry.sectors); +DRV_LOG (INFO ,"num_queues: 0x%08x" , + internal->hw.blk_cfg->num_queues); } list->internal = internal; --1.8.3.1
RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect
Hi Chenbo,
Thanks for your reply.
My reply is inline.
Quoted 19 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:53 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connectQuoted 9 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect
Quoted 4 lines
Title can be:
vdpa/ifc: add log for config space of virtio blk
Sure.
Quoted 27 lines
Quoted 24 lines
Add some log of virtio blk device config space information at VDPA launch before qemu connects.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex ca49bc3..4060a44 100644drivers/vdpa/ifc/ifcvf_vdpa.c +3
-0@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +uint64_t capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1433
Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<
VIRTIO_F_IOMMU_PLATFORM);
Quoted 13 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1443
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9
-0internal->features |= dev_info[IFCVF_BLK ].features; + +/* cannot read 64-bit register in one attempt, +* so read byte by byte. +*/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +DRV_LOG (INFO ,"capacity : %" PRIu64"G" , capacity >>21 );I believe this '21' should be calculated rather than hard-code.
/* The capacity (in 512-byte sectors). */
So right shift 1 we gwt in K. Another right shift 10 we get in M, right shift 10 more bits, we get in G. 10 + 10 + 1 = 21.
I think add some comments in code is fine, what do you think?
Quoted 2 lines
And should these log all be debug log?
These information is hardware ability. I think INFO is fine.
Quoted 25 lines
Thanks,
ChenboQuoted 21 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +16
-1+ +DRV_LOG (INFO ,"size_max : 0x%08x" , + internal->hw.blk_cfg->size_max); +DRV_LOG (INFO ,"seg_max : 0x%08x" , + internal->hw.blk_cfg->seg_max); +DRV_LOG (INFO ,"blk_size : 0x%08x" , + internal->hw.blk_cfg->blk_size); +DRV_LOG (INFO ,"geometry" ); +DRV_LOG (INFO ," cylinders: %u" , + internal->hw.blk_cfg->geometry.cylinders); +DRV_LOG (INFO ," heads : %u" , + internal->hw.blk_cfg->geometry.heads); +DRV_LOG (INFO ," sectors : %u" , + internal->hw.blk_cfg->geometry.sectors); +DRV_LOG (INFO ,"num_queues: 0x%08x" , + internal->hw.blk_cfg->num_queues); } list->internal = internal; --1.8.3.1
RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect
Quoted 79 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Friday, May 13, 2022 4:35 PM
To: Xia, Chenbo <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connectHi Chenbo,
Thanks for your reply.
My reply is inline.Quoted 7 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:53 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launchbefore
Quoted 12 lines
qemu connect
Quoted 9 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connectQuoted 4 lines
Title can be:
vdpa/ifc: add log for config space of virtio blk
Sure.
Quoted 27 lines
Quoted 24 lines
Add some log of virtio blk device config space information at VDPA launch before qemu connects.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex ca49bc3..4060a44 100644drivers/vdpa/ifc/ifcvf_vdpa.c +3
-0@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +uint64_t capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1433
Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<
VIRTIO_F_IOMMU_PLATFORM);Quoted 13 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1443
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9
-0internal->features |= dev_info[IFCVF_BLK ].features; + +/* cannot read 64-bit register in one attempt, +* so read byte by byte. +*/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { +byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; +capacity |= (uint64_t )*byte << (i *8 ); +} +DRV_LOG (INFO ,"capacity : %" PRIu64"G" , capacity >>21 );I believe this '21' should be calculated rather than hard-code.
Quoted 5 lines
/* The capacity (in 512-byte sectors). */
So right shift 1 we gwt in K. Another right shift 10 we get in M, right shift 10 more bits, we get in G.
10 + 10 + 1 = 21.
I think add some comments in code is fine, what do you think?
Ok. Work for me.
Thanks,
Chenbo
Quoted 31 lines
Quoted 3 lines
And should these log all be debug log?
These information is hardware ability. I think INFO is fine.
Quoted 25 lines
Thanks,
ChenboQuoted 21 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +16
-1+ +DRV_LOG (INFO ,"size_max : 0x%08x" , +internal->hw.blk_cfg->size_max); +DRV_LOG (INFO ,"seg_max : 0x%08x" , +internal->hw.blk_cfg->seg_max); +DRV_LOG (INFO ,"blk_size : 0x%08x" , +internal->hw.blk_cfg->blk_size); +DRV_LOG (INFO ,"geometry" ); +DRV_LOG (INFO ," cylinders: %u" , +internal->hw.blk_cfg->geometry.cylinders); +DRV_LOG (INFO ," heads : %u" , +internal->hw.blk_cfg->geometry.heads); +DRV_LOG (INFO ," sectors : %u" , +internal->hw.blk_cfg->geometry.sectors); +DRV_LOG (INFO ,"num_queues: 0x%08x" , +internal->hw.blk_cfg->num_queues); } list->internal = internal; --1.8.3.1
Re: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect
In reply to Pei, AndyQuoted 24 lines · Andy
Hi Chenbo,
Thanks for your reply.
My reply is inline.Quoted 19 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:53 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connectQuoted 9 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect
Quoted 44 lines
Quoted 4 lines
Title can be:
vdpa/ifc: add log for config space of virtio blk
Sure.
Quoted 39 lines
Quoted 24 lines
Add some log of virtio blk device config space information at VDPA launch before qemu connects.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex ca49bc3..4060a44 100644drivers/vdpa/ifc/ifcvf_vdpa.c +3
-0@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +uint64_t capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1433
Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<
VIRTIO_F_IOMMU_PLATFORM);drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1443
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9
-0internal->features |= dev_info[IFCVF_BLK ].features; + +/* cannot read 64-bit register in one attempt, +* so read byte by byte. +*/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +DRV_LOG (INFO ,"capacity : %" PRIu64"G" , capacity >>21 );I believe this '21' should be calculated rather than hard-code.
Sounds like a debug not info message.
It is meant for developer not end user.
RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect
Hi Stephen,
Thanks for your reply.
I will change to DEBUG level log in next version.
Quoted 84 lines
-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Friday, May 13, 2022 11:38 PM
To: Pei, Andy <[email protected]>
Cc: Xia, Chenbo <[email protected]>; [email protected]; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: Re: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connectQuoted 24 lines · Andy
Hi Chenbo,
Thanks for your reply.
My reply is inline.Quoted 19 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:53 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connectQuoted 9 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connectQuoted 44 lines
Quoted 4 lines
Title can be:
vdpa/ifc: add log for config space of virtio blk
Sure.
Quoted 39 lines
Quoted 24 lines
Add some log of virtio blk device config space information at VDPA launch before qemu connects.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex ca49bc3..4060a44 100644drivers/vdpa/ifc/ifcvf_vdpa.c +3
-0@@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +uint64_t capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1433
Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0@@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<
VIRTIO_F_IOMMU_PLATFORM);drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1443
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9
-0internal->features |= dev_info[IFCVF_BLK ].features; + +/* cannot read 64-bit register in one attempt, +* so read byte by byte. +*/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +DRV_LOG (INFO ,"capacity : %" PRIu64"G" , capacity >>21 );I believe this '21' should be calculated rather than hard-code.
Sounds like a debug not info message.
It is meant for developer not end user.
PATCH v7 13/18 vdpa/ifc: read virtio max queues from hardware
In reply to Andy PeiOriginal code max_queues is set to IFCVF_MAX_QUEUES.
New code max_queues is the min of IFCVF_MAX_QUEUES and hardware num_queues.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
@@ -1457,6 +1457,10 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors);DRV_LOG (INFO ,"num_queues: 0x%08x" , internal->hw.blk_cfg->num_queues); + +/* reset max_queue here, to minimum modification */ + internal->max_queues =RTE_MIN (IFCVF_MAX_QUEUES , + internal->hw.blk_cfg->num_queues); } list->internal = internal;
Signature · 1 line
1.8.3.1
RE: [PATCH v7 13/18] vdpa/ifc: read virtio max queues from hardware
Quoted 21 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 13/18] vdpa/ifc: read virtio max queues from hardwareOriginal code
max_queuesis set toIFCVF_MAX_QUEUES.
New codemax_queuesis the min ofIFCVF_MAX_QUEUESand hardwarenum_queues.Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1463
Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4
-0@@ -1457,6 +1457,10 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors);DRV_LOG (INFO ,"num_queues: 0x%08x" , internal->hw.blk_cfg->num_queues); + +/* reset max_queue here, to minimum modification */ + internal->max_queues =RTE_MIN (IFCVF_MAX_QUEUES , + internal->hw.blk_cfg->num_queues);
MQ is not supported now in this driver, should we make this change later?
Thanks,
Chenbo
Quoted 5 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0
-1} list->internal = internal; --1.8.3.1
RE: [PATCH v7 13/18] vdpa/ifc: read virtio max queues from hardware
Hi Chenbo,
Thanks for your reply.
This patch will be omitted in next version.
Quoted 52 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Thursday, May 12, 2022 9:55 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 13/18] vdpa/ifc: read virtio max queues from hardwareQuoted 10 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 13/18] vdpa/ifc: read virtio max queues from hardwareQuoted 2 lines
Original code
max_queuesis set toIFCVF_MAX_QUEUES. New codemax_queuesis the min ofIFCVF_MAX_QUEUESand hardware
num_queues.Quoted 11 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++ 1 file changed, 4 insertions(+)(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex 4060a44..5a8cf1c 100644drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1463
Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4
-0@@ -1457,6 +1457,10 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors);DRV_LOG (INFO ,"num_queues: 0x%08x" , internal->hw.blk_cfg->num_queues); + +/* reset max_queue here, to minimum modification */ + internal->max_queues =RTE_MIN (IFCVF_MAX_QUEUES , + internal->hw.blk_cfg->num_queues);MQ is not supported now in this driver, should we make this change later?
Thanks,
ChenboQuoted 5 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0
-1} list->internal = internal; --1.8.3.1
PATCH v7 14/18 vdpa/ifc: add interrupt and handle for virtio blk
In reply to Andy PeiCreate a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to info qemu.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 112 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +112 -0
@@ -53,7 +53,9 @@ struct ifcvf_internal {int vfio_group_fd;int vfio_dev_fd;pthread_t tid;/* thread for notify relay */ +pthread_t intr_tid;/* thread for intr relay */ int epfd; +int csc_fd;int vid;struct rte_vdpa_device *vdev;uint16_t max_queues;
@@ -558,6 +560,107 @@ struct rte_vdpa_dev_info {return 0 ; } +static void +virtio_interrupt_handler (struct ifcvf_internal *internal) +{ +int vid = internal->vid; +int ret; + + ret =rte_vhost_slave_config_change (vid,1 ); +if (ret) +DRV_LOG (ERR ,"failed to notify the guest about configuration space change." ); +} + +static void * +intr_relay (void *arg) +{ +struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; +struct epoll_event csc_event; +struct epoll_event ev; +uint64_t buf; +int nbytes; +int csc_fd, csc_val =0 ; + + csc_fd =epoll_create (1 ); +if (csc_fd <0 ) { +DRV_LOG (ERR ,"failed to create epoll for config space change." ); +return NULL ; + } + + ev.events =EPOLLIN |EPOLLPRI |EPOLLRDHUP |EPOLLHUP ; + ev.data.fd =rte_intr_fd_get (internal->pdev->intr_handle); +if (epoll_ctl (csc_fd,EPOLL_CTL_ADD , +rte_intr_fd_get (internal->pdev->intr_handle), &ev) <0 ) { +DRV_LOG (ERR ,"epoll add error: %s" ,strerror (errno)); +return NULL ; + } + + internal->csc_fd = csc_fd; + +for (;;) { + csc_val =epoll_wait (csc_fd, &csc_event,1 ,-1 ); +if (csc_val <0 ) { +if (errno ==EINTR ) +continue ; +DRV_LOG (ERR ,"epoll_wait return fail\n" ); +return NULL ; + }else if (csc_val ==0 ) { +continue ; + }else { +/* csc_val > 0 */ + nbytes =read (csc_event.data.fd, &buf,8 ); +if (nbytes <0 ) { +if (errno ==EINTR || errno ==EWOULDBLOCK ) +continue ; +DRV_LOG (ERR ,"Error reading from file descriptor %d: %s\n" , + csc_event.data.fd, +strerror (errno)); +return NULL ; + }else if (nbytes ==0 ) { +DRV_LOG (ERR ,"Read nothing from file descriptor %d\n" , + csc_event.data.fd); +continue ; + }else { +virtio_interrupt_handler (internal); + } + } + } + +return NULL ; +} + +static int +setup_intr_relay (struct ifcvf_internal *internal) +{ +int ret; + + ret =pthread_create (&internal->intr_tid,NULL , intr_relay, + (void *)internal); +if (ret) { +DRV_LOG (ERR ,"failed to create notify relay pthread." ); +return -1 ; + } +return 0 ; +} + +static int +unset_intr_relay (struct ifcvf_internal *internal) +{ +void *status; + +if (internal->intr_tid) { +pthread_cancel (internal->intr_tid); +pthread_join (internal->intr_tid, &status); + } + internal->intr_tid =0 ; + +if (internal->csc_fd >=0 ) +close (internal->csc_fd); + internal->csc_fd =-1 ; + +return 0 ; +} +static int update_datapath (struct ifcvf_internal *internal) {
@@ -584,10 +687,16 @@ struct rte_vdpa_dev_info {if (ret) goto err; + ret =setup_intr_relay (internal); +if (ret) + goto err; +rte_atomic32_set (&internal->running,1 ); }else if (rte_atomic32_read (&internal->running) && (!rte_atomic32_read (&internal->started) || !rte_atomic32_read (&internal->dev_attached))) { + ret =unset_intr_relay (internal); + ret =unset_notify_relay (internal);if (ret) goto err;
@@ -880,6 +989,9 @@ struct rte_vdpa_dev_info {/* stop the direct IO data path */ unset_notify_relay (internal);vdpa_ifcvf_stop (internal); + +unset_intr_relay (internal); +vdpa_disable_vfio_intr (internal); ret =rte_vhost_host_notifier_ctrl (vid,RTE_VHOST_QUEUE_ALL , false);
Signature · 1 line
1.8.3.1
RE: [PATCH v7 14/18] vdpa/ifc: add interrupt and handle for virtio blk
Quoted 8 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 14/18] vdpa/ifc: add interrupt and handle for virtio blk
Better be: vdpa/ifc: add interrupt handling for config space
Quoted 3 lines
Create a thread to poll and relay config space change interrupt. Use
VHOST_USER_SLAVE_CONFIG_CHANGE_MSGto info qemu.
Inform QEMU. You don't need to save words in commit log. The commit log should be as detailed as possible to make readers understand quickly what the commit is doing :)
Quoted 12 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 112 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+)drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0
drivers/vdpa/ifc/ifcvf_vdpa.c struct ifcvf_internal { · line 56
Quoted 4 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0@@ -53,7 +53,9 @@ struct ifcvf_internal {int vfio_group_fd;int vfio_dev_fd;pthread_t tid;/* thread for notify relay */ +pthread_t intr_tid;/* thread for intr relay */
Thread for virtio-blk config space change interrupt relay
drivers/vdpa/ifc/ifcvf_vdpa.c struct ifcvf_internal { · line 58
Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0int epfd; +int csc_fd;
csc_epfd
Quoted 32 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +24
-0int vid;struct rte_vdpa_device *vdev;uint16_t max_queues;@@ -558,6 +560,107 @@ struct rte_vdpa_dev_info {return 0 ; } +static void +virtio_interrupt_handler (struct ifcvf_internal *internal) +{ +int vid = internal->vid; +int ret; + + ret =rte_vhost_slave_config_change (vid,1 ); +if (ret) + DRV_LOG(ERR, "failed to notify the guest about configuration space change."); +} + +static void * +intr_relay (void *arg) +{ +struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; +struct epoll_event csc_event; +struct epoll_event ev; +uint64_t buf; +int nbytes; +int csc_fd, csc_val =0 ; + + csc_fd =epoll_create (1 ); +if (csc_fd <0 ) { + DRV_LOG(ERR, "failed to create epoll for config space
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 597
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9
-0change."); +return NULL ; + } + + ev.events =EPOLLIN |EPOLLPRI |EPOLLRDHUP |EPOLLHUP ; + ev.data.fd =rte_intr_fd_get (internal->pdev->intr_handle); +if (epoll_ctl (csc_fd,EPOLL_CTL_ADD , +rte_intr_fd_get (internal->pdev->intr_handle), &ev) <0 ) { +DRV_LOG (ERR ,"epoll add error: %s" ,strerror (errno)); +return NULL ;
Close the epfd and set to -1 if err.
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 605
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10
-0+ } + + internal->csc_fd = csc_fd; + +for (;;) { + csc_val =epoll_wait (csc_fd, &csc_event,1 ,-1 ); +if (csc_val <0 ) { +if (errno ==EINTR ) +continue ; +DRV_LOG (ERR ,"epoll_wait return fail\n" );
Save '\n', it's not needed for DRV_LOG. Please check other DRV_LOGs
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 613
Quoted 8 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +8
-0+return NULL ; + }else if (csc_val ==0 ) { +continue ; + }else { +/* csc_val > 0 */ + nbytes =read (csc_event.data.fd, &buf,8 ); +if (nbytes <0 ) { +if (errno ==EINTR || errno ==EWOULDBLOCK )
EAGAIN should also be this case?
Quoted 17 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7
-0+continue ; + DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n", + csc_event.data.fd, +strerror (errno)); +return NULL ; + }else if (nbytes ==0 ) { + DRV_LOG(ERR, "Read nothing from filedescriptor %d\n",
drivers/vdpa/ifc/ifcvf_vdpa.c +8
-0+ csc_event.data.fd); +continue ; + }else { +virtio_interrupt_handler (internal); + } + } + } +
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 638
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10
-0+return NULL ; +} + +static int +setup_intr_relay (struct ifcvf_internal *internal) +{ +int ret; + + ret =pthread_create (&internal->intr_tid,NULL , intr_relay, + (void *)internal);
EAL API: rte_ctrl_thread_create, will be preferred.
Quoted 29 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +25
-0+if (ret) { +DRV_LOG (ERR ,"failed to create notify relay pthread." ); +return -1 ; + } +return 0 ; +} + +static int +unset_intr_relay (struct ifcvf_internal *internal) +{ +void *status; + +if (internal->intr_tid) { +pthread_cancel (internal->intr_tid); +pthread_join (internal->intr_tid, &status); + } + internal->intr_tid =0 ; + +if (internal->csc_fd >=0 ) +close (internal->csc_fd); + internal->csc_fd =-1 ; + +return 0 ; +} +static int update_datapath (struct ifcvf_internal *internal) {
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 693
Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4
-0@@ -584,10 +687,16 @@ struct rte_vdpa_dev_info {if (ret) goto err; + ret =setup_intr_relay (internal); +if (ret) + goto err; +
But this is not needed for net, right? As I said, we should include validation for net also.
Thanks,
Chenbo
Quoted 21 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5
-1rte_atomic32_set (&internal->running,1 ); }else if (rte_atomic32_read (&internal->running) && (!rte_atomic32_read (&internal->started) || !rte_atomic32_read (&internal->dev_attached))) { + ret =unset_intr_relay (internal); + ret =unset_notify_relay (internal);if (ret) goto err;@@ -880,6 +989,9 @@ struct rte_vdpa_dev_info {/* stop the direct IO data path */ unset_notify_relay (internal);vdpa_ifcvf_stop (internal); + +unset_intr_relay (internal); +vdpa_disable_vfio_intr (internal); ret =rte_vhost_host_notifier_ctrl (vid,RTE_VHOST_QUEUE_ALL , false); --1.8.3.1
RE: [PATCH v7 14/18] vdpa/ifc: add interrupt and handle for virtio blk
HI Chenbo,
Thanks for your reply.
My reply is inline.
Quoted 19 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Friday, May 13, 2022 10:53 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 14/18] vdpa/ifc: add interrupt and handle for virtio blkQuoted 9 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 14/18] vdpa/ifc: add interrupt and handle for virtio blk
Quoted 2 lines
Better be: vdpa/ifc: add interrupt handling for config space
Sure. I will fix it in next version.
Quoted 8 lines
Quoted 3 lines
Create a thread to poll and relay config space change interrupt. Use
VHOST_USER_SLAVE_CONFIG_CHANGE_MSGto info qemu.Inform QEMU. You don't need to save words in commit log. The commit log should be as detailed as possible to make readers understand quickly what the commit is doing :)
Sure. I will fix it in next version.
Quoted 7 lines
Quoted 7 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 112 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+)
Quoted 12 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct ifcvf_internal { · line 56
Quoted 9 lines
(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex 5a8cf1c..0e94e1f 100644drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0@@ -53,7 +53,9 @@ struct ifcvf_internal {int vfio_group_fd;int vfio_dev_fd;pthread_t tid;/* thread for notify relay */ +pthread_t intr_tid;/* thread for intr relay */ Thread for virtio-blk config space change interrupt relay
Sure.
Quoted 5 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct ifcvf_internal { · line 58
Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0int epfd; +int csc_fd;
csc_epfd
OK.
Quoted 35 lines
Quoted 35 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +26
-0int vid;struct rte_vdpa_device *vdev;uint16_t max_queues;@@ -558,6 +560,107 @@ struct rte_vdpa_dev_info {return 0 ; } +static void +virtio_interrupt_handler (struct ifcvf_internal *internal) { +int vid = internal->vid; +int ret; + + ret =rte_vhost_slave_config_change (vid,1 ); +if (ret) + DRV_LOG(ERR, "failed to notify the guest about configuration space change."); +} + +static void * +intr_relay (void *arg) +{ +struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; +struct epoll_event csc_event; +struct epoll_event ev; +uint64_t buf; +int nbytes; +int csc_fd, csc_val =0 ; + + csc_fd =epoll_create (1 ); +if (csc_fd <0 ) { + DRV_LOG(ERR, "failed to create epoll for config space change."); +return NULL ; + } +
Quoted 9 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 595
Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +6
-0+ ev.events =EPOLLIN |EPOLLPRI |EPOLLRDHUP |EPOLLHUP ; + ev.data.fd =rte_intr_fd_get (internal->pdev->intr_handle); +if (epoll_ctl (csc_fd,EPOLL_CTL_ADD , +rte_intr_fd_get (internal->pdev->intr_handle), &ev) <0 ) { +DRV_LOG (ERR ,"epoll add error: %s" ,strerror (errno)); +return NULL ;Close the epfd and set to -1 if err.
I check other epoll usage in DPDK, it seems most usage do not close the epfd and set to -1. I am not sure whether it is needed.
Quoted 4 lines
Quoted 4 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4
-0+ } + + internal->csc_fd = csc_fd; +
Quoted 9 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 605
Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +6
-0+for (;;) { + csc_val =epoll_wait (csc_fd, &csc_event,1 ,-1 ); +if (csc_val <0 ) { +if (errno ==EINTR ) +continue ; +DRV_LOG (ERR ,"epoll_wait return fail\n" );Save '\n', it's not needed for
DRV_LOG. Please check otherDRV_LOGs
OK
Quoted 11 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 613
Quoted 8 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +8
-0+return NULL ; + }else if (csc_val ==0 ) { +continue ; + }else { +/* csc_val > 0 */ + nbytes =read (csc_event.data.fd, &buf,8 ); +if (nbytes <0 ) { +if (errno ==EINTR || errno ==EWOULDBLOCK )EAGAIN should also be this case?
Yes, it will be add in next version.
Quoted 20 lines
Quoted 20 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7
-0+continue ; + DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n", + csc_event.data.fd, +strerror (errno)); +return NULL ; + }else if (nbytes ==0 ) { + DRV_LOG(ERR, "Read nothing from filedescriptor %d\n",
drivers/vdpa/ifc/ifcvf_vdpa.c +11
-0+ csc_event.data.fd); +continue ; + }else { +virtio_interrupt_handler (internal); + } + } + } + +return NULL ; +} +
Quoted 9 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 638
Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +6
-0+static int +setup_intr_relay (struct ifcvf_internal *internal) { +int ret; + + ret =pthread_create (&internal->intr_tid,NULL , intr_relay, + (void *)internal);EAL API:
rte_ctrl_thread_create, will be preferred.
Sure, I will use " rte_ctrl_thread_create " in next version.
Quoted 38 lines
Quoted 24 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +24
-0+if (ret) { +DRV_LOG (ERR ,"failed to create notify relay pthread." ); +return -1 ; + } +return 0 ; +} + +static int +unset_intr_relay (struct ifcvf_internal *internal) { +void *status; + +if (internal->intr_tid) { +pthread_cancel (internal->intr_tid); +pthread_join (internal->intr_tid, &status); + } + internal->intr_tid =0 ; + +if (internal->csc_fd >=0 ) +close (internal->csc_fd); + internal->csc_fd =-1 ; + +return 0 ; +} +drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info {
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5
-0static int update_datapath(struct ifcvf_internal *internal) { @@ -584,10 +687,16 @@ struct rte_vdpa_dev_info { if (ret) goto err; + ret = setup_intr_relay(internal); + if (ret) + goto err; +But this is not needed for net, right? As I said, we should include validation for net also.
Quoted 3 lines
Thanks,
Chenbo
For net device, especially the harden virtio device, fabric plug in or out will cause config change. I think net device may also need this interrupt, but I am not sure.
Quoted 23 lines
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +2
-0rte_atomic32_set (&internal->running,1 ); }else if (rte_atomic32_read (&internal->running) && (!rte_atomic32_read (&internal->started) || !rte_atomic32_read (&internal->dev_attached))) { + ret =unset_intr_relay (internal); + ret =unset_notify_relay (internal);if (ret) goto err;drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 997
Quoted 9 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +3
-0@@ -880,6 +989,9 @@ struct rte_vdpa_dev_info {/* stop the direct IO data path */ unset_notify_relay (internal);vdpa_ifcvf_stop (internal); + +unset_intr_relay (internal); +vdpa_disable_vfio_intr (internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL,false);
Quoted 2 lines
--
1.8.3.1
PATCH v7 15/18 vdpa/ifc: add is blk flag to ifcvf HW struct
In reply to Andy PeiAdd is_blk flag to ifcvf_hw, and init is_blk during probe.
--- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++ 2 files changed, 3 insertions(+)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
@@ -149,6 +149,7 @@ struct ifcvf_hw {u8 *lm_cfg;struct vring_info vring[IFCVF_MAX_QUEUES *2 ];u8 nr_vring; +u8 is_blk;struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE ]; };
drivers/vdpa/ifc/ifcvf_vdpa.c +2 -0
@@ -1536,11 +1536,13 @@ struct rte_vdpa_dev_info dev_info[] = {if (device_id ==VIRTIO_ID_NET ) { internal->device_type =IFCVF_NET ; + internal->hw.is_blk =IFCVF_NET ; internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_NET ].features; }else if (device_id ==VIRTIO_ID_BLOCK ) { internal->device_type =IFCVF_BLK ; + internal->hw.is_blk =IFCVF_BLK ; internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_BLK ].features;
Signature · 1 line
1.8.3.1
RE: [PATCH v7 15/18] vdpa/ifc: add is blk flag to ifcvf HW struct
Quoted 33 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 15/18] vdpa/ifc: add is blk flag to ifcvf HW structAdd
is_blkflag toifcvf_hw, and initis_blkduring probe.Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++ 2 files changed, 3 insertions(+)drivers/vdpa/ifc/base/ifcvf.h +1
-0@@ -149,6 +149,7 @@ struct ifcvf_hw {u8 *lm_cfg;struct vring_info vring[IFCVF_MAX_QUEUES *2 ];u8 nr_vring; +u8 is_blk;struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE ]; };drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1539
Quoted 4 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0@@ -1536,11 +1536,13 @@ struct rte_vdpa_dev_info dev_info[] = {if (device_id ==VIRTIO_ID_NET ) { internal->device_type =IFCVF_NET ; + internal->hw.is_blk =IFCVF_NET ;
I believe it's enough to keep only device_type. Device type can be defined in internal or internal->hw. Choose the way you prefer.
Thanks,
Chenbo
Quoted 11 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1
-1internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_NET ].features; }else if (device_id ==VIRTIO_ID_BLOCK ) { internal->device_type =IFCVF_BLK ; + internal->hw.is_blk =IFCVF_BLK ; internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_BLK ].features; --1.8.3.1
RE: [PATCH v7 15/18] vdpa/ifc: add is blk flag to ifcvf HW struct
Hi Chenbo,
Quoted 51 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Friday, May 13, 2022 10:55 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 15/18] vdpa/ifc: add is blk flag to ifcvf HW structQuoted 22 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 15/18] vdpa/ifc: add is blk flag to ifcvf HW structAdd
is_blkflag toifcvf_hw, and initis_blkduring probe.Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++ 2 files changed, 3 insertions(+)(quoted diff) +0
-0
b/drivers/vdpa/ifc/base/ifcvf.hindex 769c603..8591ef1 100644drivers/vdpa/ifc/base/ifcvf.h struct ifcvf_hw { · line 153
Quoted 5 lines
drivers/vdpa/ifc/base/ifcvf.h +1
-0@@ -149,6 +149,7 @@ struct ifcvf_hw {u8 *lm_cfg;struct vring_info vring[IFCVF_MAX_QUEUES *2 ];u8 nr_vring; +u8 is_blk;struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE];
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1539
Quoted 11 lines
drivers/vdpa/ifc/base/ifcvf.h +0
-0};(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex 0e94e1f..4923bc1 100644drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0@@ -1536,11 +1536,13 @@ struct rte_vdpa_dev_info dev_info[] = {if (device_id ==VIRTIO_ID_NET ) { internal->device_type =IFCVF_NET ; + internal->hw.is_blk =IFCVF_NET ;I believe it's enough to keep only
device_type. Device type can be defined in internal orinternal->hw. Choose the way you prefer.
Quoted 3 lines
Thanks,
Chenbo
Yes, you are right. To keep API not changed, I think it is better to keep only device_type in internal->hw. So I have to make some change to other patch in this patch set.
Quoted 14 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = { · line 1541
Quoted 2 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0internal->features = features & ~(1ULL <<
VIRTIO_F_IOMMU_PLATFORM);drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info dev_info[] = {
Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0internal->features |= dev_info[IFCVF_NET ].features; }else if (device_id ==VIRTIO_ID_BLOCK ) { internal->device_type =IFCVF_BLK ; + internal->hw.is_blk =IFCVF_BLK ; internal->features = features & ~(1ULL <<
VIRTIO_F_IOMMU_PLATFORM);Quoted 3 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +0
-1internal->features |= dev_info[IFCVF_BLK ].features; --1.8.3.1
PATCH v7 16/18 vdpa/ifc/base: access correct register for blk device
In reply to Andy Pei1.last_avail_idx is lower 16 bit of the register.
2.address of ring_state register is different between net and blk device.
--- drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++------- drivers/vdpa/ifc/base/ifcvf.h | 1 + 2 files changed, 30 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +29 -7
@@ -218,10 +218,18 @@ &cfg->queue_used_hi);IFCVF_WRITE_REG16 (hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ) = - (u32 )hw->vring[i].last_avail_idx | - ((u32 )hw->vring[i].last_used_idx <<16 ); +if (hw->is_blk ==IFCVF_BLK ) { + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 ); + }else if (hw->is_blk ==IFCVF_NET ) { + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 ); + }IFCVF_WRITE_REG16 (i +1 , &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -254,9 +262,23 @@IFCVF_WRITE_REG16 (i, &cfg->queue_select);IFCVF_WRITE_REG16 (0 , &cfg->queue_enable);IFCVF_WRITE_REG16 (IFCVF_MSI_NO_VECTOR , &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ); - hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); + +if (hw->is_blk) { + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); + }else if (hw->is_blk ==IFCVF_NET ) { + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ); + } + +if (hw->is_blk ==IFCVF_BLK ) + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); +else if (hw->is_blk ==IFCVF_NET ) + hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); hw->vring[i].last_used_idx = (u16)(ring_state >>16 ); } }
drivers/vdpa/ifc/base/ifcvf.h +1 -0
@@ -65,6 +65,7 @@#define IFCVF_MEDIATED_VRING 0x200000000000#define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff#ifndef VHOST_USER_PROTOCOL_F_CONFIG #define VHOST_USER_PROTOCOL_F_CONFIG 9
Signature · 1 line
1.8.3.1
RE: [PATCH v7 16/18] vdpa/ifc/base: access correct register for blk device
Quoted 11 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 16/18] vdpa/ifc/base: access correct register for blk device1.last_avail_idx is lower 16 bit of the register.
2.address ofring_stateregister is different between net and blk device.
Not a good commit log. The commit log should illustrate more on what's the commit is doing.
Thanks,
Chenbo
Quoted 77 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++------- drivers/vdpa/ifc/base/ifcvf.h | 1 + 2 files changed, 30 insertions(+), 7 deletions(-)drivers/vdpa/ifc/base/ifcvf.c +30
-7@@ -218,10 +218,18 @@ &cfg->queue_used_hi);IFCVF_WRITE_REG16 (hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ) = - (u32 )hw->vring[i].last_avail_idx | - ((u32 )hw->vring[i].last_used_idx <<16 ); +if (hw->is_blk ==IFCVF_BLK ) { + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 ); + }else if (hw->is_blk ==IFCVF_NET ) { + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 ); + }IFCVF_WRITE_REG16 (i +1 , &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==@@ -254,9 +262,23 @@IFCVF_WRITE_REG16 (i, &cfg->queue_select);IFCVF_WRITE_REG16 (0 , &cfg->queue_enable);IFCVF_WRITE_REG16 (IFCVF_MSI_NO_VECTOR , &cfg- >queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ); - hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); + +if (hw->is_blk) { + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); + }else if (hw->is_blk ==IFCVF_NET ) { + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ); + } + +if (hw->is_blk ==IFCVF_BLK ) + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); +else if (hw->is_blk ==IFCVF_NET ) + hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); hw->vring[i].last_used_idx = (u16)(ring_state >>16 ); } }drivers/vdpa/ifc/base/ifcvf.h +1
-1@@ -65,6 +65,7 @@#define IFCVF_MEDIATED_VRING 0x200000000000#define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff#ifndef VHOST_USER_PROTOCOL_F_CONFIG #define VHOST_USER_PROTOCOL_F_CONFIG 9 --1.8.3.1
RE: [PATCH v7 16/18] vdpa/ifc/base: access correct register for blk device
HI Chenbo,
Thanks for your reply.
I will send out a new version to fix this.
Quoted 109 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Friday, May 13, 2022 10:58 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 16/18] vdpa/ifc/base: access correct register for blk deviceQuoted 10 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 16/18] vdpa/ifc/base: access correct register for blk deviceQuoted 2 lines
1.last_avail_idx is lower 16 bit of the register.
2.address ofring_stateregister is different between net and blk device.Not a good commit log. The commit log should illustrate more on what's the commit is doing.
Thanks,
ChenboQuoted 25 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++------- drivers/vdpa/ifc/base/ifcvf.h | 1 + 2 files changed, 30 insertions(+), 7 deletions(-)(quoted diff) +0
-0
b/drivers/vdpa/ifc/base/ifcvf.cindex d10c1fd..4d5881a 100644drivers/vdpa/ifc/base/ifcvf.c +5
-4@@ -218,10 +218,18 @@ &cfg->queue_used_hi);IFCVF_WRITE_REG16 (hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ) = - (u32 )hw->vring[i].last_avail_idx | - ((u32 )hw->vring[i].last_used_idx <<16 ); +if (hw->is_blk ==IFCVF_BLK ) { + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 );drivers/vdpa/ifc/base/ifcvf.c line 235
Quoted 10 lines
drivers/vdpa/ifc/base/ifcvf.c +7
-0+ }else if (hw->is_blk ==IFCVF_NET ) { + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 ); + }IFCVF_WRITE_REG16 (i +1 , &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -254,9 +262,23
drivers/vdpa/ifc/base/ifcvf.c
Quoted 6 lines
@@
drivers/vdpa/ifc/base/ifcvf.c +0
-0IFCVF_WRITE_REG16 (i, &cfg->queue_select);IFCVF_WRITE_REG16 (0 , &cfg->queue_enable); IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg-Quoted 1 line
queue_msix_vector);drivers/vdpa/ifc/base/ifcvf.c +0
-1- ring_state = *(u32 *)(hw->lm_cfg +
IFCVF_LM_RING_STATE_OFFSETQuoted 37 lines
drivers/vdpa/ifc/base/ifcvf.c +18
-2+ - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4); - hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); + +if (hw->is_blk) { + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); + }else if (hw->is_blk ==IFCVF_NET ) { + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ); + } + +if (hw->is_blk ==IFCVF_BLK ) + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); +else if (hw->is_blk ==IFCVF_NET ) + hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); hw->vring[i].last_used_idx = (u16)(ring_state >>16 ); } }(quoted diff) +0
-0
b/drivers/vdpa/ifc/base/ifcvf.hindex 8591ef1..ff11b12 100644drivers/vdpa/ifc/base/ifcvf.h +1
-1@@ -65,6 +65,7 @@#define IFCVF_MEDIATED_VRING 0x200000000000#define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff#ifndef VHOST_USER_PROTOCOL_F_CONFIG #define VHOST_USER_PROTOCOL_F_CONFIG 9 --1.8.3.1
PATCH v7 17/18 vdpa/ifc: blk device pause without no inflight IO
In reply to Andy PeiWhen virtio blk device is pause, make sure hardware last_avail_idx and last_used_idx are the same.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +17 -8
@@ -314,12 +314,12 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_stop(struct ifcvf_internal *internal) {struct ifcvf_hw *hw = &internal->hw; -struct rte_vhost_vring vq;uint32_t i;int vid;uint64_t features =0 ;uint64_t log_base =0 , log_size =0 ;uint64_t len; +u32 ring_state =0 ; vid = internal->vid;
@@ -328,13 +328,22 @@ struct rte_vdpa_dev_info {*/ if (internal->device_type ==IFCVF_BLK ) {for (i =0 ; i < hw->nr_vring; i++) { -rte_vhost_get_vhost_vring (internal->vid, i, &vq); -while (vq.avail->idx != vq.used->idx) { -ifcvf_notify_queue (hw, i); -usleep (10 ); - } - hw->vring[i].last_avail_idx = vq.avail->idx; - hw->vring[i].last_used_idx = vq.used->idx; +do { +if (hw->lm_cfg !=NULL ) + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); + hw->vring[i].last_used_idx = + (u16)(ring_state >>16 ); +if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { +ifcvf_notify_queue (hw, i); +usleep (10 ); + } + }while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); } }
Signature · 1 line
1.8.3.1
RE: [PATCH v7 17/18] vdpa/ifc: blk device pause without no inflight IO
Quoted 11 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 17/18] vdpa/ifc: blk device pause without no inflight IOWhen virtio blk device is pause, make sure hardware
last_avail_idxandlast_used_idxare the same.
Patch is good. But illustrate more on avoiding in-flight packet in commit log
Thanks,
Chenbo
Quoted 56 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-)drivers/vdpa/ifc/ifcvf_vdpa.c +17
-9@@ -314,12 +314,12 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_stop(struct ifcvf_internal *internal) {struct ifcvf_hw *hw = &internal->hw; -struct rte_vhost_vring vq;uint32_t i;int vid;uint64_t features =0 ;uint64_t log_base =0 , log_size =0 ;uint64_t len; +u32 ring_state =0 ; vid = internal->vid;@@ -328,13 +328,22 @@ struct rte_vdpa_dev_info {*/ if (internal->device_type ==IFCVF_BLK ) {for (i =0 ; i < hw->nr_vring; i++) { -rte_vhost_get_vhost_vring (internal->vid, i, &vq); -while (vq.avail->idx != vq.used->idx) { -ifcvf_notify_queue (hw, i); -usleep (10 ); - } - hw->vring[i].last_avail_idx = vq.avail->idx; - hw->vring[i].last_used_idx = vq.used->idx; +do { +if (hw->lm_cfg !=NULL ) + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); + hw->vring[i].last_used_idx = + (u16)(ring_state >>16 ); +if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { +ifcvf_notify_queue (hw, i); +usleep (10 ); + } + }while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); } } --1.8.3.1
RE: [PATCH v7 17/18] vdpa/ifc: blk device pause without no inflight IO
Hi Chenbo,
Thanks for reply.
This will be addressed in next version.
Quoted 85 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Friday, May 13, 2022 10:59 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: RE: [PATCH v7 17/18] vdpa/ifc: blk device pause without no inflight IOQuoted 10 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 17/18] vdpa/ifc: blk device pause without no inflight IOQuoted 2 lines
When virtio blk device is pause, make sure hardware
last_avail_idxandlast_used_idxare the same.Patch is good. But illustrate more on avoiding in-flight packet in commit log
Thanks,
ChenboQuoted 28 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-)(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex 4923bc1..def6adf 100644drivers/vdpa/ifc/ifcvf_vdpa.c +1
-2@@ -314,12 +314,12 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_stop(struct ifcvf_internal *internal) {struct ifcvf_hw *hw = &internal->hw; -struct rte_vhost_vring vq;uint32_t i;int vid;uint64_t features =0 ;uint64_t log_base =0 , log_size =0 ;uint64_t len; +u32 ring_state =0 ; vid = internal->vid;@@ -328,13 +328,22 @@ struct rte_vdpa_dev_info {*/ if (internal->device_type ==IFCVF_BLK ) {for (i =0 ; i < hw->nr_vring; i++) { -rte_vhost_get_vhost_vring (internal->vid, i, &vq);drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 334
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4
-6-while (vq.avail->idx != vq.used->idx) { -ifcvf_notify_queue (hw, i); -usleep (10 ); - } - hw->vring[i].last_avail_idx = vq.avail->idx; - hw->vring[i].last_used_idx = vq.used->idx; + do { + if (hw->lm_cfg != NULL) + ring_state = *(u32 *)(hw->lm_cfg + +
IFCVF_LM_RING_STATE_OFFSET+drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info {
Quoted 3 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +3
-0+ i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state &
IFCVF_16_BIT_MASK);Quoted 14 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9
-1+ hw->vring[i].last_used_idx = + (u16)(ring_state >>16 ); +if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { +ifcvf_notify_queue (hw, i); +usleep (10 ); + } + }while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); } } --1.8.3.1
PATCH v7 18/18 vhost: make sure each queue callfd is configured
In reply to Andy PeiDuring the vhost data path building process, qemu will create a call fd at first, and create another call fd in the end. The final call fd will be used to relay notify.
In the original code, after kick fd is set, dev_conf will set the first call fd. Even though the actual call fd will set, the data path will not work correctly.
--- lib/vhost/vhost_user.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
lib/vhost/vhost_user.c +14 -0
@@ -3230,12 +3230,26 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev,if (!vdpa_dev) goto out; +if (request !=VHOST_USER_SET_VRING_CALL ) + goto out; +if (!(dev->flags &VIRTIO_DEV_VDPA_CONFIGURED )) {if (vdpa_dev->ops->dev_conf (dev->vid))VHOST_LOG_CONFIG (ERR ,"(%s) failed to configure vDPA device\n" , dev->ifname);else dev->flags |=VIRTIO_DEV_VDPA_CONFIGURED ; + }else { +/* when VIRTIO_DEV_VDPA_CONFIGURED already configured +* close the device and config the device again, +* make sure the call fd of each queue is configured correctly. +*/ +if (vdpa_dev->ops->dev_close (dev->vid)) +VHOST_LOG_CONFIG (ERR , +"Failed to close vDPA device\n" ); +if (vdpa_dev->ops->dev_conf (dev->vid)) +VHOST_LOG_CONFIG (ERR , +"Failed to re-config vDPA device\n" ); } out:
Signature · 1 line
1.8.3.1
RE: [PATCH v7 18/18] vhost: make sure each queue callfd is configured
Quoted 8 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, April 27, 2022 4:30 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>
Subject: [PATCH v7 18/18] vhost: make sure each queue callfd is configured
Quoted 6 lines
During the vhost data path building process, qemu will create a call fd at first, and create another call fd in the end. The final call fd will be used to relay notify.
In the original code, after kick fd is set,dev_confwill set the first call fd. Even though the actual call fd will set, the data path will not work correctly.
I don't understand the problem. If callfd reset, we should re-configure in the set_vring_state callback, and issue can be handled there.
Thanks,
Chenbo
Quoted 41 lines
Signed-off-by Andy Pei <[email protected]>--- lib/vhost/vhost_user.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)lib/vhost/vhost_user.c +14
-0@@ -3230,12 +3230,26 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev, if (!vdpa_dev) goto out; +if (request !=VHOST_USER_SET_VRING_CALL ) + goto out; +if (!(dev->flags &VIRTIO_DEV_VDPA_CONFIGURED )) {if (vdpa_dev->ops->dev_conf (dev->vid))VHOST_LOG_CONFIG (ERR ,"(%s) failed to configure vDPA device\n", dev->ifname);else dev->flags |=VIRTIO_DEV_VDPA_CONFIGURED ; + }else { +/* when VIRTIO_DEV_VDPA_CONFIGURED already configured +* close the device and config the device again, +* make sure the call fd of each queue is configured correctly. +*/ +if (vdpa_dev->ops->dev_close (dev->vid)) +VHOST_LOG_CONFIG (ERR , +"Failed to close vDPA device\n" ); +if (vdpa_dev->ops->dev_conf (dev->vid)) +VHOST_LOG_CONFIG (ERR , +"Failed to re-config vDPA device\n" ); }out:
--
1.8.3.1
v8 cover add virtio_blk device support to vdpa/ifc
In reply to Andy PeiThis patch set add virtio_blk device support to vdpa/ifc driver. With a lot of similarities, I re-use part of vdpa/ifc driver. Distinguish the virtio net and blk device by device id, and implement specific features and ops.
Add example to vdpa to support virtio_blk device.
To support blk device live migration, some modification to vhost lib. Perform dev_conf op only under VHOST_USER_SET_VRING_CALL msg.
v8:
delete some redundant code.
fix some commit log.
v7:
Check on expected fd num in new vhost msg handler. Sanity check on vhost msg size.
Fix typo.
Add commit log to help understand code. Remove duplicated code.
Add new API to get vDPA device type.
v6:
fix some commit log.
add vhost socket in log output to make it more user-friendly. when driver ops fail, just output some log, do not break message handler. check vhost msg size in msg handler.
v5:
fix some coding style issues.
v4:
add args "isblk" to vdpa example to specify a block device, fix some issue in example.
Make sure code specify for block device does not affect net device.
v3:
Fix some compile issues.
v2:
Fix some coding style issues.
Andy Pei (13):
vdpa/ifc: add support for virtio blk device
vhost: add vDPA ops for blk device
vhost: add vhost msg support for get/set config vdpa/ifc: add blk ops for ifc device vdpa/ifc: add vDPA interrupt relay for blk device vdpa/ifc: add block device SW live-migration
vhost: add API to get vDPA device type vdpa/ifc: add get device type ops to ifc driver examples/vdpa: add add virtio blk support
usertools: add support for virtio blk device vdpa/ifc: add log for config space of virtio blk vdpa/ifc: add interrupt handling for config space vdpa/ifc/base: access correct register for blk device
doc/guides/prog_guide/vhost_lib.rst | 5 + doc/guides/rel_notes/release_22_07.rst | 4 + drivers/vdpa/ifc/base/ifcvf.c | 34 ++- drivers/vdpa/ifc/base/ifcvf.h | 21 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 408 +++++++++++++++++++++++++++++++-- examples/vdpa/main.c | 56 +++++ examples/vdpa/vdpa_blk_compact.h | 58 +++++ lib/vhost/rte_vhost.h | 17 ++ lib/vhost/socket.c | 44 ++++ lib/vhost/vdpa_driver.h | 11 +- lib/vhost/version.map | 1 + lib/vhost/vhost_user.c | 73 ++++++ lib/vhost/vhost_user.h | 13 ++ usertools/dpdk-devbind.py | 5 +- 14 files changed, 725 insertions(+), 25 deletions(-) create mode 100644 examples/vdpa/vdpa_blk_compact.h
Signature · 1 line
1.8.3.1
PATCH v8 1/13 vdpa/ifc: add support for virtio blk device
Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops.
--- drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++- drivers/vdpa/ifc/ifcvf_vdpa.c | 91 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 98 insertions(+), 9 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +15 -1
@@ -5,8 +5,17 @@#ifndef _IFCVF_H_#define _IFCVF_H_ +#include <linux/virtio_blk.h> #include "ifcvf_osdep.h" +#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +/* for BLK */ +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_DEVICE_ID 0x0002 +#define IFCVF_VENDOR_ID 0x1AF4#define IFCVF_DEVICE_ID 0x1041#define IFCVF_SUBSYS_VENDOR_ID 0x8086
@@ -126,13 +135,18 @@ struct ifcvf_hw {u8 notify_region;u32 notify_off_multiplier;struct ifcvf_pci_common_cfg *common_cfg; -struct ifcvf_net_config *dev_cfg; +union { +struct ifcvf_net_config *net_cfg; +struct virtio_blk_config *blk_cfg; +void *dev_cfg; + };u8 *isr;u16 *notify_base;u16 *notify_addr[IFCVF_MAX_QUEUES *2 ];u8 *lm_cfg;struct vring_info vring[IFCVF_MAX_QUEUES *2 ];u8 nr_vring; +int device_type;struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE ]; };
drivers/vdpa/ifc/ifcvf_vdpa.c +83 -8
127 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RE: [PATCH v8 01/13] vdpa/ifc: add support for virtio blk device
Hi Andy,
Quoted 31 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 01/13] vdpa/ifc: add support for virtio blk deviceRe-use the vdpa/ifc code, distinguish blk and net device by
pci_device_id. Blk and net device are implemented with proper feature and ops.Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++- drivers/vdpa/ifc/ifcvf_vdpa.c | 91 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 98 insertions(+), 9 deletions(-)drivers/vdpa/ifc/base/ifcvf.h +2
-0@@ -5,8 +5,17 @@#ifndef _IFCVF_H_#define _IFCVF_H_ +#include <linux/virtio_blk.h> #include "ifcvf_osdep.h" +#define IFCVF_NET 0
drivers/vdpa/ifc/base/ifcvf.h line 21
Quoted 10 lines
drivers/vdpa/ifc/base/ifcvf.h +7
-0+#define IFCVF_BLK 1 + +/* for BLK */ +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_DEVICE_ID 0x0002 +#define IFCVF_VENDOR_ID 0x1AF4#define IFCVF_DEVICE_ID 0x1041#define IFCVF_SUBSYS_VENDOR_ID 0x8086
Let's rename IFCVF_DEVICE_ID to IFCVF_NET_DEVICE_ID as it's only used for net now.
Quoted 65 lines
drivers/vdpa/ifc/base/ifcvf.h +6
-1@@ -126,13 +135,18 @@ struct ifcvf_hw {u8 notify_region;u32 notify_off_multiplier;struct ifcvf_pci_common_cfg *common_cfg; -struct ifcvf_net_config *dev_cfg; +union { +struct ifcvf_net_config *net_cfg; +struct virtio_blk_config *blk_cfg; +void *dev_cfg; + };u8 *isr;u16 *notify_base;u16 *notify_addr[IFCVF_MAX_QUEUES *2 ];u8 *lm_cfg;struct vring_info vring[IFCVF_MAX_QUEUES *2 ];u8 nr_vring; +int device_type;struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE ]; };drivers/vdpa/ifc/ifcvf_vdpa.c +30
-0@@ -75,6 +75,12 @@ struct internal_list {struct ifcvf_internal *internal; }; +/* vdpa device info includes device features and devcic operation. */ +struct rte_vdpa_dev_info { +uint64_t features; +struct rte_vdpa_dev_ops *ops; +}; +TAILQ_HEAD (internal_list_head, internal_list);static struct internal_list_head internal_list =TAILQ_HEAD_INITIALIZER (internal_list);@@ -1167,6 +1173,48 @@ struct internal_list {return 0 ; } +static int16_t +ifcvf_pci_get_device_type (struct rte_pci_device *pci_dev) +{ +uint16_t pci_device_id = pci_dev->id.device_id; +uint16_t device_id; + +if (pci_device_id <0x1000 || pci_device_id >0x107f ) { +DRV_LOG (ERR ,"Probe device is not a virtio device\n" ); +return -1 ; + } + +if (pci_device_id <0x1040 ) { +/* Transitional devices: use the PCI subsystem device id as +* virtio device id, same as legacy driver always did. +*/ + device_id = pci_dev->id.subsystem_device_id; + }else { +/* Modern devices: simply use PCI device id, +* but start from 0x1040. +*/ + device_id = pci_device_id -0x1040 ; + } + +return device_id;
drivers/vdpa/ifc/ifcvf_vdpa.c struct internal_list { · line 1209
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10
-0+} + +struct rte_vdpa_dev_info dev_info[] = { + { + .features = (1ULL <<VIRTIO_NET_F_GUEST_ANNOUNCE ) | + (1ULL <<VIRTIO_NET_F_CTRL_VQ ) | + (1ULL <<VIRTIO_NET_F_STATUS ) | + (1ULL <<VHOST_USER_F_PROTOCOL_FEATURES ) | + (1ULL <<VHOST_F_LOG_ALL ), + .ops = &ifcvf_ops,
Rename ifcvf_ops -> ifcvf_net_ops
Overall the patch LGTM. With above fixed:
Quoted 85 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +29
-8+ }, + { + .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = NULL, + }, +}; +static int ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,struct rte_pci_device *pci_dev)@@ -1178,6 +1226,7 @@ struct internal_list {int sw_fallback_lm =0 ;struct rte_kvargs *kvlist =NULL ;int ret =0 ; +int16_t device_id;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;@@ -1227,13 +1276,24 @@ struct internal_list { internal->configured =0 ; internal->max_queues =IFCVF_MAX_QUEUES ; features =ifcvf_get_features (&internal->hw); - internal->features = (features & - ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM )) | - (1ULL <<VIRTIO_NET_F_GUEST_ANNOUNCE ) | - (1ULL <<VIRTIO_NET_F_CTRL_VQ ) | - (1ULL <<VIRTIO_NET_F_STATUS ) | - (1ULL <<VHOST_USER_F_PROTOCOL_FEATURES ) | - (1ULL <<VHOST_F_LOG_ALL ); + + device_id =ifcvf_pci_get_device_type (pci_dev); +if (device_id <0 ) { +DRV_LOG (ERR ,"failed to get device %s type" , pci_dev->name); + goto error; + } + +if (device_id ==VIRTIO_ID_NET ) { + internal->hw.device_type =IFCVF_NET ; + internal->features = features & + ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); + internal->features |= dev_info[IFCVF_NET ].features; + }else if (device_id ==VIRTIO_ID_BLOCK ) { + internal->hw.device_type =IFCVF_BLK ; + internal->features = features & + ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); + internal->features |= dev_info[IFCVF_BLK ].features; + } list->internal = internal;@@ -1245,7 +1305,8 @@ struct internal_list { } internal->sw_lm = sw_fallback_lm; - internal->vdev =rte_vdpa_register_device (&pci_dev->device, &ifcvf_ops); + internal->vdev =rte_vdpa_register_device (&pci_dev->device, + dev_info[internal->hw.device_type].ops);if (internal->vdev ==NULL ) {DRV_LOG (ERR ,"failed to register device %s" , pci_dev->name);drivers/vdpa/ifc/ifcvf_vdpa.c +14
-1goto error;@@ -1313,6 +1374,20 @@ struct internal_list { .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_MODERN_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + { .vendor_id = 0, /* sentinel */ }, }; --1.8.3.1
RE: [PATCH v8 01/13] vdpa/ifc: add support for virtio blk device
Hi Chenbo,
Thanks for your reply, my reply is inline.
Quoted 42 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 11:42 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 01/13] vdpa/ifc: add support for virtio blk deviceHi Andy,
Quoted 31 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 01/13] vdpa/ifc: add support for virtio blk deviceRe-use the vdpa/ifc code, distinguish blk and net device by
pci_device_id. Blk and net device are implemented with proper feature and ops.Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++- drivers/vdpa/ifc/ifcvf_vdpa.c | 91 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 98 insertions(+), 9 deletions(-)(quoted diff) +0
-0
b/drivers/vdpa/ifc/base/ifcvf.hindex 573a35f..483d38b 100644drivers/vdpa/ifc/base/ifcvf.h +1
-0@@ -5,8 +5,17 @@#ifndef _IFCVF_H_#define _IFCVF_H_ +#include <linux/virtio_blk.h> #include "ifcvf_osdep.h"
Quoted 15 lines
drivers/vdpa/ifc/base/ifcvf.h line 21
Quoted 11 lines
drivers/vdpa/ifc/base/ifcvf.h +8
-0+#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +/* for BLK */ +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_DEVICE_ID 0x0002 +#define IFCVF_VENDOR_ID 0x1AF4#define IFCVF_DEVICE_ID 0x1041#define IFCVF_SUBSYS_VENDOR_ID 0x8086Let's rename
IFCVF_DEVICE_IDtoIFCVF_NET_DEVICE_IDas it's only used for net now.
Sure, I will do it in next version.
Quoted 78 lines
Quoted 8 lines
drivers/vdpa/ifc/base/ifcvf.h +3
-1@@ -126,13 +135,18 @@ struct ifcvf_hw {u8 notify_region;u32 notify_off_multiplier;struct ifcvf_pci_common_cfg *common_cfg; -struct ifcvf_net_config *dev_cfg; +union { +struct ifcvf_net_config *net_cfg; +struct virtio_blk_config *blk_cfg;drivers/vdpa/ifc/base/ifcvf.h struct ifcvf_hw { · line 150
Quoted 10 lines
drivers/vdpa/ifc/base/ifcvf.h +3
-0+void *dev_cfg; + };u8 *isr;u16 *notify_base;u16 *notify_addr[IFCVF_MAX_QUEUES *2 ];u8 *lm_cfg;struct vring_info vring[IFCVF_MAX_QUEUES *2 ];u8 nr_vring; +int device_type;struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE];
Quoted 46 lines
drivers/vdpa/ifc/base/ifcvf.h +0
-0};(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex 9f05595..be0efd3 100644drivers/vdpa/ifc/ifcvf_vdpa.c +6
-0@@ -75,6 +75,12 @@ struct internal_list {struct ifcvf_internal *internal; }; +/* vdpa device info includes device features and devcic operation. */ +struct rte_vdpa_dev_info { +uint64_t features; +struct rte_vdpa_dev_ops *ops; +}; + TAILQ_HEAD(internal_list_head, internal_list); static struct
internal_list_headinternal_list=drivers/vdpa/ifc/ifcvf_vdpa.c +23
-0TAILQ_HEAD_INITIALIZER (internal_list);@@ -1167,6 +1173,48 @@ struct internal_list {return 0 ; } +static int16_t +ifcvf_pci_get_device_type (struct rte_pci_device *pci_dev) { +uint16_t pci_device_id = pci_dev->id.device_id; +uint16_t device_id; + +if (pci_device_id <0x1000 || pci_device_id >0x107f ) { +DRV_LOG (ERR ,"Probe device is not a virtio device\n" ); +return -1 ; + } + +if (pci_device_id <0x1040 ) { +/* Transitional devices: use the PCI subsystem device id as +* virtio device id, same as legacy driver always did. +*/ + device_id = pci_dev->id.subsystem_device_id; + }else { +/* Modern devices: simply use PCI device id, +* but start from 0x1040. +*/ + device_id = pci_device_id -0x1040 ; + } + +return device_id;drivers/vdpa/ifc/ifcvf_vdpa.c struct internal_list { · line 1208
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10
-0+} + +struct rte_vdpa_dev_info dev_info[] = { + { + .features = (1ULL <<VIRTIO_NET_F_GUEST_ANNOUNCE ) | + (1ULL <<VIRTIO_NET_F_CTRL_VQ ) | + (1ULL <<VIRTIO_NET_F_STATUS ) | + (1ULL <<VHOST_USER_F_PROTOCOL_FEATURES ) | + (1ULL <<VHOST_F_LOG_ALL ), + .ops = &ifcvf_ops,Rename
ifcvf_ops->ifcvf_net_ops
Quoted 4 lines
Overall the patch LGTM. With above fixed:
Reviewed-by Chenbo Xia <[email protected]>
OK, I will do this in next version.
Quoted 89 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct internal_list {
Quoted 3 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +3
-0+ }, + { + .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES)|
Quoted 29 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +8
-7+ (1ULL << VHOST_F_LOG_ALL), + .ops = NULL, + }, +}; +static int ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,struct rte_pci_device *pci_dev)@@ -1178,6 +1226,7 @@ struct internal_list {int sw_fallback_lm =0 ;struct rte_kvargs *kvlist =NULL ;int ret =0 ; +int16_t device_id;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;@@ -1227,13 +1276,24 @@ struct internal_list { internal->configured =0 ; internal->max_queues =IFCVF_MAX_QUEUES ; features =ifcvf_get_features (&internal->hw); - internal->features = (features & - ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM )) | - (1ULL <<VIRTIO_NET_F_GUEST_ANNOUNCE ) | - (1ULL <<VIRTIO_NET_F_CTRL_VQ ) | - (1ULL <<VIRTIO_NET_F_STATUS ) | - (1ULL <<VHOST_USER_F_PROTOCOL_FEATURES ) | - (1ULL <<VHOST_F_LOG_ALL ); + + device_id =ifcvf_pci_get_device_type (pci_dev);drivers/vdpa/ifc/ifcvf_vdpa.c struct internal_list { · line 1290
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +9
-0+if (device_id <0 ) { +DRV_LOG (ERR ,"failed to get device %s type" , pci_dev- name); + goto error; + } + +if (device_id ==VIRTIO_ID_NET ) { + internal->hw.device_type =IFCVF_NET ; + internal->features = features & + ~(1ULL <<
VIRTIO_F_IOMMU_PLATFORM);drivers/vdpa/ifc/ifcvf_vdpa.c struct internal_list {
Quoted 5 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5
-0+ internal->features |= dev_info[IFCVF_NET ].features; + }else if (device_id ==VIRTIO_ID_BLOCK ) { + internal->hw.device_type =IFCVF_BLK ; + internal->features = features & + ~(1ULL <<
VIRTIO_F_IOMMU_PLATFORM);Quoted 39 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4
-1+ internal->features |= dev_info[IFCVF_BLK ].features; + } list->internal = internal;@@ -1245,7 +1305,8 @@ struct internal_list { } internal->sw_lm = sw_fallback_lm; - internal->vdev =rte_vdpa_register_device (&pci_dev->device, &ifcvf_ops); + internal->vdev =rte_vdpa_register_device (&pci_dev->device, + dev_info[internal->hw.device_type].ops);if (internal->vdev ==NULL ) {DRV_LOG (ERR ,"failed to register device %s" , pci_dev->name);drivers/vdpa/ifc/ifcvf_vdpa.c +14
-1goto error;@@ -1313,6 +1374,20 @@ struct internal_list { .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_MODERN_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + { .vendor_id = 0, /* sentinel */ }, }; --1.8.3.1
PATCH v8 2/13 vhost: add vDPA ops for blk device
In reply to Andy PeiGet_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops.
--- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
lib/vhost/vdpa_driver.h +6 -2
@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {/** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev,int qid); -/** Reserved for future extension */ -void *reserved[2 ]; +/** Get the device configuration space */ +int (*get_config)(int vid,uint8_t *config,uint32_t len); + +/** Set the device configuration space */ +int (*set_config)(int vid,uint8_t *config,uint32_t offset, +uint32_t size,uint32_t flags); };/**
Signature · 1 line
1.8.3.1
RE: [PATCH v8 02/13] vhost: add vDPA ops for blk device
Quoted 23 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 02/13] vhost: add vDPA ops for blk device
Get_configandset_configare necessary ops for blk device. Addget_configandset_configops to vDPA ops.Signed-off-by Andy Pei <[email protected]>Reviewed-by Maxime Coquelin <[email protected]>--- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)lib/vhost/vdpa_driver.h +0
-0
lib/vhost/vdpa_driver.h struct rte_vdpa_dev_ops { · line 76
Quoted 7 lines
lib/vhost/vdpa_driver.h +2
-2@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {/** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev,int qid); -/** Reserved for future extension */ -void *reserved[2 ]; +/** Get the device configuration space */ +int (*get_config)(int vid,uint8_t *config,uint32_t len);
One suggestion is renaming 'len' to 'size' to make it aligned to set_config.
With above fixed:
Quoted 9 lines
lib/vhost/vdpa_driver.h +4
-1+ +/** Set the device configuration space */ +int (*set_config)(int vid,uint8_t *config,uint32_t offset, +uint32_t size,uint32_t flags); };/** -- 1.8.3.1
RE: [PATCH v8 02/13] vhost: add vDPA ops for blk device
Hi Chenbo,
Thanks for your reply, I will send out a new version to address this.
Quoted 56 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 11:46 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 02/13] vhost: add vDPA ops for blk deviceQuoted 24 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 02/13] vhost: add vDPA ops for blk device
Get_configandset_configare necessary ops for blk device. Addget_configandset_configops to vDPA ops.Signed-off-by Andy Pei <[email protected]>Reviewed-by Maxime Coquelin <[email protected]>--- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)lib/vhost/vdpa_driver.h index renamed from lib/vhost/vdpa_driver.h +0
-088138be..e59a834 100644
lib/vhost/vdpa_driver.h struct rte_vdpa_dev_ops { · line 76
Quoted 7 lines
lib/vhost/vdpa_driver.h +2
-2@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {/** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev,int qid); -/** Reserved for future extension */ -void *reserved[2 ]; +/** Get the device configuration space */ +int (*get_config)(int vid,uint8_t *config,uint32_t len);One suggestion is renaming 'len' to 'size' to make it aligned to
set_config.With above fixed:
Reviewed-by Chenbo Xia <[email protected]>Quoted 9 lines
lib/vhost/vdpa_driver.h +4
-1+ +/** Set the device configuration space */ +int (*set_config)(int vid,uint8_t *config,uint32_t offset, +uint32_t size,uint32_t flags); };/** -- 1.8.3.1
PATCH v8 3/13 vhost: add vhost msg support for get/set config
In reply to Andy PeiAdd support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device.
--- lib/vhost/vhost_user.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 +++++++++ 2 files changed, 86 insertions(+)
lib/vhost/vhost_user.c +73 -0
@@ -2468,6 +2468,77 @@ static int is_vring_iotlb(struct virtio_net *dev, }static int +vhost_user_get_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config (dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); +if (ret !=0 ) { + ctx->msg.size =0 ; +VHOST_LOG_CONFIG (ERR , +"(%s) get_config() return error!\n" , + dev->ifname); + } + }else { +VHOST_LOG_CONFIG (ERR ,"(%s) get_config() not supported!\n" , + dev->ifname); + } + +return RTE_VHOST_MSG_RESULT_REPLY ; +} + +static int +vhost_user_set_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (ctx->msg.payload.cfg.size >VHOST_USER_MAX_CONFIG_SIZE ) { +VHOST_LOG_CONFIG (ERR , +"(%s) vhost_user_config size: %" PRIu32", should not be larger than %d\n" , + dev->ifname, ctx->msg.payload.cfg.size, +VHOST_USER_MAX_CONFIG_SIZE ); + goto out; + } + +if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config (dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); +if (ret) +VHOST_LOG_CONFIG (ERR , +"(%s) set_config() return error!\n" , + dev->ifname); + }else { +VHOST_LOG_CONFIG (ERR ,"(%s) set_config() not supported!\n" , + dev->ifname); + } + +return RTE_VHOST_MSG_RESULT_OK ; + +out: +return RTE_VHOST_MSG_RESULT_ERR ; +} + +static int vhost_user_iotlb_msg (struct virtio_net **pdev,struct vhu_msg_context *ctx, int main_fd __rte_unused)
@@ -2686,6 +2757,8 @@ static int is_vring_iotlb(struct virtio_net *dev, VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU, vhost_user_net_set_mtu, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD, vhost_user_set_req_fd, true) \ VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG, vhost_user_get_config, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG, vhost_user_set_config, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE, vhost_user_set_postcopy_advise, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false) \VHOST_MESSAGE_HANDLER (VHOST_USER_POSTCOPY_END , vhost_user_postcopy_end, false) \
lib/vhost/vhost_user.h +13 -0
@@ -50,6 +50,8 @@VHOST_USER_NET_SET_MTU =20 ,VHOST_USER_SET_SLAVE_REQ_FD =21 ,VHOST_USER_IOTLB_MSG =22 , +VHOST_USER_GET_CONFIG =24 , +VHOST_USER_SET_CONFIG =25 ,VHOST_USER_CRYPTO_CREATE_SESS =26 ,VHOST_USER_CRYPTO_CLOSE_SESS =27 , VHOST_USER_POSTCOPY_ADVISE = 28,
@@ -123,6 +125,16 @@uint16_t queue_size; } VhostUserInflight; +#define VHOST_USER_MAX_CONFIG_SIZE 256 + +/** Get/set config msg payload */ +struct vhost_user_config { +uint32_t offset; +uint32_t size; +uint32_t flags; +uint8_t region[VHOST_USER_MAX_CONFIG_SIZE ]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */
@@ -146,6 +158,7 @@VhostUserCryptoSessionParam crypto_session;VhostUserVringArea area;VhostUserInflight inflight; +struct vhost_user_config cfg; } payload;/* Nothing should be added after the payload */ }__rte_packed VhostUserMsg;
Signature · 1 line
1.8.3.1
RE: [PATCH v8 03/13] vhost: add vhost msg support for get/set config
Quoted 88 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 03/13] vhost: add vhost msg support for get/set configAdd support for
VHOST_USER_GET_CONFIGandVHOST_USER_SET_CONFIG.VHOST_USER_GET_CONFIGandVHOST_USER_SET_CONFIGmessage is only supported by virtio blk VDPA device.Signed-off-by Andy Pei <[email protected]>--- lib/vhost/vhost_user.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 +++++++++ 2 files changed, 86 insertions(+)lib/vhost/vhost_user.c +59
-0@@ -2468,6 +2468,77 @@ static int is_vring_iotlb(struct virtio_net *dev, }static int +vhost_user_get_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config (dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); +if (ret !=0 ) { + ctx->msg.size =0 ; +VHOST_LOG_CONFIG (ERR , +"(%s) get_config() return error!\n" , + dev->ifname); + } + }else { +VHOST_LOG_CONFIG (ERR ,"(%s) get_config() not supported!\n" , + dev->ifname); + } + +return RTE_VHOST_MSG_RESULT_REPLY ; +} + +static int +vhost_user_set_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (ctx->msg.payload.cfg.size >VHOST_USER_MAX_CONFIG_SIZE ) { +VHOST_LOG_CONFIG (ERR , + "(%s) vhost_user_config size: %"PRIu32", should not be larger than %d\n", + dev->ifname, ctx->msg.payload.cfg.size, +VHOST_USER_MAX_CONFIG_SIZE ); + goto out; + } + +if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config (dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); +if (ret) +VHOST_LOG_CONFIG (ERR , +"(%s) set_config() return error!\n" , + dev->ifname);
lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev, · line 2540
Quoted 10 lines
lib/vhost/vhost_user.c +10
-0+ }else { +VHOST_LOG_CONFIG (ERR ,"(%s) set_config() not supported!\n" , + dev->ifname); + } + +return RTE_VHOST_MSG_RESULT_OK ; + +out: +return RTE_VHOST_MSG_RESULT_ERR ; +}
I missed one thing in previous reviews: vdpa_dev pointer needed to be checked (Not Null) in both set/get config.
Thanks,
Chenbo
Quoted 62 lines
lib/vhost/vhost_user.c +2
-0+ +static int vhost_user_iotlb_msg(struct virtio_net **pdev, struct vhu_msg_context *ctx,lib/vhost/vhost_user.c +1
-0int main_fd __rte_unused)@@ -2686,6 +2757,8 @@ static int is_vring_iotlb(struct virtio_net *dev,VHOST_MESSAGE_HANDLER (VHOST_USER_NET_SET_MTU , vhost_user_net_set_mtu, false) \VHOST_MESSAGE_HANDLER (VHOST_USER_SET_SLAVE_REQ_FD , vhost_user_set_req_fd, true) \ VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false) \ +VHOST_MESSAGE_HANDLER (VHOST_USER_GET_CONFIG , vhost_user_get_config, false)\
+VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG,vhost_user_set_config, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE,vhost_user_set_postcopy_advise, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN,vhost_user_set_postcopy_listen, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END,vhost_user_postcopy_end,
false) \lib/vhost/vhost_user.h +13
-1@@ -50,6 +50,8 @@VHOST_USER_NET_SET_MTU =20 ,VHOST_USER_SET_SLAVE_REQ_FD =21 ,VHOST_USER_IOTLB_MSG =22 , +VHOST_USER_GET_CONFIG =24 , +VHOST_USER_SET_CONFIG =25 ,VHOST_USER_CRYPTO_CREATE_SESS =26 ,VHOST_USER_CRYPTO_CLOSE_SESS =27 , VHOST_USER_POSTCOPY_ADVISE = 28,@@ -123,6 +125,16 @@uint16_t queue_size; } VhostUserInflight; +#define VHOST_USER_MAX_CONFIG_SIZE 256 + +/** Get/set config msg payload */ +struct vhost_user_config { +uint32_t offset; +uint32_t size; +uint32_t flags; +uint8_t region[VHOST_USER_MAX_CONFIG_SIZE ]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */@@ -146,6 +158,7 @@VhostUserCryptoSessionParam crypto_session;VhostUserVringArea area;VhostUserInflight inflight; +struct vhost_user_config cfg; } payload;/* Nothing should be added after the payload */ }__rte_packed VhostUserMsg; --1.8.3.1
RE: [PATCH v8 03/13] vhost: add vhost msg support for get/set config
Hi Chenbo,
Thanks for your reply.
I will check it in next version patch.
Quoted 191 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 11:55 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 03/13] vhost: add vhost msg support for get/set configQuoted 11 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 03/13] vhost: add vhost msg support for get/set configQuoted 1 line
Add support for
VHOST_USER_GET_CONFIGand
VHOST_USER_SET_CONFIG.Quoted 31 lines
VHOST_USER_GET_CONFIGandVHOST_USER_SET_CONFIGmessage is only supported by virtio blk VDPA device.Signed-off-by Andy Pei <[email protected]>--- lib/vhost/vhost_user.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 +++++++++ 2 files changed, 86 insertions(+)lib/vhost/vhost_user.c index renamed from lib/vhost/vhost_user.c +0
-0850848c..1dd1e25f 100644
lib/vhost/vhost_user.c +13
-0@@ -2468,6 +2468,77 @@ static int is_vring_iotlb(struct virtio_net *dev, }static int +vhost_user_get_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config(dev->vid,lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net · line 2493
Quoted 10 lines
lib/vhost/vhost_user.c +10
-0+ ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); +if (ret !=0 ) { + ctx->msg.size =0 ; +VHOST_LOG_CONFIG (ERR , +"(%s) get_config() return error!\n" , + dev->ifname); + } + }else { + VHOST_LOG_CONFIG(ERR, "(%s) get_config() notsupported!\n",
Quoted 11 lines
lib/vhost/vhost_user.c +11
-0+ dev->ifname); + } + +return RTE_VHOST_MSG_RESULT_REPLY ; +} + +static int +vhost_user_set_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev,
Quoted 10 lines
lib/vhost/vhost_user.c +10
-0+struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + + if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) { + VHOST_LOG_CONFIG(ERR, + "(%s) vhost_user_config size: %"PRIu32", should notbe
Quoted 8 lines
lib/vhost/vhost_user.c +7
-0larger than %d\n", + dev->ifname, ctx->msg.payload.cfg.size, + VHOST_USER_MAX_CONFIG_SIZE); + goto out; + } + +if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config(dev->vid,lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev,
Quoted 10 lines
lib/vhost/vhost_user.c +10
-0+ ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); +if (ret) +VHOST_LOG_CONFIG (ERR , +"(%s) set_config() return error!\n" , + dev->ifname); + }else { + VHOST_LOG_CONFIG(ERR, "(%s) set_config() notsupported!\n",
lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev, · line 2539
Quoted 8 lines
lib/vhost/vhost_user.c +8
-0+ dev->ifname); + } + +return RTE_VHOST_MSG_RESULT_OK ; + +out: +return RTE_VHOST_MSG_RESULT_ERR ; +}I missed one thing in previous reviews:
vdpa_devpointer needed to be checked (Not Null) in both set/get config.Thanks,
Chenbolib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net
Quoted 13 lines
lib/vhost/vhost_user.c +2
-0+ +static int vhost_user_iotlb_msg(struct virtio_net **pdev, struct vhu_msg_context *ctx,lib/vhost/vhost_user.c +0
-0int main_fd __rte_unused)@@ -2686,6 +2757,8 @@ static int is_vring_iotlb(struct virtio_net *dev,VHOST_MESSAGE_HANDLER (VHOST_USER_NET_SET_MTU , vhost_user_net_set_mtu, false) \VHOST_MESSAGE_HANDLER (VHOST_USER_SET_SLAVE_REQ_FD , vhost_user_set_req_fd, true) \ VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG,
vhost_user_iotlb_msg,Quoted 2 lines
false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG,
vhost_user_get_config,Quoted 3 lines
+false)
\
+VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG,
vhost_user_set_config,Quoted 29 lines
+false)
\
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE,vhost_user_set_postcopy_advise, false) \VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false)\
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END,vhost_user_postcopy_end,
false) \lib/vhost/vhost_user.h index renamed from lib/vhost/vhost_user.h +0
-0ba1c5c7..c4d091e 100644
lib/vhost/vhost_user.h +5
-0@@ -50,6 +50,8 @@VHOST_USER_NET_SET_MTU =20 ,VHOST_USER_SET_SLAVE_REQ_FD =21 ,VHOST_USER_IOTLB_MSG =22 , +VHOST_USER_GET_CONFIG =24 , +VHOST_USER_SET_CONFIG =25 ,VHOST_USER_CRYPTO_CREATE_SESS =26 ,VHOST_USER_CRYPTO_CLOSE_SESS =27 , VHOST_USER_POSTCOPY_ADVISE = 28,@@ -123,6 +125,16 @@uint16_t queue_size; } VhostUserInflight; +#define VHOST_USER_MAX_CONFIG_SIZE 256 + +/** Get/set config msg payload */ lib/vhost/vhost_user.h line 140
Quoted 10 lines
lib/vhost/vhost_user.h +7
-0+struct vhost_user_config { +uint32_t offset; +uint32_t size; +uint32_t flags; +uint8_t region[VHOST_USER_MAX_CONFIG_SIZE ]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */ @@ -146,6+158,7 @@
Quoted 9 lines
lib/vhost/vhost_user.h +1
-0VhostUserCryptoSessionParam crypto_session;VhostUserVringArea area;VhostUserInflight inflight; +struct vhost_user_config cfg; } payload;/* Nothing should be added after the payload */ } __rte_packedVhostUserMsg;
--
1.8.3.1
PATCH v8 4/13 vdpa/ifc: add blk ops for ifc device
In reply to Andy PeiFor virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.
--- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 91 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 94 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/base/ifcvf.h +4 -0
@@ -67,6 +67,10 @@#define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif +struct ifcvf_pci_cap {u8 cap_vndr;/* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next;/* Generic PCI field: next ptr. */
drivers/vdpa/ifc/ifcvf_vdpa.c +90 -1
109 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RE: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc device
Quoted 62 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc deviceFor virtio blk device, re-use part of ifc driver ops. Implement
ifcvf_blk_get_configfor virtio blk device. SupportVHOST_USER_PROTOCOL_F_CONFIGfeature for virtio blk device.Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 91 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 94 insertions(+), 1 deletion(-)drivers/vdpa/ifc/base/ifcvf.h +4
-0@@ -67,6 +67,10 @@#define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif +struct ifcvf_pci_cap {u8 cap_vndr;/* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next;/* Generic PCI field: next ptr. */ drivers/vdpa/ifc/ifcvf_vdpa.c +4
-0@@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) +static int ifcvf_get_protocol_features (struct rte_vdpa_device *vdev,uint64_t *features){
drivers/vdpa/ifc/ifcvf_vdpa.c +6
-0@@ -1199,6 +1203,91 @@ struct rte_vdpa_dev_info {return device_id; } +static int +ifcvf_blk_get_config (int vid,uint8_t *config,uint32_t len) +{ +struct virtio_blk_config *dev_cfg; +struct ifcvf_internal *internal; +struct rte_vdpa_device *vdev;
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1221
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10
-0+struct internal_list *list; +uint32_t i; +uint64_t capacity =0 ; +uint8_t *byte; + +if (len <sizeof (struct virtio_blk_config )) { +DRV_LOG (ERR ,"Invalid len: %u, required: %u" , + len, (uint32_t )sizeof (struct virtio_blk_config )); +return -1 ; + }
I believe it should not be > sizeof(struct virtio_blk_config) too?
And one question is should we limit only reading the whole blk config. I guess we are having this check because of current usage of QEMU? Although this is fine as it's vendor-specific logic, just wondering if we should have this limit for blk device.
Thanks,
Chenbo
Quoted 84 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +69
-0+ + vdev =rte_vhost_get_vdpa_device (vid); + list =find_internal_resource_by_vdev (vdev); +if (list ==NULL ) { +DRV_LOG (ERR ,"Invalid vDPA device: %p" , vdev); +return -1 ; + } + + internal = list->internal; + +for (i =0 ; i <sizeof (struct virtio_blk_config ); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + +/* cannot read 64-bit register in one attempt, so read byte by byte. */ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +/* The capacity is number of sectors in 512-byte. +* So right shift 1 bit we get in K, +* another right shift 10 bits we get in M, +* right shift 10 more bits, we get in G. +* To show capacity in G, we right shift 21 bits in total. +*/ +DRV_LOG (DEBUG ,"capacity : %" PRIu64"G" , capacity >>21 ); + +DRV_LOG (DEBUG ,"size_max : 0x%08x" , dev_cfg->size_max); +DRV_LOG (DEBUG ,"seg_max : 0x%08x" , dev_cfg->seg_max); +DRV_LOG (DEBUG ,"blk_size : 0x%08x" , dev_cfg->blk_size); +DRV_LOG (DEBUG ,"geometry" ); +DRV_LOG (DEBUG ," cylinders: %u" , dev_cfg->geometry.cylinders); +DRV_LOG (DEBUG ," heads : %u" , dev_cfg->geometry.heads); +DRV_LOG (DEBUG ," sectors : %u" , dev_cfg->geometry.sectors); +DRV_LOG (DEBUG ,"num_queues: 0x%08x" , dev_cfg->num_queues); + +DRV_LOG (DEBUG ,"config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n" , + config[0 ], config[1 ], config[2 ], config[3 ], config[4 ], + config[5 ], config[6 ], config[7 ]); +return 0 ; +} + +static int +ifcvf_blk_get_protocol_features (struct rte_vdpa_device *vdev, +uint64_t *features) +{ +RTE_SET_USED (vdev); + + *features =VDPA_SUPPORTED_PROTOCOL_FEATURES ; + *features |=VDPA_BLK_PROTOCOL_FEATURES ; +return 0 ; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = ifcvf_set_vring_state, + .migration_done =NULL , + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; +struct rte_vdpa_dev_info dev_info[] = { {drivers/vdpa/ifc/ifcvf_vdpa.c +1
-2.features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |@@ -1211,7 +1300,7 @@ struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | (1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; --1.8.3.1
RE: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc device
HI Chenbo,
Thanks for your reply, my reply is inline.
Quoted 74 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 12:08 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc deviceQuoted 65 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc deviceFor virtio blk device, re-use part of ifc driver ops. Implement
ifcvf_blk_get_configfor virtio blk device. SupportVHOST_USER_PROTOCOL_F_CONFIGfeature for virtio blk device.Signed-off-by Andy Pei <[email protected]>---
drivers/vdpa/ifc/base/ifcvf.h| 4 ++drivers/vdpa/ifc/ifcvf_vdpa.c
| 91
++++++++++++++++++++++++++++++++++++++++++-2 files changed, 94 insertions(+), 1 deletion(-)(quoted diff) +0
-0
b/drivers/vdpa/ifc/base/ifcvf.hindex 483d38b..244de46 100644drivers/vdpa/ifc/base/ifcvf.h +4
-0@@ -67,6 +67,10 @@#define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif +struct ifcvf_pci_cap {u8 cap_vndr;/* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next;/* Generic PCI field: next ptr. */ (quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex be0efd3..350214a 100644drivers/vdpa/ifc/ifcvf_vdpa.c +4
-0@@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) +static int ifcvf_get_protocol_features (struct rte_vdpa_device *vdev,uint64_t *features){
drivers/vdpa/ifc/ifcvf_vdpa.c +9
-0@@ -1199,6 +1203,91 @@ struct rte_vdpa_dev_info {return device_id; } +static int +ifcvf_blk_get_config (int vid,uint8_t *config,uint32_t len) { +struct virtio_blk_config *dev_cfg; +struct ifcvf_internal *internal; +struct rte_vdpa_device *vdev; +struct internal_list *list; +uint32_t i; +uint64_t capacity =0 ; +uint8_t *byte;
Quoted 9 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1221
Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +6
-0+ +if (len <sizeof (struct virtio_blk_config )) { +DRV_LOG (ERR ,"Invalid len: %u, required: %u" , + len, (uint32_t )sizeof (struct virtio_blk_config )); +return -1 ; + }I believe it should not be >
sizeof(struct virtio_blk_config)too?
I think I can make it if (len != sizeof(struct virtio_blk_config))
Quoted 5 lines
And one question is should we limit only reading the whole blk config. I guess we are having this check because of current usage of QEMU? Although this is fine as it's vendor-specific logic, just wondering if we should have this limit for blk device.
I think if user want to read any part of the config space,
The args should be (int vid, uint8_t *config, uint32_t offset, uint32_t len). offset mean the offset of struct virtio_blk_config to be the start of user needed data, the len means the data size user need to read.
That will change the message between vhost and QEMU( or other software). I thinks that is another story.
Quoted 89 lines
Thanks,
ChenboQuoted 63 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +62
-0+ + vdev =rte_vhost_get_vdpa_device (vid); + list =find_internal_resource_by_vdev (vdev); +if (list ==NULL ) { +DRV_LOG (ERR ,"Invalid vDPA device: %p" , vdev); +return -1 ; + } + + internal = list->internal; + +for (i =0 ; i <sizeof (struct virtio_blk_config ); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + +/* cannot read 64-bit register in one attempt, so read byte by byte. */ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +/* The capacity is number of sectors in 512-byte. +* So right shift 1 bit we get in K, +* another right shift 10 bits we get in M, +* right shift 10 more bits, we get in G. +* To show capacity in G, we right shift 21 bits in total. +*/ +DRV_LOG (DEBUG ,"capacity : %" PRIu64"G" , capacity >>21 ); + +DRV_LOG (DEBUG ,"size_max : 0x%08x" , dev_cfg->size_max); +DRV_LOG (DEBUG ,"seg_max : 0x%08x" , dev_cfg->seg_max); +DRV_LOG (DEBUG ,"blk_size : 0x%08x" , dev_cfg->blk_size); +DRV_LOG (DEBUG ,"geometry" ); +DRV_LOG (DEBUG ," cylinders: %u" , dev_cfg->geometry.cylinders); +DRV_LOG (DEBUG ," heads : %u" , dev_cfg->geometry.heads); +DRV_LOG (DEBUG ," sectors : %u" , dev_cfg->geometry.sectors); +DRV_LOG (DEBUG ,"num_queues: 0x%08x" , dev_cfg->num_queues); + +DRV_LOG (DEBUG ,"config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n" , + config[0 ], config[1 ], config[2 ], config[3 ], config[4 ], + config[5 ], config[6 ], config[7 ]); +return 0 ; +} + +static int +ifcvf_blk_get_protocol_features (struct rte_vdpa_device *vdev, +uint64_t *features) +{ +RTE_SET_USED (vdev); + + *features =VDPA_SUPPORTED_PROTOCOL_FEATURES ; + *features |=VDPA_BLK_PROTOCOL_FEATURES ; +return 0 ; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = ifcvf_set_vring_state,drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1293
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7
-0+ .migration_done = NULL, + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; +struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |@@ -1211,7
drivers/vdpa/ifc/ifcvf_vdpa.c
Quoted 3 lines
+1300,7 @@ struct
rte_vdpa_dev_infodev_info[] = {drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0{ .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES)|
Quoted 8 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1
-2(1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; --1.8.3.1
PATCH v8 5/13 vdpa/ifc: add vDPA interrupt relay for blk device
In reply to Andy PeiFor the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +8 -1
@@ -369,6 +369,7 @@ struct rte_vdpa_dev_info { irq_set->index =VFIO_PCI_MSIX_IRQ_INDEX ; irq_set->start =0 ; fd_ptr = (int *)&irq_set->data; +/* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET ] =rte_intr_fd_get (internal->pdev->intr_handle);
@@ -378,7 +379,13 @@ struct rte_vdpa_dev_info {for (i =0 ; i < nr_vring; i++) {rte_vhost_get_vhost_vring (internal->vid, i, &vring); fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; -if ((i &1 ) ==0 && m_rx == true) { +if (m_rx == true && + ((i &1 ) ==0 || internal->hw.device_type ==IFCVF_BLK )) { +/* For the net we only need to relay rx queue, +* which will change the mem of VM. +* For the blk we need to relay all the read cmd +* of each queue +*/ fd =eventfd (0 ,EFD_NONBLOCK |EFD_CLOEXEC );if (fd <0 ) { DRV_LOG(ERR, "can't setup eventfd: %s",
Signature · 1 line
1.8.3.1
RE: [PATCH v8 05/13] vdpa/ifc: add vDPA interrupt relay for blk device
Quoted 47 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 05/13] vdpa/ifc: add vDPA interrupt relay for blk deviceFor the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0@@ -369,6 +369,7 @@ struct rte_vdpa_dev_info { irq_set->index =VFIO_PCI_MSIX_IRQ_INDEX ; irq_set->start =0 ; fd_ptr = (int *)&irq_set->data; +/* The first interrupt is for the configure space change notification */
drivers/vdpa/ifc/ifcvf_vdpa.c +7
-1fd_ptr[RTE_INTR_VEC_ZERO_OFFSET ] =rte_intr_fd_get (internal->pdev->intr_handle);@@ -378,7 +379,13 @@ struct rte_vdpa_dev_info {for (i =0 ; i < nr_vring; i++) {rte_vhost_get_vhost_vring (internal->vid, i, &vring); fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; -if ((i &1 ) ==0 && m_rx == true) { +if (m_rx == true && + ((i &1 ) ==0 || internal->hw.device_type ==IFCVF_BLK )) { +/* For the net we only need to relay rx queue, +* which will change the mem of VM. +* For the blk we need to relay all the read cmd +* of each queue +*/ fd =eventfd (0 ,EFD_NONBLOCK |EFD_CLOEXEC );if (fd <0 ) {
Quoted 3 lines
DRV_LOG(ERR, "can't setup eventfd: %s",
--
1.8.3.1
PATCH v8 6/13 vdpa/ifc: add block device SW live-migration
In reply to Andy PeiAdd SW live-migration support to block device. For block device, it is critical that no packet should be dropped. So when virtio blk device is pause, make sure hardware last_avail_idx and last_used_idx are the same. This indicates all requests have recieved acks, and no inflight IO.
--- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 39 insertions(+), 4 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
@@ -65,6 +65,7 @@#define IFCVF_MEDIATED_VRING 0x200000000000#define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff#ifndef VHOST_USER_PROTOCOL_F_CONFIG
drivers/vdpa/ifc/ifcvf_vdpa.c +38 -4
69 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RE: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration
Quoted 9 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration
Quoted 4 lines
Add SW live-migration support to block device.
For block device, it is critical that no packet
should be dropped. So when virtio blk device is
pause, make sure hardwarelast_avail_idxand
paused
Quoted 2 lines
last_used_idxare the same. This indicates all
requests have recieved acks, and no inflight IO.
received
With above fixed:
Quoted 99 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 39 insertions(+), 4 deletions(-)drivers/vdpa/ifc/base/ifcvf.h +1
-0@@ -65,6 +65,7 @@#define IFCVF_MEDIATED_VRING 0x200000000000#define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff#ifndef VHOST_USER_PROTOCOL_F_CONFIG drivers/vdpa/ifc/ifcvf_vdpa.c +38
-5@@ -316,8 +316,34 @@ struct rte_vdpa_dev_info {uint64_t features =0 ;uint64_t log_base =0 , log_size =0 ;uint64_t len; +u32 ring_state =0 ; vid = internal->vid; + +/* to make sure no packet is lost for blk device +* do not stop until last_avail_idx == last_used_idx +*/ +if (internal->hw.device_type ==IFCVF_BLK ) { +for (i =0 ; i < hw->nr_vring; i++) { +do { +if (hw->lm_cfg !=NULL ) + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); + hw->vring[i].last_used_idx = + (u16)(ring_state >>16 ); +if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { +ifcvf_notify_queue (hw, i); +usleep (10 ); + } + }while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); + } + } +ifcvf_stop_hw (hw);for (i =0 ; i < hw->nr_vring; i++)@@ -641,8 +667,10 @@ struct rte_vdpa_dev_info { } hw->vring[i].avail = gpa; -/* Direct I/O for Tx queue, relay for Rx queue */ -if (i &1 ) { +/* NET: Direct I/O for Tx queue, relay for Rx queue +* BLK: relay every queue +*/ +if ((internal->hw.device_type ==IFCVF_NET ) && (i &1 )) { gpa =hva_to_gpa (vid, (uint64_t )(uintptr_t )vq.used);if (gpa ==0 ) {DRV_LOG (ERR ,"Fail to get GPA for used ring." );@@ -692,8 +720,12 @@ struct rte_vdpa_dev_info {for (i =0 ; i < hw->nr_vring; i++) {/* synchronize remaining new used entries if any */ -if ((i &1 ) ==0 ) +if (internal->hw.device_type ==IFCVF_NET ) { +if ((i &1 ) ==0 ) +update_used_ring (internal, i); + }else if (internal->hw.device_type ==IFCVF_BLK ) {update_used_ring (internal, i); + }rte_vhost_get_vhost_vring (vid, i, &vq); len =IFCVF_USED_RING_LEN (vq.size);@@ -755,7 +787,9 @@ struct rte_vdpa_dev_info { } } -for (qid =0 ; qid < q_num; qid +=2 ) { +for (qid =0 ; qid < q_num; qid +=1 ) { +if ((internal->hw.device_type ==IFCVF_NET ) && (qid &1 )) +continue ; ev.events =EPOLLIN |EPOLLPRI ;/* leave a flag to mark it's for interrupt */ ev.data.u64 = 1 | qid << 1 | --1.8.3.1
RE: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration
Hi Chenbo,
Thanks for your reply, I will fix them in next version.
Quoted 137 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 1:25 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migrationQuoted 10 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migrationQuoted 3 lines
Add SW live-migration support to block device.
For block device, it is critical that no packet should be dropped. So when virtio blk device is pause, make sure hardwarelast_avail_idxandpaused
Quoted 2 lines
last_used_idxare the same. This indicates all requests have recieved acks, and no inflight IO.received
With above fixed:
Reviewed-by Chenbo Xia <[email protected]>Quoted 31 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 39 insertions(+), 4 deletions(-)(quoted diff) +0
-0
b/drivers/vdpa/ifc/base/ifcvf.hindex 244de46..4fb1736 100644drivers/vdpa/ifc/base/ifcvf.h +1
-0@@ -65,6 +65,7 @@#define IFCVF_MEDIATED_VRING 0x200000000000#define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff#ifndef VHOST_USER_PROTOCOL_F_CONFIG (quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex 509a1ed..3e78c7d 100644drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0@@ -316,8 +316,34 @@ struct rte_vdpa_dev_info {uint64_t features =0 ;uint64_t log_base =0 , log_size =0 ;uint64_t len; +u32 ring_state =0 ; vid = internal->vid;drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 331
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10
-0+ + /* to make sure no packet is lost for blk device + * do not stop until last_avail_idx == last_used_idx + */ + if (internal->hw.device_type == IFCVF_BLK) { + for (i = 0; i < hw->nr_vring; i++) { + do { + if (hw->lm_cfg != NULL) + ring_state = *(u32 *)(hw->lm_cfg + +
IFCVF_LM_RING_STATE_OFFSET+drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info {
Quoted 3 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +3
-0+ i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state &
IFCVF_16_BIT_MASK);Quoted 18 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +12
-0+ hw->vring[i].last_used_idx = + (u16)(ring_state >>16 ); +if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { +ifcvf_notify_queue (hw, i); +usleep (10 ); + } + }while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); + } + } +ifcvf_stop_hw (hw);for (i =0 ; i < hw->nr_vring; i++)@@ -641,8 +667,10 @@ struct rte_vdpa_dev_info { } hw->vring[i].avail = gpa;drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 676
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +4
-2-/* Direct I/O for Tx queue, relay for Rx queue */ -if (i &1 ) { +/* NET: Direct I/O for Tx queue, relay for Rx queue +* BLK: relay every queue +*/ +if ((internal->hw.device_type ==IFCVF_NET ) && (i &1 )) { gpa =hva_to_gpa (vid, (uint64_t )(uintptr_t )vq.used);if (gpa ==0 ) { DRV_LOG(ERR, "Fail to get GPA for usedring."); @@ -692,8 +720,12
Quoted 4 lines
@@ struct
rte_vdpa_dev_info{drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0for (i =0 ; i < hw->nr_vring; i++) {/* synchronize remaining new used entries if any */ drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info {
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5
-1-if ((i &1 ) ==0 ) +if (internal->hw.device_type ==IFCVF_NET ) { +if ((i &1 ) ==0 ) +update_used_ring (internal, i); + }else if (internal->hw.device_type ==IFCVF_BLK ) {update_used_ring (internal, i); + }rte_vhost_get_vhost_vring (vid, i, &vq); len = IFCVF_USED_RING_LEN(vq.size); @@ -755,7 +787,9@@ struct
Quoted 13 lines
rte_vdpa_dev_info{drivers/vdpa/ifc/ifcvf_vdpa.c +3
-2} } -for (qid =0 ; qid < q_num; qid +=2 ) { +for (qid =0 ; qid < q_num; qid +=1 ) { +if ((internal->hw.device_type ==IFCVF_NET ) && (qid &1 )) +continue ; ev.events =EPOLLIN |EPOLLPRI ;/* leave a flag to mark it's for interrupt */ ev.data.u64 = 1 | qid << 1 | --1.8.3.1
PATCH v8 7/13 vhost: add API to get vDPA device type
In reply to Andy PeiVhost backend of different devices have different features. Add a API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices.
--- doc/guides/prog_guide/vhost_lib.rst | 5 ++++ doc/guides/rel_notes/release_22_07.rst | 4 ++++ lib/vhost/rte_vhost.h | 17 +++++++++++++ lib/vhost/socket.c | 44 ++++++++++++++++++++++++++++++++++ lib/vhost/vdpa_driver.h | 3 +++ lib/vhost/version.map | 1 + 6 files changed, 74 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +5 -0
@@ -282,6 +282,11 @@ The following is an overview of some key Vhost API functions: Clear inflight packets which are submitted to DMA engine in vhost async data path. Completed packets are returned to applications through ``pkts``. +* ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` + + Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, + VDPA_DEVICE_TYPE_BLK. + Vhost-user Implementations --------------------------
doc/guides/rel_notes/release_22_07.rst +4 -0
@@ -104,6 +104,10 @@ New Features * ``RTE_EVENT_QUEUE_ATTR_WEIGHT`` * ``RTE_EVENT_QUEUE_ATTR_AFFINITY`` +* **Added vhost API to get the device type of a vDPA device.** + + Added an API which can get the device type of vDPA device. + Removed Items -------------
lib/vhost/rte_vhost.h +17 -0
@@ -117,6 +117,9 @@#define RTE_MAX_VHOST_DEVICE 1024 +#define VDPA_DEVICE_TYPE_NET 0 +#define VDPA_DEVICE_TYPE_BLK 1 +struct rte_vdpa_device ;/**
@@ -486,6 +489,20 @@ struct rte_vdpa_device * rte_vhost_driver_get_vdpa_device(const char *path);/** +* Get the device type of the vdpa device. +* +* @param path +* The vhost-user socket file path +* @param type +* the device type of the vdpa device +* @return +* 0 on success, -1 on failure +*/ +__rte_experimental +int +rte_vhost_driver_get_vdpa_dev_type (const char *path,uint32_t *type); + +/** * Set the feature bits the vhost-user driver supports. * * @param path
lib/vhost/socket.c +44 -0
50 lines not on this page · raw message
lib/vhost/vdpa_driver.h +3 -0
@@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops {/** Set the device configuration space */ int (*set_config)(int vid,uint8_t *config,uint32_t offset,uint32_t size,uint32_t flags); + +/** get device type: net device, blk device... */ +int (*get_dev_type)(struct rte_vdpa_device *dev,uint32_t *type); };/**
lib/vhost/version.map +1 -0
@@ -90,6 +90,7 @@ EXPERIMENTAL { # added in 22.07 rte_vhost_async_get_inflight_thread_unsafe; + rte_vhost_driver_get_vdpa_dev_type; };
Signature · 1 line
1.8.3.1
RE: [PATCH v8 07/13] vhost: add API to get vDPA device type
Quoted 11 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 07/13] vhost: add API to get vDPA device typeVhost backend of different devices have different features. Add a API to get vDPA device type, net device or blk device
an API
Quoted 39 lines
currently, so users can set different features for different kinds of devices.
Signed-off-by Andy Pei <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 5 ++++ doc/guides/rel_notes/release_22_07.rst | 4 ++++ lib/vhost/rte_vhost.h | 17 +++++++++++++ lib/vhost/socket.c | 44 ++++++++++++++++++++++++++++++++++ lib/vhost/vdpa_driver.h | 3 +++ lib/vhost/version.map | 1 + 6 files changed, 74 insertions(+)doc/guides/prog_guide/vhost_lib.rst +0
-0@@ -282,6 +282,11 @@ The following is an overview of some key Vhost APIfunctions:
Clear inflight packets which are submitted to DMA engine in vhost async
data
path. Completed packets are returned to applications through ``pkts``.doc/guides/prog_guide/vhost_lib.rst +5
-0+* ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` + + Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, + VDPA_DEVICE_TYPE_BLK. + Vhost-user Implementations --------------------------doc/guides/rel_notes/release_22_07.rst +0
-0
doc/guides/rel_notes/release_22_07.rst New Features · line 110
Quoted 7 lines
doc/guides/rel_notes/release_22_07.rst +4
-0@@ -104,6 +104,10 @@ New Features * ``RTE_EVENT_QUEUE_ATTR_WEIGHT`` * ``RTE_EVENT_QUEUE_ATTR_AFFINITY`` +* **Added vhost API to get the device type of a vDPA device.** + + Added an API which can get the device type of vDPA device. +
Release note has a requirement of order, check comment under 'New features' in release notes.
Based on these comments, you should put this after 'Added vhost API to get the number of in-flight packets'
lib/vhost/rte_vhost.h line 121
Quoted 13 lines
doc/guides/rel_notes/release_22_07.rst +0
-0Removed Items -------------lib/vhost/rte_vhost.h +2
-0@@ -117,6 +117,9 @@#define RTE_MAX_VHOST_DEVICE 1024 +#define VDPA_DEVICE_TYPE_NET 0 +#define VDPA_DEVICE_TYPE_BLK 1
These will be new APIs of vhost lib. I suggest to rename them by adding prefix 'RTE_VHOST_'
Thanks,
Chenbo
Quoted 109 lines
lib/vhost/rte_vhost.h +15
-0+struct rte_vdpa_device ;/** @@ -486,6 +489,20 @@ struct rte_vdpa_device * rte_vhost_driver_get_vdpa_device(const char *path);/** +* Get the device type of the vdpa device. +* +* @param path +* The vhost-user socket file path +* @param type +* the device type of the vdpa device +* @return +* 0 on success, -1 on failure +*/ +__rte_experimental +int +rte_vhost_driver_get_vdpa_dev_type (const char *path,uint32_t *type); + +/** * Set the feature bits the vhost-user driver supports. * * @param path lib/vhost/socket.c +44
-0@@ -619,6 +619,50 @@ struct rte_vdpa_device * }int +rte_vhost_driver_get_vdpa_dev_type (const char *path,uint32_t *type) +{ +struct vhost_user_socket *vsocket; +struct rte_vdpa_device *vdpa_dev; +uint32_t vdpa_type =0 ; +int ret =0 ; + +pthread_mutex_lock (&vhost_user.mutex); + vsocket =find_vhost_user_socket (path); +if (!vsocket) { +VHOST_LOG_CONFIG (ERR , +"(%s) socket file is not registered yet.\n" , + path); + ret =-1 ; + goto unlock_exit; + } + + vdpa_dev = vsocket->vdpa_dev; +if (!vdpa_dev) { + ret =-1 ; + goto unlock_exit; + } + +if (vdpa_dev->ops->get_dev_type) { + ret = vdpa_dev->ops->get_dev_type (vdpa_dev, &vdpa_type); +if (ret) { +VHOST_LOG_CONFIG (ERR , + "(%s) failed to get vdpa dev type for socket file.\n", + path); + ret =-1 ; + goto unlock_exit; + } + }else { + vdpa_type =VDPA_DEVICE_TYPE_NET ; + } + + *type = vdpa_type; + +unlock_exit: +pthread_mutex_unlock (&vhost_user.mutex); +return ret; +} + +int rte_vhost_driver_disable_features (const char *path,uint64_t features) {lib/vhost/socket.c +0
-0struct vhost_user_socket *vsocket;lib/vhost/vdpa_driver.h +3
-0@@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops {/** Set the device configuration space */ int (*set_config)(int vid,uint8_t *config,uint32_t offset,uint32_t size,uint32_t flags); + +/** get device type: net device, blk device... */ +int (*get_dev_type)(struct rte_vdpa_device *dev,uint32_t *type); };/** lib/vhost/version.map +1
-1@@ -90,6 +90,7 @@ EXPERIMENTAL { # added in 22.07 rte_vhost_async_get_inflight_thread_unsafe; + rte_vhost_driver_get_vdpa_dev_type; }; --1.8.3.1
RE: [PATCH v8 07/13] vhost: add API to get vDPA device type
Hi Chenbo,
Thanks for your reply.
I will send a new version to address these.
Quoted 207 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 3:26 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 07/13] vhost: add API to get vDPA device typeQuoted 10 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 07/13] vhost: add API to get vDPA device typeQuoted 2 lines
Vhost backend of different devices have different features. Add a API to get vDPA device type, net device or blk device
an API
Quoted 40 lines
currently, so users can set different features for different kinds of devices.
Signed-off-by Andy Pei <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 5 ++++ doc/guides/rel_notes/release_22_07.rst | 4 ++++ lib/vhost/rte_vhost.h | 17 +++++++++++++ lib/vhost/socket.c | 44 ++++++++++++++++++++++++++++++++++ lib/vhost/vdpa_driver.h | 3 +++ lib/vhost/version.map | 1 + 6 files changed, 74 insertions(+)doc/guides/prog_guide/vhost_lib.rst +0
-0@@ -282,6 +282,11 @@ The following is an overview of some key VhostAPI
functions:
Clear inflight packets which are submitted to DMA engine in vhost
async data
path. Completed packets are returned to applications through ``pkts``.doc/guides/prog_guide/vhost_lib.rst +5
-0+* ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` + + Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, + VDPA_DEVICE_TYPE_BLK. + Vhost-user Implementations --------------------------doc/guides/rel_notes/release_22_07.rst +0
-0doc/guides/rel_notes/release_22_07.rst New Features · line 110
Quoted 7 lines
doc/guides/rel_notes/release_22_07.rst +4
-0@@ -104,6 +104,10 @@ New Features * ``RTE_EVENT_QUEUE_ATTR_WEIGHT`` * ``RTE_EVENT_QUEUE_ATTR_AFFINITY`` +* **Added vhost API to get the device type of a vDPA device.** + + Added an API which can get the device type of vDPA device. +Release note has a requirement of order, check comment under 'New features'
in release notes.Based on these comments, you should put this after 'Added vhost API to get the number of in-flight packets'
lib/vhost/rte_vhost.h line 121
Quoted 13 lines
doc/guides/rel_notes/release_22_07.rst +0
-0Removed Items -------------lib/vhost/rte_vhost.h index renamed from lib/vhost/rte_vhost.h +0
-0c733f85..c977a24 100644
lib/vhost/rte_vhost.h +2
-0@@ -117,6 +117,9 @@#define RTE_MAX_VHOST_DEVICE 1024 +#define VDPA_DEVICE_TYPE_NET 0 +#define VDPA_DEVICE_TYPE_BLK 1These will be new APIs of vhost lib. I suggest to rename them by adding prefix '
RTE_VHOST_'Thanks,
ChenboQuoted 108 lines
lib/vhost/rte_vhost.h +15
-0+struct rte_vdpa_device ;/** @@ -486,6 +489,20 @@ struct rte_vdpa_device * rte_vhost_driver_get_vdpa_device(const char *path);/** +* Get the device type of the vdpa device. +* +* @param path +* The vhost-user socket file path +* @param type +* the device type of the vdpa device +* @return +* 0 on success, -1 on failure +*/ +__rte_experimental +int +rte_vhost_driver_get_vdpa_dev_type (const char *path,uint32_t *type); + +/** * Set the feature bits the vhost-user driver supports. * * @param path lib/vhost/socket.c index renamed from lib/vhost/socket.c +0
-0b304339..ef0f401 100644
lib/vhost/socket.c +44
-0@@ -619,6 +619,50 @@ struct rte_vdpa_device * }int +rte_vhost_driver_get_vdpa_dev_type (const char *path,uint32_t *type) +{ +struct vhost_user_socket *vsocket; +struct rte_vdpa_device *vdpa_dev; +uint32_t vdpa_type =0 ; +int ret =0 ; + +pthread_mutex_lock (&vhost_user.mutex); + vsocket =find_vhost_user_socket (path); +if (!vsocket) { +VHOST_LOG_CONFIG (ERR , +"(%s) socket file is not registered yet.\n" , + path); + ret =-1 ; + goto unlock_exit; + } + + vdpa_dev = vsocket->vdpa_dev; +if (!vdpa_dev) { + ret =-1 ; + goto unlock_exit; + } + +if (vdpa_dev->ops->get_dev_type) { + ret = vdpa_dev->ops->get_dev_type (vdpa_dev, &vdpa_type); +if (ret) { +VHOST_LOG_CONFIG (ERR , + "(%s) failed to get vdpa dev type for socket file.\n", + path); + ret =-1 ; + goto unlock_exit; + } + }else { + vdpa_type =VDPA_DEVICE_TYPE_NET ; + } + + *type = vdpa_type; + +unlock_exit: +pthread_mutex_unlock (&vhost_user.mutex); +return ret; +} + +int rte_vhost_driver_disable_features (const char *path,uint64_t features) {struct vhost_user_socket *vsocket;lib/vhost/vdpa_driver.h index renamed from lib/vhost/vdpa_driver.h +0
-0e59a834..9cbd7cd 100644
lib/vhost/vdpa_driver.h +3
-0@@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops {/** Set the device configuration space */ int (*set_config)(int vid,uint8_t *config,uint32_t offset,uint32_t size,uint32_t flags); + +/** get device type: net device, blk device... */ +int (*get_dev_type)(struct rte_vdpa_device *dev,uint32_t *type); };/** lib/vhost/version.map index renamed from lib/vhost/version.map +0
-05841315..583b4f3 100644
lib/vhost/version.map +1
-1@@ -90,6 +90,7 @@ EXPERIMENTAL { # added in 22.07 rte_vhost_async_get_inflight_thread_unsafe; + rte_vhost_driver_get_vdpa_dev_type; }; --1.8.3.1
PATCH v8 8/13 vdpa/ifc: add get device type ops to ifc driver
In reply to Andy PeiAdd get device type ops to ifc driver.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +25 -0
@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {return 0 ; } +static int +ifcvf_get_device_type (struct rte_vdpa_device *vdev, +uint32_t *type) +{ +struct ifcvf_internal *internal; +struct internal_list *list; + + list =find_internal_resource_by_vdev (vdev); +if (list ==NULL ) { +DRV_LOG (ERR ,"Invalid vDPA device: %p" , vdev); +return -1 ; + } + + internal = list->internal; + +if (internal->hw.device_type ==IFCVF_BLK ) + *type =VDPA_DEVICE_TYPE_BLK ; +else + *type =VDPA_DEVICE_TYPE_NET ; + +return 0 ; +} +static struct rte_vdpa_dev_ops ifcvf_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features,
@@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info { .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, + .get_dev_type = ifcvf_get_device_type, }; static inline int
@@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info { .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_get_device_type, };struct rte_vdpa_dev_info dev_info[] = {
Signature · 1 line
1.8.3.1
RE: [PATCH v8 08/13] vdpa/ifc: add get device type ops to ifc driver
Quoted 67 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 08/13] vdpa/ifc: add get device type ops to ifc driverAdd get device type ops to ifc driver.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)drivers/vdpa/ifc/ifcvf_vdpa.c +25
-1@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {return 0 ; } +static int +ifcvf_get_device_type (struct rte_vdpa_device *vdev, +uint32_t *type) +{ +struct ifcvf_internal *internal; +struct internal_list *list; + + list =find_internal_resource_by_vdev (vdev); +if (list ==NULL ) { +DRV_LOG (ERR ,"Invalid vDPA device: %p" , vdev); +return -1 ; + } + + internal = list->internal; + +if (internal->hw.device_type ==IFCVF_BLK ) + *type =VDPA_DEVICE_TYPE_BLK ; +else + *type =VDPA_DEVICE_TYPE_NET ; + +return 0 ; +} +static struct rte_vdpa_dev_ops ifcvf_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features,@@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info { .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, + .get_dev_type = ifcvf_get_device_type, }; static inline int@@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info { .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_get_device_type, };struct rte_vdpa_dev_info dev_info[] = { --
Quoted 1 line
1.8.3.1
With renaming fixed mentioned in patch 7:
RE: [PATCH v8 08/13] vdpa/ifc: add get device type ops to ifc driver
Sure
Quoted 83 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 3:30 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 08/13] vdpa/ifc: add get device type ops to ifc driverQuoted 69 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 08/13] vdpa/ifc: add get device type ops to ifc driverAdd get device type ops to ifc driver.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)(quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex 3e78c7d..0acfa8c 100644drivers/vdpa/ifc/ifcvf_vdpa.c +23
-0@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {return 0 ; } +static int +ifcvf_get_device_type (struct rte_vdpa_device *vdev, +uint32_t *type) +{ +struct ifcvf_internal *internal; +struct internal_list *list; + + list =find_internal_resource_by_vdev (vdev); +if (list ==NULL ) { +DRV_LOG (ERR ,"Invalid vDPA device: %p" , vdev); +return -1 ; + } + + internal = list->internal; + +if (internal->hw.device_type ==IFCVF_BLK ) + *type =VDPA_DEVICE_TYPE_BLK ; +else + *type =VDPA_DEVICE_TYPE_NET ; + +return 0 ; +} +static struct rte_vdpa_dev_ops ifcvf_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features, @@ -1201,6 +1224,7 @@struct
rte_vdpa_dev_info{drivers/vdpa/ifc/ifcvf_vdpa.c +2
-1.get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, + .get_dev_type = ifcvf_get_device_type, }; static inline int@@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info { .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_get_device_type, };struct rte_vdpa_dev_info dev_info[] = { --Quoted 1 line
1.8.3.1
With renaming fixed mentioned in patch 7:
Reviewed-by Chenbo Xia <[email protected]>
PATCH v8 9/13 examples/vdpa: add add virtio blk support
In reply to Andy PeiAdd virtio blk device support to vDPA example.
--- examples/vdpa/main.c | 56 ++++++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 58 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 114 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c +56 -0
@@ -20,6 +20,7 @@#include <cmdline_parse_string.h> #include <cmdline_parse_num.h> #include <cmdline.h> +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128#define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,53 @@ struct vdpa_port { };static int +vdpa_blk_device_set_features_and_protocol (const char *path) +{ +uint64_t protocol_features =0 ; +int ret; + + ret =rte_vhost_driver_set_features (path,VHOST_BLK_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_disable_features (path, +VHOST_BLK_DISABLED_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_disable_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_get_protocol_features (path, &protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_get_protocol_features for %s failed.\n" , + path); + goto out; + } + + protocol_features |=VHOST_BLK_PROTOCOL_FEATURES ; + + ret =rte_vhost_driver_set_protocol_features (path, protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_protocol_features for %s failed.\n" , + path); + } + +out: +return ret; +} + +static int start_vdpa (struct vdpa_port *vport) { +uint32_t device_type =0 ;int ret;char *socket_path = vport->ifname;
@@ -192,6 +238,16 @@ struct vdpa_port {"attach vdpa device failed: %s\n" , socket_path); + ret =rte_vhost_driver_get_vdpa_dev_type (socket_path, &device_type); +if (ret ==0 && device_type ==VDPA_DEVICE_TYPE_BLK ) { +RTE_LOG (NOTICE ,VDPA ,"%s is a blk device\n" , socket_path); + ret =vdpa_blk_device_set_features_and_protocol (socket_path); +if (ret !=0 ) +rte_exit (EXIT_FAILURE , +"set vhost blk driver features and protocol features failed: %s\n" , + socket_path); + } +if (rte_vhost_driver_start (socket_path) <0 ) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n",
examples/vdpa/vdpa_blk_compact.h new file +58 -0
58 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RE: [PATCH v8 09/13] examples/vdpa: add add virtio blk support
Quoted 8 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 09/13] examples/vdpa: add add virtio blk support
Double add in the title
Quoted 109 lines
Add virtio blk device support to vDPA example.
Signed-off-by Andy Pei <[email protected]>--- examples/vdpa/main.c | 56 ++++++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 58 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 114 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.hexamples/vdpa/main.c +56
-0@@ -20,6 +20,7 @@#include <cmdline_parse_string.h> #include <cmdline_parse_num.h> #include <cmdline.h> +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128#define MAX_VDPA_SAMPLE_PORTS 1024@@ -159,8 +160,53 @@ struct vdpa_port { };static int +vdpa_blk_device_set_features_and_protocol (const char *path) +{ +uint64_t protocol_features =0 ; +int ret; + + ret =rte_vhost_driver_set_features (path,VHOST_BLK_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_disable_features (path, +VHOST_BLK_DISABLED_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_disable_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_get_protocol_features (path, &protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , + "rte_vhost_driver_get_protocol_features for %s failed.\n", + path); + goto out; + } + + protocol_features |=VHOST_BLK_PROTOCOL_FEATURES ; + + ret =rte_vhost_driver_set_protocol_features (path, protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , + "rte_vhost_driver_set_protocol_features for %s failed.\n", + path); + } + +out: +return ret; +} + +static int start_vdpa (struct vdpa_port *vport) { +uint32_t device_type =0 ;int ret;char *socket_path = vport->ifname;@@ -192,6 +238,16 @@ struct vdpa_port {"attach vdpa device failed: %s\n" , socket_path); + ret =rte_vhost_driver_get_vdpa_dev_type (socket_path, &device_type); +if (ret ==0 && device_type ==VDPA_DEVICE_TYPE_BLK ) { +RTE_LOG (NOTICE ,VDPA ,"%s is a blk device\n" , socket_path); + ret =vdpa_blk_device_set_features_and_protocol (socket_path); +if (ret !=0 ) +rte_exit (EXIT_FAILURE , + "set vhost blk driver features and protocol features failed: %s\n", + socket_path); + } +if (rte_vhost_driver_start (socket_path) <0 ) rte_exit(EXIT_FAILURE,examples/vdpa/main.c +0
-0"start vhost driver failed: %s\n",examples/vdpa/vdpa_blk_compact.h new file +2
-0@@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2022 Intel Corporation
examples/vdpa/vdpa_blk_compact.h line 12
Quoted 10 lines
examples/vdpa/vdpa_blk_compact.h new file +10
-0+*/ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_ + +/** +* @file +* +* Device specific vhost lib +*/
You describe the file with more details, or maybe just delete it.
Quoted 13 lines
examples/vdpa/vdpa_blk_compact.h new file +5
-0+ +#include <rte_vhost.h> + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1/* Indicates maximum segment size */
+#defineVIRTIO_BLK_F_SEG_MAX2 /* Indicates maximum # of segments
*/examples/vdpa/vdpa_blk_compact.h new file +2
-0+#define VIRTIO_BLK_F_GEOMETRY 4/* Legacy geometry available */ +#define VIRTIO_BLK_F_BLK_SIZE 6/* Block size of disk is available */
+#defineVIRTIO_BLK_F_TOPOLOGY10 /* Topology information is
available */
examples/vdpa/vdpa_blk_compact.h line 25
Quoted 4 lines
examples/vdpa/vdpa_blk_compact.h new file +4
-0+#define VIRTIO_BLK_F_MQ 12/* support more than one vq */ + +/* Legacy feature bits */ +#ifndef VIRTIO_BLK_NO_LEGACY
This is always true? Is it possible VIRTIO_BLK_NO_LEGACY get defined?
Thanks,
Chenbo
Quoted 41 lines
examples/vdpa/vdpa_blk_compact.h new file +2
-0+#define VIRTIO_BLK_F_BARRIER 0/* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7/* Supports scsi command passthru */
+#defineVIRTIO_BLK_F_CONFIG_WCE11 /* Writeback mode available in
config */examples/vdpa/vdpa_blk_compact.h new file +10
-0+#endif /* !VIRTIO_BLK_NO_LEGACY */ + +#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1)) + +#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL <<
VIRTIO_F_NOTIFY_ON_EMPTY) | \examples/vdpa/vdpa_blk_compact.h new file +5
-0+ (1ULL << VIRTIO_RING_F_EVENT_IDX)) + +#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) |\
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER) |\
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL <<VIRTIO_BLK_F_SCSI ) | (1ULL <<VIRTIO_BLK_F_CONFIG_WCE )| \
examples/vdpa/vdpa_blk_compact.h new file +13
-1+ (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define VHOST_BLK_DISABLED_FEATURES (VHOST_BLK_DISABLED_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE)) + +/* Vhost-blk support protocol features */ +#define VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG)) + +#endif /* _VDPA_BLK_COMPACT_H_ */ --1.8.3.1
RE: [PATCH v8 09/13] examples/vdpa: add add virtio blk support
Hi Chenbo,
Thanks for your reply, my reply is inline.
Quoted 19 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 3:40 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 09/13] examples/vdpa: add add virtio blk supportQuoted 9 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 09/13] examples/vdpa: add add virtio blk support
Quoted 2 lines
Double add in the title
I will fix this in next version.
Quoted 112 lines
Quoted 81 lines
Add virtio blk device support to vDPA example.
Signed-off-by Andy Pei <[email protected]>--- examples/vdpa/main.c | 56 ++++++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 58 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 114 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.hexamples/vdpa/main.c index renamed from examples/vdpa/main.c +0
-05ab0765..62b6660 100644
examples/vdpa/main.c +45
-0@@ -20,6 +20,7 @@#include <cmdline_parse_string.h> #include <cmdline_parse_num.h> #include <cmdline.h> +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128#define MAX_VDPA_SAMPLE_PORTS 1024@@ -159,8 +160,53 @@ struct vdpa_port { };static int +vdpa_blk_device_set_features_and_protocol (const char *path) { +uint64_t protocol_features =0 ; +int ret; + + ret =rte_vhost_driver_set_features (path,VHOST_BLK_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_disable_features (path, +VHOST_BLK_DISABLED_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_disable_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_get_protocol_features (path, &protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , + "rte_vhost_driver_get_protocol_features for %s failed.\n", + path); + goto out; + } + + protocol_features |=VHOST_BLK_PROTOCOL_FEATURES ; + + ret =rte_vhost_driver_set_protocol_features (path, protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , + "rte_vhost_driver_set_protocol_features for %s failed.\n", + path); + } + +out: +return ret; +} + +static int start_vdpa (struct vdpa_port *vport) { +uint32_t device_type =0 ;int ret;char *socket_path = vport->ifname;examples/vdpa/main.c struct vdpa_port { · line 241
Quoted 4 lines
examples/vdpa/main.c +1
-0@@ -192,6 +238,16 @@ struct vdpa_port {"attach vdpa device failed: %s\n" , socket_path); + ret = rte_vhost_driver_get_vdpa_dev_type(socket_path,
&device_type);examples/vdpa/main.c struct vdpa_port {
Quoted 3 lines
examples/vdpa/main.c +3
-0+if (ret ==0 && device_type ==VDPA_DEVICE_TYPE_BLK ) { +RTE_LOG (NOTICE ,VDPA ,"%s is a blk device\n" , socket_path); + ret =
vdpa_blk_device_set_features_and_protocol(socket_path);Quoted 22 lines
examples/vdpa/main.c +6
-0+if (ret !=0 ) +rte_exit (EXIT_FAILURE , + "set vhost blk driver features and protocol features failed: %s\n", + socket_path); + } +if (rte_vhost_driver_start (socket_path) <0 ) rte_exit(EXIT_FAILURE,"start vhost driver failed: %s\n", diff --git
a/examples/vdpa/vdpa_blk_compact.hb/examples/vdpa/vdpa_blk_compact.h
new file mode 100644
index 0000000..4193561examples/vdpa/vdpa_blk_compact.h new file +5
-0@@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2022 Intel Corporation */ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_
Quoted 9 lines
examples/vdpa/vdpa_blk_compact.h line 12
Quoted 6 lines
examples/vdpa/vdpa_blk_compact.h new file +6
-0+ +/** +* @file +* +* Device specific vhost lib +*/ You describe the file with more details, or maybe just delete it.
I will delete it in next version.
Quoted 9 lines
examples/vdpa/vdpa_blk_compact.h
Quoted 5 lines
examples/vdpa/vdpa_blk_compact.h new file +5
-0+ +#include <rte_vhost.h> + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1/* Indicates maximum segment size
Quoted 2 lines
*/
+#defineVIRTIO_BLK_F_SEG_MAX2 /* Indicates maximum # ofsegments
Quoted 13 lines
examples/vdpa/vdpa_blk_compact.h line 25
Quoted 10 lines
*/
examples/vdpa/vdpa_blk_compact.h new file +2
-0+#define VIRTIO_BLK_F_GEOMETRY 4/* Legacy geometry available */ +#define VIRTIO_BLK_F_BLK_SIZE 6/* Block size of disk is available */
+#defineVIRTIO_BLK_F_TOPOLOGY10 /* Topology information is
available */examples/vdpa/vdpa_blk_compact.h new file +4
-0+#define VIRTIO_BLK_F_MQ 12/* support more than one vq */ + +/* Legacy feature bits */ +#ifndef VIRTIO_BLK_NO_LEGACY This is always true? Is it possible
VIRTIO_BLK_NO_LEGACYget defined?
The I will delete this condition.
Quoted 51 lines
Thanks,
ChenboQuoted 16 lines
examples/vdpa/vdpa_blk_compact.h new file +2
-0+#define VIRTIO_BLK_F_BARRIER 0/* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7/* Supports scsi command passthru */
+#defineVIRTIO_BLK_F_CONFIG_WCE11 /* Writeback mode available in
config */examples/vdpa/vdpa_blk_compact.h new file +10
-0+#endif /* !VIRTIO_BLK_NO_LEGACY */ + +#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1)) + +#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL <<
VIRTIO_F_NOTIFY_ON_EMPTY) | \examples/vdpa/vdpa_blk_compact.h
Quoted 4 lines
examples/vdpa/vdpa_blk_compact.h new file +4
-0+ (1ULL << VIRTIO_RING_F_EVENT_IDX)) + +#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL <<
VIRTIO_BLK_F_SEG_MAX) | \examples/vdpa/vdpa_blk_compact.h
Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL <<
VIRTIO_BLK_F_BLK_SIZE) |examples/vdpa/vdpa_blk_compact.h
Quoted 2 lines
\
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL <<
VIRTIO_BLK_F_BARRIER) |examples/vdpa/vdpa_blk_compact.h
Quoted 2 lines
\
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL <<
VIRTIO_BLK_F_CONFIG_WCE)examples/vdpa/vdpa_blk_compact.h
Quoted 5 lines
| \
examples/vdpa/vdpa_blk_compact.h new file +4
-0+ (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define VHOST_BLK_DISABLED_FEATURES (
VHOST_BLK_DISABLED_FEATURES_BASE| \examples/vdpa/vdpa_blk_compact.h
Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL <<
VIRTIO_BLK_F_BARRIER) | \examples/vdpa/vdpa_blk_compact.h
Quoted 1 line
examples/vdpa/vdpa_blk_compact.h new file +1
-0+ (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL <<
VIRTIO_BLK_F_CONFIG_WCE))Quoted 9 lines
examples/vdpa/vdpa_blk_compact.h new file +7
-1+ +/* Vhost-blk support protocol features */ #define +VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG)) + +#endif /* _VDPA_BLK_COMPACT_H_ */ --1.8.3.1
PATCH v8 10/13 usertools: add support for virtio blk device
In reply to Andy PeiAdd virtio blk device support to devbind.
--- usertools/dpdk-devbind.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-devbind.py +4 -1
@@ -72,6 +72,9 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class]
@@ -82,7 +85,7 @@ compress_devices = [cavium_zip] regex_devices = [cn9k_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, - intel_ntb_skx, intel_ntb_icx] + intel_ntb_skx, intel_ntb_icx, virtio_blk] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
Signature · 1 line
1.8.3.1
RE: [PATCH v8 10/13] usertools: add support for virtio blk device
Quoted 21 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 10/13] usertools: add support for virtio blk deviceAdd virtio blk device support to devbind.
Signed-off-by Andy Pei <[email protected]>--- usertools/dpdk-devbind.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)usertools/dpdk-devbind.py +0
-0
usertools/dpdk-devbind.py line 75
Quoted 4 lines
usertools/dpdk-devbind.py +1
-0@@ -72,6 +72,9 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001',
No modern device support?
Thanks,
Chenbo
Quoted 16 lines
usertools/dpdk-devbind.py +3
-2+ 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class]@@ -82,7 +85,7 @@ compress_devices = [cavium_zip] regex_devices = [cn9k_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, - intel_ntb_skx, intel_ntb_icx] + intel_ntb_skx, intel_ntb_icx, virtio_blk] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties --1.8.3.1
RE: [PATCH v8 10/13] usertools: add support for virtio blk device
Hi Chenbo,
Quoted 34 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 3:43 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 10/13] usertools: add support for virtio blk deviceQuoted 25 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 10/13] usertools: add support for virtio blk deviceAdd virtio blk device support to devbind.
Signed-off-by Andy Pei <[email protected]>--- usertools/dpdk-devbind.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)usertools/dpdk-devbind.py +0
-0@@ -72,6 +72,9 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None}
Quoted 4 lines
usertools/dpdk-devbind.py line 75
Quoted 1 line
usertools/dpdk-devbind.py +1
-0+virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001',No modern device support?
It should be supported. I will add modern device in next version.
Quoted 17 lines
Thanks,
ChenboQuoted 14 lines
usertools/dpdk-devbind.py +2
-0+ 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
baseband_devices= [acceleration_class]crypto_devices= [encryption_class,intel_processor_class] @@ -82,7 +85,7 @@compress_devices= [cavium_zip]regex_devices= [cn9k_ree]misc_devices= [cnxk_bphy,cnxk_bphy_cgx,cnxk_inl_dev,usertools/dpdk-devbind.py +1
-2- intel_ntb_skx, intel_ntb_icx] + intel_ntb_skx, intel_ntb_icx, virtio_blk] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties --1.8.3.1
PATCH v8 11/13 vdpa/ifc: add log for config space of virtio blk
In reply to Andy PeiAdd some log of virtio blk device config space information at VDPA launch before qemu connects.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +34 -0
@@ -1382,6 +1382,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +uint64_t capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;
@@ -1448,6 +1451,37 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_BLK ].features; + +/* cannot read 64-bit register in one attempt, +* so read byte by byte. +*/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +/* The capacity is number of sectors in 512-byte. +* So right shift 1 bit we get in K, +* another right shift 10 bits we get in M, +* right shift 10 more bits, we get in G. +* To show capacity in G, we right shift 21 bits in total. +*/ +DRV_LOG (DEBUG ,"capacity : %" PRIu64"G" , capacity >>21 ); + +DRV_LOG (DEBUG ,"size_max : 0x%08x" , + internal->hw.blk_cfg->size_max); +DRV_LOG (DEBUG ,"seg_max : 0x%08x" , + internal->hw.blk_cfg->seg_max); +DRV_LOG (DEBUG ,"blk_size : 0x%08x" , + internal->hw.blk_cfg->blk_size); +DRV_LOG (DEBUG ,"geometry" ); +DRV_LOG (DEBUG ," cylinders: %u" , + internal->hw.blk_cfg->geometry.cylinders); +DRV_LOG (DEBUG ," heads : %u" , + internal->hw.blk_cfg->geometry.heads); +DRV_LOG (DEBUG ," sectors : %u" , + internal->hw.blk_cfg->geometry.sectors); +DRV_LOG (DEBUG ,"num_queues: 0x%08x" , + internal->hw.blk_cfg->num_queues); } list->internal = internal;
Signature · 1 line
1.8.3.1
RE: [PATCH v8 11/13] vdpa/ifc: add log for config space of virtio blk
Quoted 70 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 11/13] vdpa/ifc: add log for config space of virtio blkAdd some log of virtio blk device config space information at VDPA launch before qemu connects.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)drivers/vdpa/ifc/ifcvf_vdpa.c +34
-1@@ -1382,6 +1382,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +uint64_t capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;@@ -1448,6 +1451,37 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_BLK ].features; + +/* cannot read 64-bit register in one attempt, +* so read byte by byte. +*/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +/* The capacity is number of sectors in 512-byte. +* So right shift 1 bit we get in K, +* another right shift 10 bits we get in M, +* right shift 10 more bits, we get in G. +* To show capacity in G, we right shift 21 bits in total. +*/ +DRV_LOG (DEBUG ,"capacity : %" PRIu64"G" , capacity >>21 ); + +DRV_LOG (DEBUG ,"size_max : 0x%08x" , + internal->hw.blk_cfg->size_max); +DRV_LOG (DEBUG ,"seg_max : 0x%08x" , + internal->hw.blk_cfg->seg_max); +DRV_LOG (DEBUG ,"blk_size : 0x%08x" , + internal->hw.blk_cfg->blk_size); +DRV_LOG (DEBUG ,"geometry" ); +DRV_LOG (DEBUG ," cylinders: %u" , + internal->hw.blk_cfg->geometry.cylinders); +DRV_LOG (DEBUG ," heads : %u" , + internal->hw.blk_cfg->geometry.heads); +DRV_LOG (DEBUG ," sectors : %u" , + internal->hw.blk_cfg->geometry.sectors); +DRV_LOG (DEBUG ,"num_queues: 0x%08x" , + internal->hw.blk_cfg->num_queues); } list->internal = internal; --
Quoted 1 line
1.8.3.1
PATCH v8 12/13 vdpa/ifc: add interrupt handling for config space
In reply to Andy PeiCreate a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 118 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 117 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +117 -1
@@ -53,7 +53,9 @@ struct ifcvf_internal {int vfio_group_fd;int vfio_dev_fd;pthread_t tid;/* thread for notify relay */ +pthread_t intr_tid;/* thread for config space change interrupt relay */ int epfd; +int csc_epfd;int vid;struct rte_vdpa_device *vdev;uint16_t max_queues;
@@ -566,6 +568,111 @@ struct rte_vdpa_dev_info {return 0 ; } +static void +virtio_interrupt_handler (struct ifcvf_internal *internal) +{ +int vid = internal->vid; +int ret; + + ret =rte_vhost_slave_config_change (vid,1 ); +if (ret) +DRV_LOG (ERR ,"failed to notify the guest about configuration space change." ); +} + +static void * +intr_relay (void *arg) +{ +struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; +struct epoll_event csc_event; +struct epoll_event ev; +uint64_t buf; +int nbytes; +int csc_epfd, csc_val =0 ; + + csc_epfd =epoll_create (1 ); +if (csc_epfd <0 ) { +DRV_LOG (ERR ,"failed to create epoll for config space change." ); +return NULL ; + } + + ev.events =EPOLLIN |EPOLLPRI |EPOLLRDHUP |EPOLLHUP ; + ev.data.fd =rte_intr_fd_get (internal->pdev->intr_handle); +if (epoll_ctl (csc_epfd,EPOLL_CTL_ADD , +rte_intr_fd_get (internal->pdev->intr_handle), &ev) <0 ) { +DRV_LOG (ERR ,"epoll add error: %s" ,strerror (errno)); +return NULL ; + } + + internal->csc_epfd = csc_epfd; + +for (;;) { + csc_val =epoll_wait (csc_epfd, &csc_event,1 ,-1 ); +if (csc_val <0 ) { +if (errno ==EINTR ) +continue ; +DRV_LOG (ERR ,"epoll_wait return fail." ); +return NULL ; + }else if (csc_val ==0 ) { +continue ; + }else { +/* csc_val > 0 */ + nbytes =read (csc_event.data.fd, &buf,8 ); +if (nbytes <0 ) { +if (errno ==EINTR || + errno ==EWOULDBLOCK || + errno ==EAGAIN ) +continue ; +DRV_LOG (ERR ,"Error reading from file descriptor %d: %s\n" , + csc_event.data.fd, +strerror (errno)); +return NULL ; + }else if (nbytes ==0 ) { +DRV_LOG (ERR ,"Read nothing from file descriptor %d\n" , + csc_event.data.fd); +continue ; + }else { +virtio_interrupt_handler (internal); + } + } + } + +return NULL ; +} + +static int +setup_intr_relay (struct ifcvf_internal *internal) +{ +char name[THREAD_NAME_LEN ]; +int ret; + +snprintf (name,sizeof (name),"ifc-intr-%d" , internal->vid); + ret =rte_ctrl_thread_create (&internal->intr_tid, name,NULL , + intr_relay, (void *)internal); +if (ret) { +DRV_LOG (ERR ,"failed to create notify relay pthread." ); +return -1 ; + } +return 0 ; +} + +static int +unset_intr_relay (struct ifcvf_internal *internal) +{ +void *status; + +if (internal->intr_tid) { +pthread_cancel (internal->intr_tid); +pthread_join (internal->intr_tid, &status); + } + internal->intr_tid =0 ; + +if (internal->csc_epfd >=0 ) +close (internal->csc_epfd); + internal->csc_epfd =-1 ; + +return 0 ; +} +static int update_datapath (struct ifcvf_internal *internal) {
@@ -592,10 +699,16 @@ struct rte_vdpa_dev_info {if (ret) goto err; + ret =setup_intr_relay (internal); +if (ret) + goto err; +rte_atomic32_set (&internal->running,1 ); }else if (rte_atomic32_read (&internal->running) && (!rte_atomic32_read (&internal->started) || !rte_atomic32_read (&internal->dev_attached))) { + ret =unset_intr_relay (internal); + ret =unset_notify_relay (internal);if (ret) goto err;
@@ -812,7 +925,7 @@ struct rte_vdpa_dev_info {if (nfds <0 ) {if (errno ==EINTR )continue ; -DRV_LOG (ERR ,"epoll_wait return fail\n" ); +DRV_LOG (ERR ,"epoll_wait return fail." );return NULL ; }
@@ -888,6 +1001,9 @@ struct rte_vdpa_dev_info {/* stop the direct IO data path */ unset_notify_relay (internal);vdpa_ifcvf_stop (internal); + +unset_intr_relay (internal); +vdpa_disable_vfio_intr (internal); ret =rte_vhost_host_notifier_ctrl (vid,RTE_VHOST_QUEUE_ALL , false);
Signature · 1 line
1.8.3.1
RE: [PATCH v8 12/13] vdpa/ifc: add interrupt handling for config space
Quoted 102 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 12/13] vdpa/ifc: add interrupt handling for config spaceCreate a thread to poll and relay config space change interrupt. Use
VHOST_USER_SLAVE_CONFIG_CHANGE_MSGto inform QEMU.Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 118 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 117 insertions(+), 1 deletion(-)drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0@@ -53,7 +53,9 @@ struct ifcvf_internal {int vfio_group_fd;int vfio_dev_fd;pthread_t tid;/* thread for notify relay */ +pthread_t intr_tid;/* thread for config space change interrupt relay */
drivers/vdpa/ifc/ifcvf_vdpa.c +61
-0int epfd; +int csc_epfd;int vid;struct rte_vdpa_device *vdev;uint16_t max_queues;@@ -566,6 +568,111 @@ struct rte_vdpa_dev_info {return 0 ; } +static void +virtio_interrupt_handler (struct ifcvf_internal *internal) +{ +int vid = internal->vid; +int ret; + + ret =rte_vhost_slave_config_change (vid,1 ); +if (ret) + DRV_LOG(ERR, "failed to notify the guest about configuration space change."); +} + +static void * +intr_relay (void *arg) +{ +struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; +struct epoll_event csc_event; +struct epoll_event ev; +uint64_t buf; +int nbytes; +int csc_epfd, csc_val =0 ; + + csc_epfd =epoll_create (1 ); +if (csc_epfd <0 ) { + DRV_LOG(ERR, "failed to create epoll for config space change."); +return NULL ; + } + + ev.events =EPOLLIN |EPOLLPRI |EPOLLRDHUP |EPOLLHUP ; + ev.data.fd =rte_intr_fd_get (internal->pdev->intr_handle); +if (epoll_ctl (csc_epfd,EPOLL_CTL_ADD , +rte_intr_fd_get (internal->pdev->intr_handle), &ev) <0 ) { +DRV_LOG (ERR ,"epoll add error: %s" ,strerror (errno)); +return NULL ; + } + + internal->csc_epfd = csc_epfd; + +for (;;) { + csc_val =epoll_wait (csc_epfd, &csc_event,1 ,-1 ); +if (csc_val <0 ) { +if (errno ==EINTR ) +continue ; +DRV_LOG (ERR ,"epoll_wait return fail." ); +return NULL ; + }else if (csc_val ==0 ) { +continue ; + }else { +/* csc_val > 0 */ + nbytes =read (csc_event.data.fd, &buf,8 ); +if (nbytes <0 ) { +if (errno ==EINTR || + errno ==EWOULDBLOCK || + errno ==EAGAIN ) +continue ; + DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n", + csc_event.data.fd, +strerror (errno)); +return NULL ; + }else if (nbytes ==0 ) { + DRV_LOG(ERR, "Read nothing from filedescriptor %d\n",
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 640
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10
-0+ csc_event.data.fd); +continue ; + }else { +virtio_interrupt_handler (internal); + } + } + } + +return NULL ; +}
I think we should not assume unset_intr_relay will help us close epfd when Error happens, so just close it when there's some error.
Quoted 24 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +24
-0+ +static int +setup_intr_relay (struct ifcvf_internal *internal) +{ +char name[THREAD_NAME_LEN ]; +int ret; + +snprintf (name,sizeof (name),"ifc-intr-%d" , internal->vid); + ret =rte_ctrl_thread_create (&internal->intr_tid, name,NULL , + intr_relay, (void *)internal); +if (ret) { +DRV_LOG (ERR ,"failed to create notify relay pthread." ); +return -1 ; + } +return 0 ; +} + +static int +unset_intr_relay (struct ifcvf_internal *internal) +{ +void *status; + +if (internal->intr_tid) { +pthread_cancel (internal->intr_tid);
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 674
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10
-0+pthread_join (internal->intr_tid, &status); + } + internal->intr_tid =0 ; + +if (internal->csc_epfd >=0 ) +close (internal->csc_epfd); + internal->csc_epfd =-1 ; + +return 0 ; +}
It will always return 0, so return type should be void
Quoted 7 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0+static int update_datapath (struct ifcvf_internal *internal) {@@ -592,10 +699,16 @@ struct rte_vdpa_dev_info {if (ret) goto err;
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 710
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5
-0+ ret =setup_intr_relay (internal); +if (ret) + goto err; +rte_atomic32_set (&internal->running,1 ); }else if (rte_atomic32_read (&internal->running) && (!rte_atomic32_read (&internal->started) || !rte_atomic32_read (&internal->dev_attached))) { + ret =unset_intr_relay (internal);
This will be changed accordingly.
Thanks,
Chenbo
Quoted 25 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +5
-2+ ret =unset_notify_relay (internal);if (ret) goto err;@@ -812,7 +925,7 @@ struct rte_vdpa_dev_info {if (nfds <0 ) {if (errno ==EINTR )continue ; -DRV_LOG (ERR ,"epoll_wait return fail\n" ); +DRV_LOG (ERR ,"epoll_wait return fail." );return NULL ; }@@ -888,6 +1001,9 @@ struct rte_vdpa_dev_info {/* stop the direct IO data path */ unset_notify_relay (internal);vdpa_ifcvf_stop (internal); + +unset_intr_relay (internal); +vdpa_disable_vfio_intr (internal); ret =rte_vhost_host_notifier_ctrl (vid,RTE_VHOST_QUEUE_ALL , false); --1.8.3.1
PATCH v8 13/13 vdpa/ifc/base: access correct register for blk device
In reply to Andy PeiRegister address is different between net and blk device. We are re-use most of the code, when register address is different, we have to check net and blk device go through different code.
--- drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +27 -7
@@ -218,10 +218,17 @@ &cfg->queue_used_hi);IFCVF_WRITE_REG16 (hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ) = - (u32 )hw->vring[i].last_avail_idx | - ((u32 )hw->vring[i].last_used_idx <<16 ); +if (hw->device_type ==IFCVF_BLK ) + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 ); +else + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 );IFCVF_WRITE_REG16 (i +1 , &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -254,9 +261,22 @@IFCVF_WRITE_REG16 (i, &cfg->queue_select);IFCVF_WRITE_REG16 (0 , &cfg->queue_enable);IFCVF_WRITE_REG16 (IFCVF_MSI_NO_VECTOR , &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ); - hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); + +if (hw->device_type ==IFCVF_BLK ) + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); +else + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ); + +if (hw->device_type ==IFCVF_BLK ) + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); +else + hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); hw->vring[i].last_used_idx = (u16)(ring_state >>16 ); } }
Signature · 1 line
1.8.3.1
RE: [PATCH v8 13/13] vdpa/ifc/base: access correct register for blk device
Quoted 10 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 13/13] vdpa/ifc/base: access correct register for blk device
Quoted 2 lines
Register address is different between net and blk device. We are re-use most of the code, when register address is
Re-using
With this fixed:
Quoted 64 lines
different, we have to check net and blk device go through different code.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)drivers/vdpa/ifc/base/ifcvf.c +28
-8@@ -218,10 +218,17 @@ &cfg->queue_used_hi);IFCVF_WRITE_REG16 (hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ) = - (u32 )hw->vring[i].last_avail_idx | - ((u32 )hw->vring[i].last_used_idx <<16 ); +if (hw->device_type ==IFCVF_BLK ) + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 ); +else + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 );IFCVF_WRITE_REG16 (i +1 , &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==@@ -254,9 +261,22 @@IFCVF_WRITE_REG16 (i, &cfg->queue_select);IFCVF_WRITE_REG16 (0 , &cfg->queue_enable);IFCVF_WRITE_REG16 (IFCVF_MSI_NO_VECTOR , &cfg- >queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ); - hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); + +if (hw->device_type ==IFCVF_BLK ) + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); +else + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ); + +if (hw->device_type ==IFCVF_BLK ) + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); +else + hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); hw->vring[i].last_used_idx = (u16)(ring_state >>16 ); } } --1.8.3.1
RE: [PATCH v8 13/13] vdpa/ifc/base: access correct register for blk device
Hi Chenbo,
Thanks for your reply.
I will address in next version.
Quoted 96 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 23, 2022 3:56 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v8 13/13] vdpa/ifc/base: access correct register for blk deviceQuoted 11 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Wednesday, May 18, 2022 8:14 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v8 13/13] vdpa/ifc/base: access correct register for blk deviceQuoted 2 lines
Register address is different between net and blk device. We are re-use most of the code, when register address is
Re-using
With this fixed:
Reviewed-by Chenbo Xia <[email protected]>Quoted 24 lines
different, we have to check net and blk device go through different code.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)(quoted diff) +0
-0
b/drivers/vdpa/ifc/base/ifcvf.cindex d10c1fd..dd475a7 100644drivers/vdpa/ifc/base/ifcvf.c +4
-4@@ -218,10 +218,17 @@ &cfg->queue_used_hi);IFCVF_WRITE_REG16 (hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ) = - (u32 )hw->vring[i].last_avail_idx | - ((u32 )hw->vring[i].last_used_idx <<16 ); +if (hw->device_type ==IFCVF_BLK ) + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ) = + (u32)hw->vring[i].last_avail_idx |drivers/vdpa/ifc/base/ifcvf.c line 234
Quoted 10 lines
drivers/vdpa/ifc/base/ifcvf.c +7
-0+ ((u32 )hw->vring[i].last_used_idx <<16 ); + else + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 );IFCVF_WRITE_REG16 (i +1 , &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -254,9 +261,22
drivers/vdpa/ifc/base/ifcvf.c
Quoted 6 lines
@@
drivers/vdpa/ifc/base/ifcvf.c +0
-0IFCVF_WRITE_REG16 (i, &cfg->queue_select);IFCVF_WRITE_REG16 (0 , &cfg->queue_enable); IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg-Quoted 1 line
queue_msix_vector);drivers/vdpa/ifc/base/ifcvf.c +0
-1- ring_state = *(u32 *)(hw->lm_cfg +
IFCVF_LM_RING_STATE_OFFSETQuoted 24 lines
drivers/vdpa/ifc/base/ifcvf.c +17
-3+ - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4); - hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); + +if (hw->device_type ==IFCVF_BLK ) + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); +else + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ); + +if (hw->device_type ==IFCVF_BLK ) + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); +else + hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); hw->vring[i].last_used_idx = (u16)(ring_state >>16 ); } } --1.8.3.1
v9 cover add virtio_blk device support to vdpa/ifc
In reply to Andy PeiThis patch set add virtio_blk device support to vdpa/ifc driver. With a lot of similarities, I re-use part of vdpa/ifc driver. Distinguish the virtio net and blk device by device id, and implement specific features and ops.
Add example to vdpa to support virtio_blk device.
To support blk device live migration, some modification to vhost lib. Perform dev_conf op only under VHOST_USER_SET_VRING_CALL msg.
v9:
fix some naming issue.
add more NULL pointer check.
close open file when errors occur.
v8:
delete some redundant code.
fix some commit log.
v7:
Check on expected fd num in new vhost msg handler. Sanity check on vhost msg size.
Fix typo.
Add commit log to help understand code. Remove duplicated code.
Add new API to get vDPA device type.
v6:
fix some commit log.
add vhost socket in log output to make it more user-friendly. when driver ops fail, just output some log, do not break message handler. check vhost msg size in msg handler.
v5:
fix some coding style issues.
v4:
add args "isblk" to vdpa example to specify a block device, fix some issue in example.
Make sure code specify for block device does not affect net device.
v3:
Fix some compile issues.
v2:
Fix some coding style issues.
Andy Pei (13):
vdpa/ifc: add support for virtio blk device
vhost: add vDPA ops for blk device
vhost: add vhost msg support for get/set config vdpa/ifc: add blk ops for ifc device vdpa/ifc: add vDPA interrupt relay for blk device vdpa/ifc: add block device SW live-migration
vhost: add API to get vDPA device type vdpa/ifc: add get device type ops to ifc driver examples/vdpa: add virtio blk support
usertools: add support for virtio blk device vdpa/ifc: add log for config space of virtio blk vdpa/ifc: add interrupt handling for config space vdpa/ifc/base: access correct register for blk device
doc/guides/prog_guide/vhost_lib.rst | 5 + doc/guides/rel_notes/release_22_07.rst | 4 + drivers/vdpa/ifc/base/ifcvf.c | 34 ++- drivers/vdpa/ifc/base/ifcvf.h | 27 ++- drivers/vdpa/ifc/ifcvf_vdpa.c | 415 +++++++++++++++++++++++++++++++-- examples/vdpa/main.c | 56 +++++ examples/vdpa/vdpa_blk_compact.h | 50 ++++ lib/vhost/rte_vhost.h | 17 ++ lib/vhost/socket.c | 44 ++++ lib/vhost/vdpa_driver.h | 11 +- lib/vhost/version.map | 1 + lib/vhost/vhost_user.c | 88 +++++++ lib/vhost/vhost_user.h | 13 ++ usertools/dpdk-devbind.py | 5 +- 14 files changed, 739 insertions(+), 31 deletions(-) create mode 100644 examples/vdpa/vdpa_blk_compact.h
Signature · 1 line
1.8.3.1
PATCH v9 1/13 vdpa/ifc: add support for virtio blk device
Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops.
--- drivers/vdpa/ifc/base/ifcvf.h | 22 +++++++--- drivers/vdpa/ifc/ifcvf_vdpa.c | 95 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 102 insertions(+), 15 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +17 -5
@@ -5,12 +5,19 @@#ifndef _IFCVF_H_#define _IFCVF_H_ +#include <linux/virtio_blk.h> #include "ifcvf_osdep.h" -#define IFCVF_VENDOR_ID 0x1AF4 -#define IFCVF_DEVICE_ID 0x1041 -#define IFCVF_SUBSYS_VENDOR_ID 0x8086 -#define IFCVF_SUBSYS_DEVICE_ID 0x001A +#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +#define IFCVF_VENDOR_ID 0x1AF4 +#define IFCVF_NET_DEVICE_ID 0x1041 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_SUBSYS_VENDOR_ID 0x8086 +#define IFCVF_SUBSYS_DEVICE_ID 0x001A +#define IFCVF_BLK_DEVICE_ID 0x0002#define IFCVF_MAX_QUEUES 1
@@ -126,13 +133,18 @@ struct ifcvf_hw {u8 notify_region;u32 notify_off_multiplier;struct ifcvf_pci_common_cfg *common_cfg; -struct ifcvf_net_config *dev_cfg; +union { +struct ifcvf_net_config *net_cfg; +struct virtio_blk_config *blk_cfg; +void *dev_cfg; + };u8 *isr;u16 *notify_base;u16 *notify_addr[IFCVF_MAX_QUEUES *2 ];u8 *lm_cfg;struct vring_info vring[IFCVF_MAX_QUEUES *2 ];u8 nr_vring; +int device_type;struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE ]; };
drivers/vdpa/ifc/ifcvf_vdpa.c +85 -10
141 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RE: [PATCH v9 01/13] vdpa/ifc: add support for virtio blk device
Quoted 12 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 01/13] vdpa/ifc: add support for virtio blk deviceRe-use the vdpa/ifc code, distinguish blk and net device by
pci_device_id. Blk and net device are implemented with proper feature and ops.
Quoted 6 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.h | 22 +++++++--- drivers/vdpa/ifc/ifcvf_vdpa.c | 95 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 102 insertions(+), 15 deletions(-)
PATCH v9 2/13 vhost: add vDPA ops for blk device
In reply to Andy PeiGet_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops.
--- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
lib/vhost/vdpa_driver.h +6 -2
@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {/** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev,int qid); -/** Reserved for future extension */ -void *reserved[2 ]; +/** Get the device configuration space */ +int (*get_config)(int vid,uint8_t *config,uint32_t size); + +/** Set the device configuration space */ +int (*set_config)(int vid,uint8_t *config,uint32_t offset, +uint32_t size,uint32_t flags); };/**
Signature · 1 line
1.8.3.1
RE: [PATCH v9 02/13] vhost: add vDPA ops for blk device
Quoted 38 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 02/13] vhost: add vDPA ops for blk device
Get_configandset_configare necessary ops for blk device. Addget_configandset_configops to vDPA ops.Signed-off-by Andy Pei <[email protected]>Reviewed-by Maxime Coquelin <[email protected]>--- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)lib/vhost/vdpa_driver.h +6
-3@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {/** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev,int qid); -/** Reserved for future extension */ -void *reserved[2 ]; +/** Get the device configuration space */ +int (*get_config)(int vid,uint8_t *config,uint32_t size); + +/** Set the device configuration space */ +int (*set_config)(int vid,uint8_t *config,uint32_t offset, +uint32_t size,uint32_t flags); };/** --
Quoted 1 line
1.8.3.1
PATCH v9 3/13 vhost: add vhost msg support for get/set config
In reply to Andy PeiAdd support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device.
--- lib/vhost/vhost_user.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 ++++++++ 2 files changed, 101 insertions(+)
lib/vhost/vhost_user.c +88 -0
@@ -2468,6 +2468,92 @@ static int is_vring_iotlb(struct virtio_net *dev, }static int +vhost_user_get_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (!vdpa_dev) { +VHOST_LOG_CONFIG (ERR , +"(%s) is not vDPA device!\n" , + dev->ifname); + goto out; + } + +if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config (dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); +if (ret !=0 ) { + ctx->msg.size =0 ; +VHOST_LOG_CONFIG (ERR , +"(%s) get_config() return error!\n" , + dev->ifname); + } + }else { +VHOST_LOG_CONFIG (ERR ,"(%s) get_config() not supported!\n" , + dev->ifname); + } + +out: +return RTE_VHOST_MSG_RESULT_REPLY ; +} + +static int +vhost_user_set_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (ctx->msg.payload.cfg.size >VHOST_USER_MAX_CONFIG_SIZE ) { +VHOST_LOG_CONFIG (ERR , +"(%s) vhost_user_config size: %" PRIu32", should not be larger than %d\n" , + dev->ifname, ctx->msg.payload.cfg.size, +VHOST_USER_MAX_CONFIG_SIZE ); + goto out; + } + +if (!vdpa_dev) { +VHOST_LOG_CONFIG (ERR , +"(%s) is not vDPA device!\n" , + dev->ifname); + goto out; + } + +if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config (dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); +if (ret) +VHOST_LOG_CONFIG (ERR , +"(%s) set_config() return error!\n" , + dev->ifname); + }else { +VHOST_LOG_CONFIG (ERR ,"(%s) set_config() not supported!\n" , + dev->ifname); + } + +return RTE_VHOST_MSG_RESULT_OK ; + +out: +return RTE_VHOST_MSG_RESULT_ERR ; +} + +static int vhost_user_iotlb_msg (struct virtio_net **pdev,struct vhu_msg_context *ctx, int main_fd __rte_unused)
@@ -2686,6 +2772,8 @@ static int is_vring_iotlb(struct virtio_net *dev, VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU, vhost_user_net_set_mtu, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD, vhost_user_set_req_fd, true) \ VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG, vhost_user_get_config, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG, vhost_user_set_config, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE, vhost_user_set_postcopy_advise, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false) \VHOST_MESSAGE_HANDLER (VHOST_USER_POSTCOPY_END , vhost_user_postcopy_end, false) \
lib/vhost/vhost_user.h +13 -0
@@ -50,6 +50,8 @@VHOST_USER_NET_SET_MTU =20 ,VHOST_USER_SET_SLAVE_REQ_FD =21 ,VHOST_USER_IOTLB_MSG =22 , +VHOST_USER_GET_CONFIG =24 , +VHOST_USER_SET_CONFIG =25 ,VHOST_USER_CRYPTO_CREATE_SESS =26 ,VHOST_USER_CRYPTO_CLOSE_SESS =27 , VHOST_USER_POSTCOPY_ADVISE = 28,
@@ -123,6 +125,16 @@uint16_t queue_size; } VhostUserInflight; +#define VHOST_USER_MAX_CONFIG_SIZE 256 + +/** Get/set config msg payload */ +struct vhost_user_config { +uint32_t offset; +uint32_t size; +uint32_t flags; +uint8_t region[VHOST_USER_MAX_CONFIG_SIZE ]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */
@@ -146,6 +158,7 @@VhostUserCryptoSessionParam crypto_session;VhostUserVringArea area;VhostUserInflight inflight; +struct vhost_user_config cfg; } payload;/* Nothing should be added after the payload */ }__rte_packed VhostUserMsg;
Signature · 1 line
1.8.3.1
RE: [PATCH v9 03/13] vhost: add vhost msg support for get/set config
Quoted 35 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 03/13] vhost: add vhost msg support for get/set configAdd support for
VHOST_USER_GET_CONFIGandVHOST_USER_SET_CONFIG.VHOST_USER_GET_CONFIGandVHOST_USER_SET_CONFIGmessage is only supported by virtio blk VDPA device.Signed-off-by Andy Pei <[email protected]>--- lib/vhost/vhost_user.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 ++++++++ 2 files changed, 101 insertions(+)lib/vhost/vhost_user.c +7
-0@@ -2468,6 +2468,92 @@ static int is_vring_iotlb(struct virtio_net *dev, }static int +vhost_user_get_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ;
lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev, · line 2487
Quoted 10 lines
lib/vhost/vhost_user.c +10
-0+ +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (!vdpa_dev) { +VHOST_LOG_CONFIG (ERR , +"(%s) is not vDPA device!\n" , + dev->ifname); + goto out; + }
In my understanding, this message is only targeted for vDPA device. Traditional SW back-end will not use it. So if vdpa_dev is not attached, you can just return RTE_VHOST_MSG_RESULT_ERR here. Another way is setting msg size to 0 and reply, but I think in this case. We should just break the msg handling.
Besides, two lines are enough for the log above:
+ VHOST_LOG_CONFIG(ERR, "(%s) is not vDPA device!\n",
+ dev->ifname);
Thanks,
Chenbo
Quoted 130 lines
lib/vhost/vhost_user.c +69
-0+ +if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config (dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); +if (ret !=0 ) { + ctx->msg.size =0 ; +VHOST_LOG_CONFIG (ERR , +"(%s) get_config() return error!\n" , + dev->ifname); + } + }else { +VHOST_LOG_CONFIG (ERR ,"(%s) get_config() not supported!\n" , + dev->ifname); + } + +out: +return RTE_VHOST_MSG_RESULT_REPLY ; +} + +static int +vhost_user_set_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (ctx->msg.payload.cfg.size >VHOST_USER_MAX_CONFIG_SIZE ) { +VHOST_LOG_CONFIG (ERR , + "(%s) vhost_user_config size: %"PRIu32", should not be larger than %d\n", + dev->ifname, ctx->msg.payload.cfg.size, +VHOST_USER_MAX_CONFIG_SIZE ); + goto out; + } + +if (!vdpa_dev) { +VHOST_LOG_CONFIG (ERR , +"(%s) is not vDPA device!\n" , + dev->ifname); + goto out; + } + +if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config (dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); +if (ret) +VHOST_LOG_CONFIG (ERR , +"(%s) set_config() return error!\n" , + dev->ifname); + }else { +VHOST_LOG_CONFIG (ERR ,"(%s) set_config() not supported!\n" , + dev->ifname); + } + +return RTE_VHOST_MSG_RESULT_OK ; + +out: +return RTE_VHOST_MSG_RESULT_ERR ; +} + +static int vhost_user_iotlb_msg(struct virtio_net **pdev, struct vhu_msg_context *ctx,lib/vhost/vhost_user.c +1
-0int main_fd __rte_unused)@@ -2686,6 +2772,8 @@ static int is_vring_iotlb(struct virtio_net *dev,VHOST_MESSAGE_HANDLER (VHOST_USER_NET_SET_MTU , vhost_user_net_set_mtu, false) \VHOST_MESSAGE_HANDLER (VHOST_USER_SET_SLAVE_REQ_FD , vhost_user_set_req_fd, true) \ VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false) \ +VHOST_MESSAGE_HANDLER (VHOST_USER_GET_CONFIG , vhost_user_get_config, false)\
+VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG,vhost_user_set_config, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE,vhost_user_set_postcopy_advise, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN,vhost_user_set_postcopy_listen, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END,vhost_user_postcopy_end,
false) \lib/vhost/vhost_user.h +13
-1@@ -50,6 +50,8 @@VHOST_USER_NET_SET_MTU =20 ,VHOST_USER_SET_SLAVE_REQ_FD =21 ,VHOST_USER_IOTLB_MSG =22 , +VHOST_USER_GET_CONFIG =24 , +VHOST_USER_SET_CONFIG =25 ,VHOST_USER_CRYPTO_CREATE_SESS =26 ,VHOST_USER_CRYPTO_CLOSE_SESS =27 , VHOST_USER_POSTCOPY_ADVISE = 28,@@ -123,6 +125,16 @@uint16_t queue_size; } VhostUserInflight; +#define VHOST_USER_MAX_CONFIG_SIZE 256 + +/** Get/set config msg payload */ +struct vhost_user_config { +uint32_t offset; +uint32_t size; +uint32_t flags; +uint8_t region[VHOST_USER_MAX_CONFIG_SIZE ]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */@@ -146,6 +158,7 @@VhostUserCryptoSessionParam crypto_session;VhostUserVringArea area;VhostUserInflight inflight; +struct vhost_user_config cfg; } payload;/* Nothing should be added after the payload */ }__rte_packed VhostUserMsg; --1.8.3.1
RE: [PATCH v9 03/13] vhost: add vhost msg support for get/set config
Quoted 58 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, May 24, 2022 10:52 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v9 03/13] vhost: add vhost msg support for get/set configQuoted 11 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 03/13] vhost: add vhost msg support for get/set configQuoted 1 line
Add support for
VHOST_USER_GET_CONFIGand
VHOST_USER_SET_CONFIG.Quoted 33 lines
VHOST_USER_GET_CONFIGandVHOST_USER_SET_CONFIGmessage is only supported by virtio blk VDPA device.Signed-off-by Andy Pei <[email protected]>--- lib/vhost/vhost_user.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 ++++++++ 2 files changed, 101 insertions(+)lib/vhost/vhost_user.c index renamed from lib/vhost/vhost_user.c +0
-0850848c..b37d814 100644
lib/vhost/vhost_user.c +15
-0@@ -2468,6 +2468,92 @@ static int is_vring_iotlb(struct virtio_net *dev, }static int +vhost_user_get_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (!vdpa_dev) { +VHOST_LOG_CONFIG (ERR , +"(%s) is not vDPA device!\n" , + dev->ifname);
Quoted 9 lines
lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net · line 2487
Quoted 2 lines
lib/vhost/vhost_user.c +2
-0+ goto out; + }In my understanding, this message is only targeted for vDPA device. Traditional SW back-end will not use it. So if
vdpa_devis not attached, you can just returnRTE_VHOST_MSG_RESULT_ERRhere. Another way is setting msg size to 0 and reply, but I think in this case. We should just break the msg handling.
OK. I will just return RTE_VHOST_MSG_RESULT_ERR here.
Review
Quoted 5 lines
Besides, two lines are enough for the log above:
(quoted diff) +2
-0+ VHOST_LOG_CONFIG(ERR, "(%s) is not vDPA device!\n", + dev->ifname);
Sure.
Quoted 144 lines
Thanks,
Chenbolib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev,
Quoted 13 lines
lib/vhost/vhost_user.c +13
-0+ +if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config (dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); +if (ret !=0 ) { + ctx->msg.size =0 ; +VHOST_LOG_CONFIG (ERR , +"(%s) get_config() return error!\n" , + dev->ifname); + } + }else { + VHOST_LOG_CONFIG(ERR, "(%s) get_config() notsupported!\n",
Quoted 12 lines
lib/vhost/vhost_user.c +12
-0+ dev->ifname); + } + +out: +return RTE_VHOST_MSG_RESULT_REPLY ; +} + +static int +vhost_user_set_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev,
Quoted 10 lines
lib/vhost/vhost_user.c +10
-0+struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + + if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) { + VHOST_LOG_CONFIG(ERR, + "(%s) vhost_user_config size: %"PRIu32", should notbe
Quoted 15 lines
lib/vhost/vhost_user.c +14
-0larger than %d\n", + dev->ifname, ctx->msg.payload.cfg.size, + VHOST_USER_MAX_CONFIG_SIZE); + goto out; + } + +if (!vdpa_dev) { +VHOST_LOG_CONFIG (ERR , +"(%s) is not vDPA device!\n" , + dev->ifname); + goto out; + } + +if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config(dev->vid,lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net *dev,
Quoted 10 lines
lib/vhost/vhost_user.c +10
-0+ ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); +if (ret) +VHOST_LOG_CONFIG (ERR , +"(%s) set_config() return error!\n" , + dev->ifname); + }else { + VHOST_LOG_CONFIG(ERR, "(%s) set_config() notsupported!\n",
Quoted 14 lines
lib/vhost/vhost_user.c +10
-0+ dev->ifname); + } + +return RTE_VHOST_MSG_RESULT_OK ; + +out: +return RTE_VHOST_MSG_RESULT_ERR ; +} + +static int vhost_user_iotlb_msg(struct virtio_net **pdev, struct vhu_msg_context *ctx,lib/vhost/vhost_user.c +0
-0int main_fd __rte_unused)lib/vhost/vhost_user.c static int is_vring_iotlb(struct virtio_net
Quoted 7 lines
lib/vhost/vhost_user.c +0
-0@@ -2686,6 +2772,8 @@ static int is_vring_iotlb(struct virtio_net *dev,VHOST_MESSAGE_HANDLER (VHOST_USER_NET_SET_MTU , vhost_user_net_set_mtu, false) \VHOST_MESSAGE_HANDLER (VHOST_USER_SET_SLAVE_REQ_FD , vhost_user_set_req_fd, true) \ VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG,
vhost_user_iotlb_msg,Quoted 2 lines
false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG,
vhost_user_get_config,Quoted 3 lines
+false)
\
+VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG,
vhost_user_set_config,Quoted 29 lines
+false)
\
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE,vhost_user_set_postcopy_advise, false) \VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false)\
VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END,vhost_user_postcopy_end,
false) \lib/vhost/vhost_user.h index renamed from lib/vhost/vhost_user.h +0
-0ba1c5c7..c4d091e 100644
lib/vhost/vhost_user.h +5
-0@@ -50,6 +50,8 @@VHOST_USER_NET_SET_MTU =20 ,VHOST_USER_SET_SLAVE_REQ_FD =21 ,VHOST_USER_IOTLB_MSG =22 , +VHOST_USER_GET_CONFIG =24 , +VHOST_USER_SET_CONFIG =25 ,VHOST_USER_CRYPTO_CREATE_SESS =26 ,VHOST_USER_CRYPTO_CLOSE_SESS =27 , VHOST_USER_POSTCOPY_ADVISE = 28,@@ -123,6 +125,16 @@uint16_t queue_size; } VhostUserInflight; +#define VHOST_USER_MAX_CONFIG_SIZE 256 + +/** Get/set config msg payload */ lib/vhost/vhost_user.h line 140
Quoted 10 lines
lib/vhost/vhost_user.h +7
-0+struct vhost_user_config { +uint32_t offset; +uint32_t size; +uint32_t flags; +uint8_t region[VHOST_USER_MAX_CONFIG_SIZE ]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */ @@ -146,6+158,7 @@
Quoted 9 lines
lib/vhost/vhost_user.h +1
-0VhostUserCryptoSessionParam crypto_session;VhostUserVringArea area;VhostUserInflight inflight; +struct vhost_user_config cfg; } payload;/* Nothing should be added after the payload */ } __rte_packedVhostUserMsg;
--
1.8.3.1
PATCH v9 4/13 vdpa/ifc: add blk ops for ifc device
In reply to Andy PeiFor virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.
--- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 91 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 94 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/base/ifcvf.h +4 -0
@@ -65,6 +65,10 @@#define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif +struct ifcvf_pci_cap {u8 cap_vndr;/* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next;/* Generic PCI field: next ptr. */
drivers/vdpa/ifc/ifcvf_vdpa.c +90 -1
109 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RE: [PATCH v9 04/13] vdpa/ifc: add blk ops for ifc device
Quoted 64 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 04/13] vdpa/ifc: add blk ops for ifc deviceFor virtio blk device, re-use part of ifc driver ops. Implement
ifcvf_blk_get_configfor virtio blk device. SupportVHOST_USER_PROTOCOL_F_CONFIGfeature for virtio blk device.Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 91 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 94 insertions(+), 1 deletion(-)drivers/vdpa/ifc/base/ifcvf.h +4
-0@@ -65,6 +65,10 @@#define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif +struct ifcvf_pci_cap {u8 cap_vndr;/* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next;/* Generic PCI field: next ptr. */ drivers/vdpa/ifc/ifcvf_vdpa.c +4
-0@@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) +static int ifcvf_get_protocol_features (struct rte_vdpa_device *vdev,uint64_t *features){
drivers/vdpa/ifc/ifcvf_vdpa.c +8
-0@@ -1199,6 +1203,91 @@ struct rte_vdpa_dev_info {return device_id; } +static int +ifcvf_blk_get_config (int vid,uint8_t *config,uint32_t size) +{ +struct virtio_blk_config *dev_cfg; +struct ifcvf_internal *internal; +struct rte_vdpa_device *vdev; +struct internal_list *list; +uint32_t i;
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1223
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +10
-0+uint64_t capacity =0 ; +uint8_t *byte; + +if (size !=sizeof (struct virtio_blk_config )) { +DRV_LOG (ERR ,"Invalid len: %u, required: %u" , + size, (uint32_t )sizeof (struct virtio_blk_config )); +return -1 ; + } + + vdev =rte_vhost_get_vdpa_device (vid);
Check vdev is not NULL here.
Thanks
Chenbo
Quoted 82 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +67
-0+ list =find_internal_resource_by_vdev (vdev); +if (list ==NULL ) { +DRV_LOG (ERR ,"Invalid vDPA device: %p" , vdev); +return -1 ; + } + + internal = list->internal; + +for (i =0 ; i <sizeof (struct virtio_blk_config ); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + +/* cannot read 64-bit register in one attempt, so read byte by byte. */ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +/* The capacity is number of sectors in 512-byte. +* So right shift 1 bit we get in K, +* another right shift 10 bits we get in M, +* right shift 10 more bits, we get in G. +* To show capacity in G, we right shift 21 bits in total. +*/ +DRV_LOG (DEBUG ,"capacity : %" PRIu64"G" , capacity >>21 ); + +DRV_LOG (DEBUG ,"size_max : 0x%08x" , dev_cfg->size_max); +DRV_LOG (DEBUG ,"seg_max : 0x%08x" , dev_cfg->seg_max); +DRV_LOG (DEBUG ,"blk_size : 0x%08x" , dev_cfg->blk_size); +DRV_LOG (DEBUG ,"geometry" ); +DRV_LOG (DEBUG ," cylinders: %u" , dev_cfg->geometry.cylinders); +DRV_LOG (DEBUG ," heads : %u" , dev_cfg->geometry.heads); +DRV_LOG (DEBUG ," sectors : %u" , dev_cfg->geometry.sectors); +DRV_LOG (DEBUG ,"num_queues: 0x%08x" , dev_cfg->num_queues); + +DRV_LOG (DEBUG ,"config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n" , + config[0 ], config[1 ], config[2 ], config[3 ], config[4 ], + config[5 ], config[6 ], config[7 ]); +return 0 ; +} + +static int +ifcvf_blk_get_protocol_features (struct rte_vdpa_device *vdev, +uint64_t *features) +{ +RTE_SET_USED (vdev); + + *features =VDPA_SUPPORTED_PROTOCOL_FEATURES ; + *features |=VDPA_BLK_PROTOCOL_FEATURES ; +return 0 ; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = ifcvf_set_vring_state, + .migration_done =NULL , + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; +struct rte_vdpa_dev_info dev_info[] = { {drivers/vdpa/ifc/ifcvf_vdpa.c +1
-2.features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |@@ -1211,7 +1300,7 @@ struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | (1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; --1.8.3.1
RE: [PATCH v9 04/13] vdpa/ifc: add blk ops for ifc device
Quoted 78 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, May 24, 2022 10:55 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v9 04/13] vdpa/ifc: add blk ops for ifc deviceQuoted 67 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 04/13] vdpa/ifc: add blk ops for ifc deviceFor virtio blk device, re-use part of ifc driver ops. Implement
ifcvf_blk_get_configfor virtio blk device. SupportVHOST_USER_PROTOCOL_F_CONFIGfeature for virtio blk device.Signed-off-by Andy Pei <[email protected]>---
drivers/vdpa/ifc/base/ifcvf.h| 4 ++drivers/vdpa/ifc/ifcvf_vdpa.c
| 91
++++++++++++++++++++++++++++++++++++++++++-2 files changed, 94 insertions(+), 1 deletion(-)(quoted diff) +0
-0
b/drivers/vdpa/ifc/base/ifcvf.hindex a761d49..b6fdfdb 100644drivers/vdpa/ifc/base/ifcvf.h +4
-0@@ -65,6 +65,10 @@#define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif +struct ifcvf_pci_cap {u8 cap_vndr;/* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next;/* Generic PCI field: next ptr. */ (quoted diff) +0
-0
b/drivers/vdpa/ifc/ifcvf_vdpa.cindex 1eed90b..c1fc1d7 100644drivers/vdpa/ifc/ifcvf_vdpa.c +4
-0@@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) +static int ifcvf_get_protocol_features (struct rte_vdpa_device *vdev,uint64_t *features){
drivers/vdpa/ifc/ifcvf_vdpa.c +11
-0@@ -1199,6 +1203,91 @@ struct rte_vdpa_dev_info {return device_id; } +static int +ifcvf_blk_get_config (int vid,uint8_t *config,uint32_t size) { +struct virtio_blk_config *dev_cfg; +struct ifcvf_internal *internal; +struct rte_vdpa_device *vdev; +struct internal_list *list; +uint32_t i; +uint64_t capacity =0 ; +uint8_t *byte; + +if (size !=sizeof (struct virtio_blk_config )) {
Quoted 9 lines
drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1223
Quoted 6 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +6
-0+DRV_LOG (ERR ,"Invalid len: %u, required: %u" , + size, (uint32_t )sizeof (struct virtio_blk_config )); +return -1 ; + } + + vdev =rte_vhost_get_vdpa_device (vid);Check vdev is not NULL here.
Sure, I will fix it in next version.
Quoted 87 lines
Thanks
ChenboQuoted 61 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +60
-0+ list =find_internal_resource_by_vdev (vdev); +if (list ==NULL ) { +DRV_LOG (ERR ,"Invalid vDPA device: %p" , vdev); +return -1 ; + } + + internal = list->internal; + +for (i =0 ; i <sizeof (struct virtio_blk_config ); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + +/* cannot read 64-bit register in one attempt, so read byte by byte. */ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +/* The capacity is number of sectors in 512-byte. +* So right shift 1 bit we get in K, +* another right shift 10 bits we get in M, +* right shift 10 more bits, we get in G. +* To show capacity in G, we right shift 21 bits in total. +*/ +DRV_LOG (DEBUG ,"capacity : %" PRIu64"G" , capacity >>21 ); + +DRV_LOG (DEBUG ,"size_max : 0x%08x" , dev_cfg->size_max); +DRV_LOG (DEBUG ,"seg_max : 0x%08x" , dev_cfg->seg_max); +DRV_LOG (DEBUG ,"blk_size : 0x%08x" , dev_cfg->blk_size); +DRV_LOG (DEBUG ,"geometry" ); +DRV_LOG (DEBUG ," cylinders: %u" , dev_cfg->geometry.cylinders); +DRV_LOG (DEBUG ," heads : %u" , dev_cfg->geometry.heads); +DRV_LOG (DEBUG ," sectors : %u" , dev_cfg->geometry.sectors); +DRV_LOG (DEBUG ,"num_queues: 0x%08x" , dev_cfg->num_queues); + +DRV_LOG (DEBUG ,"config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n" , + config[0 ], config[1 ], config[2 ], config[3 ], config[4 ], + config[5 ], config[6 ], config[7 ]); +return 0 ; +} + +static int +ifcvf_blk_get_protocol_features (struct rte_vdpa_device *vdev, +uint64_t *features) +{ +RTE_SET_USED (vdev); + + *features =VDPA_SUPPORTED_PROTOCOL_FEATURES ; + *features |=VDPA_BLK_PROTOCOL_FEATURES ; +return 0 ; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = ifcvf_set_vring_state,drivers/vdpa/ifc/ifcvf_vdpa.c struct rte_vdpa_dev_info { · line 1293
Quoted 10 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +7
-0+ .migration_done = NULL, + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; +struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |@@ -1211,7
drivers/vdpa/ifc/ifcvf_vdpa.c
Quoted 3 lines
+1300,7 @@ struct
rte_vdpa_dev_infodev_info[] = {drivers/vdpa/ifc/ifcvf_vdpa.c +0
-0{ .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES)|
Quoted 8 lines
drivers/vdpa/ifc/ifcvf_vdpa.c +1
-2(1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; --1.8.3.1
PATCH v9 5/13 vdpa/ifc: add vDPA interrupt relay for blk device
In reply to Andy PeiFor the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +8 -1
@@ -369,6 +369,7 @@ struct rte_vdpa_dev_info { irq_set->index =VFIO_PCI_MSIX_IRQ_INDEX ; irq_set->start =0 ; fd_ptr = (int *)&irq_set->data; +/* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET ] =rte_intr_fd_get (internal->pdev->intr_handle);
@@ -378,7 +379,13 @@ struct rte_vdpa_dev_info {for (i =0 ; i < nr_vring; i++) {rte_vhost_get_vhost_vring (internal->vid, i, &vring); fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; -if ((i &1 ) ==0 && m_rx == true) { +if (m_rx == true && + ((i &1 ) ==0 || internal->hw.device_type ==IFCVF_BLK )) { +/* For the net we only need to relay rx queue, +* which will change the mem of VM. +* For the blk we need to relay all the read cmd +* of each queue +*/ fd =eventfd (0 ,EFD_NONBLOCK |EFD_CLOEXEC );if (fd <0 ) { DRV_LOG(ERR, "can't setup eventfd: %s",
Signature · 1 line
1.8.3.1
RE: [PATCH v9 05/13] vdpa/ifc: add vDPA interrupt relay for blk device
Quoted 47 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 05/13] vdpa/ifc: add vDPA interrupt relay for blk deviceFor the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)drivers/vdpa/ifc/ifcvf_vdpa.c +1
-0@@ -369,6 +369,7 @@ struct rte_vdpa_dev_info { irq_set->index =VFIO_PCI_MSIX_IRQ_INDEX ; irq_set->start =0 ; fd_ptr = (int *)&irq_set->data; +/* The first interrupt is for the configure space change notification */
drivers/vdpa/ifc/ifcvf_vdpa.c +7
-1fd_ptr[RTE_INTR_VEC_ZERO_OFFSET ] =rte_intr_fd_get (internal->pdev->intr_handle);@@ -378,7 +379,13 @@ struct rte_vdpa_dev_info {for (i =0 ; i < nr_vring; i++) {rte_vhost_get_vhost_vring (internal->vid, i, &vring); fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; -if ((i &1 ) ==0 && m_rx == true) { +if (m_rx == true && + ((i &1 ) ==0 || internal->hw.device_type ==IFCVF_BLK )) { +/* For the net we only need to relay rx queue, +* which will change the mem of VM. +* For the blk we need to relay all the read cmd +* of each queue +*/ fd =eventfd (0 ,EFD_NONBLOCK |EFD_CLOEXEC );if (fd <0 ) {
Quoted 3 lines
DRV_LOG(ERR, "can't setup eventfd: %s",
--
1.8.3.1
Btw: add my R-by in next version as I see in this version, they are missed.
PATCH v9 6/13 vdpa/ifc: add block device SW live-migration
In reply to Andy PeiAdd SW live-migration support to block device. For block device, it is critical that no packet should be dropped. So when virtio blk device is paused, make sure hardware last_avail_idx and last_used_idx are the same. This indicates all requests have received acks, and no inflight IO.
--- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 39 insertions(+), 4 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
@@ -63,6 +63,7 @@#define IFCVF_MEDIATED_VRING 0x200000000000#define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff#ifndef VHOST_USER_PROTOCOL_F_CONFIG
drivers/vdpa/ifc/ifcvf_vdpa.c +38 -4
69 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RE: [PATCH v9 06/13] vdpa/ifc: add block device SW live-migration
Quoted 16 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 06/13] vdpa/ifc: add block device SW live-migrationAdd SW live-migration support to block device.
For block device, it is critical that no packet
should be dropped. So when virtio blk device is
paused, make sure hardwarelast_avail_idxandlast_used_idxare the same. This indicates all
requests have received acks, and no inflight IO.
Quoted 6 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 39 insertions(+), 4 deletions(-)
PATCH v9 7/13 vhost: add API to get vDPA device type
In reply to Andy PeiVhost backend of different devices have different features. Add an API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices.
--- doc/guides/prog_guide/vhost_lib.rst | 5 ++++ doc/guides/rel_notes/release_22_07.rst | 4 ++++ lib/vhost/rte_vhost.h | 17 +++++++++++++ lib/vhost/socket.c | 44 ++++++++++++++++++++++++++++++++++ lib/vhost/vdpa_driver.h | 3 +++ lib/vhost/version.map | 1 + 6 files changed, 74 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +5 -0
@@ -282,6 +282,11 @@ The following is an overview of some key Vhost API functions: Clear inflight packets which are submitted to DMA engine in vhost async data path. Completed packets are returned to applications through ``pkts``. +* ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` + + Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, + VDPA_DEVICE_TYPE_BLK. + Vhost-user Implementations --------------------------
doc/guides/rel_notes/release_22_07.rst +4 -0
@@ -60,6 +60,10 @@ New Features Added an API which can get the number of in-flight packets in vhost async data path without using lock. +* **Added vhost API to get the device type of a vDPA device.** + + Added an API which can get the device type of vDPA device. + * **Updated Intel iavf driver.** * Added Tx QoS queue rate limitation support.
lib/vhost/rte_vhost.h +17 -0
@@ -117,6 +117,9 @@#define RTE_MAX_VHOST_DEVICE 1024 +#define RTE_VHOST_VDPA_DEVICE_TYPE_NET 0 +#define RTE_VHOST_VDPA_DEVICE_TYPE_BLK 1 +struct rte_vdpa_device ;/**
@@ -486,6 +489,20 @@ struct rte_vdpa_device * rte_vhost_driver_get_vdpa_device(const char *path);/** +* Get the device type of the vdpa device. +* +* @param path +* The vhost-user socket file path +* @param type +* the device type of the vdpa device +* @return +* 0 on success, -1 on failure +*/ +__rte_experimental +int +rte_vhost_driver_get_vdpa_dev_type (const char *path,uint32_t *type); + +/** * Set the feature bits the vhost-user driver supports. * * @param path
lib/vhost/socket.c +44 -0
50 lines not on this page · raw message
lib/vhost/vdpa_driver.h +3 -0
@@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops {/** Set the device configuration space */ int (*set_config)(int vid,uint8_t *config,uint32_t offset,uint32_t size,uint32_t flags); + +/** get device type: net device, blk device... */ +int (*get_dev_type)(struct rte_vdpa_device *dev,uint32_t *type); };/**
lib/vhost/version.map +1 -0
@@ -90,6 +90,7 @@ EXPERIMENTAL { # added in 22.07 rte_vhost_async_get_inflight_thread_unsafe; + rte_vhost_driver_get_vdpa_dev_type; };
Signature · 1 line
1.8.3.1
RE: [PATCH v9 07/13] vhost: add API to get vDPA device type
Quoted 15 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 07/13] vhost: add API to get vDPA device typeVhost backend of different devices have different features. Add an API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices.
Signed-off-by Andy Pei <[email protected]>
Quoted 9 lines
--- doc/guides/prog_guide/vhost_lib.rst | 5 ++++ doc/guides/rel_notes/release_22_07.rst | 4 ++++ lib/vhost/rte_vhost.h | 17 +++++++++++++ lib/vhost/socket.c | 44 ++++++++++++++++++++++++++++++++++ lib/vhost/vdpa_driver.h | 3 +++ lib/vhost/version.map | 1 + 6 files changed, 74 insertions(+)
PATCH v9 8/13 vdpa/ifc: add get device type ops to ifc driver
In reply to Andy PeiAdd get device type ops to ifc driver.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +25 -0
@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {return 0 ; } +static int +ifcvf_get_device_type (struct rte_vdpa_device *vdev, +uint32_t *type) +{ +struct ifcvf_internal *internal; +struct internal_list *list; + + list =find_internal_resource_by_vdev (vdev); +if (list ==NULL ) { +DRV_LOG (ERR ,"Invalid vDPA device: %p" , vdev); +return -1 ; + } + + internal = list->internal; + +if (internal->hw.device_type ==IFCVF_BLK ) + *type =RTE_VHOST_VDPA_DEVICE_TYPE_BLK ; +else + *type =RTE_VHOST_VDPA_DEVICE_TYPE_NET ; + +return 0 ; +} +static struct rte_vdpa_dev_ops ifcvf_net_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features,
@@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info { .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, + .get_dev_type = ifcvf_get_device_type, }; static inline int
@@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info { .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_get_device_type, };struct rte_vdpa_dev_info dev_info[] = {
Signature · 1 line
1.8.3.1
RE: [PATCH v9 08/13] vdpa/ifc: add get device type ops to ifc driver
Quoted 67 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 08/13] vdpa/ifc: add get device type ops to ifc driverAdd get device type ops to ifc driver.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)drivers/vdpa/ifc/ifcvf_vdpa.c +25
-1@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {return 0 ; } +static int +ifcvf_get_device_type (struct rte_vdpa_device *vdev, +uint32_t *type) +{ +struct ifcvf_internal *internal; +struct internal_list *list; + + list =find_internal_resource_by_vdev (vdev); +if (list ==NULL ) { +DRV_LOG (ERR ,"Invalid vDPA device: %p" , vdev); +return -1 ; + } + + internal = list->internal; + +if (internal->hw.device_type ==IFCVF_BLK ) + *type =RTE_VHOST_VDPA_DEVICE_TYPE_BLK ; +else + *type =RTE_VHOST_VDPA_DEVICE_TYPE_NET ; + +return 0 ; +} +static struct rte_vdpa_dev_ops ifcvf_net_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features,@@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info { .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, + .get_dev_type = ifcvf_get_device_type, }; static inline int@@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info { .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_get_device_type, };struct rte_vdpa_dev_info dev_info[] = { --
Quoted 1 line
1.8.3.1
PATCH v9 9/13 examples/vdpa: add virtio blk support
In reply to Andy PeiAdd virtio blk device support to vDPA example.
--- examples/vdpa/main.c | 56 ++++++++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 50 +++++++++++++++++++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c +56 -0
@@ -20,6 +20,7 @@#include <cmdline_parse_string.h> #include <cmdline_parse_num.h> #include <cmdline.h> +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128#define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,53 @@ struct vdpa_port { };static int +vdpa_blk_device_set_features_and_protocol (const char *path) +{ +uint64_t protocol_features =0 ; +int ret; + + ret =rte_vhost_driver_set_features (path,VHOST_BLK_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_disable_features (path, +VHOST_BLK_DISABLED_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_disable_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_get_protocol_features (path, &protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_get_protocol_features for %s failed.\n" , + path); + goto out; + } + + protocol_features |=VHOST_BLK_PROTOCOL_FEATURES ; + + ret =rte_vhost_driver_set_protocol_features (path, protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_protocol_features for %s failed.\n" , + path); + } + +out: +return ret; +} + +static int start_vdpa (struct vdpa_port *vport) { +uint32_t device_type =0 ;int ret;char *socket_path = vport->ifname;
@@ -192,6 +238,16 @@ struct vdpa_port {"attach vdpa device failed: %s\n" , socket_path); + ret =rte_vhost_driver_get_vdpa_dev_type (socket_path, &device_type); +if (ret ==0 && device_type ==RTE_VHOST_VDPA_DEVICE_TYPE_BLK ) { +RTE_LOG (NOTICE ,VDPA ,"%s is a blk device\n" , socket_path); + ret =vdpa_blk_device_set_features_and_protocol (socket_path); +if (ret !=0 ) +rte_exit (EXIT_FAILURE , +"set vhost blk driver features and protocol features failed: %s\n" , + socket_path); + } +if (rte_vhost_driver_start (socket_path) <0 ) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n",
examples/vdpa/vdpa_blk_compact.h new file +50 -0
50 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RE: [PATCH v9 09/13] examples/vdpa: add virtio blk support
Quoted 11 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 09/13] examples/vdpa: add virtio blk supportAdd virtio blk device support to vDPA example.
Quoted 7 lines
Signed-off-by Andy Pei <[email protected]>--- examples/vdpa/main.c | 56 ++++++++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 50 +++++++++++++++++++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.h
PATCH v9 10/13 usertools: add support for virtio blk device
In reply to Andy PeiAdd virtio blk device support to devbind.
--- usertools/dpdk-devbind.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-devbind.py +4 -1
@@ -72,6 +72,9 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001,1042', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class]
@@ -82,7 +85,7 @@ compress_devices = [cavium_zip] regex_devices = [cn9k_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, - intel_ntb_skx, intel_ntb_icx] + intel_ntb_skx, intel_ntb_icx, virtio_blk] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
Signature · 1 line
1.8.3.1
RE: [PATCH v9 10/13] usertools: add support for virtio blk device
Quoted 40 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 10/13] usertools: add support for virtio blk deviceAdd virtio blk device support to devbind.
Signed-off-by Andy Pei <[email protected]>--- usertools/dpdk-devbind.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)usertools/dpdk-devbind.py +4
-2@@ -72,6 +72,9 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001,1042', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class]@@ -82,7 +85,7 @@ compress_devices = [cavium_zip] regex_devices = [cn9k_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, - intel_ntb_skx, intel_ntb_icx] + intel_ntb_skx, intel_ntb_icx, virtio_blk] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties --
Quoted 1 line
1.8.3.1
PATCH v9 11/13 vdpa/ifc: add log for config space of virtio blk
In reply to Andy PeiAdd some log of virtio blk device config space information at VDPA launch before qemu connects.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +34 -0
@@ -1382,6 +1382,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +uint64_t capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;
@@ -1448,6 +1451,37 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_BLK ].features; + +/* cannot read 64-bit register in one attempt, +* so read byte by byte. +*/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +/* The capacity is number of sectors in 512-byte. +* So right shift 1 bit we get in K, +* another right shift 10 bits we get in M, +* right shift 10 more bits, we get in G. +* To show capacity in G, we right shift 21 bits in total. +*/ +DRV_LOG (DEBUG ,"capacity : %" PRIu64"G" , capacity >>21 ); + +DRV_LOG (DEBUG ,"size_max : 0x%08x" , + internal->hw.blk_cfg->size_max); +DRV_LOG (DEBUG ,"seg_max : 0x%08x" , + internal->hw.blk_cfg->seg_max); +DRV_LOG (DEBUG ,"blk_size : 0x%08x" , + internal->hw.blk_cfg->blk_size); +DRV_LOG (DEBUG ,"geometry" ); +DRV_LOG (DEBUG ," cylinders: %u" , + internal->hw.blk_cfg->geometry.cylinders); +DRV_LOG (DEBUG ," heads : %u" , + internal->hw.blk_cfg->geometry.heads); +DRV_LOG (DEBUG ," sectors : %u" , + internal->hw.blk_cfg->geometry.sectors); +DRV_LOG (DEBUG ,"num_queues: 0x%08x" , + internal->hw.blk_cfg->num_queues); } list->internal = internal;
Signature · 1 line
1.8.3.1
RE: [PATCH v9 11/13] vdpa/ifc: add log for config space of virtio blk
Quoted 70 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 11/13] vdpa/ifc: add log for config space of virtio blkAdd some log of virtio blk device config space information at VDPA launch before qemu connects.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)drivers/vdpa/ifc/ifcvf_vdpa.c +34
-1@@ -1382,6 +1382,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +uint64_t capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;@@ -1448,6 +1451,37 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_BLK ].features; + +/* cannot read 64-bit register in one attempt, +* so read byte by byte. +*/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +/* The capacity is number of sectors in 512-byte. +* So right shift 1 bit we get in K, +* another right shift 10 bits we get in M, +* right shift 10 more bits, we get in G. +* To show capacity in G, we right shift 21 bits in total. +*/ +DRV_LOG (DEBUG ,"capacity : %" PRIu64"G" , capacity >>21 ); + +DRV_LOG (DEBUG ,"size_max : 0x%08x" , + internal->hw.blk_cfg->size_max); +DRV_LOG (DEBUG ,"seg_max : 0x%08x" , + internal->hw.blk_cfg->seg_max); +DRV_LOG (DEBUG ,"blk_size : 0x%08x" , + internal->hw.blk_cfg->blk_size); +DRV_LOG (DEBUG ,"geometry" ); +DRV_LOG (DEBUG ," cylinders: %u" , + internal->hw.blk_cfg->geometry.cylinders); +DRV_LOG (DEBUG ," heads : %u" , + internal->hw.blk_cfg->geometry.heads); +DRV_LOG (DEBUG ," sectors : %u" , + internal->hw.blk_cfg->geometry.sectors); +DRV_LOG (DEBUG ,"num_queues: 0x%08x" , + internal->hw.blk_cfg->num_queues); } list->internal = internal; --
Quoted 1 line
1.8.3.1
PATCH v9 12/13 vdpa/ifc: add interrupt handling for config space
In reply to Andy PeiCreate a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 121 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +120 -1
@@ -53,7 +53,9 @@ struct ifcvf_internal {int vfio_group_fd;int vfio_dev_fd;pthread_t tid;/* thread for notify relay */ +pthread_t intr_tid;/* thread for config space change interrupt relay */ int epfd; +int csc_epfd;int vid;struct rte_vdpa_device *vdev;uint16_t max_queues;
@@ -566,6 +568,114 @@ struct rte_vdpa_dev_info {return 0 ; } +static void +virtio_interrupt_handler (struct ifcvf_internal *internal) +{ +int vid = internal->vid; +int ret; + + ret =rte_vhost_slave_config_change (vid,1 ); +if (ret) +DRV_LOG (ERR ,"failed to notify the guest about configuration space change." ); +} + +static void * +intr_relay (void *arg) +{ +struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; +struct epoll_event csc_event; +struct epoll_event ev; +uint64_t buf; +int nbytes; +int csc_epfd, csc_val =0 ; + + csc_epfd =epoll_create (1 ); +if (csc_epfd <0 ) { +DRV_LOG (ERR ,"failed to create epoll for config space change." ); +return NULL ; + } + + ev.events =EPOLLIN |EPOLLPRI |EPOLLRDHUP |EPOLLHUP ; + ev.data.fd =rte_intr_fd_get (internal->pdev->intr_handle); +if (epoll_ctl (csc_epfd,EPOLL_CTL_ADD , +rte_intr_fd_get (internal->pdev->intr_handle), &ev) <0 ) { +DRV_LOG (ERR ,"epoll add error: %s" ,strerror (errno)); + goto out; + } + + internal->csc_epfd = csc_epfd; + +for (;;) { + csc_val =epoll_wait (csc_epfd, &csc_event,1 ,-1 ); +if (csc_val <0 ) { +if (errno ==EINTR ) +continue ; +DRV_LOG (ERR ,"epoll_wait return fail." ); + goto out; + }else if (csc_val ==0 ) { +continue ; + }else { +/* csc_val > 0 */ + nbytes =read (csc_event.data.fd, &buf,8 ); +if (nbytes <0 ) { +if (errno ==EINTR || + errno ==EWOULDBLOCK || + errno ==EAGAIN ) +continue ; +DRV_LOG (ERR ,"Error reading from file descriptor %d: %s\n" , + csc_event.data.fd, +strerror (errno)); + goto out; + }else if (nbytes ==0 ) { +DRV_LOG (ERR ,"Read nothing from file descriptor %d\n" , + csc_event.data.fd); +continue ; + }else { +virtio_interrupt_handler (internal); + } + } + } + +out: +if (csc_epfd >=0 ) +close (csc_epfd); + internal->csc_epfd =-1 ; + +return NULL ; +} + +static int +setup_intr_relay (struct ifcvf_internal *internal) +{ +char name[THREAD_NAME_LEN ]; +int ret; + +snprintf (name,sizeof (name),"ifc-intr-%d" , internal->vid); + ret =rte_ctrl_thread_create (&internal->intr_tid, name,NULL , + intr_relay, (void *)internal); +if (ret) { +DRV_LOG (ERR ,"failed to create notify relay pthread." ); +return -1 ; + } +return 0 ; +} + +static void +unset_intr_relay (struct ifcvf_internal *internal) +{ +void *status; + +if (internal->intr_tid) { +pthread_cancel (internal->intr_tid); +pthread_join (internal->intr_tid, &status); + } + internal->intr_tid =0 ; + +if (internal->csc_epfd >=0 ) +close (internal->csc_epfd); + internal->csc_epfd =-1 ; +} +static int update_datapath (struct ifcvf_internal *internal) {
@@ -592,10 +702,16 @@ struct rte_vdpa_dev_info {if (ret) goto err; + ret =setup_intr_relay (internal); +if (ret) + goto err; +rte_atomic32_set (&internal->running,1 ); }else if (rte_atomic32_read (&internal->running) && (!rte_atomic32_read (&internal->started) || !rte_atomic32_read (&internal->dev_attached))) { +unset_intr_relay (internal); + ret =unset_notify_relay (internal);if (ret) goto err;
@@ -812,7 +928,7 @@ struct rte_vdpa_dev_info {if (nfds <0 ) {if (errno ==EINTR )continue ; -DRV_LOG (ERR ,"epoll_wait return fail\n" ); +DRV_LOG (ERR ,"epoll_wait return fail." );return NULL ; }
@@ -888,6 +1004,9 @@ struct rte_vdpa_dev_info {/* stop the direct IO data path */ unset_notify_relay (internal);vdpa_ifcvf_stop (internal); + +unset_intr_relay (internal); +vdpa_disable_vfio_intr (internal); ret =rte_vhost_host_notifier_ctrl (vid,RTE_VHOST_QUEUE_ALL , false);
Signature · 1 line
1.8.3.1
RE: [PATCH v9 12/13] vdpa/ifc: add interrupt handling for config space
Quoted 10 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 12/13] vdpa/ifc: add interrupt handling for config space
Quoted 8 lines
Create a thread to poll and relay config space change interrupt. Use
VHOST_USER_SLAVE_CONFIG_CHANGE_MSGto inform QEMU.Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/ifcvf_vdpa.c | 121 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 1 deletion(-)
PATCH v9 13/13 vdpa/ifc/base: access correct register for blk device
In reply to Andy PeiRegister address is different between net and blk device. We are re-useing most of the code, when register address is different, we have to check net and blk device go through different code.
--- drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +27 -7
@@ -218,10 +218,17 @@ &cfg->queue_used_hi);IFCVF_WRITE_REG16 (hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ) = - (u32 )hw->vring[i].last_avail_idx | - ((u32 )hw->vring[i].last_used_idx <<16 ); +if (hw->device_type ==IFCVF_BLK ) + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 ); +else + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 );IFCVF_WRITE_REG16 (i +1 , &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -254,9 +261,22 @@IFCVF_WRITE_REG16 (i, &cfg->queue_select);IFCVF_WRITE_REG16 (0 , &cfg->queue_enable);IFCVF_WRITE_REG16 (IFCVF_MSI_NO_VECTOR , &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ); - hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); + +if (hw->device_type ==IFCVF_BLK ) + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); +else + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ); + +if (hw->device_type ==IFCVF_BLK ) + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); +else + hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); hw->vring[i].last_used_idx = (u16)(ring_state >>16 ); } }
Signature · 1 line
1.8.3.1
RE: [PATCH v9 13/13] vdpa/ifc/base: access correct register for blk device
Quoted 10 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 13/13] vdpa/ifc/base: access correct register for blk device
Quoted 2 lines
Register address is different between net and blk device. We are re-useing most of the code, when register address is
re-using
With this fixed:
Quoted 8 lines
different, we have to check net and blk device go through different code.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)
RE: [PATCH v9 13/13] vdpa/ifc/base: access correct register for blk device
Hi Chenbo,
Sure.
Thanks for your review.
I will address in V10.
Quoted 38 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, May 24, 2022 11:10 AM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v9 13/13] vdpa/ifc/base: access correct register for blk deviceQuoted 11 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Monday, May 23, 2022 5:35 PM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v9 13/13] vdpa/ifc/base: access correct register for blk deviceQuoted 2 lines
Register address is different between net and blk device. We are re-useing most of the code, when register address is
re-using
With this fixed:
Reviewed-by Chenbo Xia <[email protected]>Quoted 8 lines
different, we have to check net and blk device go through different code.
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)
v10 cover add virtio_blk device support to vdpa/ifc
In reply to Andy PeiThis patch set add virtio_blk device support to vdpa/ifc driver. With a lot of similarities, I re-use part of vdpa/ifc driver. Distinguish the virtio net and blk device by device id, and implement specific features and ops.
Add example to vdpa to support virtio_blk device.
To support blk device live migration, some modification to vhost lib. Perform dev_conf op only under VHOST_USER_SET_VRING_CALL msg.
v10:
fix typo.
add more NULL pointer check.
fix a error handler.
v9:
fix some naming issue.
add more NULL pointer check.
close open file when errors occur.
v8:
delete some redundant code.
fix some commit log.
v7:
Check on expected fd num in new vhost msg handler. Sanity check on vhost msg size.
Fix typo.
Add commit log to help understand code. Remove duplicated code.
Add new API to get vDPA device type.
v6:
fix some commit log.
add vhost socket in log output to make it more user-friendly. when driver ops fail, just output some log, do not break message handler. check vhost msg size in msg handler.
v5:
fix some coding style issues.
v4:
add args "isblk" to vdpa example to specify a block device, fix some issue in example.
Make sure code specify for block device does not affect net device.
v3:
Fix some compile issues.
v2:
Fix some coding style issues.
Andy Pei (13):
vdpa/ifc: add support for virtio blk device
vhost: add vDPA ops for blk device
vhost: add vhost msg support for get/set config vdpa/ifc: add blk ops for ifc device vdpa/ifc: add vDPA interrupt relay for blk device vdpa/ifc: add block device SW live-migration
vhost: add API to get vDPA device type vdpa/ifc: add get device type ops to ifc driver examples/vdpa: add virtio blk support
usertools: add support for virtio blk device vdpa/ifc: add log for config space of virtio blk vdpa/ifc: add interrupt handling for config space vdpa/ifc/base: access correct register for blk device
doc/guides/prog_guide/vhost_lib.rst | 5 + doc/guides/rel_notes/release_22_07.rst | 4 + drivers/vdpa/ifc/base/ifcvf.c | 34 ++- drivers/vdpa/ifc/base/ifcvf.h | 27 ++- drivers/vdpa/ifc/ifcvf_vdpa.c | 420 +++++++++++++++++++++++++++++++-- examples/vdpa/main.c | 56 +++++ examples/vdpa/vdpa_blk_compact.h | 50 ++++ lib/vhost/rte_vhost.h | 17 ++ lib/vhost/socket.c | 44 ++++ lib/vhost/vdpa_driver.h | 11 +- lib/vhost/version.map | 1 + lib/vhost/vhost_user.c | 85 +++++++ lib/vhost/vhost_user.h | 13 + usertools/dpdk-devbind.py | 5 +- 14 files changed, 741 insertions(+), 31 deletions(-) create mode 100644 examples/vdpa/vdpa_blk_compact.h
Signature · 1 line
1.8.3.1
PATCH v10 1/13 vdpa/ifc: add support for virtio blk device
Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops.
--- drivers/vdpa/ifc/base/ifcvf.h | 22 +++++++--- drivers/vdpa/ifc/ifcvf_vdpa.c | 95 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 102 insertions(+), 15 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +17 -5
@@ -5,12 +5,19 @@#ifndef _IFCVF_H_#define _IFCVF_H_ +#include <linux/virtio_blk.h> #include "ifcvf_osdep.h" -#define IFCVF_VENDOR_ID 0x1AF4 -#define IFCVF_DEVICE_ID 0x1041 -#define IFCVF_SUBSYS_VENDOR_ID 0x8086 -#define IFCVF_SUBSYS_DEVICE_ID 0x001A +#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +#define IFCVF_VENDOR_ID 0x1AF4 +#define IFCVF_NET_DEVICE_ID 0x1041 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_SUBSYS_VENDOR_ID 0x8086 +#define IFCVF_SUBSYS_DEVICE_ID 0x001A +#define IFCVF_BLK_DEVICE_ID 0x0002#define IFCVF_MAX_QUEUES 1
@@ -126,13 +133,18 @@ struct ifcvf_hw {u8 notify_region;u32 notify_off_multiplier;struct ifcvf_pci_common_cfg *common_cfg; -struct ifcvf_net_config *dev_cfg; +union { +struct ifcvf_net_config *net_cfg; +struct virtio_blk_config *blk_cfg; +void *dev_cfg; + };u8 *isr;u16 *notify_base;u16 *notify_addr[IFCVF_MAX_QUEUES *2 ];u8 *lm_cfg;struct vring_info vring[IFCVF_MAX_QUEUES *2 ];u8 nr_vring; +int device_type;struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE ]; };
drivers/vdpa/ifc/ifcvf_vdpa.c +85 -10
141 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v10 2/13 vhost: add vDPA ops for blk device
In reply to Andy PeiGet_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops.
--- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
lib/vhost/vdpa_driver.h +6 -2
@@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops {/** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev,int qid); -/** Reserved for future extension */ -void *reserved[2 ]; +/** Get the device configuration space */ +int (*get_config)(int vid,uint8_t *config,uint32_t size); + +/** Set the device configuration space */ +int (*set_config)(int vid,uint8_t *config,uint32_t offset, +uint32_t size,uint32_t flags); };/**
Signature · 1 line
1.8.3.1
PATCH v10 3/13 vhost: add vhost msg support for get/set config
In reply to Andy PeiAdd support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device.
--- lib/vhost/vhost_user.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 ++++++++ 2 files changed, 98 insertions(+)
lib/vhost/vhost_user.c +85 -0
@@ -2468,6 +2468,89 @@ static int is_vring_iotlb(struct virtio_net *dev, }static int +vhost_user_get_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (!vdpa_dev) { +VHOST_LOG_CONFIG (ERR ,"(%s) is not vDPA device!\n" , + dev->ifname); +return RTE_VHOST_MSG_RESULT_ERR ; + } + +if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config (dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); +if (ret !=0 ) { + ctx->msg.size =0 ; +VHOST_LOG_CONFIG (ERR , +"(%s) get_config() return error!\n" , + dev->ifname); + } + }else { +VHOST_LOG_CONFIG (ERR ,"(%s) get_config() not supported!\n" , + dev->ifname); + } + +return RTE_VHOST_MSG_RESULT_REPLY ; +} + +static int +vhost_user_set_config (struct virtio_net **pdev, +struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ +struct virtio_net *dev = *pdev; +struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; +int ret =0 ; + +if (validate_msg_fds (dev, ctx,0 ) !=0 ) +return RTE_VHOST_MSG_RESULT_ERR ; + +if (ctx->msg.payload.cfg.size >VHOST_USER_MAX_CONFIG_SIZE ) { +VHOST_LOG_CONFIG (ERR , +"(%s) vhost_user_config size: %" PRIu32", should not be larger than %d\n" , + dev->ifname, ctx->msg.payload.cfg.size, +VHOST_USER_MAX_CONFIG_SIZE ); + goto out; + } + +if (!vdpa_dev) { +VHOST_LOG_CONFIG (ERR ,"(%s) is not vDPA device!\n" , + dev->ifname); + goto out; + } + +if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config (dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); +if (ret) +VHOST_LOG_CONFIG (ERR , +"(%s) set_config() return error!\n" , + dev->ifname); + }else { +VHOST_LOG_CONFIG (ERR ,"(%s) set_config() not supported!\n" , + dev->ifname); + } + +return RTE_VHOST_MSG_RESULT_OK ; + +out: +return RTE_VHOST_MSG_RESULT_ERR ; +} + +static int vhost_user_iotlb_msg (struct virtio_net **pdev,struct vhu_msg_context *ctx, int main_fd __rte_unused)
@@ -2686,6 +2769,8 @@ static int is_vring_iotlb(struct virtio_net *dev, VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU, vhost_user_net_set_mtu, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD, vhost_user_set_req_fd, true) \ VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG, vhost_user_get_config, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG, vhost_user_set_config, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE, vhost_user_set_postcopy_advise, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false) \VHOST_MESSAGE_HANDLER (VHOST_USER_POSTCOPY_END , vhost_user_postcopy_end, false) \
lib/vhost/vhost_user.h +13 -0
@@ -50,6 +50,8 @@VHOST_USER_NET_SET_MTU =20 ,VHOST_USER_SET_SLAVE_REQ_FD =21 ,VHOST_USER_IOTLB_MSG =22 , +VHOST_USER_GET_CONFIG =24 , +VHOST_USER_SET_CONFIG =25 ,VHOST_USER_CRYPTO_CREATE_SESS =26 ,VHOST_USER_CRYPTO_CLOSE_SESS =27 , VHOST_USER_POSTCOPY_ADVISE = 28,
@@ -123,6 +125,16 @@uint16_t queue_size; } VhostUserInflight; +#define VHOST_USER_MAX_CONFIG_SIZE 256 + +/** Get/set config msg payload */ +struct vhost_user_config { +uint32_t offset; +uint32_t size; +uint32_t flags; +uint8_t region[VHOST_USER_MAX_CONFIG_SIZE ]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */
@@ -146,6 +158,7 @@VhostUserCryptoSessionParam crypto_session;VhostUserVringArea area;VhostUserInflight inflight; +struct vhost_user_config cfg; } payload;/* Nothing should be added after the payload */ }__rte_packed VhostUserMsg;
Signature · 1 line
1.8.3.1
RE: [PATCH v10 03/13] vhost: add vhost msg support for get/set config
Quoted 13 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Tuesday, May 24, 2022 10:48 AM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v10 03/13] vhost: add vhost msg support for get/set configAdd support for
VHOST_USER_GET_CONFIGandVHOST_USER_SET_CONFIG.VHOST_USER_GET_CONFIGandVHOST_USER_SET_CONFIGmessage is only supported by virtio blk VDPA device.
Quoted 6 lines
Signed-off-by Andy Pei <[email protected]>--- lib/vhost/vhost_user.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 ++++++++ 2 files changed, 98 insertions(+)
PATCH v10 4/13 vdpa/ifc: add blk ops for ifc device
In reply to Andy PeiFor virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.
--- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 96 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 99 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/base/ifcvf.h +4 -0
@@ -65,6 +65,10 @@#define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif +struct ifcvf_pci_cap {u8 cap_vndr;/* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next;/* Generic PCI field: next ptr. */
drivers/vdpa/ifc/ifcvf_vdpa.c +95 -1
114 lines not on this page · raw message
Signature · 1 line
1.8.3.1
RE: [PATCH v10 04/13] vdpa/ifc: add blk ops for ifc device
Quoted 14 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Tuesday, May 24, 2022 10:48 AM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v10 04/13] vdpa/ifc: add blk ops for ifc deviceFor virtio blk device, re-use part of ifc driver ops. Implement
ifcvf_blk_get_configfor virtio blk device. SupportVHOST_USER_PROTOCOL_F_CONFIGfeature for virtio blk device.
Quoted 6 lines
Signed-off-by Andy Pei <[email protected]>--- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 96 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 99 insertions(+), 1 deletion(-)
RE: [PATCH v10 04/13] vdpa/ifc: add blk ops for ifc device
Hi Chenbo,
Thanks for your effort.
Quoted 32 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, May 24, 2022 2:28 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: RE: [PATCH v10 04/13] vdpa/ifc: add blk ops for ifc deviceQuoted 14 lines
-----Original Message-----
From: Pei, Andy <[email protected]>
Sent: Tuesday, May 24, 2022 10:48 AM
To: [email protected]
Cc: Xia, Chenbo <[email protected]>; [email protected]; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: [PATCH v10 04/13] vdpa/ifc: add blk ops for ifc deviceFor virtio blk device, re-use part of ifc driver ops. Implement
ifcvf_blk_get_configfor virtio blk device. SupportVHOST_USER_PROTOCOL_F_CONFIGfeature for virtio blk device.Quoted 6 lines
Signed-off-by Andy Pei <[email protected]>---
drivers/vdpa/ifc/base/ifcvf.h| 4 ++drivers/vdpa/ifc/ifcvf_vdpa.c
| 96
++++++++++++++++++++++++++++++++++++++++++-2 files changed, 99 insertions(+), 1 deletion(-)Reviewed-by Chenbo Xia <[email protected]>
PATCH v10 5/13 vdpa/ifc: add vDPA interrupt relay for blk device
In reply to Andy PeiFor the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +8 -1
@@ -369,6 +369,7 @@ struct rte_vdpa_dev_info { irq_set->index =VFIO_PCI_MSIX_IRQ_INDEX ; irq_set->start =0 ; fd_ptr = (int *)&irq_set->data; +/* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET ] =rte_intr_fd_get (internal->pdev->intr_handle);
@@ -378,7 +379,13 @@ struct rte_vdpa_dev_info {for (i =0 ; i < nr_vring; i++) {rte_vhost_get_vhost_vring (internal->vid, i, &vring); fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; -if ((i &1 ) ==0 && m_rx == true) { +if (m_rx == true && + ((i &1 ) ==0 || internal->hw.device_type ==IFCVF_BLK )) { +/* For the net we only need to relay rx queue, +* which will change the mem of VM. +* For the blk we need to relay all the read cmd +* of each queue +*/ fd =eventfd (0 ,EFD_NONBLOCK |EFD_CLOEXEC );if (fd <0 ) { DRV_LOG(ERR, "can't setup eventfd: %s",
Signature · 1 line
1.8.3.1
PATCH v10 6/13 vdpa/ifc: add block device SW live-migration
In reply to Andy PeiAdd SW live-migration support to block device. For block device, it is critical that no packet should be dropped. So when virtio blk device is paused, make sure hardware last_avail_idx and last_used_idx are the same. This indicates all requests have received acks, and no inflight IO.
--- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 39 insertions(+), 4 deletions(-)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
@@ -63,6 +63,7 @@#define IFCVF_MEDIATED_VRING 0x200000000000#define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff#ifndef VHOST_USER_PROTOCOL_F_CONFIG
drivers/vdpa/ifc/ifcvf_vdpa.c +38 -4
69 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v10 7/13 vhost: add API to get vDPA device type
In reply to Andy PeiVhost backend of different devices have different features. Add an API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices.
--- doc/guides/prog_guide/vhost_lib.rst | 5 ++++ doc/guides/rel_notes/release_22_07.rst | 4 ++++ lib/vhost/rte_vhost.h | 17 +++++++++++++ lib/vhost/socket.c | 44 ++++++++++++++++++++++++++++++++++ lib/vhost/vdpa_driver.h | 3 +++ lib/vhost/version.map | 1 + 6 files changed, 74 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +5 -0
@@ -282,6 +282,11 @@ The following is an overview of some key Vhost API functions: Clear inflight packets which are submitted to DMA engine in vhost async data path. Completed packets are returned to applications through ``pkts``. +* ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` + + Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, + VDPA_DEVICE_TYPE_BLK. + Vhost-user Implementations --------------------------
doc/guides/rel_notes/release_22_07.rst +4 -0
@@ -60,6 +60,10 @@ New Features Added an API which can get the number of in-flight packets in vhost async data path without using lock. +* **Added vhost API to get the device type of a vDPA device.** + + Added an API which can get the device type of vDPA device. + * **Updated Intel iavf driver.** * Added Tx QoS queue rate limitation support.
lib/vhost/rte_vhost.h +17 -0
@@ -117,6 +117,9 @@#define RTE_MAX_VHOST_DEVICE 1024 +#define RTE_VHOST_VDPA_DEVICE_TYPE_NET 0 +#define RTE_VHOST_VDPA_DEVICE_TYPE_BLK 1 +struct rte_vdpa_device ;/**
@@ -486,6 +489,20 @@ struct rte_vdpa_device * rte_vhost_driver_get_vdpa_device(const char *path);/** +* Get the device type of the vdpa device. +* +* @param path +* The vhost-user socket file path +* @param type +* the device type of the vdpa device +* @return +* 0 on success, -1 on failure +*/ +__rte_experimental +int +rte_vhost_driver_get_vdpa_dev_type (const char *path,uint32_t *type); + +/** * Set the feature bits the vhost-user driver supports. * * @param path
lib/vhost/socket.c +44 -0
50 lines not on this page · raw message
lib/vhost/vdpa_driver.h +3 -0
@@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops {/** Set the device configuration space */ int (*set_config)(int vid,uint8_t *config,uint32_t offset,uint32_t size,uint32_t flags); + +/** get device type: net device, blk device... */ +int (*get_dev_type)(struct rte_vdpa_device *dev,uint32_t *type); };/**
lib/vhost/version.map +1 -0
@@ -90,6 +90,7 @@ EXPERIMENTAL { # added in 22.07 rte_vhost_async_get_inflight_thread_unsafe; + rte_vhost_driver_get_vdpa_dev_type; };
Signature · 1 line
1.8.3.1
PATCH v10 8/13 vdpa/ifc: add get device type ops to ifc driver
In reply to Andy PeiAdd get device type ops to ifc driver.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +25 -0
@@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info {return 0 ; } +static int +ifcvf_get_device_type (struct rte_vdpa_device *vdev, +uint32_t *type) +{ +struct ifcvf_internal *internal; +struct internal_list *list; + + list =find_internal_resource_by_vdev (vdev); +if (list ==NULL ) { +DRV_LOG (ERR ,"Invalid vDPA device: %p" , vdev); +return -1 ; + } + + internal = list->internal; + +if (internal->hw.device_type ==IFCVF_BLK ) + *type =RTE_VHOST_VDPA_DEVICE_TYPE_BLK ; +else + *type =RTE_VHOST_VDPA_DEVICE_TYPE_NET ; + +return 0 ; +} +static struct rte_vdpa_dev_ops ifcvf_net_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features,
@@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info { .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, + .get_dev_type = ifcvf_get_device_type, }; static inline int
@@ -1332,6 +1356,7 @@ struct rte_vdpa_dev_info { .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_get_device_type, };struct rte_vdpa_dev_info dev_info[] = {
Signature · 1 line
1.8.3.1
PATCH v10 9/13 examples/vdpa: add virtio blk support
In reply to Andy PeiAdd virtio blk device support to vDPA example.
--- examples/vdpa/main.c | 56 ++++++++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 50 +++++++++++++++++++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.h
examples/vdpa/main.c +56 -0
@@ -20,6 +20,7 @@#include <cmdline_parse_string.h> #include <cmdline_parse_num.h> #include <cmdline.h> +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128#define MAX_VDPA_SAMPLE_PORTS 1024
@@ -159,8 +160,53 @@ struct vdpa_port { };static int +vdpa_blk_device_set_features_and_protocol (const char *path) +{ +uint64_t protocol_features =0 ; +int ret; + + ret =rte_vhost_driver_set_features (path,VHOST_BLK_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_disable_features (path, +VHOST_BLK_DISABLED_FEATURES ); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_disable_features for %s failed.\n" , + path); + goto out; + } + + ret =rte_vhost_driver_get_protocol_features (path, &protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_get_protocol_features for %s failed.\n" , + path); + goto out; + } + + protocol_features |=VHOST_BLK_PROTOCOL_FEATURES ; + + ret =rte_vhost_driver_set_protocol_features (path, protocol_features); +if (ret !=0 ) { +RTE_LOG (ERR ,VDPA , +"rte_vhost_driver_set_protocol_features for %s failed.\n" , + path); + } + +out: +return ret; +} + +static int start_vdpa (struct vdpa_port *vport) { +uint32_t device_type =0 ;int ret;char *socket_path = vport->ifname;
@@ -192,6 +238,16 @@ struct vdpa_port {"attach vdpa device failed: %s\n" , socket_path); + ret =rte_vhost_driver_get_vdpa_dev_type (socket_path, &device_type); +if (ret ==0 && device_type ==RTE_VHOST_VDPA_DEVICE_TYPE_BLK ) { +RTE_LOG (NOTICE ,VDPA ,"%s is a blk device\n" , socket_path); + ret =vdpa_blk_device_set_features_and_protocol (socket_path); +if (ret !=0 ) +rte_exit (EXIT_FAILURE , +"set vhost blk driver features and protocol features failed: %s\n" , + socket_path); + } +if (rte_vhost_driver_start (socket_path) <0 ) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n",
examples/vdpa/vdpa_blk_compact.h new file +50 -0
50 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v10 10/13 usertools: add support for virtio blk device
In reply to Andy PeiAdd virtio blk device support to devbind.
--- usertools/dpdk-devbind.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-devbind.py +4 -1
@@ -72,6 +72,9 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001,1042', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class]
@@ -82,7 +85,7 @@ compress_devices = [cavium_zip] regex_devices = [cn9k_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, - intel_ntb_skx, intel_ntb_icx] + intel_ntb_skx, intel_ntb_icx, virtio_blk] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
Signature · 1 line
1.8.3.1
PATCH v10 11/13 vdpa/ifc: add log for config space of virtio blk
In reply to Andy PeiAdd some log of virtio blk device config space information at VDPA launch before qemu connects.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +34 -0
@@ -1387,6 +1387,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +uint64_t capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;
@@ -1453,6 +1456,37 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_BLK ].features; + +/* cannot read 64-bit register in one attempt, +* so read byte by byte. +*/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t )*byte << (i *8 ); + } +/* The capacity is number of sectors in 512-byte. +* So right shift 1 bit we get in K, +* another right shift 10 bits we get in M, +* right shift 10 more bits, we get in G. +* To show capacity in G, we right shift 21 bits in total. +*/ +DRV_LOG (DEBUG ,"capacity : %" PRIu64"G" , capacity >>21 ); + +DRV_LOG (DEBUG ,"size_max : 0x%08x" , + internal->hw.blk_cfg->size_max); +DRV_LOG (DEBUG ,"seg_max : 0x%08x" , + internal->hw.blk_cfg->seg_max); +DRV_LOG (DEBUG ,"blk_size : 0x%08x" , + internal->hw.blk_cfg->blk_size); +DRV_LOG (DEBUG ,"geometry" ); +DRV_LOG (DEBUG ," cylinders: %u" , + internal->hw.blk_cfg->geometry.cylinders); +DRV_LOG (DEBUG ," heads : %u" , + internal->hw.blk_cfg->geometry.heads); +DRV_LOG (DEBUG ," sectors : %u" , + internal->hw.blk_cfg->geometry.sectors); +DRV_LOG (DEBUG ,"num_queues: 0x%08x" , + internal->hw.blk_cfg->num_queues); } list->internal = internal;
Signature · 1 line
1.8.3.1
PATCH v10 12/13 vdpa/ifc: add interrupt handling for config space
In reply to Andy PeiCreate a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 121 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +120 -1
@@ -53,7 +53,9 @@ struct ifcvf_internal {int vfio_group_fd;int vfio_dev_fd;pthread_t tid;/* thread for notify relay */ +pthread_t intr_tid;/* thread for config space change interrupt relay */ int epfd; +int csc_epfd;int vid;struct rte_vdpa_device *vdev;uint16_t max_queues;
@@ -566,6 +568,114 @@ struct rte_vdpa_dev_info {return 0 ; } +static void +virtio_interrupt_handler (struct ifcvf_internal *internal) +{ +int vid = internal->vid; +int ret; + + ret =rte_vhost_slave_config_change (vid,1 ); +if (ret) +DRV_LOG (ERR ,"failed to notify the guest about configuration space change." ); +} + +static void * +intr_relay (void *arg) +{ +struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; +struct epoll_event csc_event; +struct epoll_event ev; +uint64_t buf; +int nbytes; +int csc_epfd, csc_val =0 ; + + csc_epfd =epoll_create (1 ); +if (csc_epfd <0 ) { +DRV_LOG (ERR ,"failed to create epoll for config space change." ); +return NULL ; + } + + ev.events =EPOLLIN |EPOLLPRI |EPOLLRDHUP |EPOLLHUP ; + ev.data.fd =rte_intr_fd_get (internal->pdev->intr_handle); +if (epoll_ctl (csc_epfd,EPOLL_CTL_ADD , +rte_intr_fd_get (internal->pdev->intr_handle), &ev) <0 ) { +DRV_LOG (ERR ,"epoll add error: %s" ,strerror (errno)); + goto out; + } + + internal->csc_epfd = csc_epfd; + +for (;;) { + csc_val =epoll_wait (csc_epfd, &csc_event,1 ,-1 ); +if (csc_val <0 ) { +if (errno ==EINTR ) +continue ; +DRV_LOG (ERR ,"epoll_wait return fail." ); + goto out; + }else if (csc_val ==0 ) { +continue ; + }else { +/* csc_val > 0 */ + nbytes =read (csc_event.data.fd, &buf,8 ); +if (nbytes <0 ) { +if (errno ==EINTR || + errno ==EWOULDBLOCK || + errno ==EAGAIN ) +continue ; +DRV_LOG (ERR ,"Error reading from file descriptor %d: %s\n" , + csc_event.data.fd, +strerror (errno)); + goto out; + }else if (nbytes ==0 ) { +DRV_LOG (ERR ,"Read nothing from file descriptor %d\n" , + csc_event.data.fd); +continue ; + }else { +virtio_interrupt_handler (internal); + } + } + } + +out: +if (csc_epfd >=0 ) +close (csc_epfd); + internal->csc_epfd =-1 ; + +return NULL ; +} + +static int +setup_intr_relay (struct ifcvf_internal *internal) +{ +char name[THREAD_NAME_LEN ]; +int ret; + +snprintf (name,sizeof (name),"ifc-intr-%d" , internal->vid); + ret =rte_ctrl_thread_create (&internal->intr_tid, name,NULL , + intr_relay, (void *)internal); +if (ret) { +DRV_LOG (ERR ,"failed to create notify relay pthread." ); +return -1 ; + } +return 0 ; +} + +static void +unset_intr_relay (struct ifcvf_internal *internal) +{ +void *status; + +if (internal->intr_tid) { +pthread_cancel (internal->intr_tid); +pthread_join (internal->intr_tid, &status); + } + internal->intr_tid =0 ; + +if (internal->csc_epfd >=0 ) +close (internal->csc_epfd); + internal->csc_epfd =-1 ; +} +static int update_datapath (struct ifcvf_internal *internal) {
@@ -592,10 +702,16 @@ struct rte_vdpa_dev_info {if (ret) goto err; + ret =setup_intr_relay (internal); +if (ret) + goto err; +rte_atomic32_set (&internal->running,1 ); }else if (rte_atomic32_read (&internal->running) && (!rte_atomic32_read (&internal->started) || !rte_atomic32_read (&internal->dev_attached))) { +unset_intr_relay (internal); + ret =unset_notify_relay (internal);if (ret) goto err;
@@ -812,7 +928,7 @@ struct rte_vdpa_dev_info {if (nfds <0 ) {if (errno ==EINTR )continue ; -DRV_LOG (ERR ,"epoll_wait return fail\n" ); +DRV_LOG (ERR ,"epoll_wait return fail." );return NULL ; }
@@ -888,6 +1004,9 @@ struct rte_vdpa_dev_info {/* stop the direct IO data path */ unset_notify_relay (internal);vdpa_ifcvf_stop (internal); + +unset_intr_relay (internal); +vdpa_disable_vfio_intr (internal); ret =rte_vhost_host_notifier_ctrl (vid,RTE_VHOST_QUEUE_ALL , false);
Signature · 1 line
1.8.3.1
PATCH v10 13/13 vdpa/ifc/base: access correct register for blk device
In reply to Andy PeiRegister address is different between net and blk device. We are re-using most of the code, when register address is different, we have to check net and blk device go through different code.
--- drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +27 -7
@@ -218,10 +218,17 @@ &cfg->queue_used_hi);IFCVF_WRITE_REG16 (hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ) = - (u32 )hw->vring[i].last_avail_idx | - ((u32 )hw->vring[i].last_used_idx <<16 ); +if (hw->device_type ==IFCVF_BLK ) + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 ); +else + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 );IFCVF_WRITE_REG16 (i +1 , &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -254,9 +261,22 @@IFCVF_WRITE_REG16 (i, &cfg->queue_select);IFCVF_WRITE_REG16 (0 , &cfg->queue_enable);IFCVF_WRITE_REG16 (IFCVF_MSI_NO_VECTOR , &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ); - hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); + +if (hw->device_type ==IFCVF_BLK ) + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); +else + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ); + +if (hw->device_type ==IFCVF_BLK ) + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); +else + hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); hw->vring[i].last_used_idx = (u16)(ring_state >>16 ); } }
Signature · 1 line
1.8.3.1
Maxime Coquelin
AppliedQuoted 77 lines · Andy Pei
This patch set add
virtio_blkdevice support to vdpa/ifc driver. With a lot of similarities, I re-use part of vdpa/ifc driver. Distinguish the virtio net and blk device by device id, and implement specific features and ops.
Add example to vdpa to supportvirtio_blkdevice.
To support blk device live migration, some modification to vhost lib. Performdev_confop only underVHOST_USER_SET_VRING_CALLmsg.v10:
fix typo.
add more NULL pointer check.
fix a error handler.v9:
fix some naming issue.
add more NULL pointer check.
close open file when errors occur.v8:
delete some redundant code.
fix some commit log.v7:
Check on expected fd num in new vhost msg handler. Sanity check on vhost msg size.
Fix typo.
Add commit log to help understand code. Remove duplicated code.
Add new API to get vDPA device type.v6:
fix some commit log.
add vhost socket in log output to make it more user-friendly. when driver ops fail, just output some log, do not break message handler. check vhost msg size in msg handler.
v5:
fix some coding style issues.
v4:
add args "isblk" to vdpa example to specify a block device, fix some issue in example.
Make sure code specify for block device does not affect net device.
v3:
Fix some compile issues.
v2:
Fix some coding style issues.Andy Pei (13):
vdpa/ifc: add support for virtio blk device
vhost: add vDPA ops for blk device
vhost: add vhost msg support for get/set config vdpa/ifc: add blk ops for ifc device vdpa/ifc: add vDPA interrupt relay for blk device vdpa/ifc: add block device SW live-migration
vhost: add API to get vDPA device type vdpa/ifc: add get device type ops to ifc driver examples/vdpa: add virtio blk support
usertools: add support for virtio blk device vdpa/ifc: add log for config space of virtio blk vdpa/ifc: add interrupt handling for config space vdpa/ifc/base: access correct register for blk devicedoc/guides/prog_guide/vhost_lib.rst | 5 + doc/guides/rel_notes/release_22_07.rst | 4 + drivers/vdpa/ifc/base/ifcvf.c | 34 ++- drivers/vdpa/ifc/base/ifcvf.h | 27 ++- drivers/vdpa/ifc/ifcvf_vdpa.c | 420 +++++++++++++++++++++++++++++++-- examples/vdpa/main.c | 56 +++++ examples/vdpa/vdpa_blk_compact.h | 50 ++++ lib/vhost/rte_vhost.h | 17 ++ lib/vhost/socket.c | 44 ++++ lib/vhost/vdpa_driver.h | 11 +- lib/vhost/version.map | 1 + lib/vhost/vhost_user.c | 85 +++++++ lib/vhost/vhost_user.h | 13 + usertools/dpdk-devbind.py | 5 +- 14 files changed, 741 insertions(+), 31 deletions(-)
Quoted 2 lines
create mode 100644
examples/vdpa/vdpa_blk_compact.h
Applied to dpdk-next-virtio/main.
Thanks,
Maxime
Thanks Maxime
Quoted 97 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Wednesday, June 1, 2022 6:11 PM
To: Pei, Andy <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>; Cao, Gang <[email protected]>; Liu, Changpeng <[email protected]>; Xu, Rosen <[email protected]>; Xiao, QimaiX <[email protected]>
Subject: Re: [PATCH v10 00/13] addvirtio_blkdevice support to vdpa/ifcQuoted 62 lines · Andy Pei
This patch set add
virtio_blkdevice support to vdpa/ifc driver. With a lot of similarities, I re-use part of vdpa/ifc driver. Distinguish the virtio net and blk device by device id, and implement specific features and ops.
Add example to vdpa to supportvirtio_blkdevice.
To support blk device live migration, some modification to vhost lib. Performdev_confop only underVHOST_USER_SET_VRING_CALLmsg.v10:
fix typo.
add more NULL pointer check.
fix a error handler.v9:
fix some naming issue.
add more NULL pointer check.
close open file when errors occur.v8:
delete some redundant code.
fix some commit log.v7:
Check on expected fd num in new vhost msg handler. Sanity check on vhost msg size.
Fix typo.
Add commit log to help understand code. Remove duplicated code.
Add new API to get vDPA device type.v6:
fix some commit log.
add vhost socket in log output to make it more user-friendly. when driver ops fail, just output some log, do not break message handler. check vhost msg size in msg handler.
v5:
fix some coding style issues.
v4:
add args "isblk" to vdpa example to specify a block device, fix some issue in example.
Make sure code specify for block device does not affect net device.
v3:
Fix some compile issues.
v2:
Fix some coding style issues.Andy Pei (13):
vdpa/ifc: add support for virtio blk device
vhost: add vDPA ops for blk device
vhost: add vhost msg support for get/set config vdpa/ifc: add blk ops for ifc device vdpa/ifc: add vDPA interrupt relay for blk device vdpa/ifc: add block device SW live-migration
vhost: add API to get vDPA device type vdpa/ifc: add get device type ops to ifc driver examples/vdpa: add virtio blk support
usertools: add support for virtio blk device vdpa/ifc: add log for config space of virtio blk vdpa/ifc: add interrupt handling for config space vdpa/ifc/base: access correct register for blk deviceQuoted 5 lines
doc/guides/prog_guide/vhost_lib.rst | 5 + doc/guides/rel_notes/release_22_07.rst | 4 + drivers/vdpa/ifc/base/ifcvf.c | 34 ++- drivers/vdpa/ifc/base/ifcvf.h | 27 ++- drivers/vdpa/ifc/ifcvf_vdpa.c | 420+++++++++++++++++++++++++++++++--
Quoted 10 lines
examples/vdpa/main.c | 56 +++++ examples/vdpa/vdpa_blk_compact.h | 50 ++++ lib/vhost/rte_vhost.h | 17 ++ lib/vhost/socket.c | 44 ++++ lib/vhost/vdpa_driver.h | 11 +- lib/vhost/version.map | 1 + lib/vhost/vhost_user.c | 85 +++++++ lib/vhost/vhost_user.h | 13 + usertools/dpdk-devbind.py | 5 +- 14 files changed, 741 insertions(+), 31 deletions(-)Quoted 2 lines
create mode 100644
examples/vdpa/vdpa_blk_compact.hApplied to dpdk-next-virtio/main.
Thanks,
Maxime
PATCH 2/15 vhost: add vdpa ops for blk device
In reply to Andy PeiGet_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vdpa ops.
--- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
lib/vhost/vdpa_driver.h +6 -2
@@ -65,8 +65,12 @@ struct rte_vdpa_dev_ops {/** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev,int qid); -/** Reserved for future extension */ -void *reserved[2 ]; +/** Get the device configuration space */ +int (*get_config)(int vid,uint8_t *config,uint32_t len); + +/** Set the device configuration space */ +int (*set_config)(int vid,uint8_t *config,uint32_t offset, +uint32_t size,uint32_t flags); };/**
Signature · 1 line
1.8.3.1
PATCH 3/15 vdpa/ifc: add blk ops for ifc device
In reply to Andy PeiFor virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device.
--- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 88 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 91 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/base/ifcvf.h +4 -0
@@ -66,6 +66,10 @@#define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif +struct ifcvf_pci_cap {u8 cap_vndr;/* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next;/* Generic PCI field: next ptr. */
drivers/vdpa/ifc/ifcvf_vdpa.c +87 -1
106 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH 4/15 vdpa/ifc: add vdpa interrupt for blk device
In reply to Andy PeiFor the blk we need to relay all the cmd of each queue.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 48 +++++++++++++++++++++++++++++++++---------- 1 file changed, 37 insertions(+), 11 deletions(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +37 -11
@@ -374,24 +374,50 @@ struct rte_vdpa_dev_info { irq_set->index =VFIO_PCI_MSIX_IRQ_INDEX ; irq_set->start =0 ; fd_ptr = (int *)&irq_set->data; +/* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET ] =rte_intr_fd_get (internal->pdev->intr_handle);for (i =0 ; i < nr_vring; i++) internal->intr_fd[i] =-1 ; -for (i =0 ; i < nr_vring; i++) { -rte_vhost_get_vhost_vring (internal->vid, i, &vring); - fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; -if ((i &1 ) ==0 && m_rx == true) { - fd =eventfd (0 ,EFD_NONBLOCK |EFD_CLOEXEC ); -if (fd <0 ) { -DRV_LOG (ERR ,"can't setup eventfd: %s" , -strerror (errno)); -return -1 ; +if (internal->device_type ==IFCVF_NET ) { +for (i =0 ; i < nr_vring; i++) { +rte_vhost_get_vhost_vring (internal->vid, i, &vring); + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; +if ((i &1 ) ==0 && m_rx == true) { +/** +** For the net we only need to relay rx queue, +** which will change the mem of VM. +**/ + fd =eventfd (0 ,EFD_NONBLOCK |EFD_CLOEXEC ); +if (fd <0 ) { +DRV_LOG (ERR ,"can't setup eventfd: %s" , +strerror (errno)); +return -1 ; + } + internal->intr_fd[i] = fd; + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; + } + } + }else if (internal->device_type ==IFCVF_BLK ) { +for (i =0 ; i < nr_vring; i++) { +rte_vhost_get_vhost_vring (internal->vid, i, &vring); + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; +if (m_rx == true) { +/** +** For the blk we need to relay all the read cmd +** of each queue +**/ + fd =eventfd (0 ,EFD_NONBLOCK |EFD_CLOEXEC ); +if (fd <0 ) { +DRV_LOG (ERR ,"can't setup eventfd: %s" , +strerror (errno)); +return -1 ; + } + internal->intr_fd[i] = fd; + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; } - internal->intr_fd[i] = fd; - fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; } }
Signature · 1 line
1.8.3.1
PATCH 5/15 vdpa/ifc: add blk dev sw live migration
In reply to Andy PeiEnable virtio blk sw live migration relay callfd and log the dirty page.
In this version we ignore the write cmd and still mark it dirty. Maybe we can improve it later.
--- drivers/vdpa/ifc/base/ifcvf.c | 4 +- drivers/vdpa/ifc/base/ifcvf.h | 6 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 130 +++++++++++++++++++++++++++++++++++------- 3 files changed, 118 insertions(+), 22 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +2 -2
@@ -189,7 +189,7 @@IFCVF_WRITE_REG32 (val >>32 , hi); } -STATIC int +int ifcvf_hw_enable (struct ifcvf_hw *hw) {struct ifcvf_pci_common_cfg *cfg;
@@ -238,7 +238,7 @@return 0 ; } -STATIC void +void ifcvf_hw_disable (struct ifcvf_hw *hw) {u32 i;
drivers/vdpa/ifc/base/ifcvf.h +6 -0
@@ -179,4 +179,10 @@ struct ifcvf_hw {u64 ifcvf_get_queue_notify_off (struct ifcvf_hw *hw,int qid); +int +ifcvf_hw_enable (struct ifcvf_hw *hw); + +void +ifcvf_hw_disable (struct ifcvf_hw *hw); +#endif /* _IFCVF_H_ */
drivers/vdpa/ifc/ifcvf_vdpa.c +110 -20
180 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH 6/15 example/vdpa:add vdpa blk support in example
In reply to Andy Pei--- examples/vdpa/Makefile | 2 +- examples/vdpa/main.c | 8 ++ examples/vdpa/meson.build | 1 + examples/vdpa/vdpa_blk_compact.c | 152 +++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 118 ++++++++++++++++++++++++ examples/vdpa/vhost_user.h | 190 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 470 insertions(+), 1 deletion(-) create mode 100644 examples/vdpa/vdpa_blk_compact.c create mode 100644 examples/vdpa/vdpa_blk_compact.h create mode 100644 examples/vdpa/vhost_user.h
examples/vdpa/Makefile +1 -1
@@ -5,7 +5,7 @@ APP = vdpa # all source are stored in SRCS-y -SRCS-y := main.c +SRCS-y := main.c vdpa_blk_compact.c CFLAGS += -DALLOW_EXPERIMENTAL_API PKGCONF ?= pkg-config
examples/vdpa/main.c +8 -0
@@ -20,6 +20,7 @@#include <cmdline_parse_string.h> #include <cmdline_parse_num.h> #include <cmdline.h> +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128#define MAX_VDPA_SAMPLE_PORTS 1024
@@ -156,6 +157,7 @@ struct vdpa_port {static const struct rte_vhost_device_ops vdpa_sample_devops = { .new_device = new_device, .destroy_device = destroy_device, + .new_connection = rte_vhost_blk_session_install_rte_compat_hooks, }; static int
@@ -192,6 +194,12 @@ struct vdpa_port {"attach vdpa device failed: %s\n" , socket_path); +if (vdpa_blk_device_set_features_and_protocol (socket_path, vport->dev) + <0 ) +rte_exit (EXIT_FAILURE , +"set vhost blk driver features and protocal features failed: %s\n" , + socket_path); +if (rte_vhost_driver_start (socket_path) <0 ) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n",
examples/vdpa/meson.build +1 -0
@@ -15,4 +15,5 @@ deps += 'vhost' allow_experimental_apis = true sources = files( 'main.c', + 'vdpa_blk_compact.c', )
examples/vdpa/vdpa_blk_compact.c new file +152 -0
152 lines not on this page · raw message
examples/vdpa/vdpa_blk_compact.h new file +118 -0
118 lines not on this page · raw message
examples/vdpa/vhost_user.h new file +190 -0
190 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH 7/15 usertools: add support for virtio blk device
In reply to Andy Pei--- usertools/dpdk-devbind.py | 8 ++++++++ 1 file changed, 8 insertions(+)
usertools/dpdk-devbind.py +8 -0
@@ -14,6 +14,8 @@ from os.path import join as path_join # The PCI base class for all devices +virtio_blk_class = {'Class': '01', 'Vendor': "1af4", 'Device': '1001', + 'SVendor': '8086', 'SDevice': '0002'} network_class = {'Class': '02', 'Vendor': None, 'Device': None, 'SVendor': None, 'SDevice': None} acceleration_class = {'Class': '12', 'Vendor': None, 'Device': None,
@@ -72,6 +74,7 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk_devices = [virtio_blk_class] network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class]
@@ -587,6 +590,9 @@ def show_status(): Displays to the user what devices are bound to the igb_uio driver, the kernel driver or to no driver''' + if status_dev in ["virtio_blk", "all"]: + show_device_status(virtio_blk_devices, "virtio_blk") + if status_dev in ["net", "all"]: show_device_status(network_devices, "Network", if_field=True)
@@ -746,6 +752,7 @@ def do_arg_actions(): if b_flag is not None: clear_data() # refresh if we have changed anything + get_device_details(virtio_blk_devices) get_device_details(network_devices) get_device_details(baseband_devices) get_device_details(crypto_devices)
@@ -769,6 +776,7 @@ def main(): parse_args() check_modules() clear_data() + get_device_details(virtio_blk_devices) get_device_details(network_devices) get_device_details(baseband_devices) get_device_details(crypto_devices)
Signature · 1 line
1.8.3.1
PATCH 8/15 vdpa/ifc: set_vring_state op is mandatory, add set_vring_state for blk device
In reply to Andy Pei--- drivers/vdpa/ifc/ifcvf_vdpa.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/vdpa/ifc/ifcvf_vdpa.c +11 -1
@@ -1384,6 +1384,16 @@ struct rte_vdpa_dev_info { }static int +ifcvf_blk_set_vring_state (int vid,int vring,int state) +{ +RTE_SET_USED (vid); +RTE_SET_USED (vring); +RTE_SET_USED (state); + +return 0 ; +} + +static int ifcvf_blk_get_protocol_features (struct rte_vdpa_device *vdev,uint64_t *features) {
@@ -1401,7 +1411,7 @@ struct rte_vdpa_dev_info { .get_protocol_features = ifcvf_blk_get_protocol_features, .dev_conf = ifcvf_dev_config, .dev_close = ifcvf_dev_close, - .set_vring_state = NULL, + .set_vring_state = ifcvf_blk_set_vring_state, .migration_done = NULL, .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd,
Signature · 1 line
1.8.3.1
PATCH 9/15 vdpa/ifc: add some log at VDPA lauch before qemu connect
In reply to Andy Pei--- drivers/vdpa/ifc/ifcvf_vdpa.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +29 -0
@@ -1447,6 +1447,9 @@ struct rte_vdpa_dev_info dev_info[] = {struct rte_kvargs *kvlist =NULL ;int ret =0 ;int16_t device_id; +__u64 capacity =0 ; +uint8_t *byte; +uint32_t i;if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ;
@@ -1513,6 +1516,32 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_BLK ].features; + +/** +** cannot read 64-bit register in one attempt, +** so read byte by byte. +**/ +for (i =0 ; i <sizeof (internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (__u64)*byte << (i *8 ); + } +DRV_LOG (INFO ,"capacity : %quG" , capacity >>21 ); + +DRV_LOG (INFO ,"size_max : 0x%08x" , + internal->hw.blk_cfg->size_max); +DRV_LOG (INFO ,"seg_max : 0x%08x" , + internal->hw.blk_cfg->seg_max); +DRV_LOG (INFO ,"blk_size : 0x%08x" , + internal->hw.blk_cfg->blk_size); +DRV_LOG (INFO ,"geometry" ); +DRV_LOG (INFO ," cylinders: %u" , + internal->hw.blk_cfg->geometry.cylinders); +DRV_LOG (INFO ," heads : %u" , + internal->hw.blk_cfg->geometry.heads); +DRV_LOG (INFO ," sectors : %u" , + internal->hw.blk_cfg->geometry.sectors); +DRV_LOG (INFO ,"num_queues: 0x%08x" , + internal->hw.blk_cfg->num_queues); } list->internal = internal;
Signature · 1 line
1.8.3.1
PATCH 10/15 vdpa/ifc: read virtio max_queues from hardware
In reply to Andy Peioriginal code max_queues is set to IFCVF_MAX_QUEUES.
New code max_queues is the min of IFCVF_MAX_QUEUES and hardware num_queues.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +4 -0
@@ -1542,6 +1542,10 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors);DRV_LOG (INFO ,"num_queues: 0x%08x" , internal->hw.blk_cfg->num_queues); + +/* reset max_queue here, to minimum modification */ + internal->max_queues =RTE_MIN (IFCVF_MAX_QUEUES , + internal->hw.blk_cfg->num_queues); } list->internal = internal;
Signature · 1 line
1.8.3.1
PATCH 11/15 vdpa: add config space change interrupt register and handle for virtio_blk
In reply to Andy PeiCreate a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to info qemu.
--- drivers/vdpa/ifc/ifcvf_vdpa.c | 113 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +113 -0
@@ -53,7 +53,9 @@ struct ifcvf_internal {int vfio_group_fd;int vfio_dev_fd;pthread_t tid;/* thread for notify relay */ +pthread_t intr_tid;/* thread for intr relay */ int epfd; +int csc_fd;int vid;struct rte_vdpa_device *vdev;uint16_t max_queues;
@@ -622,6 +624,108 @@ struct rte_vdpa_dev_info {return 0 ; } +static void +virtio_interrupt_handler (struct ifcvf_internal *internal) +{ +int vid = internal->vid; +int ret; + + ret =rte_vhost_slave_config_change (vid,1 ); +if (ret) +DRV_LOG (ERR ,"failed to notify the guest about configuration space change." ); + +return ; +} + +static void * +intr_relay (void *arg) +{ +struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; +struct epoll_event csc_event; +struct epoll_event ev; +uint64_t buf; +int nbytes; +int csc_fd, csc_val =0 ; + + csc_fd =epoll_create (1 ); +if (csc_fd <0 ) { +DRV_LOG (ERR ,"failed to create epoll for config space change." ); +return NULL ; + } + + ev.events =EPOLLIN |EPOLLPRI |EPOLLRDHUP |EPOLLHUP ; + ev.data.fd =rte_intr_fd_get (internal->pdev->intr_handle); +if (epoll_ctl (csc_fd,EPOLL_CTL_ADD , +rte_intr_fd_get (internal->pdev->intr_handle), &ev) <0 ) { +DRV_LOG (ERR ,"epoll add error: %s" ,strerror (errno)); +return NULL ; + } + + internal->csc_fd = csc_fd; + +for (;;) { + csc_val =epoll_wait (csc_fd, &csc_event,1 ,-1 ); +if (csc_val <0 ) { +if (errno ==EINTR ) +continue ; +DRV_LOG (ERR ,"epoll_wait return fail\n" ); +return NULL ; + }else if (csc_val ==0 ) { +continue ; + }else { +/* csc_val > 0 */ + nbytes =read (csc_event.data.fd, &buf,8 ); +if (nbytes <0 ) { +if (errno ==EINTR || errno ==EWOULDBLOCK ) +continue ; +DRV_LOG (ERR ,"Error reading from file descriptor %d: %s\n" , + csc_event.data.fd, +strerror (errno)); +return NULL ; + }else if (nbytes ==0 ) { +DRV_LOG (ERR ,"Read nothing from file descriptor %d\n" , + csc_event.data.fd); +continue ; + }else { +virtio_interrupt_handler (internal); + } + } + } +return NULL ; +} + +static int +setup_intr_relay (struct ifcvf_internal *internal) +{ +int ret; + + ret =pthread_create (&internal->intr_tid,NULL , intr_relay, + (void *)internal); +if (ret) { +DRV_LOG (ERR ,"failed to create notify relay pthread." ); +return -1 ; + } +return 0 ; +} + +static int +unset_intr_relay (struct ifcvf_internal *internal) +{ +void *status; + +if (internal->intr_tid) { +pthread_cancel (internal->intr_tid); +pthread_join (internal->intr_tid, &status); + } + internal->intr_tid =0 ; + +if (internal->csc_fd >=0 ) +close (internal->csc_fd); + internal->csc_fd =-1 ; + +return 0 ; +} +static int update_datapath (struct ifcvf_internal *internal) {
@@ -648,10 +752,16 @@ struct rte_vdpa_dev_info {if (ret) goto err; + ret =setup_intr_relay (internal); +if (ret) + goto err; +rte_atomic32_set (&internal->running,1 ); }else if (rte_atomic32_read (&internal->running) && (!rte_atomic32_read (&internal->started) || !rte_atomic32_read (&internal->dev_attached))) { + ret =unset_intr_relay (internal); + ret =unset_notify_relay (internal);if (ret) goto err;
@@ -969,6 +1079,9 @@ struct rte_vdpa_dev_info {vdpa_ifcvf_stop (internal); else if (internal->device_type == IFCVF_BLK) vdpa_ifcvf_blk_pause(internal); + +unset_intr_relay (internal ); +vdpa_disable_vfio_intr (internal); ret =rte_vhost_host_notifier_ctrl (vid,RTE_VHOST_QUEUE_ALL , false);
Signature · 1 line
1.8.3.1
PATCH 12/15 vdpa/ifc: add is_blk flag to ifcvf_hw, and init is_blk during probe
In reply to Andy Pei--- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++ 2 files changed, 3 insertions(+)
drivers/vdpa/ifc/base/ifcvf.h +1 -0
@@ -149,6 +149,7 @@ struct ifcvf_hw {u8 *lm_cfg;struct vring_info vring[IFCVF_MAX_QUEUES *2 ];u8 nr_vring; +u8 is_blk;struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE ]; };
drivers/vdpa/ifc/ifcvf_vdpa.c +2 -0
@@ -1621,11 +1621,13 @@ struct rte_vdpa_dev_info dev_info[] = {if (device_id ==VIRTIO_ID_NET ) { internal->device_type =IFCVF_NET ; + internal->hw.is_blk =IFCVF_NET ; internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_NET ].features; }else if (device_id ==VIRTIO_ID_BLOCK ) { internal->device_type =IFCVF_BLK ; + internal->hw.is_blk =IFCVF_BLK ; internal->features = features & ~(1ULL <<VIRTIO_F_IOMMU_PLATFORM ); internal->features |= dev_info[IFCVF_BLK ].features;
Signature · 1 line
1.8.3.1
PATCH 13/15 vdpa/ifc/base: for blk device, live migration register is different from net device
In reply to Andy Pei1.last_avail_idx is lower 16 bit of the register.
2.address of ring_state register is different between net and blk device.
--- drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++------- drivers/vdpa/ifc/base/ifcvf.h | 1 + 2 files changed, 30 insertions(+), 7 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +29 -7
@@ -216,10 +216,18 @@ &cfg->queue_used_hi);IFCVF_WRITE_REG16 (hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ) = - (u32 )hw->vring[i].last_avail_idx | - ((u32 )hw->vring[i].last_used_idx <<16 ); +if (hw->is_blk ==IFCVF_BLK ) { + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 ); + }else if (hw->is_blk ==IFCVF_NET ) { + *(u32 *)(lm_cfg +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ) = + (u32 )hw->vring[i].last_avail_idx | + ((u32 )hw->vring[i].last_used_idx <<16 ); + }IFCVF_WRITE_REG16 (i +1 , &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==
@@ -252,9 +260,23 @@IFCVF_WRITE_REG16 (i, &cfg->queue_select);IFCVF_WRITE_REG16 (0 , &cfg->queue_enable);IFCVF_WRITE_REG16 (IFCVF_MSI_NO_VECTOR , &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg +IFCVF_LM_RING_STATE_OFFSET + - (i /2 ) *IFCVF_LM_CFG_SIZE + (i %2 ) *4 ); - hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); + +if (hw->is_blk) { + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); + }else if (hw->is_blk ==IFCVF_NET ) { + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + (i /2 ) *IFCVF_LM_CFG_SIZE + + (i %2 ) *4 ); + } + +if (hw->is_blk ==IFCVF_BLK ) + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); +else if (hw->is_blk ==IFCVF_NET ) + hw->vring[i].last_avail_idx = (u16)(ring_state >>16 ); hw->vring[i].last_used_idx = (u16)(ring_state >>16 ); } }
drivers/vdpa/ifc/base/ifcvf.h +1 -0
@@ -65,6 +65,7 @@#define IFCVF_MEDIATED_VRING 0x200000000000#define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff#ifndef VHOST_USER_PROTOCOL_F_CONFIG #define VHOST_USER_PROTOCOL_F_CONFIG 9
Signature · 1 line
1.8.3.1
PATCH 14/15 vdpa/ifc: make sure hardware last_avail_idx and last_used_idx is the same when blk device pause
In reply to Andy Pei--- drivers/vdpa/ifc/base/ifcvf.c | 2 +- drivers/vdpa/ifc/base/ifcvf.h | 3 +++ drivers/vdpa/ifc/ifcvf_vdpa.c | 32 +++++++++++++++++++++++--------- 3 files changed, 27 insertions(+), 10 deletions(-)
drivers/vdpa/ifc/base/ifcvf.c +1 -1
@@ -116,7 +116,7 @@IFCVF_WRITE_REG8 (status, &hw->common_cfg->device_status); } -STATIC void +void ifcvf_reset (struct ifcvf_hw *hw) {ifcvf_set_status (hw,0 );
drivers/vdpa/ifc/base/ifcvf.h +3 -0
@@ -157,6 +157,9 @@ struct ifcvf_hw {int ifcvf_init_hw (struct ifcvf_hw *hw,PCI_DEV *dev); +void +ifcvf_reset (struct ifcvf_hw *hw); +u64 ifcvf_get_features (struct ifcvf_hw *hw);
drivers/vdpa/ifc/ifcvf_vdpa.c +23 -9
@@ -359,23 +359,32 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_blk_pause(struct ifcvf_internal *internal) {struct ifcvf_hw *hw = &internal->hw; -struct rte_vhost_vring vq;int i, vid;uint64_t features =0 ;uint64_t log_base =0 , log_size =0 ;uint64_t len; +u32 ring_state =0 ; vid = internal->vid;if (internal->device_type ==IFCVF_BLK ) {for (i =0 ; i < hw->nr_vring; i++) { -rte_vhost_get_vhost_vring (internal->vid, i, &vq); -while (vq.avail->idx != vq.used->idx) { -ifcvf_notify_queue (hw, i); -usleep (10 ); - } - hw->vring[i].last_avail_idx = vq.avail->idx; - hw->vring[i].last_used_idx = vq.used->idx; +do { +if (hw->lm_cfg !=NULL ) + ring_state = *(u32 *)(hw->lm_cfg + +IFCVF_LM_RING_STATE_OFFSET + + i *IFCVF_LM_CFG_SIZE ); + hw->vring[i].last_avail_idx = + (u16)(ring_state &IFCVF_16_BIT_MASK ); + hw->vring[i].last_used_idx = + (u16)(ring_state >>16 ); +if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { +ifcvf_notify_queue (hw, i); +usleep (10 ); + } + }while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); } }
@@ -766,7 +775,12 @@ struct rte_vdpa_dev_info {if (ret) goto err; -vdpa_ifcvf_stop (internal); +if (internal->device_type ==IFCVF_BLK ) { +vdpa_ifcvf_blk_pause (internal); +ifcvf_reset (&internal->hw); + }else { +vdpa_ifcvf_stop (internal); + } ret =vdpa_disable_vfio_intr (internal);if (ret)
Signature · 1 line
1.8.3.1
PATCH 15/15 vhost: make sure each queue callfd is configured
In reply to Andy PeiDuring the vhost data path building process, qemu will create a call fd at first, and create another call fd in the end. The final call fd will be used to relay notify. In the original code, after kick fd is set, dev_conf will set the first call fd. Even though the actual call fd will set, the data path will not work correctly.
--- lib/vhost/vhost_user.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
lib/vhost/vhost_user.c +15 -0
@@ -3137,12 +3137,27 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev,if (!vdpa_dev) goto out; +if (request !=VHOST_USER_SET_VRING_CALL ) + goto out; +if (!(dev->flags &VIRTIO_DEV_VDPA_CONFIGURED )) {if (vdpa_dev->ops->dev_conf (dev->vid))VHOST_LOG_CONFIG (ERR ,"Failed to configure vDPA device\n" );else dev->flags |=VIRTIO_DEV_VDPA_CONFIGURED ; + }else { +/** +** when VIRTIO_DEV_VDPA_CONFIGURED already configured +** close the device and config the device again, +** make sure the call fd of each queue is configed correctly. +**/ +if (vdpa_dev->ops->dev_close (dev->vid)) +VHOST_LOG_CONFIG (ERR , +"Failed to close vDPA device\n" ); +if (vdpa_dev->ops->dev_conf (dev->vid)) +VHOST_LOG_CONFIG (ERR , +"Failed to re-config vDPA device\n" ); } out:
Signature · 1 line
1.8.3.1