example: vdpa: support run in nested virtualization environment
PATCH 1/1 example: vdpa: support run in nested virtualization environment
When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.
This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.
--- examples/vdpa/main.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
examples/vdpa/main.c +14 -5
@@ -43,16 +43,20 @@ static char iface[MAX_PATH_LEN];static int devcnt;static int interactive;static int client_mode; +static int iommu_support;/* display usage */ static void vdpa_usage (const char *prgname) { -printf ("Usage: %s [EAL options] -- " -" --interactive|-i: run in interactive mode.\n" -" --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n" -" --client: register a vhost-user socket as client mode.\n" , - prgname); +printf ("Usage: %s [EAL options] --\n" , prgname); +const char *usage_str =" --interactive|-i: run in interactive mode.\n" +" --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n" +" --client: register a vhost-user socket as client mode.\n" +" --iommu-support: use guest vIOMMU to protect vhost.\n" ; + +fprintf (stderr,"%s" , usage_str); +exit (1 ); } static int
@@ -63,6 +67,7 @@ parse_args(int argc, char **argv) {"iface", required_argument, NULL, 0}, {"interactive", no_argument, &interactive, 1}, {"client", no_argument, &client_mode, 1}, + {"iommu-support", no_argument, &iommu_support, 1}, {NULL, 0, 0, 0}, };int opt, idx;
@@ -220,6 +225,10 @@ start_vdpa(struct vdpa_port *vport) socket_path);return -1 ; } + +if (iommu_support) + vport->flags |=RTE_VHOST_USER_IOMMU_SUPPORT ; + ret =rte_vhost_driver_register (socket_path, vport->flags);if (ret !=0 ) rte_exit(EXIT_FAILURE,
Signature · 1 line
2.27.0
Quoted 7 lines
-----Original Message-----
From: Hao Chen <[email protected]>
Sent: Tuesday, October 11, 2022 10:56 AM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: [PATCH] example: vdpa: support run in nested virtualization environment
Title should be:
examples/vdpa: support running in nested virtualization environment
Quoted 47 lines
When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.
This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.
Signed-off-by Hao Chen <[email protected]>--- examples/vdpa/main.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)examples/vdpa/main.c +1
-2@@ -43,16 +43,20 @@ static char iface[MAX_PATH_LEN];static int devcnt;static int interactive;static int client_mode; +static int iommu_support;/* display usage */ static void vdpa_usage (const char *prgname) { - printf("Usage: %s [EAL options] -- " - " --interactive|-i: run in interactivemode.\n"
examples/vdpa/main.c +0
-1- " --iface <path>: specify the path prefix ofthe socket files, e.g. /tmp/vhost-user-.\n"
examples/vdpa/main.c +0
-1- " --client: register a vhost-user socket asclient mode.\n",
examples/vdpa/main.c +2
-1- prgname); +printf ("Usage: %s [EAL options] --\n" , prgname); + const char *usage_str = " --interactive|-i: run in interactivemode.\n"
examples/vdpa/main.c +1
-0+ " --iface <path>: specify the path prefix ofthe socket files, e.g. /tmp/vhost-user-.\n"
examples/vdpa/main.c +1
-0+ " --client: register a vhost-user socket asclient mode.\n"
examples/vdpa/main.c +1
-0+ " --iommu-support: use guest vIOMMU to protectvhost.\n";
examples/vdpa/main.c static char iface[MAX_PATH_LEN]; · line 58
Quoted 2 lines
examples/vdpa/main.c +2
-0+ +fprintf (stderr,"%s" , usage_str);
Why not use fprintf all string including the eal one?
examples/vdpa/main.c static char iface[MAX_PATH_LEN]; · line 59
Quoted 1 line
examples/vdpa/main.c +1
-0+exit (1 );
I believe the exit should happen outside of the function.
Thanks,
Chenbo
Quoted 24 lines
}
static int
examples/vdpa/main.c +5
-1@@ -63,6 +67,7 @@ parse_args(int argc, char **argv) {"iface", required_argument, NULL, 0}, {"interactive", no_argument, &interactive, 1}, {"client", no_argument, &client_mode, 1}, + {"iommu-support", no_argument, &iommu_support, 1}, {NULL, 0, 0, 0}, };int opt, idx;@@ -220,6 +225,10 @@ start_vdpa(struct vdpa_port *vport) socket_path);return -1 ; } + +if (iommu_support) + vport->flags |=RTE_VHOST_USER_IOMMU_SUPPORT ; + ret =rte_vhost_driver_register (socket_path, vport->flags);if (ret !=0 ) rte_exit(EXIT_FAILURE, --2.27.0
PATCH v2 1/1 example: vdpa: support run in nested virtualization environment
In reply to Hao ChenWhen we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.
This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.
---
v2:
*fprintf all string including the eal one.
*remove exit(1).
examples/vdpa/main.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
examples/vdpa/main.c +12 -5
@@ -43,16 +43,18 @@ static char iface[MAX_PATH_LEN];static int devcnt;static int interactive;static int client_mode; +static int iommu_support;/* display usage */ static void vdpa_usage (const char *prgname) { -printf ("Usage: %s [EAL options] -- " -" --interactive|-i: run in interactive mode.\n" -" --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n" -" --client: register a vhost-user socket as client mode.\n" , - prgname); +const char *usage_str =" --interactive|-i: run in interactive mode.\n" +" --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n" +" --client: register a vhost-user socket as client mode.\n" +" --iommu-support: use guest vIOMMU to protect vhost.\n" ; + +fprintf (stderr,"Usage: %s [EAL options] --\n%s" , prgname, usage_str); } static int
@@ -63,6 +65,7 @@ parse_args(int argc, char **argv) {"iface", required_argument, NULL, 0}, {"interactive", no_argument, &interactive, 1}, {"client", no_argument, &client_mode, 1}, + {"iommu-support", no_argument, &iommu_support, 1}, {NULL, 0, 0, 0}, };int opt, idx;
@@ -220,6 +223,10 @@ start_vdpa(struct vdpa_port *vport) socket_path);return -1 ; } + +if (iommu_support) + vport->flags |=RTE_VHOST_USER_IOMMU_SUPPORT ; + ret =rte_vhost_driver_register (socket_path, vport->flags);if (ret !=0 ) rte_exit(EXIT_FAILURE,
Signature · 1 line
2.27.0
PATCH v3 1/1 examples/vdpa: support running in nested virtualization environment
When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.
This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.
---
v3:
*Modify mail title.
v2:
*fprintf all string including the eal one.
*remove exit(1).
examples/vdpa/main.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
examples/vdpa/main.c +12 -5
@@ -43,16 +43,18 @@ static char iface[MAX_PATH_LEN];static int devcnt;static int interactive;static int client_mode; +static int iommu_support;/* display usage */ static void vdpa_usage (const char *prgname) { -printf ("Usage: %s [EAL options] -- " -" --interactive|-i: run in interactive mode.\n" -" --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n" -" --client: register a vhost-user socket as client mode.\n" , - prgname); +const char *usage_str =" --interactive|-i: run in interactive mode.\n" +" --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n" +" --client: register a vhost-user socket as client mode.\n" +" --iommu-support: use guest vIOMMU to protect vhost.\n" ; + +fprintf (stderr,"Usage: %s [EAL options] --\n%s" , prgname, usage_str); } static int
@@ -63,6 +65,7 @@ parse_args(int argc, char **argv) {"iface", required_argument, NULL, 0}, {"interactive", no_argument, &interactive, 1}, {"client", no_argument, &client_mode, 1}, + {"iommu-support", no_argument, &iommu_support, 1}, {NULL, 0, 0, 0}, };int opt, idx;
@@ -220,6 +223,10 @@ start_vdpa(struct vdpa_port *vport) socket_path);return -1 ; } + +if (iommu_support) + vport->flags |=RTE_VHOST_USER_IOMMU_SUPPORT ; + ret =rte_vhost_driver_register (socket_path, vport->flags);if (ret !=0 ) rte_exit(EXIT_FAILURE,
Signature · 1 line
2.27.0
Re: [PATCH v3] examples/vdpa: support running in nested virtualization environment
Hi Hao
Quoted 2 lines · Hao Chen
When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is
NG == not good? Please don't use abbreviations.
Quoted 7 lines
that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.
This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.
I'm wondering whether the flag shouldn't just be set by default, as the feature negotiation will be done between the vDPA driver & the guest driver anyways?
Quoted 59 lines
Signed-off-by Hao Chen <[email protected]>---
v3:
*Modify mail title.v2:
*fprintf all string including the eal one.
*remove exit(1).examples/vdpa/main.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)examples/vdpa/main.c +12
-5@@ -43,16 +43,18 @@ static char iface[MAX_PATH_LEN];static int devcnt;static int interactive;static int client_mode; +static int iommu_support;/* display usage */ static void vdpa_usage (const char *prgname) { -printf ("Usage: %s [EAL options] -- " -" --interactive|-i: run in interactive mode.\n" -" --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n" -" --client: register a vhost-user socket as client mode.\n" , - prgname); +const char *usage_str =" --interactive|-i: run in interactive mode.\n" +" --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n" +" --client: register a vhost-user socket as client mode.\n" +" --iommu-support: use guest vIOMMU to protect vhost.\n" ; + +fprintf (stderr,"Usage: %s [EAL options] --\n%s" , prgname, usage_str); } static int@@ -63,6 +65,7 @@ parse_args(int argc, char **argv) {"iface", required_argument, NULL, 0}, {"interactive", no_argument, &interactive, 1}, {"client", no_argument, &client_mode, 1}, + {"iommu-support", no_argument, &iommu_support, 1}, {NULL, 0, 0, 0}, };int opt, idx;@@ -220,6 +223,10 @@ start_vdpa(struct vdpa_port *vport) socket_path);return -1 ; } + +if (iommu_support) + vport->flags |=RTE_VHOST_USER_IOMMU_SUPPORT ; + ret =rte_vhost_driver_register (socket_path, vport->flags);if (ret !=0 ) rte_exit(EXIT_FAILURE,
Re: [PATCH v3] examples/vdpa: support running in nested virtualization environment
Quoted 8 lines · Maxime Coquelin
Hi Hao
Quoted 2 lines · Hao Chen
When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is
NG == not good? Please don't use abbreviations.
Yes, I will pay attention to this later.
Quoted 4 lines
Quoted 4 lines
that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.
Quoted 8 lines
Quoted 3 lines
This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.
I'm wondering whether the flag shouldn't just be set by default, as the feature negotiation will be done between the vDPA driver & the guest driver anyways?
Yes, It can be set by default rather than as an option. VIRTIO_F_IOMMU_PLATFORM feature will not be negotiated successfully if it is not in a nested virtualization environment.
I will simplify the patch.
Thanks.
Quoted 66 lines
Quoted 65 lines
Signed-off-by Hao Chen <[email protected]>---
v3:
*Modify mail title.v2:
*fprintf all string including the eal one.
*remove exit(1).
examples/vdpa/main.c| 17 ++++++++++++-----1 file changed, 12 insertions(+), 5 deletions(-)examples/vdpa/main.c +1
-0@@ -43,16 +43,18 @@ static char iface[MAX_PATH_LEN]; static int devcnt; static int interactive; static int client_mode; +static int iommu_support; /* display usage */static void
vdpa_usage(const char *prgname)
{
- printf("Usage: %s [EAL options] -- "
- " --interactive|-i: run in interactive mode.\n" - " --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
- " --client: register a vhost-user socket as client mode.\n",
- prgname);
+ const char*usage_str= " --interactive|-i: run in interactive mode.\n"
+ " --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+ " --client: register a vhost-user socket as client mode.\n"
+ " --iommu-support: use guest vIOMMU to protect vhost.\n";
+
+fprintf(stderr, "Usage: %s [EAL options] --\n%s", prgname, usage_str);
}
static intexamples/vdpa/main.c +5
-0@@ -63,6 +65,7 @@ parse_args(int argc, char **argv) {"iface", required_argument, NULL, 0}, {"interactive", no_argument, &interactive, 1}, {"client", no_argument, &client_mode, 1}, + {"iommu-support", no_argument, &iommu_support, 1}, {NULL, 0, 0, 0}, }; int opt, idx;@@ -220,6 +223,10 @@ start_vdpa(struct vdpa_port *vport) socket_path); return -1; } + + if (iommu_support) + vport->flags |= RTE_VHOST_USER_IOMMU_SUPPORT; + ret = rte_vhost_driver_register(socket_path, vport->flags); if (ret != 0) rte_exit(EXIT_FAILURE,
PATCH v4 1/1 examples/vdpa: support running in nested virtualization environment
In reply to Hao ChenWhen we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is not good. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.
This patch add flag RTE_VHOST_USER_IOMMU_SUPPORT to use vhost vIOMMU , VIRTIO_F_IOMMU_PLATFORM feature will be negotiated successfully if virtio iommu is used in a nested virtualization environment.
The configuration is as follows:
The host starts iommu, and the kernel parameter is added with 'intel_iommu=on iommu=pt'.
VM-L1's xml add viommu and virtio device adds iommu='on' ats='on'. VM-L2's xml enables viommu, and adds parameters 'intel_iommu=on iommu=pt' to kernel.
Then the ping test in vm-L2 is OK.
---
v4:
*Simplify the patch. Set the flags RTE_VHOST_USER_IOMMU_SUPPORT default.
v3:
*Modify mail title.
v2:
*fprintf all string including the eal one.
*remove exit(1).
examples/vdpa/main.c | 2 ++ 1 file changed, 2 insertions(+)
examples/vdpa/main.c +2 -0
@@ -214,6 +214,8 @@ start_vdpa(struct vdpa_port *vport)if (client_mode) vport->flags |=RTE_VHOST_USER_CLIENT ; + vport->flags |=RTE_VHOST_USER_IOMMU_SUPPORT ; +if (access (socket_path,F_OK ) !=-1 && !client_mode) { RTE_LOG(ERR, VDPA, "%s exists, please remove it or specify another file and try again.\n",
Signature · 1 line
2.27.0
Re: [PATCH v4] examples/vdpa: support running in nested virtualization environment
Quoted 39 lines · Hao Chen
When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is not good. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.
This patch add flag
RTE_VHOST_USER_IOMMU_SUPPORTto use vhost vIOMMU ,VIRTIO_F_IOMMU_PLATFORMfeature will be negotiated successfully if virtio iommu is used in a nested virtualization environment.The configuration is as follows:
The host starts iommu, and the kernel parameter is added with 'intel_iommu=on iommu=pt'.
VM-L1's xml add viommu and virtio device adds iommu='on' ats='on'. VM-L2's xml enables viommu, and adds parameters 'intel_iommu=on iommu=pt' to kernel.Then the ping test in vm-L2 is OK.
Signed-off-by Hao Chen <[email protected]>---
v4:
*Simplify the patch. Set the flagsRTE_VHOST_USER_IOMMU_SUPPORTdefault.v3:
*Modify mail title.v2:
*fprintf all string including the eal one.
*remove exit(1).examples/vdpa/main.c | 2 ++ 1 file changed, 2 insertions(+)examples/vdpa/main.c +0
-0
examples/vdpa/main.c start_vdpa(struct vdpa_port *vport) · line 221
Quoted 8 lines
examples/vdpa/main.c +2
-0@@ -214,6 +214,8 @@ start_vdpa(struct vdpa_port *vport)if (client_mode) vport->flags |=RTE_VHOST_USER_CLIENT ; + vport->flags |=RTE_VHOST_USER_IOMMU_SUPPORT ; +if (access (socket_path,F_OK ) !=-1 && !client_mode) { RTE_LOG(ERR, VDPA, "%s exists, please remove it or specify another file and try again.\n",
Thanks,
Maxime
Maxime Coquelin
AppliedRe: [PATCH v4] examples/vdpa: support running in nested virtualization environment
In reply to Hao ChenQuoted 39 lines · Hao Chen
When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is not good. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.
This patch add flag
RTE_VHOST_USER_IOMMU_SUPPORTto use vhost vIOMMU ,VIRTIO_F_IOMMU_PLATFORMfeature will be negotiated successfully if virtio iommu is used in a nested virtualization environment.The configuration is as follows:
The host starts iommu, and the kernel parameter is added with 'intel_iommu=on iommu=pt'.
VM-L1's xml add viommu and virtio device adds iommu='on' ats='on'. VM-L2's xml enables viommu, and adds parameters 'intel_iommu=on iommu=pt' to kernel.Then the ping test in vm-L2 is OK.
Signed-off-by Hao Chen <[email protected]>---
v4:
*Simplify the patch. Set the flagsRTE_VHOST_USER_IOMMU_SUPPORTdefault.v3:
*Modify mail title.v2:
*fprintf all string including the eal one.
*remove exit(1).examples/vdpa/main.c | 2 ++ 1 file changed, 2 insertions(+)examples/vdpa/main.c +0
-0
examples/vdpa/main.c start_vdpa(struct vdpa_port *vport) · line 221
Quoted 8 lines
examples/vdpa/main.c +2
-0@@ -214,6 +214,8 @@ start_vdpa(struct vdpa_port *vport)if (client_mode) vport->flags |=RTE_VHOST_USER_CLIENT ; + vport->flags |=RTE_VHOST_USER_IOMMU_SUPPORT ; +if (access (socket_path,F_OK ) !=-1 && !client_mode) { RTE_LOG(ERR, VDPA, "%s exists, please remove it or specify another file and try again.\n",
Applied to dpdk-next-virtio/main.
Thanks,
Maxime