Skip to contentSkip to conversation

example: vdpa: support run in nested virtualization environment

PATCH Superseded Hao Chen 3 participants 9 messages Oct 11, 2022 – Feb 3, 2023

lore
Share & tools
lore

Hao Chen

author
#0 lore raw

PATCH 1/1 example: vdpa: support run in nested virtualization environment

When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.

This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.

---
 examples/vdpa/main.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
examples/vdpa/main.c +14 -5
index 4c7e81d7b6..a24c78d7a4 100644
@@ -43,16 +43,20 @@ static char iface[MAX_PATH_LEN];
 static int devcnt;
 static int interactive;
 static int client_mode;
+static int iommu_support;
 
 /* display usage */
 static void
 vdpa_usage(const char *prgname)
 {
-	printf("Usage: %s [EAL options] -- "
-				 "	--interactive|-i: run in interactive mode.\n"
-				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
-				 "	--client: register a vhost-user socket as client mode.\n",
-				 prgname);
+	printf("Usage: %s [EAL options] --\n", prgname);
+	const char *usage_str = "	--interactive|-i: run in interactive mode.\n"
+				"	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+				"	--client: register a vhost-user socket as client mode.\n"
+				"	--iommu-support: use guest vIOMMU to protect vhost.\n";
+
+	fprintf(stderr, "%s", usage_str);
+	exit(1);
 }
 
 static int
@@ -63,6 +67,7 @@ parse_args(int argc, char **argv)
 		{"iface", required_argument, NULL, 0},
 		{"interactive", no_argument, &interactive, 1},
 		{"client", no_argument, &client_mode, 1},
+		{"iommu-support", no_argument, &iommu_support, 1},
 		{NULL, 0, 0, 0},
 	};
 	int opt, idx;
@@ -220,6 +225,10 @@ start_vdpa(struct vdpa_port *vport)
 			socket_path);
 		return -1;
 	}
+
+	if (iommu_support)
+		vport->flags |= RTE_VHOST_USER_IOMMU_SUPPORT;
+
 	ret = rte_vhost_driver_register(socket_path, vport->flags);
 	if (ret != 0)
 		rte_exit(EXIT_FAILURE,
Signature · 1 line
2.27.0
#1 lore raw
Quoted 7 lines

-----Original Message-----
From: Hao Chen <[email protected]>
Sent: Tuesday, October 11, 2022 10:56 AM
To: [email protected]; Xia, Chenbo <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: [PATCH] example: vdpa: support run in nested virtualization environment

Title should be:

examples/vdpa: support running in nested virtualization environment

Quoted 47 lines

When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.

This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.

---
 examples/vdpa/main.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
examples/vdpa/main.c +1 -2
index 4c7e81d7b6..a24c78d7a4 100644
@@ -43,16 +43,20 @@ static char iface[MAX_PATH_LEN];
 static int devcnt;
 static int interactive;
 static int client_mode;
+static int iommu_support;
 
 /* display usage */
 static void
 vdpa_usage(const char *prgname)
 {
-	printf("Usage: %s [EAL options] -- "
-				 "	--interactive|-i: run in interactive

mode.\n"

examples/vdpa/main.c +0 -1
-				 "	--iface <path>: specify the path prefix of

the socket files, e.g. /tmp/vhost-user-.\n"

examples/vdpa/main.c +0 -1
-				 "	--client: register a vhost-user socket as

client mode.\n",

examples/vdpa/main.c +2 -1
-				 prgname);
+	printf("Usage: %s [EAL options] --\n", prgname);
+	const char *usage_str = "	--interactive|-i: run in interactive

mode.\n"

examples/vdpa/main.c +1 -0
+				"	--iface <path>: specify the path prefix of

the socket files, e.g. /tmp/vhost-user-.\n"

examples/vdpa/main.c +1 -0
+				"	--client: register a vhost-user socket as

client mode.\n"

examples/vdpa/main.c +1 -0
+				"	--iommu-support: use guest vIOMMU to protect

vhost.\n";

examples/vdpa/main.c static char iface[MAX_PATH_LEN]; · line 58

Quoted 2 lines
examples/vdpa/main.c +2 -0
+
+	fprintf(stderr, "%s", usage_str);

Why not use fprintf all string including the eal one?

examples/vdpa/main.c static char iface[MAX_PATH_LEN]; · line 59

Quoted 1 line
examples/vdpa/main.c +1 -0
+	exit(1);

I believe the exit should happen outside of the function.

Thanks,
Chenbo

Quoted 24 lines

}

static int

examples/vdpa/main.c +5 -1
@@ -63,6 +67,7 @@ parse_args(int argc, char **argv)
 		{"iface", required_argument, NULL, 0},
 		{"interactive", no_argument, &interactive, 1},
 		{"client", no_argument, &client_mode, 1},
+		{"iommu-support", no_argument, &iommu_support, 1},
 		{NULL, 0, 0, 0},
 	};
 	int opt, idx;
@@ -220,6 +225,10 @@ start_vdpa(struct vdpa_port *vport)
 			socket_path);
 		return -1;
 	}
+
+	if (iommu_support)
+		vport->flags |= RTE_VHOST_USER_IOMMU_SUPPORT;
+
 	ret = rte_vhost_driver_register(socket_path, vport->flags);
 	if (ret != 0)
 		rte_exit(EXIT_FAILURE,
--

2.27.0

author
#2 lore raw

PATCH v2 1/1 example: vdpa: support run in nested virtualization environment

In reply to Hao Chen

When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.

This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.

---
v2:
*fprintf all string including the eal one.
*remove exit(1).

 examples/vdpa/main.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
examples/vdpa/main.c +12 -5
index 4c7e81d7b6..71149461c6 100644
@@ -43,16 +43,18 @@ static char iface[MAX_PATH_LEN];
 static int devcnt;
 static int interactive;
 static int client_mode;
+static int iommu_support;
 
 /* display usage */
 static void
 vdpa_usage(const char *prgname)
 {
-	printf("Usage: %s [EAL options] -- "
-				 "	--interactive|-i: run in interactive mode.\n"
-				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
-				 "	--client: register a vhost-user socket as client mode.\n",
-				 prgname);
+	const char *usage_str = "	--interactive|-i: run in interactive mode.\n"
+				"	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+				"	--client: register a vhost-user socket as client mode.\n"
+				"	--iommu-support: use guest vIOMMU to protect vhost.\n";
+
+	fprintf(stderr, "Usage: %s [EAL options] --\n%s", prgname, usage_str);
 }
 
 static int
@@ -63,6 +65,7 @@ parse_args(int argc, char **argv)
 		{"iface", required_argument, NULL, 0},
 		{"interactive", no_argument, &interactive, 1},
 		{"client", no_argument, &client_mode, 1},
+		{"iommu-support", no_argument, &iommu_support, 1},
 		{NULL, 0, 0, 0},
 	};
 	int opt, idx;
@@ -220,6 +223,10 @@ start_vdpa(struct vdpa_port *vport)
 			socket_path);
 		return -1;
 	}
+
+	if (iommu_support)
+		vport->flags |= RTE_VHOST_USER_IOMMU_SUPPORT;
+
 	ret = rte_vhost_driver_register(socket_path, vport->flags);
 	if (ret != 0)
 		rte_exit(EXIT_FAILURE,
Signature · 1 line
2.27.0
author
#3 lore raw

PATCH v3 1/1 examples/vdpa: support running in nested virtualization environment

When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.

This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.

---
v3:
*Modify mail title.

v2:
*fprintf all string including the eal one.
*remove exit(1).

 examples/vdpa/main.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
examples/vdpa/main.c +12 -5
index 4c7e81d7b6..71149461c6 100644
@@ -43,16 +43,18 @@ static char iface[MAX_PATH_LEN];
 static int devcnt;
 static int interactive;
 static int client_mode;
+static int iommu_support;
 
 /* display usage */
 static void
 vdpa_usage(const char *prgname)
 {
-	printf("Usage: %s [EAL options] -- "
-				 "	--interactive|-i: run in interactive mode.\n"
-				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
-				 "	--client: register a vhost-user socket as client mode.\n",
-				 prgname);
+	const char *usage_str = "	--interactive|-i: run in interactive mode.\n"
+				"	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+				"	--client: register a vhost-user socket as client mode.\n"
+				"	--iommu-support: use guest vIOMMU to protect vhost.\n";
+
+	fprintf(stderr, "Usage: %s [EAL options] --\n%s", prgname, usage_str);
 }
 
 static int
@@ -63,6 +65,7 @@ parse_args(int argc, char **argv)
 		{"iface", required_argument, NULL, 0},
 		{"interactive", no_argument, &interactive, 1},
 		{"client", no_argument, &client_mode, 1},
+		{"iommu-support", no_argument, &iommu_support, 1},
 		{NULL, 0, 0, 0},
 	};
 	int opt, idx;
@@ -220,6 +223,10 @@ start_vdpa(struct vdpa_port *vport)
 			socket_path);
 		return -1;
 	}
+
+	if (iommu_support)
+		vport->flags |= RTE_VHOST_USER_IOMMU_SUPPORT;
+
 	ret = rte_vhost_driver_register(socket_path, vport->flags);
 	if (ret != 0)
 		rte_exit(EXIT_FAILURE,
Signature · 1 line
2.27.0
#4 lore raw

Re: [PATCH v3] examples/vdpa: support running in nested virtualization environment

Hi Hao

Quoted 2 lines · Hao Chen

When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is

NG == not good? Please don't use abbreviations.

Quoted 7 lines

that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.

This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.

I'm wondering whether the flag shouldn't just be set by default, as the feature negotiation will be done between the vDPA driver & the guest driver anyways?

Quoted 59 lines

---
v3:
*Modify mail title.

v2:
*fprintf all string including the eal one.
*remove exit(1).

  examples/vdpa/main.c | 17 ++++++++++++-----
  1 file changed, 12 insertions(+), 5 deletions(-)
examples/vdpa/main.c +12 -5
index 4c7e81d7b6..71149461c6 100644
@@ -43,16 +43,18 @@ static char iface[MAX_PATH_LEN];
  static int devcnt;
  static int interactive;
  static int client_mode;
+static int iommu_support;
 
  /* display usage */
  static void
  vdpa_usage(const char *prgname)
  {
-	printf("Usage: %s [EAL options] -- "
-				 "	--interactive|-i: run in interactive mode.\n"
-				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
-				 "	--client: register a vhost-user socket as client mode.\n",
-				 prgname);
+	const char *usage_str = "	--interactive|-i: run in interactive mode.\n"
+				"	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+				"	--client: register a vhost-user socket as client mode.\n"
+				"	--iommu-support: use guest vIOMMU to protect vhost.\n";
+
+	fprintf(stderr, "Usage: %s [EAL options] --\n%s", prgname, usage_str);
  }
 
  static int
@@ -63,6 +65,7 @@ parse_args(int argc, char **argv)
  		{"iface", required_argument, NULL, 0},
  		{"interactive", no_argument, &interactive, 1},
  		{"client", no_argument, &client_mode, 1},
+		{"iommu-support", no_argument, &iommu_support, 1},
  		{NULL, 0, 0, 0},
  	};
  	int opt, idx;
@@ -220,6 +223,10 @@ start_vdpa(struct vdpa_port *vport)
  			socket_path);
  		return -1;
  	}
+
+	if (iommu_support)
+		vport->flags |= RTE_VHOST_USER_IOMMU_SUPPORT;
+
  	ret = rte_vhost_driver_register(socket_path, vport->flags);
  	if (ret != 0)
  		rte_exit(EXIT_FAILURE,
author
#5 lore raw

Re: [PATCH v3] examples/vdpa: support running in nested virtualization environment

Quoted 8 lines · Maxime Coquelin

Hi Hao

Quoted 2 lines · Hao Chen

When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is NG. The reason for troubleshooting is

NG == not good? Please don't use abbreviations.

Yes, I will pay attention to this later.

Quoted 4 lines
Quoted 4 lines

that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.

Quoted 8 lines
Quoted 3 lines

This patch add option '--iommu-support' to use guest vIOMMU to protect vhost, then the ping test in vm-L2 is OK. This option is required in a nested virtualization environment.

I'm wondering whether the flag shouldn't just be set by default, as the feature negotiation will be done between the vDPA driver & the guest driver anyways?

Yes, It can be set by default rather than as an option. VIRTIO_F_IOMMU_PLATFORM feature will not be negotiated successfully if it is not in a nested virtualization environment.

I will simplify the patch.

Thanks.

Quoted 66 lines
Quoted 65 lines

---
v3:
*Modify mail title.

v2:
*fprintf all string including the eal one.
*remove exit(1).

  examples/vdpa/main.c | 17 ++++++++++++-----

  1 file changed, 12 insertions(+), 5 deletions(-)
examples/vdpa/main.c +1 -0
index 4c7e81d7b6..71149461c6 100644
@@ -43,16 +43,18 @@ static char iface[MAX_PATH_LEN];
   static int devcnt;
   static int interactive;
   static int client_mode;
+static int iommu_support;
     /* display usage */

  static void
  vdpa_usage(const char *prgname)
  {
-    printf("Usage: %s [EAL options] -- "
-                 "    --interactive|-i: run in interactive mode.\n" -                 "    --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
-                 "    --client: register a vhost-user socket as client mode.\n",
-                 prgname);
+    const char *usage_str = "    --interactive|-i: run in interactive mode.\n"
+                "    --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+                "    --client: register a vhost-user socket as client mode.\n"
+                "    --iommu-support: use guest vIOMMU to protect vhost.\n";
+
+    fprintf(stderr, "Usage: %s [EAL options] --\n%s", prgname, usage_str);
  }
    static int

examples/vdpa/main.c +5 -0
@@ -63,6 +65,7 @@ parse_args(int argc, char **argv)
           {"iface", required_argument, NULL, 0},
           {"interactive", no_argument, &interactive, 1},
           {"client", no_argument, &client_mode, 1},
+        {"iommu-support", no_argument, &iommu_support, 1},
           {NULL, 0, 0, 0},
       };
       int opt, idx;
@@ -220,6 +223,10 @@ start_vdpa(struct vdpa_port *vport)
               socket_path);
           return -1;
       }
+
+    if (iommu_support)
+        vport->flags |= RTE_VHOST_USER_IOMMU_SUPPORT;
+
       ret = rte_vhost_driver_register(socket_path, vport->flags);
       if (ret != 0)
           rte_exit(EXIT_FAILURE,
author
#6 lore raw

PATCH v4 1/1 examples/vdpa: support running in nested virtualization environment

In reply to Hao Chen

When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is not good. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.

This patch add flag RTE_VHOST_USER_IOMMU_SUPPORT to use vhost vIOMMU , VIRTIO_F_IOMMU_PLATFORM feature will be negotiated successfully if virtio iommu is used in a nested virtualization environment.

The configuration is as follows:
The host starts iommu, and the kernel parameter is added with 'intel_iommu=on iommu=pt'.
VM-L1's xml add viommu and virtio device adds iommu='on' ats='on'. VM-L2's xml enables viommu, and adds parameters 'intel_iommu=on iommu=pt' to kernel.

Then the ping test in vm-L2 is OK.

---
v4:
*Simplify the patch. Set the flags RTE_VHOST_USER_IOMMU_SUPPORT default.

v3:
*Modify mail title.

v2:
*fprintf all string including the eal one.
*remove exit(1).

 examples/vdpa/main.c | 2 ++
 1 file changed, 2 insertions(+)
examples/vdpa/main.c +2 -0
index 4c7e81d7b6..4d3203f3a7 100644
@@ -214,6 +214,8 @@ start_vdpa(struct vdpa_port *vport)
 	if (client_mode)
 		vport->flags |= RTE_VHOST_USER_CLIENT;
 
+	vport->flags |= RTE_VHOST_USER_IOMMU_SUPPORT;
+
 	if (access(socket_path, F_OK) != -1 && !client_mode) {
 		RTE_LOG(ERR, VDPA,
 			"%s exists, please remove it or specify another file and try again.\n",
Signature · 1 line
2.27.0
Reviewed-by on v4 1/1
#7 lore raw

Re: [PATCH v4] examples/vdpa: support running in nested virtualization environment

Quoted 39 lines · Hao Chen

When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is not good. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.

This patch add flag RTE_VHOST_USER_IOMMU_SUPPORT to use vhost vIOMMU , VIRTIO_F_IOMMU_PLATFORM feature will be negotiated successfully if virtio iommu is used in a nested virtualization environment.

The configuration is as follows:
The host starts iommu, and the kernel parameter is added with 'intel_iommu=on iommu=pt'.
VM-L1's xml add viommu and virtio device adds iommu='on' ats='on'. VM-L2's xml enables viommu, and adds parameters 'intel_iommu=on iommu=pt' to kernel.

Then the ping test in vm-L2 is OK.

---
v4:
*Simplify the patch. Set the flags RTE_VHOST_USER_IOMMU_SUPPORT default.

v3:
*Modify mail title.

v2:
*fprintf all string including the eal one.
*remove exit(1).

  examples/vdpa/main.c | 2 ++
  1 file changed, 2 insertions(+)
examples/vdpa/main.c +0 -0
index 4c7e81d7b6..4d3203f3a7 100644

examples/vdpa/main.c start_vdpa(struct vdpa_port *vport) · line 221

Quoted 8 lines
examples/vdpa/main.c +2 -0
@@ -214,6 +214,8 @@ start_vdpa(struct vdpa_port *vport)
  	if (client_mode)
  		vport->flags |= RTE_VHOST_USER_CLIENT;
 
+	vport->flags |= RTE_VHOST_USER_IOMMU_SUPPORT;
+
  	if (access(socket_path, F_OK) != -1 && !client_mode) {
  		RTE_LOG(ERR, VDPA,
  			"%s exists, please remove it or specify another file and try again.\n",

Thanks,
Maxime

#8 lore raw

Re: [PATCH v4] examples/vdpa: support running in nested virtualization environment

In reply to Hao Chen
Quoted 39 lines · Hao Chen

When we run dpdk vdpa in the nested virtual machine vm-L1 and ping test in vm-L2, the ping is not good. The reason for troubleshooting is that the virtio net in vm-L2 sends control information to the vring, and the qemu back-end device in vm-L1 cannot obtain correct data from the vring. This problem is related to the opening of the vIOMMU.

This patch add flag RTE_VHOST_USER_IOMMU_SUPPORT to use vhost vIOMMU , VIRTIO_F_IOMMU_PLATFORM feature will be negotiated successfully if virtio iommu is used in a nested virtualization environment.

The configuration is as follows:
The host starts iommu, and the kernel parameter is added with 'intel_iommu=on iommu=pt'.
VM-L1's xml add viommu and virtio device adds iommu='on' ats='on'. VM-L2's xml enables viommu, and adds parameters 'intel_iommu=on iommu=pt' to kernel.

Then the ping test in vm-L2 is OK.

---
v4:
*Simplify the patch. Set the flags RTE_VHOST_USER_IOMMU_SUPPORT default.

v3:
*Modify mail title.

v2:
*fprintf all string including the eal one.
*remove exit(1).

  examples/vdpa/main.c | 2 ++
  1 file changed, 2 insertions(+)
examples/vdpa/main.c +0 -0
index 4c7e81d7b6..4d3203f3a7 100644

examples/vdpa/main.c start_vdpa(struct vdpa_port *vport) · line 221

Quoted 8 lines
examples/vdpa/main.c +2 -0
@@ -214,6 +214,8 @@ start_vdpa(struct vdpa_port *vport)
  	if (client_mode)
  		vport->flags |= RTE_VHOST_USER_CLIENT;
 
+	vport->flags |= RTE_VHOST_USER_IOMMU_SUPPORT;
+
  	if (access(socket_path, F_OK) != -1 && !client_mode) {
  		RTE_LOG(ERR, VDPA,
  			"%s exists, please remove it or specify another file and try again.\n",

Applied to dpdk-next-virtio/main.

Thanks,
Maxime