Skip to contentSkip to conversation

vhost: add vDPA resource cleanup callback

PATCH Superseded Xueming Li 3 participants 13 messages Oct 19, 2021 – Feb 10, 2022

lore
Share & tools
lore

Xueming Li

author
#0 lore raw

PATCH 1/1 vhost: add vDPA resource cleanup callback

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
 lib/vhost/rte_vdpa_dev.h | 3 +++
 lib/vhost/vhost_user.c   | 6 ++++++
 2 files changed, 9 insertions(+)
lib/vhost/rte_vdpa_dev.h +3 -0
index b0f494815fa..2711004fe05 100644
@@ -32,6 +32,9 @@ struct rte_vdpa_dev_ops {
 	/** Driver close the device (Mandatory) */
 	int (*dev_close)(int vid);
 
+	/** Connection closed, clean up resources */
+	int (*dev_cleanup)(int vid);
+
 	/** Enable/disable this vring (Mandatory) */
 	int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +6 -0
index 5a894ca0cc7..032b621c86c 100644
@@ -162,6 +162,12 @@ free_mem_region(struct virtio_net *dev)
 void
 vhost_backend_cleanup(struct virtio_net *dev)
 {
+	struct rte_vdpa_device *vdpa_dev;
+
+	vdpa_dev = dev->vdpa_dev;
+	if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)
+		vdpa_dev->ops->dev_cleanup(dev->vid);
+
 	if (dev->mem) {
 		free_mem_region(dev);
 		rte_free(dev->mem);
Signature · 1 line
2.33.0
#1 lore raw

Hi Xueming,

Quoted 31 lines · Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
  lib/vhost/rte_vdpa_dev.h | 3 +++
  lib/vhost/vhost_user.c   | 6 ++++++
  2 files changed, 9 insertions(+)
lib/vhost/rte_vdpa_dev.h +3 -0
index b0f494815fa..2711004fe05 100644
@@ -32,6 +32,9 @@ struct rte_vdpa_dev_ops {
  	/** Driver close the device (Mandatory) */
  	int (*dev_close)(int vid);
 
+	/** Connection closed, clean up resources */
+	int (*dev_cleanup)(int vid);
+
  	/** Enable/disable this vring (Mandatory) */
  	int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +0 -0
index 5a894ca0cc7..032b621c86c 100644
@@ -162,6 +162,12 @@ free_mem_region(struct virtio_net *dev)
  void
  vhost_backend_cleanup(struct virtio_net *dev)

lib/vhost/vhost_user.c free_mem_region(struct virtio_net *dev) · line 173

Quoted 10 lines
lib/vhost/vhost_user.c +6 -0
  {
+	struct rte_vdpa_device *vdpa_dev;
+
+	vdpa_dev = dev->vdpa_dev;
+	if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)
+		vdpa_dev->ops->dev_cleanup(dev->vid);
+
  	if (dev->mem) {
  		free_mem_region(dev);
  		rte_free(dev->mem);

What will be done there that cannot be done in .dev_close()? Having the mlx5 implementation of this callback alongside this patch may help to understand.

Thanks,
Maxime

#2 lore raw
Quoted 38 lines · Maxime Coquelin

Hi Xueming,

Quoted 35 lines · Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
  lib/vhost/rte_vdpa_dev.h | 3 +++
  lib/vhost/vhost_user.c   | 6 ++++++
  2 files changed, 9 insertions(+)
lib/vhost/rte_vdpa_dev.h +3 -0
index b0f494815fa..2711004fe05 100644
@@ -32,6 +32,9 @@ struct rte_vdpa_dev_ops {
  	/** Driver close the device (Mandatory) */
  	int (*dev_close)(int vid);
 
+	/** Connection closed, clean up resources */
+	int (*dev_cleanup)(int vid);
+
  	/** Enable/disable this vring (Mandatory) */
  	int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +4 -0
index 5a894ca0cc7..032b621c86c 100644
@@ -162,6 +162,12 @@ free_mem_region(struct virtio_net *dev)
  void
  vhost_backend_cleanup(struct virtio_net *dev)
  {
+	struct rte_vdpa_device *vdpa_dev;
+
+	vdpa_dev = dev->vdpa_dev;
+	if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)
Quoted 8 lines

lib/vhost/vhost_user.c free_mem_region(struct virtio_net *dev) · line 173

Quoted 6 lines
lib/vhost/vhost_user.c +2 -0
+		vdpa_dev->ops->dev_cleanup(dev->vid);
+
  	if (dev->mem) {
  		free_mem_region(dev);
  		rte_free(dev->mem);

What will be done there that cannot be done in .dev_close()?

.dev_close() mainly handles VM suspend and driver reset. If release everything inside dev_close(), the suspend and resume takes longer time if number of VQs are huge. Customer want to upgrade VM configuration using suspend and resume, pause customer VM too long can't be accepted. So the idea is to cache and reuse resource between dev_close() and dev_conf(). Actually, the two functions looks more like dev_stop() and dev_start().

dev_cleanup hooks to vhost backend cleanup which called when socket closed for both client and server mode, a safe point to cleanup all cached resources.

Quoted 2 lines

Having the mlx5 implementation of this callback alongside this patch may help to understand.

The mlx5 implementation still a prototype, pending on internal review. So I just post the vhost part to get suggestion/comment. Let me know if the ugly code does help :)

Quoted 4 lines

Thanks,
Maxime

Reviewed-by Maxime Coquelin Reviewed-by
#3 lore raw

Hi Xueming,

Quoted 55 lines

-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Thursday, October 21, 2021 8:36 PM
To: [email protected]; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: add vDPA resource cleanup callback

Quoted 38 lines · Maxime Coquelin

Hi Xueming,

Quoted 35 lines · Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
  lib/vhost/rte_vdpa_dev.h | 3 +++
  lib/vhost/vhost_user.c   | 6 ++++++
  2 files changed, 9 insertions(+)
lib/vhost/rte_vdpa_dev.h +3 -0
index b0f494815fa..2711004fe05 100644
@@ -32,6 +32,9 @@ struct rte_vdpa_dev_ops {
  	/** Driver close the device (Mandatory) */
  	int (*dev_close)(int vid);
 
+	/** Connection closed, clean up resources */
+	int (*dev_cleanup)(int vid);
+
  	/** Enable/disable this vring (Mandatory) */
  	int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +4 -0
index 5a894ca0cc7..032b621c86c 100644
@@ -162,6 +162,12 @@ free_mem_region(struct virtio_net *dev)
  void
  vhost_backend_cleanup(struct virtio_net *dev)
  {
+	struct rte_vdpa_device *vdpa_dev;
+
+	vdpa_dev = dev->vdpa_dev;
+	if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)
Quoted 8 lines

lib/vhost/vhost_user.c free_mem_region(struct virtio_net *dev) · line 173

Quoted 6 lines
lib/vhost/vhost_user.c +2 -0
+		vdpa_dev->ops->dev_cleanup(dev->vid);
+
  	if (dev->mem) {
  		free_mem_region(dev);
  		rte_free(dev->mem);

What will be done there that cannot be done in .dev_close()?

Quoted 4 lines

.dev_close() mainly handles VM suspend and driver reset. If release everything inside dev_close(), the suspend and resume takes longer time if number of VQs are huge. Customer want to upgrade VM configuration using suspend and resume, pause customer VM too long can't be accepted.

By saying 'upgrade VM configuration', do you mean VM memory hotplug? Or something more?

Is this patch a next-step improvement of this commit?

commit 127f9c6f7b78a47b73b3e1c39e021cc81a30b4c9

Author: Matan Azrad <[email protected]>
Date:   Mon Jun 29 14:08:19 2020 +0000

vhost: handle memory hotplug with vDPA devices

Some vDPA drivers' basic configurations should be updated when the guest memory is hotplugged.

Close vDPA device before hotplug operation and recreate it after the hotplug operation is done.

Quoted 8 lines

So the idea is to cache and reuse resource between dev_close() and dev_conf(). Actually, the two functions looks more like dev_stop() and dev_start().

dev_cleanup hooks to vhost backend cleanup which called when socket closed for both client and server mode, a safe point to cleanup all cached resources.

Quoted 6 lines
Quoted 2 lines

Having the mlx5 implementation of this callback alongside this patch may help to understand.

The mlx5 implementation still a prototype, pending on internal review. So I just post the vhost part to get suggestion/comment. Let me know if the ugly code does help :)

I would prefer to see the mlx implementation with this patch in the same patchset to understand the problem. A new callback is fine if the problem itself makes sense :)

Thanks,
Chenbo

Quoted 5 lines
Quoted 5 lines

Thanks,
Maxime

#4 lore raw
Quoted 94 lines · Chenbo

Hi Xueming,

Quoted 55 lines

-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Thursday, October 21, 2021 8:36 PM
To: [email protected]; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: add vDPA resource cleanup callback

Quoted 38 lines · Maxime Coquelin

Hi Xueming,

Quoted 35 lines · Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
   lib/vhost/rte_vdpa_dev.h | 3 +++
   lib/vhost/vhost_user.c   | 6 ++++++
   2 files changed, 9 insertions(+)
lib/vhost/rte_vdpa_dev.h +3 -0
index b0f494815fa..2711004fe05 100644
@@ -32,6 +32,9 @@ struct rte_vdpa_dev_ops {
   	/** Driver close the device (Mandatory) */
   	int (*dev_close)(int vid);
 
+	/** Connection closed, clean up resources */
+	int (*dev_cleanup)(int vid);
+
   	/** Enable/disable this vring (Mandatory) */
   	int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +4 -0
index 5a894ca0cc7..032b621c86c 100644
@@ -162,6 +162,12 @@ free_mem_region(struct virtio_net *dev)
   void
   vhost_backend_cleanup(struct virtio_net *dev)
   {
+	struct rte_vdpa_device *vdpa_dev;
+
+	vdpa_dev = dev->vdpa_dev;
+	if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)
Quoted 8 lines

lib/vhost/vhost_user.c free_mem_region(struct virtio_net *dev) · line 173

Quoted 6 lines
lib/vhost/vhost_user.c +2 -0
+		vdpa_dev->ops->dev_cleanup(dev->vid);
+
   	if (dev->mem) {
   		free_mem_region(dev);
   		rte_free(dev->mem);

What will be done there that cannot be done in .dev_close()?

Quoted 4 lines

.dev_close() mainly handles VM suspend and driver reset. If release everything inside dev_close(), the suspend and resume takes longer time if number of VQs are huge. Customer want to upgrade VM configuration using suspend and resume, pause customer VM too long can't be accepted.

By saying 'upgrade VM configuration', do you mean VM memory hotplug? Or something more?

Is this patch a next-step improvement of this commit?

commit 127f9c6f7b78a47b73b3e1c39e021cc81a30b4c9

Author: Matan Azrad <[email protected]>
Date:   Mon Jun 29 14:08:19 2020 +0000

vhost: handle memory hotplug with vDPA devices

Some vDPA drivers' basic configurations should be updated when the guest memory is hotplugged.

Close vDPA device before hotplug operation and recreate it after the hotplug operation is done.

Quoted 11 lines

So the idea is to cache and reuse resource between dev_close() and dev_conf(). Actually, the two functions looks more like dev_stop() and dev_start().

dev_cleanup hooks to vhost backend cleanup which called when socket closed for both client and server mode, a safe point to cleanup all cached resources.

Quoted 2 lines

Having the mlx5 implementation of this callback alongside this patch may help to understand.

Quoted 7 lines
Quoted 3 lines

The mlx5 implementation still a prototype, pending on internal review. So I just post the vhost part to get suggestion/comment. Let me know if the ugly code does help :)

I would prefer to see the mlx implementation with this patch in the same patchset to understand the problem. A new callback is fine if the problem itself makes sense :)

FYI, I'm about to apply a patch that marks the vDPA driver API as internal, when you will submit a new version please apply on top of it.

Thanks,
Maxime

Quoted 9 lines

Thanks,
Chenbo

Quoted 5 lines
Quoted 5 lines

Thanks,
Maxime

#5 lore raw
Quoted 100 lines · Maxime Coquelin
Quoted 98 lines · Chenbo

Hi Xueming,

Quoted 55 lines

-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Thursday, October 21, 2021 8:36 PM
To: [email protected]; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: add vDPA resource cleanup callback

Quoted 38 lines · Maxime Coquelin

Hi Xueming,

Quoted 35 lines · Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
   lib/vhost/rte_vdpa_dev.h | 3 +++
   lib/vhost/vhost_user.c   | 6 ++++++
   2 files changed, 9 insertions(+)
lib/vhost/rte_vdpa_dev.h +3 -0
index b0f494815fa..2711004fe05 100644
@@ -32,6 +32,9 @@ struct rte_vdpa_dev_ops {
   	/** Driver close the device (Mandatory) */
   	int (*dev_close)(int vid);
 
+	/** Connection closed, clean up resources */
+	int (*dev_cleanup)(int vid);
+
   	/** Enable/disable this vring (Mandatory) */
   	int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +4 -0
index 5a894ca0cc7..032b621c86c 100644
@@ -162,6 +162,12 @@ free_mem_region(struct virtio_net *dev)
   void
   vhost_backend_cleanup(struct virtio_net *dev)
   {
+	struct rte_vdpa_device *vdpa_dev;
+
+	vdpa_dev = dev->vdpa_dev;
+	if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)
Quoted 8 lines

lib/vhost/vhost_user.c free_mem_region(struct virtio_net *dev) · line 173

Quoted 6 lines
lib/vhost/vhost_user.c +2 -0
+		vdpa_dev->ops->dev_cleanup(dev->vid);
+
   	if (dev->mem) {
   		free_mem_region(dev);
   		rte_free(dev->mem);

What will be done there that cannot be done in .dev_close()?

Quoted 4 lines

.dev_close() mainly handles VM suspend and driver reset. If release everything inside dev_close(), the suspend and resume takes longer time if number of VQs are huge. Customer want to upgrade VM configuration using suspend and resume, pause customer VM too long can't be accepted.

By saying 'upgrade VM configuration', do you mean VM memory hotplug? Or something more?

Is this patch a next-step improvement of this commit?

commit 127f9c6f7b78a47b73b3e1c39e021cc81a30b4c9

Author: Matan Azrad <[email protected]>
Date:   Mon Jun 29 14:08:19 2020 +0000

vhost: handle memory hotplug with vDPA devices

Some vDPA drivers' basic configurations should be updated when the guest memory is hotplugged.

Close vDPA device before hotplug operation and recreate it after the hotplug operation is done.

Quoted 14 lines

So the idea is to cache and reuse resource between dev_close() and dev_conf(). Actually, the two functions looks more like dev_stop() and dev_start().

dev_cleanup hooks to vhost backend cleanup which called when socket closed for both client and server mode, a safe point to cleanup all cached resources.

Quoted 2 lines

Having the mlx5 implementation of this callback alongside this patch may help to understand.

The mlx5 implementation still a prototype, pending on internal review. So I just post the vhost part to get suggestion/comment. Let me know if the ugly code does help :)

Quoted 6 lines
Quoted 3 lines

I would prefer to see the mlx implementation with this patch in the same patchset to understand the problem. A new callback is fine if the problem itself makes sense :)

FYI, I'm about to apply a patch that marks the vDPA driver API as internal, when you will submit a new version please apply on top of it.

Haven't check your patch yet, but sounds good form subject :)

Quoted 14 lines

Thanks,
Maxime

Quoted 9 lines

Thanks,
Chenbo

Quoted 5 lines
Quoted 5 lines

Thanks,
Maxime

#6 lore raw
Quoted 101 lines · Xueming(Steven) Li
Quoted 100 lines · Maxime Coquelin
Quoted 98 lines · Chenbo

Hi Xueming,

Quoted 55 lines

-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Thursday, October 21, 2021 8:36 PM
To: [email protected]; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: add vDPA resource cleanup callback

Quoted 38 lines · Maxime Coquelin

Hi Xueming,

Quoted 35 lines · Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
    lib/vhost/rte_vdpa_dev.h | 3 +++
    lib/vhost/vhost_user.c   | 6 ++++++
    2 files changed, 9 insertions(+)
lib/vhost/rte_vdpa_dev.h +3 -0
index b0f494815fa..2711004fe05 100644
@@ -32,6 +32,9 @@ struct rte_vdpa_dev_ops {
    	/** Driver close the device (Mandatory) */
    	int (*dev_close)(int vid);
 
+	/** Connection closed, clean up resources */
+	int (*dev_cleanup)(int vid);
+
    	/** Enable/disable this vring (Mandatory) */
    	int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +4 -0
index 5a894ca0cc7..032b621c86c 100644
@@ -162,6 +162,12 @@ free_mem_region(struct virtio_net *dev)
    void
    vhost_backend_cleanup(struct virtio_net *dev)
    {
+	struct rte_vdpa_device *vdpa_dev;
+
+	vdpa_dev = dev->vdpa_dev;
+	if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)
Quoted 8 lines

lib/vhost/vhost_user.c free_mem_region(struct virtio_net *dev) · line 173

Quoted 6 lines
lib/vhost/vhost_user.c +2 -0
+		vdpa_dev->ops->dev_cleanup(dev->vid);
+
    	if (dev->mem) {
    		free_mem_region(dev);
    		rte_free(dev->mem);

What will be done there that cannot be done in .dev_close()?

Quoted 4 lines

.dev_close() mainly handles VM suspend and driver reset. If release everything inside dev_close(), the suspend and resume takes longer time if number of VQs are huge. Customer want to upgrade VM configuration using suspend and resume, pause customer VM too long can't be accepted.

By saying 'upgrade VM configuration', do you mean VM memory hotplug? Or something more?

Is this patch a next-step improvement of this commit?

commit 127f9c6f7b78a47b73b3e1c39e021cc81a30b4c9

Author: Matan Azrad <[email protected]>
Date:   Mon Jun 29 14:08:19 2020 +0000

vhost: handle memory hotplug with vDPA devices

Some vDPA drivers' basic configurations should be updated when the guest memory is hotplugged.

Close vDPA device before hotplug operation and recreate it after the hotplug operation is done.

Quoted 14 lines

So the idea is to cache and reuse resource between dev_close() and dev_conf(). Actually, the two functions looks more like dev_stop() and dev_start().

dev_cleanup hooks to vhost backend cleanup which called when socket closed for both client and server mode, a safe point to cleanup all cached resources.

Quoted 2 lines

Having the mlx5 implementation of this callback alongside this patch may help to understand.

The mlx5 implementation still a prototype, pending on internal review. So I just post the vhost part to get suggestion/comment. Let me know if the ugly code does help :)

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I would prefer to see the mlx implementation with this patch in the same patchset to understand the problem. A new callback is fine if the problem itself makes sense :)

FYI, I'm about to apply a patch that marks the vDPA driver API as internal, when you will submit a new version please apply on top of it.

Haven't check your patch yet, but sounds good form subject :)

The branch is now ready, you can rebase your patch on top of it:
git://dpdk.org/next/dpdk-next-virtio main

Maxime

Quoted 15 lines
Quoted 14 lines

Thanks,
Maxime

Quoted 9 lines

Thanks,
Chenbo

Quoted 5 lines
Quoted 5 lines

Thanks,
Maxime

#7 lore raw

Hi Xueming,

Quoted 108 lines · Maxime Coquelin
Quoted 105 lines · Xueming(Steven) Li
Quoted 104 lines · Maxime Coquelin
Quoted 101 lines · Chenbo

Hi Xueming,

Quoted 55 lines

-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Thursday, October 21, 2021 8:36 PM
To: [email protected]; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: add vDPA resource cleanup callback

Quoted 38 lines · Maxime Coquelin

Hi Xueming,

Quoted 35 lines · Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
    lib/vhost/rte_vdpa_dev.h | 3 +++
    lib/vhost/vhost_user.c   | 6 ++++++

    2 files changed, 9 insertions(+)
lib/vhost/rte_vdpa_dev.h +3 -0
index b0f494815fa..2711004fe05 100644
@@ -32,6 +32,9 @@ struct rte_vdpa_dev_ops {
         /** Driver close the device (Mandatory) */
         int (*dev_close)(int vid);
 
+    /** Connection closed, clean up resources */
+    int (*dev_cleanup)(int vid);
+
         /** Enable/disable this vring (Mandatory) */
         int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +0 -0
index 5a894ca0cc7..032b621c86c 100644
@@ -162,6 +162,12 @@ free_mem_region(struct virtio_net *dev)

    void
    vhost_backend_cleanup(struct virtio_net *dev) {
+    struct rte_vdpa_device *vdpa_dev;
+
+    vdpa_dev = dev->vdpa_dev;
+    if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)

Quoted 8 lines
Quoted 6 lines

+        vdpa_dev->ops->dev_cleanup(dev->vid);
+
        if (dev->mem) {
            free_mem_region(dev);
            rte_free(dev->mem);

What will be done there that cannot be done in .dev_close()?

Quoted 4 lines

.dev_close() mainly handles VM suspend and driver reset. If release everything inside dev_close(), the suspend and resume takes longer time if number of VQs are huge. Customer want to upgrade VM configuration using suspend and resume, pause customer VM too long can't be accepted.

By saying 'upgrade VM configuration', do you mean VM memory hotplug? Or something more?

Is this patch a next-step improvement of this commit?

commit 127f9c6f7b78a47b73b3e1c39e021cc81a30b4c9
Author: Matan Azrad <[email protected]>
Date:   Mon Jun 29 14:08:19 2020 +0000

      vhost: handle memory hotplug with vDPA devices

      Some vDPA drivers' basic configurations should be updated when the guest memory is hotplugged.

      Close vDPA device before hotplug operation and recreate it after the hotplug operation is done.

Quoted 8 lines

So the idea is to cache and reuse resource between dev_close() and dev_conf(). Actually, the two functions looks more like dev_stop() and dev_start().

dev_cleanup hooks to vhost backend cleanup which called when socket closed for both client and server mode, a safe point to cleanup all cached resources.

Quoted 6 lines
Quoted 2 lines

Having the mlx5 implementation of this callback alongside this patch may help to understand.

The mlx5 implementation still a prototype, pending on internal review. So I just post the vhost part to get suggestion/comment. Let me know if the ugly code does help :)

I would prefer to see the mlx implementation with this patch in the same patchset to understand the problem. A new callback is fine if the problem itself makes sense :)

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

FYI, I'm about to apply a patch that marks the vDPA driver API as internal, when you will submit a new version please apply on top of it.

Haven't check your patch yet, but sounds good form subject :)

The branch is now ready, you can rebase your patch on top of it:
git://dpdk.org/next/dpdk-next-virtio main

Could you please rebase your patch if you want it in v22.03?

Thanks!
Maxime

Quoted 17 lines

Maxime

Quoted 15 lines
Quoted 14 lines

Thanks,
Maxime

Quoted 9 lines

Thanks,
Chenbo

Quoted 5 lines
Quoted 5 lines

Thanks,
Maxime

#8 lore raw
Quoted 123 lines · Maxime Coquelin

Hi Xueming,

Quoted 120 lines · Maxime Coquelin
Quoted 117 lines · Xueming(Steven) Li
Quoted 115 lines · Maxime Coquelin
Quoted 105 lines · Chenbo

Hi Xueming,

Quoted 55 lines

-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Thursday, October 21, 2021 8:36 PM
To: [email protected]; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: add vDPA resource cleanup callback

Quoted 38 lines · Maxime Coquelin

Hi Xueming,

Quoted 35 lines · Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
    lib/vhost/rte_vdpa_dev.h | 3 +++
    lib/vhost/vhost_user.c   | 6 ++++++

    2 files changed, 9 insertions(+)
lib/vhost/rte_vdpa_dev.h +3 -0
index b0f494815fa..2711004fe05 100644
@@ -32,6 +32,9 @@ struct rte_vdpa_dev_ops {
         /** Driver close the device (Mandatory) */
         int (*dev_close)(int vid);
 
+    /** Connection closed, clean up resources */
+    int (*dev_cleanup)(int vid);
+
         /** Enable/disable this vring (Mandatory) */
         int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +0 -0
index 5a894ca0cc7..032b621c86c 100644
@@ -162,6 +162,12 @@ free_mem_region(struct virtio_net *dev)

    void
    vhost_backend_cleanup(struct virtio_net *dev) {
+    struct rte_vdpa_device *vdpa_dev;
+
+    vdpa_dev = dev->vdpa_dev;
+    if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)

Quoted 8 lines
Quoted 6 lines

+        vdpa_dev->ops->dev_cleanup(dev->vid);
+
        if (dev->mem) {
            free_mem_region(dev);
            rte_free(dev->mem);

What will be done there that cannot be done in .dev_close()?

Quoted 6 lines

.dev_close() mainly handles VM suspend and driver reset. If release everything inside dev_close(), the suspend and resume takes longer time
if number of VQs are huge. Customer want to upgrade VM configuration using suspend and resume, pause customer VM too long can't be accepted.

By saying 'upgrade VM configuration', do you mean VM memory hotplug? Or something
more?

Is this patch a next-step improvement of this commit?

commit 127f9c6f7b78a47b73b3e1c39e021cc81a30b4c9
Author: Matan Azrad <[email protected]>
Date:   Mon Jun 29 14:08:19 2020 +0000

      vhost: handle memory hotplug with vDPA devices

      Some vDPA drivers' basic configurations should be updated when the
      guest memory is hotplugged.

      Close vDPA device before hotplug operation and recreate it after the
      hotplug operation is done.

Quoted 8 lines

So the idea is to cache and reuse resource between dev_close() and dev_conf(). Actually, the two functions looks more like dev_stop() and dev_start().

dev_cleanup hooks to vhost backend cleanup which called when socket closed for both client and server mode, a safe point to cleanup all cached resources.

Quoted 8 lines
Quoted 3 lines

Having the mlx5 implementation of this callback alongside this patch may
help to understand.

The mlx5 implementation still a prototype, pending on internal review. So I just post the vhost part to get suggestion/comment. Let me know if
the ugly code does help :)

Quoted 5 lines

I would prefer to see the mlx implementation with this patch in the same
patchset to understand the problem. A new callback is fine if the problem
itself makes sense :)

FYI, I'm about to apply a patch that marks the vDPA driver API as internal, when you will submit a new version please apply on top of it.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Haven't check your patch yet, but sounds good form subject :)

The branch is now ready, you can rebase your patch on top of it:
git://dpdk.org/next/dpdk-next-virtio main

Could you please rebase your patch if you want it in v22.03?

Thanks for reminding, new version sent.
Mlx5 PMD still WIP.

Quoted 22 lines

Thanks!
Maxime

Quoted 17 lines

Maxime

Quoted 15 lines
Quoted 14 lines

Thanks,
Maxime

Quoted 9 lines

Thanks,
Chenbo

Quoted 5 lines
Quoted 5 lines

Thanks,
Maxime

#9 lore raw
In reply to Xia, Chenbo
Quoted 62 lines · Chenbo

Hi Xueming,

Quoted 55 lines

-----Original Message-----
From: Xueming(Steven) Li <[email protected]>
Sent: Thursday, October 21, 2021 8:36 PM
To: [email protected]; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: add vDPA resource cleanup callback

Quoted 38 lines · Maxime Coquelin

Hi Xueming,

Quoted 35 lines · Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
  lib/vhost/rte_vdpa_dev.h | 3 +++
  lib/vhost/vhost_user.c   | 6 ++++++
  2 files changed, 9 insertions(+)
lib/vhost/rte_vdpa_dev.h +3 -0
index b0f494815fa..2711004fe05 100644
@@ -32,6 +32,9 @@ struct rte_vdpa_dev_ops {
  	/** Driver close the device (Mandatory) */
  	int (*dev_close)(int vid);
 
+	/** Connection closed, clean up resources */
+	int (*dev_cleanup)(int vid);
+
  	/** Enable/disable this vring (Mandatory) */
  	int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +4 -0
index 5a894ca0cc7..032b621c86c 100644
@@ -162,6 +162,12 @@ free_mem_region(struct virtio_net *dev)
  void
  vhost_backend_cleanup(struct virtio_net *dev)
  {
+	struct rte_vdpa_device *vdpa_dev;
+
+	vdpa_dev = dev->vdpa_dev;
+	if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)
Quoted 8 lines

lib/vhost/vhost_user.c free_mem_region(struct virtio_net *dev) · line 173

Quoted 6 lines
lib/vhost/vhost_user.c +2 -0
+		vdpa_dev->ops->dev_cleanup(dev->vid);
+
  	if (dev->mem) {
  		free_mem_region(dev);
  		rte_free(dev->mem);

What will be done there that cannot be done in .dev_close()?

Quoted 4 lines

.dev_close() mainly handles VM suspend and driver reset. If release everything inside dev_close(), the suspend and resume takes longer time if number of VQs are huge. Customer want to upgrade VM configuration using suspend and resume, pause customer VM too long can't be accepted.

Quoted 4 lines

By saying 'upgrade VM configuration', do you mean VM memory hotplug? Or something more?

Is this patch a next-step improvement of this commit?

Hi Chenbo,

Basically irrelevant IIUC, VM could be paused for more reasons like disk upgrade or something similar. To speed up device close, resources that could be reused in resume is not released. That's why we need a callback to free resources completely.

Quoted 28 lines

commit 127f9c6f7b78a47b73b3e1c39e021cc81a30b4c9

Author: Matan Azrad <[email protected]>
Date:   Mon Jun 29 14:08:19 2020 +0000

vhost: handle memory hotplug with vDPA devices

Some vDPA drivers' basic configurations should be updated when the guest memory is hotplugged.

Close vDPA device before hotplug operation and recreate it after the hotplug operation is done.

Quoted 11 lines

So the idea is to cache and reuse resource between dev_close() and dev_conf(). Actually, the two functions looks more like dev_stop() and dev_start().

dev_cleanup hooks to vhost backend cleanup which called when socket closed for both client and server mode, a safe point to cleanup all cached resources.

Quoted 2 lines

Having the mlx5 implementation of this callback alongside this patch may help to understand.

Quoted 7 lines
Quoted 3 lines

The mlx5 implementation still a prototype, pending on internal review. So I just post the vhost part to get suggestion/comment. Let me know if the ugly code does help :)

I would prefer to see the mlx implementation with this patch in the same patchset to understand the problem. A new callback is fine if the problem itself makes sense :)

Will post once ready, thanks!

Quoted 10 lines

Thanks,
Chenbo

Quoted 5 lines
Quoted 5 lines

Thanks,
Maxime

author
#10 lore raw

PATCH v2 1/1 vhost: add vDPA resource cleanup callback

In reply to Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
 lib/vhost/vdpa_driver.h | 3 +++
 lib/vhost/vhost_user.c  | 6 ++++++
 2 files changed, 9 insertions(+)
lib/vhost/vdpa_driver.h +3 -0
index fc2d6acedd1..fddbd506523 100644
@@ -34,6 +34,9 @@ struct rte_vdpa_dev_ops {
 	/** Driver close the device (Mandatory) */
 	int (*dev_close)(int vid);
 
+	/** Connection closed, clean up resources */
+	int (*dev_cleanup)(int vid);
+
 	/** Enable/disable this vring (Mandatory) */
 	int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +6 -0
index 5eb1dd68123..798b0ca4c0d 100644
@@ -220,6 +220,12 @@ free_mem_region(struct virtio_net *dev)
 void
 vhost_backend_cleanup(struct virtio_net *dev)
 {
+	struct rte_vdpa_device *vdpa_dev;
+
+	vdpa_dev = dev->vdpa_dev;
+	if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)
+		vdpa_dev->ops->dev_cleanup(dev->vid);
+
 	if (dev->mem) {
 		free_mem_region(dev);
 		rte_free(dev->mem);
Signature · 1 line
2.34.1
Reviewed-by on v2 1/1
#11 lore raw
Quoted 30 lines · Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
  lib/vhost/vdpa_driver.h | 3 +++
  lib/vhost/vhost_user.c  | 6 ++++++
  2 files changed, 9 insertions(+)
lib/vhost/vdpa_driver.h +3 -0
index fc2d6acedd1..fddbd506523 100644
@@ -34,6 +34,9 @@ struct rte_vdpa_dev_ops {
  	/** Driver close the device (Mandatory) */
  	int (*dev_close)(int vid);
 
+	/** Connection closed, clean up resources */
+	int (*dev_cleanup)(int vid);
+
  	/** Enable/disable this vring (Mandatory) */
  	int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +0 -0
index 5eb1dd68123..798b0ca4c0d 100644
@@ -220,6 +220,12 @@ free_mem_region(struct virtio_net *dev)
  void
  vhost_backend_cleanup(struct virtio_net *dev)

lib/vhost/vhost_user.c free_mem_region(struct virtio_net *dev) · line 231

Quoted 10 lines
lib/vhost/vhost_user.c +6 -0
  {
+	struct rte_vdpa_device *vdpa_dev;
+
+	vdpa_dev = dev->vdpa_dev;
+	if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)
+		vdpa_dev->ops->dev_cleanup(dev->vid);
+
  	if (dev->mem) {
  		free_mem_region(dev);
  		rte_free(dev->mem);

Thanks,
Maxime

#12 lore raw
In reply to Xueming Li
Quoted 30 lines · Xueming Li

This patch adds vDPA device cleanup callback to release resources on vhost user connection close.

---
  lib/vhost/vdpa_driver.h | 3 +++
  lib/vhost/vhost_user.c  | 6 ++++++
  2 files changed, 9 insertions(+)
lib/vhost/vdpa_driver.h +3 -0
index fc2d6acedd1..fddbd506523 100644
@@ -34,6 +34,9 @@ struct rte_vdpa_dev_ops {
  	/** Driver close the device (Mandatory) */
  	int (*dev_close)(int vid);
 
+	/** Connection closed, clean up resources */
+	int (*dev_cleanup)(int vid);
+
  	/** Enable/disable this vring (Mandatory) */
  	int (*set_vring_state)(int vid, int vring, int state);
 
lib/vhost/vhost_user.c +0 -0
index 5eb1dd68123..798b0ca4c0d 100644
@@ -220,6 +220,12 @@ free_mem_region(struct virtio_net *dev)
  void
  vhost_backend_cleanup(struct virtio_net *dev)

lib/vhost/vhost_user.c free_mem_region(struct virtio_net *dev) · line 231

Quoted 10 lines
lib/vhost/vhost_user.c +6 -0
  {
+	struct rte_vdpa_device *vdpa_dev;
+
+	vdpa_dev = dev->vdpa_dev;
+	if (vdpa_dev && vdpa_dev->ops->dev_cleanup != NULL)
+		vdpa_dev->ops->dev_cleanup(dev->vid);
+
  	if (dev->mem) {
  		free_mem_region(dev);
  		rte_free(dev->mem);

Applied to dpdk-next-virtio/main.

Thanks,
Maxime