Skip to contentSkip to conversation

Don't destroy the netdev until the vif is shut down

PATCH Superseded Paul Durrant 2 participants 5 messages Sep 12, 2013

Subsystems (3)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 Don't destroy the netdev until the vif is shut down

Without this patch, if a frontend cycles through states Closing and Closed (which Windows frontends need to do) then the netdev will be destroyed and requires re-invocation of hotplug scripts to restore state before the frontend can move to Connected. Thus when udev is not in use the backend gets stuck in InitWait.

With this patch, the netdev is left alone whilst the backend is still online and is only de-registered and freed just prior to destroying the vif (which is also nicely symmetrical with the netdev allocation and registration being done during probe) so no re-invocation of hotplug scripts is required.

Cc David Vrabel <[email protected]>
Cc Wei Liu <[email protected]>
Cc Ian Campbell <[email protected]>

---
v2:
- Modify netback_remove() - bug only seemed to manifest with linux guest

 drivers/net/xen-netback/common.h    |    1 +
 drivers/net/xen-netback/interface.c |   13 ++++++++-----
 drivers/net/xen-netback/xenbus.c    |   17 ++++++++++++-----
 3 files changed, 21 insertions(+), 10 deletions(-)
drivers/net/xen-netback/common.h +1 -0
index a197743..5715318 100644
@@ -184,6 +184,7 @@ int xenvif_connect(struct xenvif *vif, unsigned long tx_ring_ref,
 		   unsigned long rx_ring_ref, unsigned int tx_evtchn,
 		   unsigned int rx_evtchn);
 void xenvif_disconnect(struct xenvif *vif);
+void xenvif_free(struct xenvif *vif);
 
 int xenvif_xenbus_init(void);
 void xenvif_xenbus_fini(void);
drivers/net/xen-netback/interface.c +8 -5
index 625c6f4..65e78f9 100644
@@ -477,14 +477,17 @@ void xenvif_disconnect(struct xenvif *vif)
 	if (vif->task)
 		kthread_stop(vif->task);
 
+	xenvif_unmap_frontend_rings(vif);
+
+	if (need_module_put)
+		module_put(THIS_MODULE);
+}
+
+void xenvif_free(struct xenvif *vif)
+{
 	netif_napi_del(&vif->napi);
 
 	unregister_netdev(vif->dev);
 
-	xenvif_unmap_frontend_rings(vif);
-
 	free_netdev(vif->dev);
-
-	if (need_module_put)
-		module_put(THIS_MODULE);
 }
drivers/net/xen-netback/xenbus.c +12 -5
index 1fe48fe3..a53782e 100644
@@ -42,7 +42,7 @@ static int netback_remove(struct xenbus_device *dev)
 	if (be->vif) {
 		kobject_uevent(&dev->dev.kobj, KOBJ_OFFLINE);
 		xenbus_rm(XBT_NIL, dev->nodename, "hotplug-status");
-		xenvif_disconnect(be->vif);
+		xenvif_free(be->vif);
 		be->vif = NULL;
 	}
 	kfree(be);
@@ -213,9 +213,18 @@ static void disconnect_backend(struct xenbus_device *dev)
 {
 	struct backend_info *be = dev_get_drvdata(&dev->dev);
 
+	if (be->vif)
+		xenvif_disconnect(be->vif);
+}
+
+static void destroy_backend(struct xenbus_device *dev)
+{
+	struct backend_info *be = dev_get_drvdata(&dev->dev);
+
 	if (be->vif) {
+		kobject_uevent(&dev->dev.kobj, KOBJ_OFFLINE);
 		xenbus_rm(XBT_NIL, dev->nodename, "hotplug-status");
-		xenvif_disconnect(be->vif);
+		xenvif_free(be->vif);
 		be->vif = NULL;
 	}
 }
@@ -246,14 +255,11 @@ static void frontend_changed(struct xenbus_device *dev,
 	case XenbusStateConnected:
 		if (dev->state == XenbusStateConnected)
 			break;
-		backend_create_xenvif(be);
 		if (be->vif)
 			connect(be);
 		break;
 
 	case XenbusStateClosing:
-		if (be->vif)
-			kobject_uevent(&dev->dev.kobj, KOBJ_OFFLINE);
 		disconnect_backend(dev);
 		xenbus_switch_state(dev, XenbusStateClosing);
 		break;
@@ -262,6 +268,7 @@ static void frontend_changed(struct xenbus_device *dev,
 		xenbus_switch_state(dev, XenbusStateClosed);
 		if (xenbus_dev_is_online(dev))
 			break;
+		destroy_backend(dev);
 		/* fall through if not online */
 	case XenbusStateUnknown:
 		device_unregister(&dev->dev);
Signature · 1 line
1.7.10.4
#1 lore raw

The title should start with "[PATCH net-next]".

Quoted 30 lines · Paul Durrant

Without this patch, if a frontend cycles through states Closing and Closed (which Windows frontends need to do) then the netdev will be destroyed and requires re-invocation of hotplug scripts to restore state before the frontend can move to Connected. Thus when udev is not in use the backend gets stuck in InitWait.

With this patch, the netdev is left alone whilst the backend is still online and is only de-registered and freed just prior to destroying the vif (which is also nicely symmetrical with the netdev allocation and registration being done during probe) so no re-invocation of hotplug scripts is required.

Cc David Vrabel <[email protected]>
Cc Wei Liu <[email protected]>
Cc Ian Campbell <[email protected]>

---
v2:
- Modify netback_remove() - bug only seemed to manifest with linux guest

 drivers/net/xen-netback/common.h    |    1 +
 drivers/net/xen-netback/interface.c |   13 ++++++++-----
 drivers/net/xen-netback/xenbus.c    |   17 ++++++++++++-----
 3 files changed, 21 insertions(+), 10 deletions(-)
drivers/net/xen-netback/common.h +0 -0
index a197743..5715318 100644

drivers/net/xen-netback/common.h int xenvif_connect(struct xenvif *vif, unsigned long tx_ring_ref, · line 188

Quoted 5 lines
drivers/net/xen-netback/common.h +1 -0
@@ -184,6 +184,7 @@ int xenvif_connect(struct xenvif *vif, unsigned long tx_ring_ref,
 		   unsigned long rx_ring_ref, unsigned int tx_evtchn,
 		   unsigned int rx_evtchn);
 void xenvif_disconnect(struct xenvif *vif);
+void xenvif_free(struct xenvif *vif);
 

This can be moved a few lines above to group with xenvif_alloc() IMHO.

Quoted 7 lines
drivers/net/xen-netback/common.h +0 -0
 int xenvif_xenbus_init(void);
 void xenvif_xenbus_fini(void);
drivers/net/xen-netback/interface.c +0 -0
index 625c6f4..65e78f9 100644

drivers/net/xen-netback/interface.c void xenvif_disconnect(struct xenvif *vif) · line 485

Quoted 9 lines
drivers/net/xen-netback/interface.c +6 -0
@@ -477,14 +477,17 @@ void xenvif_disconnect(struct xenvif *vif)
 	if (vif->task)
 		kthread_stop(vif->task);
 
+	xenvif_unmap_frontend_rings(vif);
+
+	if (need_module_put)
+		module_put(THIS_MODULE);
+}
+

The original thought for this module_get/put thing is that admin can just disconnect all frontends then replace netback module. With this patch that doesn't work anymore. We leak memory when all frontends are disconnected and admin runs "modprobe -r xen-netback". That happened to work before ecause everytime a frontend is disconnected the netdev structure is already freed.

The ability to unload netback is still quite handy for developer / admin so I would like to keep it. My suggestion would be, move module_get/put to the point where netdev is allocated / freed. The impact of moving module_get/put is that we cannot just simply disconnect all frontends then replace netback, we need to actually shutdown / migrate all VMs before replacing netback.

Wei.

#2 lore raw
In reply to Paul Durrant

The title should start with "[PATCH net-next]".

Quoted 30 lines · Paul Durrant

Without this patch, if a frontend cycles through states Closing and Closed (which Windows frontends need to do) then the netdev will be destroyed and requires re-invocation of hotplug scripts to restore state before the frontend can move to Connected. Thus when udev is not in use the backend gets stuck in InitWait.

With this patch, the netdev is left alone whilst the backend is still online and is only de-registered and freed just prior to destroying the vif (which is also nicely symmetrical with the netdev allocation and registration being done during probe) so no re-invocation of hotplug scripts is required.

Cc David Vrabel <[email protected]>
Cc Wei Liu <[email protected]>
Cc Ian Campbell <[email protected]>

---
v2:
- Modify netback_remove() - bug only seemed to manifest with linux guest

 drivers/net/xen-netback/common.h    |    1 +
 drivers/net/xen-netback/interface.c |   13 ++++++++-----
 drivers/net/xen-netback/xenbus.c    |   17 ++++++++++++-----
 3 files changed, 21 insertions(+), 10 deletions(-)
drivers/net/xen-netback/common.h +0 -0
index a197743..5715318 100644

drivers/net/xen-netback/common.h int xenvif_connect(struct xenvif *vif, unsigned long tx_ring_ref, · line 188

Quoted 5 lines
drivers/net/xen-netback/common.h +1 -0
@@ -184,6 +184,7 @@ int xenvif_connect(struct xenvif *vif, unsigned long tx_ring_ref,
 		   unsigned long rx_ring_ref, unsigned int tx_evtchn,
 		   unsigned int rx_evtchn);
 void xenvif_disconnect(struct xenvif *vif);
+void xenvif_free(struct xenvif *vif);
 

This can be moved a few lines above to group with xenvif_alloc() IMHO.

Quoted 7 lines
drivers/net/xen-netback/common.h +0 -0
 int xenvif_xenbus_init(void);
 void xenvif_xenbus_fini(void);
drivers/net/xen-netback/interface.c +0 -0
index 625c6f4..65e78f9 100644

drivers/net/xen-netback/interface.c void xenvif_disconnect(struct xenvif *vif) · line 485

Quoted 9 lines
drivers/net/xen-netback/interface.c +6 -0
@@ -477,14 +477,17 @@ void xenvif_disconnect(struct xenvif *vif)
 	if (vif->task)
 		kthread_stop(vif->task);
 
+	xenvif_unmap_frontend_rings(vif);
+
+	if (need_module_put)
+		module_put(THIS_MODULE);
+}
+

The original thought for this module_get/put thing is that admin can just disconnect all frontends then replace netback module. With this patch that doesn't work anymore. We leak memory when all frontends are disconnected and admin runs "modprobe -r xen-netback". That happened to work before ecause everytime a frontend is disconnected the netdev structure is already freed.

The ability to unload netback is still quite handy for developer / admin so I would like to keep it. My suggestion would be, move module_get/put to the point where netdev is allocated / freed. The impact of moving module_get/put is that we cannot just simply disconnect all frontends then replace netback, we need to actually shutdown / migrate all VMs before replacing netback.

Wei.

#3 lore raw
Quoted 10 lines

-----Original Message-----
From: Wei Liu [mailto:[email protected]]
Sent: 12 September 2013 16:21
To: Paul Durrant
Cc: [email protected]; [email protected]; David Vrabel; Wei Liu; Ian Campbell
Subject: Re: [PATCH v2] Don't destroy the netdev until the vif is shut down

The title should start with "[PATCH net-next]".

Ok.

Quoted 34 lines · Paul Durrant
Quoted 12 lines

Without this patch, if a frontend cycles through states Closing and Closed (which Windows frontends need to do) then the netdev will be destroyed and requires re-invocation of hotplug scripts to restore state before the frontend can move to Connected. Thus when udev is not in use the backend gets stuck in InitWait.

With this patch, the netdev is left alone whilst the backend is still online and is only de-registered and freed just prior to destroying the vif (which is also nicely symmetrical with the netdev allocation and registration being done during probe) so no re-invocation of hotplug scripts is required.

Quoted 7 lines
Cc David Vrabel <[email protected]>
Cc Wei Liu <[email protected]>
Cc Ian Campbell <[email protected]>

---
v2:
- Modify netback_remove() - bug only seemed to manifest with linux

guest

drivers/net/xen-

Quoted 7 lines
 drivers/net/xen-netback/common.h    |    1 +
 drivers/net/xen-netback/interface.c |   13 ++++++++-----
 drivers/net/xen-netback/xenbus.c    |   17 ++++++++++++-----
 3 files changed, 21 insertions(+), 10 deletions(-)
drivers/net/xen- renamed from drivers/net/xen-netback/common.h +0 -0

netback/common.h

drivers/net/xen-netback/common.h int xenvif_connect(struct xenvif *vif, unsigned long

Quoted 4 lines

index a197743..5715318 100644

drivers/net/xen-netback/common.h +0 -0
@@ -184,6 +184,7 @@ int xenvif_connect(struct xenvif *vif, unsigned long

tx_ring_ref,

Quoted 8 lines

drivers/net/xen-netback/common.h int xenvif_connect(struct xenvif *vif, unsigned long tx_ring_ref, · line 188

Quoted 5 lines
drivers/net/xen-netback/common.h +1 -0
 		   unsigned long rx_ring_ref, unsigned int tx_evtchn,
 		   unsigned int rx_evtchn);
 void xenvif_disconnect(struct xenvif *vif);
+void xenvif_free(struct xenvif *vif);
 

This can be moved a few lines above to group with xenvif_alloc() IMHO.

A matter of personal taste I think.

Quoted 27 lines

drivers/net/xen-

Quoted 3 lines
drivers/net/xen-netback/common.h +0 -0
 int xenvif_xenbus_init(void);
 void xenvif_xenbus_fini(void);
drivers/net/xen- renamed from drivers/net/xen-netback/interface.c +0 -0

netback/interface.c

drivers/net/xen-netback/interface.c void xenvif_disconnect(struct xenvif *vif) · line 485

Quoted 13 lines

index 625c6f4..65e78f9 100644

drivers/net/xen-netback/interface.c +6 -0
@@ -477,14 +477,17 @@ void xenvif_disconnect(struct xenvif *vif)
 	if (vif->task)
 		kthread_stop(vif->task);
 
+	xenvif_unmap_frontend_rings(vif);
+
+	if (need_module_put)
+		module_put(THIS_MODULE);
+}
+

The original thought for this module_get/put thing is that admin can just disconnect all frontends then replace netback module. With this patch that doesn't work anymore. We leak memory when all frontends are disconnected and admin runs "modprobe -r xen-netback". That happened to
work before ecause everytime a frontend is disconnected the netdev structure
is already freed.

Quoted 7 lines

The ability to unload netback is still quite handy for developer / admin so I would like to keep it. My suggestion would be, move module_get/put to the point where netdev is allocated / freed. The impact of moving module_get/put is that we cannot just simply disconnect all frontends then replace netback, we need to actually shutdown / migrate all VMs before replacing netback.

Presumably you still had the problem of needing to manually re-run the hotplug scripts if not using udev though. I'll move the module_get/put as you suggest.

Paul

#4 lore raw
In reply to Wei Liu
Quoted 10 lines

-----Original Message-----
From: Wei Liu [mailto:[email protected]]
Sent: 12 September 2013 16:21
To: Paul Durrant
Cc: [email protected]; [email protected]; David Vrabel; Wei Liu; Ian Campbell
Subject: Re: [PATCH v2] Don't destroy the netdev until the vif is shut down

The title should start with "[PATCH net-next]".

Ok.

Quoted 34 lines · Paul Durrant
Quoted 12 lines

Without this patch, if a frontend cycles through states Closing and Closed (which Windows frontends need to do) then the netdev will be destroyed and requires re-invocation of hotplug scripts to restore state before the frontend can move to Connected. Thus when udev is not in use the backend gets stuck in InitWait.

With this patch, the netdev is left alone whilst the backend is still online and is only de-registered and freed just prior to destroying the vif (which is also nicely symmetrical with the netdev allocation and registration being done during probe) so no re-invocation of hotplug scripts is required.

Quoted 7 lines
Cc David Vrabel <[email protected]>
Cc Wei Liu <[email protected]>
Cc Ian Campbell <[email protected]>

---
v2:
- Modify netback_remove() - bug only seemed to manifest with linux

guest

drivers/net/xen-

Quoted 7 lines
 drivers/net/xen-netback/common.h    |    1 +
 drivers/net/xen-netback/interface.c |   13 ++++++++-----
 drivers/net/xen-netback/xenbus.c    |   17 ++++++++++++-----
 3 files changed, 21 insertions(+), 10 deletions(-)
drivers/net/xen- renamed from drivers/net/xen-netback/common.h +0 -0

netback/common.h

drivers/net/xen-netback/common.h int xenvif_connect(struct xenvif *vif, unsigned long

Quoted 4 lines

index a197743..5715318 100644

drivers/net/xen-netback/common.h +0 -0
@@ -184,6 +184,7 @@ int xenvif_connect(struct xenvif *vif, unsigned long

tx_ring_ref,

Quoted 8 lines

drivers/net/xen-netback/common.h int xenvif_connect(struct xenvif *vif, unsigned long tx_ring_ref, · line 188

Quoted 5 lines
drivers/net/xen-netback/common.h +1 -0
 		   unsigned long rx_ring_ref, unsigned int tx_evtchn,
 		   unsigned int rx_evtchn);
 void xenvif_disconnect(struct xenvif *vif);
+void xenvif_free(struct xenvif *vif);
 

This can be moved a few lines above to group with xenvif_alloc() IMHO.

A matter of personal taste I think.

Quoted 27 lines

drivers/net/xen-

Quoted 3 lines
drivers/net/xen-netback/common.h +0 -0
 int xenvif_xenbus_init(void);
 void xenvif_xenbus_fini(void);
drivers/net/xen- renamed from drivers/net/xen-netback/interface.c +0 -0

netback/interface.c

drivers/net/xen-netback/interface.c void xenvif_disconnect(struct xenvif *vif) · line 485

Quoted 13 lines

index 625c6f4..65e78f9 100644

drivers/net/xen-netback/interface.c +6 -0
@@ -477,14 +477,17 @@ void xenvif_disconnect(struct xenvif *vif)
 	if (vif->task)
 		kthread_stop(vif->task);
 
+	xenvif_unmap_frontend_rings(vif);
+
+	if (need_module_put)
+		module_put(THIS_MODULE);
+}
+

The original thought for this module_get/put thing is that admin can just disconnect all frontends then replace netback module. With this patch that doesn't work anymore. We leak memory when all frontends are disconnected and admin runs "modprobe -r xen-netback". That happened to
work before ecause everytime a frontend is disconnected the netdev structure
is already freed.

Quoted 7 lines

The ability to unload netback is still quite handy for developer / admin so I would like to keep it. My suggestion would be, move module_get/put to the point where netdev is allocated / freed. The impact of moving module_get/put is that we cannot just simply disconnect all frontends then replace netback, we need to actually shutdown / migrate all VMs before replacing netback.

Presumably you still had the problem of needing to manually re-run the hotplug scripts if not using udev though. I'll move the module_get/put as you suggest.

Paul