From: "Ky Srinivasan" Subject: [PATCH] netback: netif cleanup Date: Sun, 06 Dec 2009 10:18:23 -0700 Message-ID: <4B1B84EF020000300007B120@sinclair.provo.novell.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=__Part90BACD4F.0__=" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org --=__Part90BACD4F.0__= Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Content-Disposition: inline We have seen cases where, when a guest is shutdown during heavy network = traffic, the backend vif hangs around for considerable amounts of time = holding up the xenbus thread for the duration of the cleanup. We have seen = cases (HP, Novell) where it has taken hours to cleanup the interface and = for that duration we cannot launch new guests. We have traced this problem = to the fact that the front-end can shutdown with transmit resources still = in transit on the back-end side. On the back-end side, the moment the = carrier is turned off on the interface, the interface is not eligible for = further scheduling in the backend. Since turning off the carrier happens = completely asynchronously with the rest of the packet processing in the = back-end (via tasklets today), we can have situations where the vif cannot = be cleaned up for a significant amount of time. I am attaching a patch for addressing this issue. I have tried to fix the = problem with minimal changes to the current code. The main idea is to = cleanup the guest transmit resources quickly and aggressively. With this = patch, the cleanup time has been dramatically reduced. Even in this patch, = the xenbus thread is held up for the duration of the cleanup - however we = attempt to cleanup aggressively. This is in comparison to the current code = where the xenbus thread just waits for the refcnt on the interface to = drop to zero. An obvious improvement would be to not tie up the xenbus = thread for the duration of cleaning up the interface. If there is = interest, I can submit a patch for doing this. This patch has been tested = on 2.6.16 (SLES10 sp2, sp3 kernels) as well as 2.6.27 (sles11) kernel. = This patch is however against the 2.6.18-xen.hg sources. Signed-off-by: K. Y. Srinivasan Regards, K. Y --=__Part90BACD4F.0__= Content-Type: text/plain; name="netback-vif-cleanup.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="netback-vif-cleanup.patch" From: K. Y. Srinivasan =0ASubject: Fix issues in = netif cleanup. =0A=0A=0ADecember 6, 2009: Aggressively cleanup transmit = resources held in netback=0Awhen the guest has exited. =0A=0ASigned-off-by:= K. Y. Srinivasan =0A=0A=0AIndex: linux-2.6.18-xen.= hg/drivers/xen/netback/common.h=0A=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=0A--- linux-2.6.18-xen.hg.orig/drivers/xen/netback/common.h = 2009-12-06 11:34:54.000000000 -0500=0A+++ linux-2.6.18-xen.hg/drivers/xen/n= etback/common.h 2009-12-06 11:36:52.000000000 -0500=0A@@ -105,7 +105,6 @@ = typedef struct netif_st {=0A =0A unsigned int carrier;=0A =0A- = wait_queue_head_t waiting_to_free;=0A } netif_t;=0A =0A /*=0A@@ -187,13 = +186,10 @@ int netif_map(netif_t *netif, unsigned l=0A unsigned = long rx_ring_ref, unsigned int evtchn);=0A =0A #define netif_get(_b) = (atomic_inc(&(_b)->refcnt))=0A-#define netif_put(_b) = \=0A- do { = \=0A- if ( atomic_dec_and_test(&(_b)->refcnt) ) \=0A- = wake_up(&(_b)->waiting_to_free); \=0A- } while (0)=0A+#def= ine netif_put(_b) (atomic_dec(&(_b)->refcnt))=0A =0A void netif_xenbus_init= (void);=0A+void netif_cleanup(netif_t *netif);=0A =0A #define netif_schedul= able(netif) \=0A (netif_running((netif)->dev= ) && netback_carrier_ok(netif))=0AIndex: linux-2.6.18-xen.hg/drivers/xen/ne= tback/interface.c=0A=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A= --- linux-2.6.18-xen.hg.orig/drivers/xen/netback/interface.c 2009-12-06 = 11:34:54.000000000 -0500=0A+++ linux-2.6.18-xen.hg/drivers/xen/netback/inte= rface.c 2009-12-06 11:36:52.000000000 -0500=0A@@ -196,7 +196,6 @@ netif_t = *netif_alloc(struct device *pare=0A netif->domid =3D domid;=0A = netif->handle =3D handle;=0A atomic_set(&netif->refcnt, 1);=0A- = init_waitqueue_head(&netif->waiting_to_free);=0A netif->dev =3D = dev;=0A =0A netback_carrier_off(netif);=0A@@ -371,9 +370,8 @@ void = netif_disconnect(netif_t *netif)=0A rtnl_unlock();=0A = netif_put(netif);=0A }=0A-=0A+ netif_cleanup(netif);=0A = atomic_dec(&netif->refcnt);=0A- wait_event(netif->waiting_to_free, = atomic_read(&netif->refcnt) =3D=3D 0);=0A =0A del_timer_sync(&netif->cred= it_timeout);=0A del_timer_sync(&netif->tx_queue_timeout);=0AIndex: = linux-2.6.18-xen.hg/drivers/xen/netback/netback.c=0A=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A--- linux-2.6.18-xen.hg.orig/drivers/xen/n= etback/netback.c 2009-12-06 11:34:54.000000000 -0500=0A+++ = linux-2.6.18-xen.hg/drivers/xen/netback/netback.c 2009-12-06 = 11:36:52.000000000 -0500=0A@@ -934,9 +934,14 @@ inline static void = net_tx_action_dealloc=0A list_empty(&pending_inuse_head)= )=0A break;=0A =0A- /* Copy any entries that = have been pending for too long. */=0A+ /* Copy any entries that = have been pending for too long; or=0A+ * if the interface is = being shutdown.=0A+ */=0A list_for_each_entry_safe(in= use, n, &pending_inuse_head, list) {=0A- if = (time_after(inuse->alloc_time + HZ / 2, jiffies))=0A+ = pending_idx =3D inuse - pending_inuse;=0A+ netif =3D = pending_tx_info[pending_idx].netif;=0A+ if ((time_after(inu= se->alloc_time + HZ / 2, jiffies)) &&=0A+ = netback_carrier_ok(netif))=0A break;=0A =0A = pending_idx =3D inuse - pending_inuse;=0A@@ -983,6 +988,33 @@ = inline static void net_tx_action_dealloc=0A }=0A }=0A =0A+void = msleep(unsigned int msecs);=0A+void netif_cleanup(netif_t *netif)=0A+{=0A+c= leanup_again:=0A+ if (atomic_read(&netif->refcnt) =3D=3D 1)=0A+ = return;=0A+ if (!__on_net_schedule_list(netif)) {=0A+ /* = Still outstanding refrences and we are not on the=0A+ * netif = scheduler list. Since we have turned the carrier off,=0A+ * = we are considered ineligible for getting on the scheduler=0A+ * = list! Go on the scheduler list to cleanup.=0A+ */=0A+ = spin_lock_irq(&net_schedule_list_lock);=0A+ list_add_tail(&neti= f->list, &net_schedule_list);=0A+ netif_get(netif);=0A+ = spin_unlock_irq(&net_schedule_list_lock);=0A+ tasklet_schedule(&n= et_tx_tasklet);=0A+ }=0A+ /*=0A+ * We are still cleaning up the = netif in the context of the =0A+ * xenbus thread; it might be = better to decouple this.=0A+ */=0A+ msleep(1);=0A+ goto cleanup_again;= =0A+}=0A+=0A+=0A static void netbk_tx_err(netif_t *netif, netif_tx_request_= t *txp, RING_IDX end)=0A {=0A RING_IDX cons =3D netif->tx.req_cons;=0A@@ = -1233,6 +1265,7 @@ static void net_tx_action(unsigned long =0A unsigned = int data_len;=0A int ret, work_to_do;=0A =0A+ smp_rmb();=0A if = (dealloc_cons !=3D dealloc_prod)=0A net_tx_action_dealloc();=0A= =0A@@ -1247,6 +1280,12 @@ static void net_tx_action(unsigned long =0A =0A = RING_FINAL_CHECK_FOR_REQUESTS(&netif->tx, work_to_do);=0A = if (!work_to_do) {=0A+ /*=0A+ * = If netif is being shutdown; try to cleanup.=0A+ */=0A+ = if (!netback_carrier_ok(netif))=0A+ = net_tx_action_dealloc();=0A+=0A netif_put(netif);= =0A continue;=0A }=0A --=__Part90BACD4F.0__= Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --=__Part90BACD4F.0__=--