Skip to contentSkip to conversation

netback: netif cleanup

PATCH In review Ky Srinivasan 2 participants 5 messages Dec 6, 2009 – Dec 7, 2009 Xen hypervisor interface

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 netback: netif cleanup

We have seen cases where, when a guest is shutdown during heavy network traffic, the backend vif hangs around for considerable amounts of time holding up the xenbus thread for the duration of the cleanup. We have seen cases (HP, Novell) where it has taken hours to cleanup the interface and for that duration we cannot launch new guests. We have traced this problem to the fact that the front-end can shutdown with transmit resources still in transit on the back-end side. On the back-end side, the moment the carrier is turned off on the interface, the interface is not eligible for further scheduling in the backend. Since turning off the carrier happens completely asynchronously with the rest of the packet processing in the back-end (via tasklets today), we can have situations where the vif cannot be cleaned up for a significant amount of time.

I am attaching a patch for addressing this issue. I have tried to fix the problem with minimal changes to the current code. The main idea is to cleanup the guest transmit resources quickly and aggressively. With this patch, the cleanup time has been dramatically reduced. Even in this patch, the xenbus thread is held up for the duration of the cleanup - however we attempt to cleanup aggressively. This is in comparison to the current code where the xenbus thread just waits for the refcnt on the interface to drop to zero. An obvious improvement would be to not tie up the xenbus thread for the duration of cleaning up the interface. If there is interest, I can submit a patch for doing this. This patch has been tested on 2.6.16 (SLES10 sp2, sp3 kernels) as well as 2.6.27 (sles11) kernel. This patch is however against the 2.6.18-xen.hg sources.

Regards,

K. Y

From: K. Y. Srinivasan <[email protected]>
Subject: Fix issues in netif cleanup.

December 6, 2009: Aggressively cleanup transmit resources held in netback when the guest has exited.

drivers/xen/netback/common.h +2 -6
Index: linux-2.6.18-xen.hg/drivers/xen/netback/common.h
===================================================================
@@ -105,7 +105,6 @@ typedef struct netif_st {
 
 	unsigned int carrier;
 
-	wait_queue_head_t waiting_to_free;
 } netif_t;
 
 /*
@@ -187,13 +186,10 @@ int netif_map(netif_t *netif, unsigned l
 	      unsigned long rx_ring_ref, unsigned int evtchn);
 
 #define netif_get(_b) (atomic_inc(&(_b)->refcnt))
-#define netif_put(_b)						\
-	do {							\
-		if ( atomic_dec_and_test(&(_b)->refcnt) )	\
-			wake_up(&(_b)->waiting_to_free);	\
-	} while (0)
+#define netif_put(_b) (atomic_dec(&(_b)->refcnt))
 
 void netif_xenbus_init(void);
+void netif_cleanup(netif_t *netif);
 
 #define netif_schedulable(netif)				\
 	(netif_running((netif)->dev) && netback_carrier_ok(netif))
drivers/xen/netback/interface.c +1 -3
Index: linux-2.6.18-xen.hg/drivers/xen/netback/interface.c
===================================================================
@@ -196,7 +196,6 @@ netif_t *netif_alloc(struct device *pare
 	netif->domid  = domid;
 	netif->handle = handle;
 	atomic_set(&netif->refcnt, 1);
-	init_waitqueue_head(&netif->waiting_to_free);
 	netif->dev = dev;
 
 	netback_carrier_off(netif);
@@ -371,9 +370,8 @@ void netif_disconnect(netif_t *netif)
 		rtnl_unlock();
 		netif_put(netif);
 	}
-
+	netif_cleanup(netif);
 	atomic_dec(&netif->refcnt);
-	wait_event(netif->waiting_to_free, atomic_read(&netif->refcnt) == 0);
 
 	del_timer_sync(&netif->credit_timeout);
 	del_timer_sync(&netif->tx_queue_timeout);
drivers/xen/netback/netback.c +41 -2

68 lines not on this page · raw message

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#1 lore raw

This looks like a hack. I think you'd get most of the benefit by simply always calling net_tx_action_dealloc() at the top of net_tx_action() (i.e., remove the dealloc_cons!=dealloc_prod conditional). I think the presence of that conditional is actually a bug, and removing it should get interface shutdowns to a reasonable time with no other changes.

-- Keir

Quoted 34 lines · Ky Srinivasan

We have seen cases where, when a guest is shutdown during heavy network traffic, the backend vif hangs around for considerable amounts of time holding up the xenbus thread for the duration of the cleanup. We have seen cases (HP, Novell) where it has taken hours to cleanup the interface and for that duration we cannot launch new guests. We have traced this problem to the fact that the front-end can shutdown with transmit resources still in transit on the back-end side. On the back-end side, the moment the carrier is turned off on the interface, the interface is not eligible for further scheduling in the backend. Since turning off the carrier happens completely asynchronously with the rest of the packet processing in the back-end (via tasklets today), we can have situations where the vif cannot be cleaned up for a significant amount of time.

I am attaching a patch for addressing this issue. I have tried to fix the problem with minimal changes to the current code. The main idea is to cleanup the guest transmit resources quickly and aggressively. With this patch, the cleanup time has been dramatically reduced. Even in this patch, the xenbus thread is held up for the duration of the cleanup - however we attempt to cleanup aggressively. This is in comparison to the current code where the xenbus thread just waits for the refcnt on the interface to drop to zero. An obvious improvement would be to not tie up the xenbus thread for the duration of cleaning up the interface. If there is interest, I can submit a patch for doing this. This patch has been tested on 2.6.16 (SLES10 sp2, sp3 kernels) as well as 2.6.27 (sles11) kernel. This patch is however against the 2.6.18-xen.hg sources.

Regards,

K. Y

#2 lore raw
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

On 12/6/2009 at 4:44 PM, in message <C741D648.3B7A%[email protected]>,

Quoted 5 lines · Keir Fraser

This looks like a hack. I think you'd get most of the benefit by simply always calling net_tx_action_dealloc() at the top of net_tx_action() (i.e., remove the dealloc_cons!=dealloc_prod conditional). I think the presence of that conditional is actually a bug, and removing it should get interface shutdowns to a reasonable time with no other changes.

That was my first impression as well and was the first patch we tested. It does not work. Take the case where we have only one netif that netback is handling that gets into this weird state, then we would never call net_tx_action_dealloc() if the carrier is turned off on the netif that is being cleaned up. This patch, fixes that problem by forcefully scheduling the netif whose carrier has been turned off. In the interest of minimizing the code changes, I went about making these changes one at a time, and the combination of what I have in the patch appears to reduce the cleanup time the most. Even with these hacks, I am not particularly happy about the maximum time we have seen the cleanup take, and for this duration the physical host is unusable for launching new guests. So, as I noted in my earlier email, we may still want to not tie up the xenbus thread for the duration of this cleanup.

Regards,

K. Y

Quoted 50 lines

-- Keir

Quoted 2 lines · Ky Srinivasan

We have seen cases where, when a guest is shutdown during heavy network traffic, the backend vif hangs around for considerable amounts of time

holding

Quoted 1 line

up the xenbus thread for the duration of the cleanup. We have seen cases

(HP,

Quoted 2 lines

Novell) where it has taken hours to cleanup the interface and for that duration we cannot launch new guests. We have traced this problem to the

fact

Quoted 3 lines

that the front-end can shutdown with transmit resources still in transit on the back-end side. On the back-end side, the moment the carrier is turned off on the interface, the interface is not eligible for further scheduling in

the

Quoted 1 line

backend. Since turning off the carrier happens completely asynchronously

with

Quoted 1 line

the rest of the packet processing in the back-end (via tasklets today), we

can

Quoted 1 line

have situations where the vif cannot be cleaned up for a significant amount

of

Quoted 4 lines

time.

I am attaching a patch for addressing this issue. I have tried to fix the problem with minimal changes to the current code. The main idea is to

cleanup

Quoted 5 lines

the guest transmit resources quickly and aggressively. With this patch, the cleanup time has been dramatically reduced. Even in this patch, the xenbus thread is held up for the duration of the cleanup - however we attempt to cleanup aggressively. This is in comparison to the current code where the xenbus thread just waits for the refcnt on the interface to drop to zero.

An

Quoted 1 line

obvious improvement would be to not tie up the xenbus thread for the

duration

Quoted 2 lines

of cleaning up the interface. If there is interest, I can submit a patch for doing this. This patch has been tested on 2.6.16 (SLES10 sp2, sp3 kernels)

as

Quoted 11 lines

well as 2.6.27 (sles11) kernel. This patch is however against the 2.6.18-xen.hg sources.

Regards,

K. Y

#3 lore raw
Quoted 11 lines · Ky Srinivasan
Quoted 5 lines

This looks like a hack. I think you'd get most of the benefit by simply always calling net_tx_action_dealloc() at the top of net_tx_action() (i.e., remove the dealloc_cons!=dealloc_prod conditional). I think the presence of that conditional is actually a bug, and removing it should get interface shutdowns to a reasonable time with no other changes.

That was my first impression as well and was the first patch we tested. It does not work. Take the case where we have only one netif that netback is handling that gets into this weird state, then we would never call net_tx_action_dealloc() if the carrier is turned off on the netif that is being cleaned up. This patch, fixes that problem by forcefully scheduling the netif whose carrier has been turned off.

I think that this is simply another bug. There is an early 'return' in net_tx_action() which should actually goto the end of the function which checks the pending list and conditionally calls mod_timer(). Otherwise the pending list can get partially flushed but the timer doesn't get re-mod()ed.

Quoted 7 lines

In the interest of minimizing the
code changes, I went about making these changes one at a time, and the combination of what I have in the patch appears to reduce the cleanup time the most. Even with these hacks, I am not particularly happy about the maximum time we have seen the cleanup take, and for this duration the physical host is unusable for launching new guests. So, as I noted in my earlier email, we may still want to not tie up the xenbus thread for the duration of this cleanup.

Perhaps we could decouple the whole thing from xenbus thread. In any case, the above bug fixes are to me preferable to the original patch, and will get the orders-of-magnitude low-hanging fruit (delay xenbus thread for maybe a second, rather than hours!).

-- Keir

#4 lore raw
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

On 12/7/2009 at 3:04 AM, in message <C7426771.3BA0%[email protected]>,

Quoted 12 lines · Keir Fraser
Quoted 10 lines · Ky Srinivasan
Quoted 5 lines

This looks like a hack. I think you'd get most of the benefit by simply always calling net_tx_action_dealloc() at the top of net_tx_action() (i.e., remove the dealloc_cons!=dealloc_prod conditional). I think the presence of that conditional is actually a bug, and removing it should get interface shutdowns to a reasonable time with no other changes.

That was my first impression as well and was the first patch we tested. It does not work. Take the case where we have only one netif that netback is handling that gets into this weird state, then we would never call net_tx_action_dealloc() if the carrier is turned off on the netif that is being cleaned up. This patch, fixes that problem by forcefully scheduling

Quoted 8 lines

the

Quoted 1 line

netif whose carrier has been turned off.

I think that this is simply another bug. There is an early 'return' in net_tx_action() which should actually goto the end of the function which checks the pending list and conditionally calls mod_timer(). Otherwise the pending list can get partially flushed but the timer doesn't get re-mod()ed.

I suspect you are referring to the conditional early return ( if (mop==tx_map_ops)) . Anna Fischer of HP has a test environment where we can see this problem most easily. I will give her a netback with these changes and report back on the cleanup times.

Thanks,

K. Y

Quoted 17 lines
Quoted 3 lines

In the interest of minimizing the
code changes, I went about making these changes one at a time, and the combination of what I have in the patch appears to reduce the cleanup time

the

Quoted 2 lines

most. Even with these hacks, I am not particularly happy about the maximum time we have seen the cleanup take, and for this duration the physical host

is

Quoted 1 line

unusable for launching new guests. So, as I noted in my earlier email, we

may

Quoted 1 line

still want to not tie up the xenbus thread for the duration of this cleanup.

Perhaps we could decouple the whole thing from xenbus thread. In any case, the above bug fixes are to me preferable to the original patch, and will get the orders-of-magnitude low-hanging fruit (delay xenbus thread for maybe a second, rather than hours!).

-- Keir