netback: netif cleanup
PATCH 1/1 netback: netif cleanup
We have seen cases where, when a guest is shutdown during heavy network traffic, the backend vif hangs around for considerable amounts of time holding up the xenbus thread for the duration of the cleanup. We have seen cases (HP, Novell) where it has taken hours to cleanup the interface and for that duration we cannot launch new guests. We have traced this problem to the fact that the front-end can shutdown with transmit resources still in transit on the back-end side. On the back-end side, the moment the carrier is turned off on the interface, the interface is not eligible for further scheduling in the backend. Since turning off the carrier happens completely asynchronously with the rest of the packet processing in the back-end (via tasklets today), we can have situations where the vif cannot be cleaned up for a significant amount of time.
I am attaching a patch for addressing this issue. I have tried to fix the problem with minimal changes to the current code. The main idea is to cleanup the guest transmit resources quickly and aggressively. With this patch, the cleanup time has been dramatically reduced. Even in this patch, the xenbus thread is held up for the duration of the cleanup - however we attempt to cleanup aggressively. This is in comparison to the current code where the xenbus thread just waits for the refcnt on the interface to drop to zero. An obvious improvement would be to not tie up the xenbus thread for the duration of cleaning up the interface. If there is interest, I can submit a patch for doing this. This patch has been tested on 2.6.16 (SLES10 sp2, sp3 kernels) as well as 2.6.27 (sles11) kernel. This patch is however against the 2.6.18-xen.hg sources.
Regards,
K. Y
From: K. Y. Srinivasan <[email protected]>
Subject: Fix issues in netif cleanup.
December 6, 2009: Aggressively cleanup transmit resources held in netback when the guest has exited.
drivers/xen/netback/common.h +2 -6
@@ -105,7 +105,6 @@ typedef struct netif_st {unsigned int carrier; -wait_queue_head_t waiting_to_free; } netif_t;/*
@@ -187,13 +186,10 @@ int netif_map(netif_t *netif, unsigned l unsigned long rx_ring_ref, unsigned int evtchn);#define netif_get (_b) (atomic_inc(&(_b)->refcnt)) -#define netif_put (_b) \ - do { \ - if ( atomic_dec_and_test(&(_b)->refcnt) ) \ - wake_up(&(_b)->waiting_to_free); \ - } while (0) +#define netif_put (_b) (atomic_dec(&(_b)->refcnt))void netif_xenbus_init (void ); +void netif_cleanup (netif_t *netif);#define netif_schedulable (netif) \ (netif_running((netif)->dev) && netback_carrier_ok(netif))
drivers/xen/netback/interface.c +1 -3
@@ -196,7 +196,6 @@ netif_t *netif_alloc(struct device *pare netif->domid = domid; netif->handle = handle;atomic_set (&netif->refcnt,1 ); -init_waitqueue_head (&netif->waiting_to_free); netif->dev = dev;netback_carrier_off (netif);
@@ -371,9 +370,8 @@ void netif_disconnect(netif_t *netif)rtnl_unlock ();netif_put (netif); } - +netif_cleanup (netif);atomic_dec (&netif->refcnt); -wait_event (netif->waiting_to_free,atomic_read (&netif->refcnt) ==0 );del_timer_sync (&netif->credit_timeout);del_timer_sync (&netif->tx_queue_timeout);
drivers/xen/netback/netback.c +41 -2
68 lines not on this page · raw message
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
This looks like a hack. I think you'd get most of the benefit by simply always calling net_tx_action_dealloc() at the top of net_tx_action() (i.e., remove the dealloc_cons!=dealloc_prod conditional). I think the presence of that conditional is actually a bug, and removing it should get interface shutdowns to a reasonable time with no other changes.
-- Keir
Quoted 34 lines · Ky Srinivasan
We have seen cases where, when a guest is shutdown during heavy network traffic, the backend vif hangs around for considerable amounts of time holding up the xenbus thread for the duration of the cleanup. We have seen cases (HP, Novell) where it has taken hours to cleanup the interface and for that duration we cannot launch new guests. We have traced this problem to the fact that the front-end can shutdown with transmit resources still in transit on the back-end side. On the back-end side, the moment the carrier is turned off on the interface, the interface is not eligible for further scheduling in the backend. Since turning off the carrier happens completely asynchronously with the rest of the packet processing in the back-end (via tasklets today), we can have situations where the vif cannot be cleaned up for a significant amount of time.
I am attaching a patch for addressing this issue. I have tried to fix the problem with minimal changes to the current code. The main idea is to cleanup the guest transmit resources quickly and aggressively. With this patch, the cleanup time has been dramatically reduced. Even in this patch, the xenbus thread is held up for the duration of the cleanup - however we attempt to cleanup aggressively. This is in comparison to the current code where the xenbus thread just waits for the refcnt on the interface to drop to zero. An obvious improvement would be to not tie up the xenbus thread for the duration of cleaning up the interface. If there is interest, I can submit a patch for doing this. This patch has been tested on 2.6.16 (SLES10 sp2, sp3 kernels) as well as 2.6.27 (sles11) kernel. This patch is however against the 2.6.18-xen.hg sources.
Signed-off-by K. Y. Srinivasan <[email protected]>Regards,
K. Y
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
On 12/6/2009 at 4:44 PM, in message <C741D648.3B7A%[email protected]>,
Quoted 5 lines · Keir Fraser
This looks like a hack. I think you'd get most of the benefit by simply always calling
net_tx_action_dealloc()at the top ofnet_tx_action()(i.e., remove the dealloc_cons!=dealloc_prod conditional). I think the presence of that conditional is actually a bug, and removing it should get interface shutdowns to a reasonable time with no other changes.
That was my first impression as well and was the first patch we tested. It does not work. Take the case where we have only one netif that netback is handling that gets into this weird state, then we would never call net_tx_action_dealloc() if the carrier is turned off on the netif that is being cleaned up. This patch, fixes that problem by forcefully scheduling the netif whose carrier has been turned off. In the interest of minimizing the code changes, I went about making these changes one at a time, and the combination of what I have in the patch appears to reduce the cleanup time the most. Even with these hacks, I am not particularly happy about the maximum time we have seen the cleanup take, and for this duration the physical host is unusable for launching new guests. So, as I noted in my earlier email, we may still want to not tie up the xenbus thread for the duration of this cleanup.
Regards,
K. Y
Quoted 50 lines
-- Keir
Quoted 2 lines · Ky Srinivasan
We have seen cases where, when a guest is shutdown during heavy network traffic, the backend vif hangs around for considerable amounts of time
holding
Quoted 1 line
up the xenbus thread for the duration of the cleanup. We have seen cases
(HP,
Quoted 2 lines
Novell) where it has taken hours to cleanup the interface and for that duration we cannot launch new guests. We have traced this problem to the
fact
Quoted 3 lines
that the front-end can shutdown with transmit resources still in transit on the back-end side. On the back-end side, the moment the carrier is turned off on the interface, the interface is not eligible for further scheduling in
the
Quoted 1 line
backend. Since turning off the carrier happens completely asynchronously
with
Quoted 1 line
the rest of the packet processing in the back-end (via tasklets today), we
can
Quoted 1 line
have situations where the vif cannot be cleaned up for a significant amount
of
Quoted 4 lines
time.
I am attaching a patch for addressing this issue. I have tried to fix the problem with minimal changes to the current code. The main idea is to
cleanup
Quoted 5 lines
the guest transmit resources quickly and aggressively. With this patch, the cleanup time has been dramatically reduced. Even in this patch, the xenbus thread is held up for the duration of the cleanup - however we attempt to cleanup aggressively. This is in comparison to the current code where the xenbus thread just waits for the refcnt on the interface to drop to zero.
An
Quoted 1 line
obvious improvement would be to not tie up the xenbus thread for the
duration
Quoted 2 lines
of cleaning up the interface. If there is interest, I can submit a patch for doing this. This patch has been tested on 2.6.16 (SLES10 sp2, sp3 kernels)
as
Quoted 11 lines
well as 2.6.27 (sles11) kernel. This patch is however against the 2.6.18-xen.hg sources.
Signed-off-by K. Y. Srinivasan <[email protected]>Regards,
K. Y
Quoted 11 lines · Ky Srinivasan
Quoted 5 lines
This looks like a hack. I think you'd get most of the benefit by simply always calling
net_tx_action_dealloc()at the top ofnet_tx_action()(i.e., remove the dealloc_cons!=dealloc_prod conditional). I think the presence of that conditional is actually a bug, and removing it should get interface shutdowns to a reasonable time with no other changes.That was my first impression as well and was the first patch we tested. It does not work. Take the case where we have only one netif that netback is handling that gets into this weird state, then we would never call
net_tx_action_dealloc()if the carrier is turned off on the netif that is being cleaned up. This patch, fixes that problem by forcefully scheduling the netif whose carrier has been turned off.
I think that this is simply another bug. There is an early 'return' in net_tx_action() which should actually goto the end of the function which checks the pending list and conditionally calls mod_timer(). Otherwise the pending list can get partially flushed but the timer doesn't get re-mod()ed.
Quoted 7 lines
In the interest of minimizing the
code changes, I went about making these changes one at a time, and the combination of what I have in the patch appears to reduce the cleanup time the most. Even with these hacks, I am not particularly happy about the maximum time we have seen the cleanup take, and for this duration the physical host is unusable for launching new guests. So, as I noted in my earlier email, we may still want to not tie up the xenbus thread for the duration of this cleanup.
Perhaps we could decouple the whole thing from xenbus thread. In any case, the above bug fixes are to me preferable to the original patch, and will get the orders-of-magnitude low-hanging fruit (delay xenbus thread for maybe a second, rather than hours!).
-- Keir
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
On 12/7/2009 at 3:04 AM, in message <C7426771.3BA0%[email protected]>,
Quoted 12 lines · Keir Fraser
Quoted 10 lines · Ky Srinivasan
Quoted 5 lines
This looks like a hack. I think you'd get most of the benefit by simply always calling
net_tx_action_dealloc()at the top ofnet_tx_action()(i.e., remove the dealloc_cons!=dealloc_prod conditional). I think the presence of that conditional is actually a bug, and removing it should get interface shutdowns to a reasonable time with no other changes.That was my first impression as well and was the first patch we tested. It does not work. Take the case where we have only one netif that netback is handling that gets into this weird state, then we would never call
net_tx_action_dealloc()if the carrier is turned off on the netif that is being cleaned up. This patch, fixes that problem by forcefully scheduling
Quoted 8 lines
the
Quoted 1 line
netif whose carrier has been turned off.
I think that this is simply another bug. There is an early 'return' in
net_tx_action()which should actually goto the end of the function which checks the pending list and conditionally callsmod_timer(). Otherwise the pending list can get partially flushed but the timer doesn't get re-mod()ed.
I suspect you are referring to the conditional early return ( if (mop==tx_map_ops)) . Anna Fischer of HP has a test environment where we can see this problem most easily. I will give her a netback with these changes and report back on the cleanup times.
Thanks,
K. Y
Quoted 17 lines
Quoted 3 lines
In the interest of minimizing the
code changes, I went about making these changes one at a time, and the combination of what I have in the patch appears to reduce the cleanup timethe
Quoted 2 lines
most. Even with these hacks, I am not particularly happy about the maximum time we have seen the cleanup take, and for this duration the physical host
is
Quoted 1 line
unusable for launching new guests. So, as I noted in my earlier email, we
may
Quoted 1 line
still want to not tie up the xenbus thread for the duration of this cleanup.
Perhaps we could decouple the whole thing from xenbus thread. In any case, the above bug fixes are to me preferable to the original patch, and will get the orders-of-magnitude low-hanging fruit (delay xenbus thread for maybe a second, rather than hours!).
-- Keir