concurrency control issues in netback
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/1 concurrency control issues in netback
I have been looking at the netback code (netback.c) and I think there may be a bug in the way concurrency is managed here. Specifically, the function that cleans up completed requests (net_tx_action_dealloc()) is executed only from within a tasklet and hence does not acquire any locks to manipulate global state that could potentially be modified from other cpus. The code seems to assume writes are ordered without explicitly forcing the ordering. The problematic global state is the index that keeps track of the last valid index to be dealloced (dealloc_prod). This
could potentially be manipulated outside of the tasklet.
The dealloc_ring[] is populated and the dealloc_prod variable is advanced in the function netif_idx_release() under the protection of a spin lock. However, if the two memory writes (assigning at the current dealloc_prod index and advancing the dealloc_prod index) in netif_idx_release() are seen out of order on the cpu executing net_tx_action_dealloc(), we can get into a situation where we could attempt to operate on an index that has invalid data. The fix would be fairly simple. We could snapshot the current value of dealloc_prod in net_tx_action_dealloc() under the protection of the same spin lock that is used to advance the dealloc_prod index in netif_idx_release. This would give a consistent snapshot of the dealloc_prod index and deal with the problem described above. The attached patch does just that.
Signed- off- by K. Y. Srinivasan <[email protected]
drivers/xen/netback/netback.c +10 -1
@@ -100,6 +100,7 @@static unsigned long mfn_list[MAX_MFN_ALLOC ];static unsigned int alloc_index =0 ;static spinlock_t mfn_lock =SPIN_LOCK_UNLOCKED ; +static spinlock_t _lock =SPIN_LOCK_UNLOCKED ;static unsigned long alloc_mfn (void ) {
@@ -448,9 +449,18 @@PEND_RING_IDX dc, dp;netif_t *netif;int ret; +unsigned long flags; dc = dealloc_cons; +/* +* dealloc_prod is updated under the protection of the spin lock +* in the function netif_idx_release(). We need to see the updates to +* dealloc_prod in the order it is updated in netif_idx_release. To +* force this, snapshot dealloc_prod under the protection of the lock. +*/ +spin_lock_irqsave (&_lock, flags); dp = dealloc_prod; +spin_unlock_irqrestore (&_lock, flags);/* * Free up any grants we have finished using
@@ -685,7 +695,6 @@static void netif_idx_release (u16 pending_idx) { -static spinlock_t _lock =SPIN_LOCK_UNLOCKED ;unsigned long flags;spin_lock_irqsave (&_lock, flags);
drivers/xen/netback/netback.c +11 -1
@@ -100,6 +100,7 @@static unsigned long mfn_list[MAX_MFN_ALLOC ];static unsigned int alloc_index =0 ;static DEFINE_SPINLOCK (mfn_lock); +static DEFINE_SPINLOCK (_lock);static unsigned long alloc_mfn (void ) {
@@ -454,9 +455,19 @@PEND_RING_IDX dc, dp;netif_t *netif;int ret; +unsigned long flags; dc = dealloc_cons; + +/* +* dealloc_prod is updated under the protection of the spin lock +* in the function netif_idx_release(). We need to see the updates to +* dealloc_prod in the order it is updated in netif_idx_release. To +* force this, snapshot dealloc_prod under the protection of the lock. +*/ +spin_lock_irqsave (&_lock, flags); dp = dealloc_prod; +spin_unlock_irqrestore (&_lock, flags);/* * Free up any grants we have finished using
@@ -691,7 +702,6 @@static void netif_idx_release (u16 pending_idx) { -static DEFINE_SPINLOCK (_lock);unsigned long flags;spin_lock_irqsave (&_lock, flags);
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 9 lines · Ky Srinivasan
I have been looking at the netback code (
netback.c) and I think there may be a bug in the way concurrency is managed here. Specifically, the function that cleans up completed requests (net_tx_action_dealloc()) is executed only from within a tasklet and hence does not acquire any locks to manipulate global state that could potentially be modified from other cpus. The code seems to assume writes are ordered without explicitly forcing the ordering. The problematic global state is the index that keeps track of the last valid index to be dealloced (dealloc_prod). This
could potentially be manipulated outside of the tasklet.
That's a good catch! Does your patch fix the network problems that you guys have been seeing?
The patch needs cleaning up: at least to rename the lock, but actually just explicitly forcing the ordering we need is probably better. I'll sort that out and check in a suitable fix.
Thanks!
Keir
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
On Mon, Jun 5, 2006 at 9:48 AM, in message
<[email protected]>, Keir Fraser
Quoted 3 lines · <[email protected]> wrote:
Quoted 1 line · Ky Srinivasan
I have been looking at the netback code (
netback.c) and I think
there
Quoted 1 line
Quoted 1 line
may be a bug in the way concurrency is managed here. Specifically,
the
Quoted 1 line
Quoted 1 line
function that cleans up completed requests (
net_tx_action_dealloc())
is
Quoted 5 lines
Quoted 5 lines
executed only from within a tasklet and hence does not acquire any locks
to manipulate global state that could potentially be modified from other
cpus. The code seems to assume writes are ordered without
explicitly
Quoted 1 line
Quoted 1 line
forcing the ordering. The problematic global state is the index
that
Quoted 6 lines
Quoted 4 lines
keeps track of the last valid index to be dealloced (
dealloc_prod).This
could potentially be manipulated outside of the tasklet.That's a good catch! Does your patch fix the network problems that
you
Quoted 1 line
guys have been seeing?
I am hoping it will fix the problem! The machines needed to reproduce the problem are in our Provo labs. We will get the tests going this morning. We will keep you posted.
Quoted 2 lines
The patch needs cleaning up: at least to rename the lock, but
actually
Quoted 1 line
just explicitly forcing the ordering we need is probably better. I'll
Agreed.
Quoted 5 lines
sort that out and check in a suitable fix.
Thanks!
Keir
Regards,
K. Y
_______________________________________________
Quoted 3 lines
Xen- devel mailing list
Xen- [email protected]
http://lists.xensource.com/xen- devel