Skip to contentSkip to conversation

concurrency control issues in netback

PATCH In review Ky Srinivasan 2 participants 3 messages Jun 5, 2006 2 messages not in the archive Xen hypervisor interface

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

PATCH 1/1 concurrency control issues in netback

I have been looking at the netback code (netback.c) and I think there may be a bug in the way concurrency is managed here. Specifically, the function that cleans up completed requests (net_tx_action_dealloc()) is executed only from within a tasklet and hence does not acquire any locks to manipulate global state that could potentially be modified from other cpus. The code seems to assume writes are ordered without explicitly forcing the ordering. The problematic global state is the index that keeps track of the last valid index to be dealloced (dealloc_prod). This
could potentially be manipulated outside of the tasklet.

The dealloc_ring[] is populated and the dealloc_prod variable is advanced in the function netif_idx_release() under the protection of a spin lock. However, if the two memory writes (assigning at the current dealloc_prod index and advancing the dealloc_prod index) in netif_idx_release() are seen out of order on the cpu executing net_tx_action_dealloc(), we can get into a situation where we could attempt to operate on an index that has invalid data. The fix would be fairly simple. We could snapshot the current value of dealloc_prod in net_tx_action_dealloc() under the protection of the same spin lock that is used to advance the dealloc_prod index in netif_idx_release. This would give a consistent snapshot of the dealloc_prod index and deal with the problem described above. The attached patch does just that.

Signed- off- by K. Y. Srinivasan <[email protected]

drivers/xen/netback/netback.c +10 -1
@@ -100,6 +100,7 @@
 static unsigned long mfn_list[MAX_MFN_ALLOC];
 static unsigned int alloc_index = 0;
 static spinlock_t mfn_lock = SPIN_LOCK_UNLOCKED;
+static spinlock_t _lock = SPIN_LOCK_UNLOCKED;
 
 static unsigned long alloc_mfn(void)
 {
@@ -448,9 +449,18 @@
 	PEND_RING_IDX dc, dp;
 	netif_t *netif;
 	int ret;
+	unsigned long flags;
 
 	dc = dealloc_cons;
+	/*
+	 * dealloc_prod is updated under the protection of the spin lock
+	 * in the function netif_idx_release(). We need to see the updates to
+	 * dealloc_prod in the order it is updated in netif_idx_release. To 
+	 * force this, snapshot dealloc_prod under the protection of the lock.
+	 */
+	spin_lock_irqsave(&_lock, flags);
 	dp = dealloc_prod;
+	spin_unlock_irqrestore(&_lock, flags);
 
 	/*
 	 * Free up any grants we have finished using
@@ -685,7 +695,6 @@
 
 static void netif_idx_release(u16 pending_idx)
 {
-	static spinlock_t _lock = SPIN_LOCK_UNLOCKED;
 	unsigned long flags;
 
 	spin_lock_irqsave(&_lock, flags);
drivers/xen/netback/netback.c +11 -1
@@ -100,6 +100,7 @@
 static unsigned long mfn_list[MAX_MFN_ALLOC];
 static unsigned int alloc_index = 0;
 static DEFINE_SPINLOCK(mfn_lock);
+static DEFINE_SPINLOCK(_lock);
 
 static unsigned long alloc_mfn(void)
 {
@@ -454,9 +455,19 @@
 	PEND_RING_IDX dc, dp;
 	netif_t *netif;
 	int ret;
+	unsigned long flags;
 
 	dc = dealloc_cons;
+
+	/*
+	 * dealloc_prod is updated under the protection of the spin lock
+	 * in the function netif_idx_release(). We need to see the updates to
+	 * dealloc_prod in the order it is updated in netif_idx_release. To 
+	 * force this, snapshot dealloc_prod under the protection of the lock.
+	 */
+	spin_lock_irqsave(&_lock, flags);
 	dp = dealloc_prod;
+	spin_unlock_irqrestore(&_lock, flags);
 
 	/*
 	 * Free up any grants we have finished using
@@ -691,7 +702,6 @@
 
 static void netif_idx_release(u16 pending_idx)
 {
-	static DEFINE_SPINLOCK(_lock);
 	unsigned long flags;
 
 	spin_lock_irqsave(&_lock, flags);

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#3 lore raw
Quoted 9 lines · Ky Srinivasan

I have been looking at the netback code (netback.c) and I think there may be a bug in the way concurrency is managed here. Specifically, the function that cleans up completed requests (net_tx_action_dealloc()) is executed only from within a tasklet and hence does not acquire any locks to manipulate global state that could potentially be modified from other cpus. The code seems to assume writes are ordered without explicitly forcing the ordering. The problematic global state is the index that keeps track of the last valid index to be dealloced (dealloc_prod). This
could potentially be manipulated outside of the tasklet.

That's a good catch! Does your patch fix the network problems that you guys have been seeing?

The patch needs cleaning up: at least to rename the lock, but actually just explicitly forcing the ordering we need is probably better. I'll sort that out and check in a suitable fix.

Thanks!
Keir

#4 lore raw
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

On Mon, Jun 5, 2006 at 9:48 AM, in message

<[email protected]>, Keir Fraser

Quoted 3 lines · <[email protected]> wrote:
Quoted 1 line · Ky Srinivasan

I have been looking at the netback code (netback.c) and I think

there

Quoted 1 line
Quoted 1 line

may be a bug in the way concurrency is managed here. Specifically,

the

Quoted 1 line
Quoted 1 line

function that cleans up completed requests (net_tx_action_dealloc())

is

Quoted 5 lines
Quoted 5 lines

executed only from within a tasklet and hence does not acquire any locks
to manipulate global state that could potentially be modified from other
cpus. The code seems to assume writes are ordered without

explicitly

Quoted 1 line
Quoted 1 line

forcing the ordering. The problematic global state is the index

that

Quoted 6 lines
Quoted 4 lines

keeps track of the last valid index to be dealloced (dealloc_prod).

This
could potentially be manipulated outside of the tasklet.

That's a good catch! Does your patch fix the network problems that

you

Quoted 1 line

guys have been seeing?

I am hoping it will fix the problem! The machines needed to reproduce the problem are in our Provo labs. We will get the tests going this morning. We will keep you posted.

Quoted 2 lines

The patch needs cleaning up: at least to rename the lock, but

actually

Quoted 1 line

just explicitly forcing the ordering we need is probably better. I'll

Agreed.

Quoted 5 lines

sort that out and check in a suitable fix.

Thanks!
Keir

Regards,

K. Y
_______________________________________________

Quoted 3 lines

Xen- devel mailing list
Xen- [email protected]
http://lists.xensource.com/xen- devel