Skip to contentSkip to conversation

netpoll: fix race between skb_queue_len and skb_dequeue

PATCH In review DDD 2 participants 3 messages Sep 8, 2009 – Sep 9, 2009

Subsystems (2)
lore
Share & tools
lore

DDD

author
#0 lore raw

PATCH 1/1 netpoll: fix race between skb_queue_len and skb_dequeue

This race will break the messages order.

Sequence of events in the problem case:

Assume there is one skb "A" in skb_queue and the next action of netpoll_send_skb() is: sending out "B" skb.
The right order of messages should be: send A first, then send B.

But as following orders, it will send B first, then send A.

CPU0 CPU1
queue_process() netpoll_send_skb(B)
A = skb_dequeue()
/****
A was removed from skb_queue,
so skb_queue_len = 0
****/

                                   Check skb_queue_len == 0 Yes
                                   sendout (B)

sendout (A)

The solution is to introduce the txq_skbuff_len in netpoll_info struct. The txq_skbuff_len will keep the skb queue lenght utill the skb was sent out, so that this race condition will gone.

---
 include/linux/netpoll.h |    1 +
 net/core/netpoll.c      |    8 ++++++--
 2 files changed, 7 insertions(+), 2 deletions(-)
include/linux/netpoll.h +1 -0
index 2524267..3757141 100644
@@ -27,6 +27,7 @@ struct netpoll_info {
 	atomic_t refcnt;
 	int rx_flags;
 	spinlock_t rx_lock;
+	__u32 txq_skbuff_len;
 	struct netpoll *rx_np; /* netpoll that registered an rx_hook */
 	struct sk_buff_head arp_tx; /* list of arp requests to reply to */
 	struct sk_buff_head txq;
net/core/netpoll.c +6 -2
index 1b76eb1..6d22426 100644
@@ -78,11 +78,14 @@ static void queue_process(struct work_struct *work)
 			__netif_tx_unlock(txq);
 			local_irq_restore(flags);
 
+			npinfo->txq_skbuff_len = skb_queue_len(&npinfo->txq);
 			schedule_delayed_work(&npinfo->tx_work, HZ/10);
 			return;
 		}
 		__netif_tx_unlock(txq);
 		local_irq_restore(flags);
+
+		npinfo->txq_skbuff_len = skb_queue_len(&npinfo->txq);
 	}
 }
 
@@ -291,7 +294,7 @@ static void netpoll_send_skb(struct netpoll *np, struct sk_buff *skb)
 	}
 
 	/* don't get messages out of order, and no recursion */
-	if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
+	if (npinfo->txq_skbuff_len == 0 && !netpoll_owner_active(dev)) {
 		struct netdev_queue *txq;
 		unsigned long flags;
 
@@ -329,7 +332,8 @@ static void netpoll_send_skb(struct netpoll *np, struct sk_buff *skb)
 
 	if (status != NETDEV_TX_OK) {
 		skb_queue_tail(&npinfo->txq, skb);
-		schedule_delayed_work(&npinfo->tx_work,0);
+		npinfo->txq_skbuff_len = skb_queue_len(&npinfo->txq);
+		schedule_delayed_work(&npinfo->tx_work, 0);
 	}
 }
 
Signature · 1 line
1.6.0.4
#1 lore raw
Quoted 9 lines · DDD

This race will break the messages order.

Sequence of events in the problem case:

Assume there is one skb "A" in skb_queue and the next action of netpoll_send_skb() is: sending out "B" skb.
The right order of messages should be: send A first, then send B.

But as following orders, it will send B first, then send A.

I would say no, the order of messages A and B queued on different CPUs is undefined. The only issue is that we can queue a message A on CPU0, then causally trigger a message on CPU1 B that arrives first. But bear in mind that the message A >>may never arrive<< because the message is about a lockup that kills processing of delayed work.

Generally speaking, queueing should be a last ditch effort. We should instead aim to deliver all messages immediately, even if they might be out of order. Because out of order is better than not arriving at all.

Signature · 1 line
http://selenic.com : development and support for Mercurial and Linux
author
#2 lore raw
Quoted 17 lines · Matt Mackall
Quoted 9 lines · DDD

This race will break the messages order.

Sequence of events in the problem case:

Assume there is one skb "A" in skb_queue and the next action of netpoll_send_skb() is: sending out "B" skb.
The right order of messages should be: send A first, then send B.

But as following orders, it will send B first, then send A.

I would say no, the order of messages A and B queued on different CPUs is undefined. The only issue is that we can queue a message A on CPU0, then causally trigger a message on CPU1 B that arrives first. But bear in mind that the message A >>may never arrive<< because the message is about a lockup that kills processing of delayed work.

Quoted 3 lines

Generally speaking, queueing should be a last ditch effort. We should instead aim to deliver all messages immediately, even if they might be out of order. Because out of order is better than not arriving at all.

Ah, yes, out of order is better than not arriving at all. :-)

Especially it is based on UDP, so I don't think the order of messages is important. :-)

I take back this patch.

Thank you very much,
Dongdong