Skip to contentSkip to conversation

net: skb_orphan on dev_hard_start_xmit

PATCH In review Rusty Russell 5 participants 45 messages May 29, 2009 – Aug 19, 2009

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

PATCH 1/4 net: skb_orphan on dev_hard_start_xmit

Various drivers do skb_orphan() because they do not free transmitted skbs in a timely manner (causing apps which hit their socket limits to stall, socket close to hang, etc.).

DaveM points out that there are advantages to doing it generally (it's more likely to be on same CPU than after xmit), and I couldn't find any new starvation issues in simple benchmarking here.

This patch adds skb_orphan to the start of dev_hard_start_xmit(): it can be premature in the NETDEV_TX_BUSY case, but that's uncommon.

I removed the drivers' skb_orphan(), though it's harmless.

Cc Divy Le Ray <[email protected]>
Cc Roland Dreier <[email protected]>
Cc Pavel Emelianov <[email protected]>
Cc Dan Williams <[email protected]>
---
 drivers/net/cxgb3/sge.c            |   27 ---------------------------
 drivers/net/loopback.c             |    2 --
 drivers/net/mlx4/en_tx.c           |    4 ----
 drivers/net/niu.c                  |    3 +--
 drivers/net/veth.c                 |    2 --
 drivers/net/wireless/libertas/tx.c |    3 ---
 net/core/dev.c                     |   21 +++++----------------
 7 files changed, 6 insertions(+), 56 deletions(-)
drivers/net/cxgb3/sge.c +0 -27
@@ -1288,33 +1288,6 @@ int t3_eth_xmit(struct sk_buff *skb, str
 	dev->trans_start = jiffies;
 	spin_unlock(&q->lock);
 
-	/*
-	 * We do not use Tx completion interrupts to free DMAd Tx packets.
-	 * This is good for performamce but means that we rely on new Tx
-	 * packets arriving to run the destructors of completed packets,
-	 * which open up space in their sockets' send queues.  Sometimes
-	 * we do not get such new packets causing Tx to stall.  A single
-	 * UDP transmitter is a good example of this situation.  We have
-	 * a clean up timer that periodically reclaims completed packets
-	 * but it doesn't run often enough (nor do we want it to) to prevent
-	 * lengthy stalls.  A solution to this problem is to run the
-	 * destructor early, after the packet is queued but before it's DMAd.
-	 * A cons is that we lie to socket memory accounting, but the amount
-	 * of extra memory is reasonable (limited by the number of Tx
-	 * descriptors), the packets do actually get freed quickly by new
-	 * packets almost always, and for protocols like TCP that wait for
-	 * acks to really free up the data the extra memory is even less.
-	 * On the positive side we run the destructors on the sending CPU
-	 * rather than on a potentially different completing CPU, usually a
-	 * good thing.  We also run them without holding our Tx queue lock,
-	 * unlike what reclaim_completed_tx() would otherwise do.
-	 *
-	 * Run the destructor before telling the DMA engine about the packet
-	 * to make sure it doesn't complete and get freed prematurely.
-	 */
-	if (likely(!skb_shared(skb)))
-		skb_orphan(skb);
-
 	write_tx_pkt_wr(adap, skb, pi, pidx, gen, q, ndesc, compl);
 	check_ring_tx_db(adap, q);
 	return NETDEV_TX_OK;
drivers/net/loopback.c +0 -2
@@ -72,8 +72,6 @@ static int loopback_xmit(struct sk_buff 
 {
 	struct pcpu_lstats *pcpu_lstats, *lb_stats;
 
-	skb_orphan(skb);
-
 	skb->protocol = eth_type_trans(skb,dev);
 
 	/* it's OK to use per_cpu_ptr() because BHs are off */
drivers/net/mlx4/en_tx.c +0 -4
@@ -807,10 +807,6 @@ int mlx4_en_xmit(struct sk_buff *skb, st
 	if (tx_desc == (struct mlx4_en_tx_desc *) ring->bounce_buf)
 		tx_desc = mlx4_en_bounce_to_desc(priv, ring, index, desc_size);
 
-	/* Run destructor before passing skb to HW */
-	if (likely(!skb_shared(skb)))
-		skb_orphan(skb);
-
 	/* Ensure new descirptor hits memory
 	 * before setting ownership of this descriptor to HW */
 	wmb();
drivers/net/niu.c +1 -2
@@ -6667,8 +6667,7 @@ static int niu_start_xmit(struct sk_buff
 		}
 		kfree_skb(skb);
 		skb = skb_new;
-	} else
-		skb_orphan(skb);
+	}
 
 	align = ((unsigned long) skb->data & (16 - 1));
 	headroom = align + sizeof(struct tx_pkt_hdr);
drivers/net/veth.c +0 -2
@@ -155,8 +155,6 @@ static int veth_xmit(struct sk_buff *skb
 	struct veth_net_stats *stats, *rcv_stats;
 	int length, cpu;
 
-	skb_orphan(skb);
-
 	priv = netdev_priv(dev);
 	rcv = priv->peer;
 	rcv_priv = netdev_priv(rcv);
(quoted diff) +0 -0

b/drivers/net/wireless/libertas/tx.c

drivers/net/wireless/libertas/tx.c +0 -3
@@ -154,9 +154,6 @@ int lbs_hard_start_xmit(struct sk_buff *
 	if (priv->monitormode) {
 		/* Keep the skb to echo it back once Tx feedback is
 		   received from FW */
-		skb_orphan(skb);
-
-		/* Keep the skb around for when we get feedback */
 		priv->currenttxskb = skb;
 	} else {
  free:
net/core/dev.c +5 -16
@@ -1677,6 +1677,10 @@ int dev_hard_start_xmit(struct sk_buff *
 	const struct net_device_ops *ops = dev->netdev_ops;
 	int rc;
 
+	/* Call destructor here.  It's SMP-cache-friendly and avoids issues
+	 * with drivers which can hold transmitted skbs for long times */
+	skb_orphan(skb);
+
 	if (likely(!skb->next)) {
 		if (!list_empty(&ptype_all))
 			dev_queue_xmit_nit(skb, dev);
@@ -1688,22 +1692,7 @@ int dev_hard_start_xmit(struct sk_buff *
 				goto gso;
 		}
 
-		rc = ops->ndo_start_xmit(skb, dev);
-		/*
-		 * TODO: if skb_orphan() was called by
-		 * dev->hard_start_xmit() (for example, the unmodified
-		 * igb driver does that; bnx2 doesn't), then
-		 * skb_tx_software_timestamp() will be unable to send
-		 * back the time stamp.
-		 *
-		 * How can this be prevented? Always create another
-		 * reference to the socket before calling
-		 * dev->hard_start_xmit()? Prevent that skb_orphan()
-		 * does anything in dev->hard_start_xmit() by clearing
-		 * the skb destructor before the call and restoring it
-		 * afterwards, then doing the skb_orphan() ourselves?
-		 */
-		return rc;
+		return ops->ndo_start_xmit(skb, dev);
 	}
 
 gso:
#1 lore raw
Quoted 9 lines · Rusty Russell

Various drivers do skb_orphan() because they do not free transmitted skbs in a timely manner (causing apps which hit their socket limits to stall, socket close to hang, etc.).

DaveM points out that there are advantages to doing it generally (it's more likely to be on same CPU than after xmit), and I couldn't find any new starvation issues in simple benchmarking here.

If really no starvations are possible at all, I really wonder why some guys added memory accounting to UDP flows. Maybe they dont run "simple benchmarks" but real apps ? :)

For TCP, I agree your patch is a huge benefit, since its paced by remote ACKS and window control, but an UDP sender will likely be able to saturate a link.

Maybe we can try to selectively call skb_orphan() only for tcp packets ?

I understand your motivations are the driver simplifications, so you want all packets being orphaned... hmm...

This patch adds skb_orphan to the start of dev_hard_start_xmit(): it

Quoted 162 lines

can be premature in the NETDEV_TX_BUSY case, but that's uncommon.

I removed the drivers' skb_orphan(), though it's harmless.

Cc Divy Le Ray <[email protected]>
Cc Roland Dreier <[email protected]>
Cc Pavel Emelianov <[email protected]>
Cc Dan Williams <[email protected]>
---
 drivers/net/cxgb3/sge.c            |   27 ---------------------------
 drivers/net/loopback.c             |    2 --
 drivers/net/mlx4/en_tx.c           |    4 ----
 drivers/net/niu.c                  |    3 +--
 drivers/net/veth.c                 |    2 --
 drivers/net/wireless/libertas/tx.c |    3 ---
 net/core/dev.c                     |   21 +++++----------------
 7 files changed, 6 insertions(+), 56 deletions(-)
drivers/net/cxgb3/sge.c +0 -27
@@ -1288,33 +1288,6 @@ int t3_eth_xmit(struct sk_buff *skb, str
 	dev->trans_start = jiffies;
 	spin_unlock(&q->lock);
 
-	/*
-	 * We do not use Tx completion interrupts to free DMAd Tx packets.
-	 * This is good for performamce but means that we rely on new Tx
-	 * packets arriving to run the destructors of completed packets,
-	 * which open up space in their sockets' send queues.  Sometimes
-	 * we do not get such new packets causing Tx to stall.  A single
-	 * UDP transmitter is a good example of this situation.  We have
-	 * a clean up timer that periodically reclaims completed packets
-	 * but it doesn't run often enough (nor do we want it to) to prevent
-	 * lengthy stalls.  A solution to this problem is to run the
-	 * destructor early, after the packet is queued but before it's DMAd.
-	 * A cons is that we lie to socket memory accounting, but the amount
-	 * of extra memory is reasonable (limited by the number of Tx
-	 * descriptors), the packets do actually get freed quickly by new
-	 * packets almost always, and for protocols like TCP that wait for
-	 * acks to really free up the data the extra memory is even less.
-	 * On the positive side we run the destructors on the sending CPU
-	 * rather than on a potentially different completing CPU, usually a
-	 * good thing.  We also run them without holding our Tx queue lock,
-	 * unlike what reclaim_completed_tx() would otherwise do.
-	 *
-	 * Run the destructor before telling the DMA engine about the packet
-	 * to make sure it doesn't complete and get freed prematurely.
-	 */
-	if (likely(!skb_shared(skb)))
-		skb_orphan(skb);
-
 	write_tx_pkt_wr(adap, skb, pi, pidx, gen, q, ndesc, compl);
 	check_ring_tx_db(adap, q);
 	return NETDEV_TX_OK;
drivers/net/loopback.c +0 -2
@@ -72,8 +72,6 @@ static int loopback_xmit(struct sk_buff 
 {
 	struct pcpu_lstats *pcpu_lstats, *lb_stats;
 
-	skb_orphan(skb);
-
 	skb->protocol = eth_type_trans(skb,dev);
 
 	/* it's OK to use per_cpu_ptr() because BHs are off */
drivers/net/mlx4/en_tx.c +0 -4
@@ -807,10 +807,6 @@ int mlx4_en_xmit(struct sk_buff *skb, st
 	if (tx_desc == (struct mlx4_en_tx_desc *) ring->bounce_buf)
 		tx_desc = mlx4_en_bounce_to_desc(priv, ring, index, desc_size);
 
-	/* Run destructor before passing skb to HW */
-	if (likely(!skb_shared(skb)))
-		skb_orphan(skb);
-
 	/* Ensure new descirptor hits memory
 	 * before setting ownership of this descriptor to HW */
 	wmb();
drivers/net/niu.c +1 -2
@@ -6667,8 +6667,7 @@ static int niu_start_xmit(struct sk_buff
 		}
 		kfree_skb(skb);
 		skb = skb_new;
-	} else
-		skb_orphan(skb);
+	}
 
 	align = ((unsigned long) skb->data & (16 - 1));
 	headroom = align + sizeof(struct tx_pkt_hdr);
drivers/net/veth.c +0 -2
@@ -155,8 +155,6 @@ static int veth_xmit(struct sk_buff *skb
 	struct veth_net_stats *stats, *rcv_stats;
 	int length, cpu;
 
-	skb_orphan(skb);
-
 	priv = netdev_priv(dev);
 	rcv = priv->peer;
 	rcv_priv = netdev_priv(rcv);
drivers/net/wireless/libertas/tx.c +0 -3
@@ -154,9 +154,6 @@ int lbs_hard_start_xmit(struct sk_buff *
 	if (priv->monitormode) {
 		/* Keep the skb to echo it back once Tx feedback is
 		   received from FW */
-		skb_orphan(skb);
-
-		/* Keep the skb around for when we get feedback */
 		priv->currenttxskb = skb;
 	} else {
  free:
net/core/dev.c +5 -16
@@ -1677,6 +1677,10 @@ int dev_hard_start_xmit(struct sk_buff *
 	const struct net_device_ops *ops = dev->netdev_ops;
 	int rc;
 
+	/* Call destructor here.  It's SMP-cache-friendly and avoids issues
+	 * with drivers which can hold transmitted skbs for long times */
+	skb_orphan(skb);
+
 	if (likely(!skb->next)) {
 		if (!list_empty(&ptype_all))
 			dev_queue_xmit_nit(skb, dev);
@@ -1688,22 +1692,7 @@ int dev_hard_start_xmit(struct sk_buff *
 				goto gso;
 		}
 
-		rc = ops->ndo_start_xmit(skb, dev);
-		/*
-		 * TODO: if skb_orphan() was called by
-		 * dev->hard_start_xmit() (for example, the unmodified
-		 * igb driver does that; bnx2 doesn't), then
-		 * skb_tx_software_timestamp() will be unable to send
-		 * back the time stamp.
-		 *
-		 * How can this be prevented? Always create another
-		 * reference to the socket before calling
-		 * dev->hard_start_xmit()? Prevent that skb_orphan()
-		 * does anything in dev->hard_start_xmit() by clearing
-		 * the skb destructor before the call and restoring it
-		 * afterwards, then doing the skb_orphan() ourselves?
-		 */
-		return rc;
+		return ops->ndo_start_xmit(skb, dev);
 	}
 
 gso:
#2 lore raw
In reply to Rusty Russell
Quoted 9 lines · Rusty Russell

Various drivers do skb_orphan() because they do not free transmitted skbs in a timely manner (causing apps which hit their socket limits to stall, socket close to hang, etc.).

DaveM points out that there are advantages to doing it generally (it's more likely to be on same CPU than after xmit), and I couldn't find any new starvation issues in simple benchmarking here.

If really no starvations are possible at all, I really wonder why some guys added memory accounting to UDP flows. Maybe they dont run "simple benchmarks" but real apps ? :)

For TCP, I agree your patch is a huge benefit, since its paced by remote ACKS and window control, but an UDP sender will likely be able to saturate a link.

Maybe we can try to selectively call skb_orphan() only for tcp packets ?

I understand your motivations are the driver simplifications, so you want all packets being orphaned... hmm...

This patch adds skb_orphan to the start of dev_hard_start_xmit(): it

Quoted 162 lines

can be premature in the NETDEV_TX_BUSY case, but that's uncommon.

I removed the drivers' skb_orphan(), though it's harmless.

Cc Divy Le Ray <[email protected]>
Cc Roland Dreier <[email protected]>
Cc Pavel Emelianov <[email protected]>
Cc Dan Williams <[email protected]>
---
 drivers/net/cxgb3/sge.c            |   27 ---------------------------
 drivers/net/loopback.c             |    2 --
 drivers/net/mlx4/en_tx.c           |    4 ----
 drivers/net/niu.c                  |    3 +--
 drivers/net/veth.c                 |    2 --
 drivers/net/wireless/libertas/tx.c |    3 ---
 net/core/dev.c                     |   21 +++++----------------
 7 files changed, 6 insertions(+), 56 deletions(-)
drivers/net/cxgb3/sge.c +0 -27
@@ -1288,33 +1288,6 @@ int t3_eth_xmit(struct sk_buff *skb, str
 	dev->trans_start = jiffies;
 	spin_unlock(&q->lock);
 
-	/*
-	 * We do not use Tx completion interrupts to free DMAd Tx packets.
-	 * This is good for performamce but means that we rely on new Tx
-	 * packets arriving to run the destructors of completed packets,
-	 * which open up space in their sockets' send queues.  Sometimes
-	 * we do not get such new packets causing Tx to stall.  A single
-	 * UDP transmitter is a good example of this situation.  We have
-	 * a clean up timer that periodically reclaims completed packets
-	 * but it doesn't run often enough (nor do we want it to) to prevent
-	 * lengthy stalls.  A solution to this problem is to run the
-	 * destructor early, after the packet is queued but before it's DMAd.
-	 * A cons is that we lie to socket memory accounting, but the amount
-	 * of extra memory is reasonable (limited by the number of Tx
-	 * descriptors), the packets do actually get freed quickly by new
-	 * packets almost always, and for protocols like TCP that wait for
-	 * acks to really free up the data the extra memory is even less.
-	 * On the positive side we run the destructors on the sending CPU
-	 * rather than on a potentially different completing CPU, usually a
-	 * good thing.  We also run them without holding our Tx queue lock,
-	 * unlike what reclaim_completed_tx() would otherwise do.
-	 *
-	 * Run the destructor before telling the DMA engine about the packet
-	 * to make sure it doesn't complete and get freed prematurely.
-	 */
-	if (likely(!skb_shared(skb)))
-		skb_orphan(skb);
-
 	write_tx_pkt_wr(adap, skb, pi, pidx, gen, q, ndesc, compl);
 	check_ring_tx_db(adap, q);
 	return NETDEV_TX_OK;
drivers/net/loopback.c +0 -2
@@ -72,8 +72,6 @@ static int loopback_xmit(struct sk_buff 
 {
 	struct pcpu_lstats *pcpu_lstats, *lb_stats;
 
-	skb_orphan(skb);
-
 	skb->protocol = eth_type_trans(skb,dev);
 
 	/* it's OK to use per_cpu_ptr() because BHs are off */
drivers/net/mlx4/en_tx.c +0 -4
@@ -807,10 +807,6 @@ int mlx4_en_xmit(struct sk_buff *skb, st
 	if (tx_desc == (struct mlx4_en_tx_desc *) ring->bounce_buf)
 		tx_desc = mlx4_en_bounce_to_desc(priv, ring, index, desc_size);
 
-	/* Run destructor before passing skb to HW */
-	if (likely(!skb_shared(skb)))
-		skb_orphan(skb);
-
 	/* Ensure new descirptor hits memory
 	 * before setting ownership of this descriptor to HW */
 	wmb();
drivers/net/niu.c +1 -2
@@ -6667,8 +6667,7 @@ static int niu_start_xmit(struct sk_buff
 		}
 		kfree_skb(skb);
 		skb = skb_new;
-	} else
-		skb_orphan(skb);
+	}
 
 	align = ((unsigned long) skb->data & (16 - 1));
 	headroom = align + sizeof(struct tx_pkt_hdr);
drivers/net/veth.c +0 -2
@@ -155,8 +155,6 @@ static int veth_xmit(struct sk_buff *skb
 	struct veth_net_stats *stats, *rcv_stats;
 	int length, cpu;
 
-	skb_orphan(skb);
-
 	priv = netdev_priv(dev);
 	rcv = priv->peer;
 	rcv_priv = netdev_priv(rcv);
drivers/net/wireless/libertas/tx.c +0 -3
@@ -154,9 +154,6 @@ int lbs_hard_start_xmit(struct sk_buff *
 	if (priv->monitormode) {
 		/* Keep the skb to echo it back once Tx feedback is
 		   received from FW */
-		skb_orphan(skb);
-
-		/* Keep the skb around for when we get feedback */
 		priv->currenttxskb = skb;
 	} else {
  free:
net/core/dev.c +5 -16
@@ -1677,6 +1677,10 @@ int dev_hard_start_xmit(struct sk_buff *
 	const struct net_device_ops *ops = dev->netdev_ops;
 	int rc;
 
+	/* Call destructor here.  It's SMP-cache-friendly and avoids issues
+	 * with drivers which can hold transmitted skbs for long times */
+	skb_orphan(skb);
+
 	if (likely(!skb->next)) {
 		if (!list_empty(&ptype_all))
 			dev_queue_xmit_nit(skb, dev);
@@ -1688,22 +1692,7 @@ int dev_hard_start_xmit(struct sk_buff *
 				goto gso;
 		}
 
-		rc = ops->ndo_start_xmit(skb, dev);
-		/*
-		 * TODO: if skb_orphan() was called by
-		 * dev->hard_start_xmit() (for example, the unmodified
-		 * igb driver does that; bnx2 doesn't), then
-		 * skb_tx_software_timestamp() will be unable to send
-		 * back the time stamp.
-		 *
-		 * How can this be prevented? Always create another
-		 * reference to the socket before calling
-		 * dev->hard_start_xmit()? Prevent that skb_orphan()
-		 * does anything in dev->hard_start_xmit() by clearing
-		 * the skb destructor before the call and restoring it
-		 * afterwards, then doing the skb_orphan() ourselves?
-		 */
-		return rc;
+		return ops->ndo_start_xmit(skb, dev);
 	}
 
 gso:
#3 lore raw
Quoted 8 lines · Eric Dumazet
Quoted 3 lines · Rusty Russell

DaveM points out that there are advantages to doing it generally (it's more likely to be on same CPU than after xmit), and I couldn't find any new starvation issues in simple benchmarking here.

If really no starvations are possible at all, I really wonder why some guys added memory accounting to UDP flows. Maybe they dont run "simple benchmarks" but real apps ? :)

Well, without any accounting at all you could use quite a lot of memory as there are many places packets can be queued.

Quoted 2 lines

For TCP, I agree your patch is a huge benefit, since its paced by remote ACKS and window control

I doubt that. There'll be some cache friendliness, but I'm not sure it'll be measurable, let alone "huge". It's the win to drivers which don't have a timely and batching tx free mechanism which I aim for.

Quoted 2 lines

, but an UDP sender will likely be able to saturate a link.

I couldn't see any difference in saturation here (with default scheduler and an 100MBit e1000e). Two reasons come to mind: firstly, only the hardware queue is
unregulated: the tx queue is still accounted. And when you add scheduling to the mix, I can't in practice cause starvation of other senders.

Hope that clarifies,
Rusty.

#4 lore raw
Quoted 12 lines · Rusty Russell
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Rusty Russell

DaveM points out that there are advantages to doing it generally (it's more likely to be on same CPU than after xmit), and I couldn't find any new starvation issues in simple benchmarking here.

If really no starvations are possible at all, I really wonder why some guys added memory accounting to UDP flows. Maybe they dont run "simple benchmarks" but real apps ? :)

Well, without any accounting at all you could use quite a lot of memory as there are many places packets can be queued.

Quoted 6 lines
Quoted 2 lines

For TCP, I agree your patch is a huge benefit, since its paced by remote ACKS and window control

I doubt that. There'll be some cache friendliness, but I'm not sure it'll be measurable, let alone "huge". It's the win to drivers which don't have a timely and batching tx free mechanism which I aim for.

At 250.000 packets/second on a Gigabit link, this is huge, I can tell you. (250.000 incoming packets and 250.000 outgoing packets per second, 700 Mbit/s)

According to this oprofile on CPU0 (dedicated to softirqs on one bnx2 eth adapter)

We can see sock_wfree() being number 2 on the profile, because it touches three cache lines per socket and transmited packet in TX completion handler.

Also, taking a reference on socket for each xmit packet in flight is very expensive, since it slows down receiver in __udp4_lib_lookup(). Several cpus are fighting for sk->refcnt cache line.

CPU: Core 2, speed 3000.24 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  cum. samples  %        cum. %     symbol name
21215    21215         11.8847  11.8847    bnx2_poll_work
17239    38454          9.6573  21.5420    sock_wfree      << effect of udp memory accounting >>
14817    53271          8.3005  29.8425    __slab_free
14635    67906          8.1986  38.0411    __udp4_lib_lookup
11425    79331          6.4003  44.4414    __alloc_skb
9710     89041          5.4396  49.8810    __slab_alloc
8095     97136          4.5348  54.4158    __udp4_lib_rcv
7831     104967         4.3869  58.8027    sock_def_write_space
7586     112553         4.2497  63.0524    ip_rcv
7518     120071         4.2116  67.2640    skb_dma_unmap
6711     126782         3.7595  71.0235    netif_receive_skb
6272     133054         3.5136  74.5371    udp_queue_rcv_skb
5262     138316         2.9478  77.4849    skb_release_data
5023     143339         2.8139  80.2988    __kmalloc_track_caller
4070     147409         2.2800  82.5788    kmem_cache_alloc
3216     150625         1.8016  84.3804    ipt_do_table
2576     153201         1.4431  85.8235    skb_queue_tail
#5 lore raw
In reply to Rusty Russell
Quoted 12 lines · Rusty Russell
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Rusty Russell

DaveM points out that there are advantages to doing it generally (it's more likely to be on same CPU than after xmit), and I couldn't find any new starvation issues in simple benchmarking here.

If really no starvations are possible at all, I really wonder why some guys added memory accounting to UDP flows. Maybe they dont run "simple benchmarks" but real apps ? :)

Well, without any accounting at all you could use quite a lot of memory as there are many places packets can be queued.

Quoted 6 lines
Quoted 2 lines

For TCP, I agree your patch is a huge benefit, since its paced by remote ACKS and window control

I doubt that. There'll be some cache friendliness, but I'm not sure it'll be measurable, let alone "huge". It's the win to drivers which don't have a timely and batching tx free mechanism which I aim for.

At 250.000 packets/second on a Gigabit link, this is huge, I can tell you. (250.000 incoming packets and 250.000 outgoing packets per second, 700 Mbit/s)

According to this oprofile on CPU0 (dedicated to softirqs on one bnx2 eth adapter)

We can see sock_wfree() being number 2 on the profile, because it touches three cache lines per socket and transmited packet in TX completion handler.

Also, taking a reference on socket for each xmit packet in flight is very expensive, since it slows down receiver in __udp4_lib_lookup(). Several cpus are fighting for sk->refcnt cache line.

CPU: Core 2, speed 3000.24 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  cum. samples  %        cum. %     symbol name
21215    21215         11.8847  11.8847    bnx2_poll_work
17239    38454          9.6573  21.5420    sock_wfree      << effect of udp memory accounting >>
14817    53271          8.3005  29.8425    __slab_free
14635    67906          8.1986  38.0411    __udp4_lib_lookup
11425    79331          6.4003  44.4414    __alloc_skb
9710     89041          5.4396  49.8810    __slab_alloc
8095     97136          4.5348  54.4158    __udp4_lib_rcv
7831     104967         4.3869  58.8027    sock_def_write_space
7586     112553         4.2497  63.0524    ip_rcv
7518     120071         4.2116  67.2640    skb_dma_unmap
6711     126782         3.7595  71.0235    netif_receive_skb
6272     133054         3.5136  74.5371    udp_queue_rcv_skb
5262     138316         2.9478  77.4849    skb_release_data
5023     143339         2.8139  80.2988    __kmalloc_track_caller
4070     147409         2.2800  82.5788    kmem_cache_alloc
3216     150625         1.8016  84.3804    ipt_do_table
2576     153201         1.4431  85.8235    skb_queue_tail
#6 lore raw
Quoted 7 lines · Eric Dumazet
Quoted 6 lines · Rusty Russell
Quoted 2 lines · Eric Dumazet

For TCP, I agree your patch is a huge benefit, since its paced by remote ACKS and window control

I doubt that. There'll be some cache friendliness, but I'm not sure it'll be measurable, let alone "huge".

...

Quoted 3 lines

We can see sock_wfree() being number 2 on the profile, because it touches three cache lines per socket and transmited packet in TX completion handler.

Interesting, I take it back: got some "after" stats as well?

Quoted 3 lines

Also, taking a reference on socket for each xmit packet in flight is very expensive, since it slows down receiver in __udp4_lib_lookup(). Several cpus are fighting for sk->refcnt cache line.

Now we have decent dynamic per-cpu, we can finally implement bigrefs. More obvious for device counts than sockets, but perhaps applicable here as well?

Rusty.

#7 lore raw
In reply to Eric Dumazet
Quoted 7 lines · Eric Dumazet
Quoted 6 lines · Rusty Russell
Quoted 2 lines · Eric Dumazet

For TCP, I agree your patch is a huge benefit, since its paced by remote ACKS and window control

I doubt that. There'll be some cache friendliness, but I'm not sure it'll be measurable, let alone "huge".

...

Quoted 3 lines

We can see sock_wfree() being number 2 on the profile, because it touches three cache lines per socket and transmited packet in TX completion handler.

Interesting, I take it back: got some "after" stats as well?

Quoted 3 lines

Also, taking a reference on socket for each xmit packet in flight is very expensive, since it slows down receiver in __udp4_lib_lookup(). Several cpus are fighting for sk->refcnt cache line.

Now we have decent dynamic per-cpu, we can finally implement bigrefs. More obvious for device counts than sockets, but perhaps applicable here as well?

Rusty.

maintainer
#8 lore raw

From: Rusty Russell <[email protected]>
Date: Thu, 4 Jun 2009 13:24:57 +0930

Quoted 7 lines · Eric Dumazet
Quoted 3 lines

Also, taking a reference on socket for each xmit packet in flight is very expensive, since it slows down receiver in __udp4_lib_lookup(). Several cpus are fighting for sk->refcnt cache line.

Now we have decent dynamic per-cpu, we can finally implement bigrefs. More obvious for device counts than sockets, but perhaps applicable here as well?

It might be very beneficial for longer lasting, active, connections, but for high connection rates it's going to be a lose in my estimation.

#9 lore raw
Quoted 4 lines · David Miller

From: Rusty Russell <[email protected]>
Date: Thu, 4 Jun 2009 13:24:57 +0930

Quoted 1 line · Eric Dumazet
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Also, taking a reference on socket for each xmit packet in flight is very expensive, since it slows down receiver in __udp4_lib_lookup(). Several cpus are fighting for sk->refcnt cache line.

Now we have decent dynamic per-cpu, we can finally implement bigrefs. More obvious for device counts than sockets, but perhaps applicable here as well?

It might be very beneficial for longer lasting, active, connections, but for high connection rates it's going to be a lose in my estimation.

Agreed.

We also can avoid the sock_put()/sock_hold() pair for each tx packet, to only touch sk_wmem_alloc (with appropriate atomic_sub_return() in sock_wfree() and atomic_dec_test in sk_free

We could initialize sk->sk_wmem_alloc to one instead of 0, so that sock_wfree() could just synchronize itself with sk_free()

void sk_free(struct sock *sk)

{
	if (atomic_dec_test(&sk->sk_wmem_alloc))
		__sk_free(sk)
}

static inline void skb_set_owner_w(struct sk_buff *skb, struct sock *sk) {

-       sock_hold(sk);
        skb->sk = sk;
        skb->destructor = sock_wfree;
        atomic_add(skb->truesize, &sk->sk_wmem_alloc);

}

void sock_wfree(struct sk_buff *skb)

 {
        struct sock *sk = skb->sk;

+ int res;

        /* In case it might be waiting for more memory. */

- atomic_sub(skb->truesize, &sk->sk_wmem_alloc);
+ res = atomic_sub_return(skb->truesize, &sk->sk_wmem_alloc);
if (!sock_flag(sk, SOCK_USE_WRITE_QUEUE))
sk->sk_write_space(sk);
- sock_put(sk);
+ if (res == 0)
+ __sk_free(sk);
}

Patch will follow after some testing

maintainer
#10 lore raw

From: Eric Dumazet <[email protected]>
Date: Thu, 04 Jun 2009 06:54:24 +0200

Quoted 6 lines

We also can avoid the sock_put()/sock_hold() pair for each tx packet, to only touch sk_wmem_alloc (with appropriate atomic_sub_return() in sock_wfree() and atomic_dec_test in sk_free

We could initialize sk->sk_wmem_alloc to one instead of 0, so that sock_wfree() could just synchronize itself with sk_free()

Excellent idea Eric.

Quoted 1 line

Patch will follow after some testing

I look forward to it :-)

#11 lore raw

PATCH 1/1 net: No more expensive sock_hold()/sock_put() on each tx

Quoted 11 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Thu, 04 Jun 2009 06:54:24 +0200

Quoted 6 lines

We also can avoid the sock_put()/sock_hold() pair for each tx packet, to only touch sk_wmem_alloc (with appropriate atomic_sub_return() in sock_wfree() and atomic_dec_test in sk_free

We could initialize sk->sk_wmem_alloc to one instead of 0, so that sock_wfree() could just synchronize itself with sk_free()

Excellent idea Eric.

Thanks !

Quoted 4 lines
Quoted 2 lines

Patch will follow after some testing

I look forward to it :-)

Here it is, based on net-next-2.6

CC list trimmed down...

Next step would be to get rid of sk_callback_lock, I dont remember if we already discussed about that rwlock, after RCUification of sockets...

Thanks

[PATCH] net: No more expensive sock_hold()/sock_put() on each tx

One of the problem with sock memory accounting is it uses a pair of sock_hold()/sock_put() for each transmitted packet.

This slows down bidirectional flows because the receive path also needs to take a refcount on socket and might use a different cpu than transmit path or transmit completion path. So these two atomic operations also trigger cache line bounces.

We can see this in tx or tx/rx workloads (media gateways for example), where sock_wfree() can be in top five functions in profiles.

We use this sock_hold()/sock_put() so that sock freeing is delayed until all tx packets are completed.

As we also update sk_wmem_alloc, we could offset sk_wmem_alloc by one unit at init time, until sk_free() is called. Once sk_free() is called, we atomic_dec_and_test(sk_wmem_alloc) to decrement initial offset and atomicaly check if any packets are in flight.

skb_set_owner_w() doesnt call sock_hold() anymore

sock_wfree() doesnt call sock_put() anymore, but check if sk_wmem_alloc reached 0 to perform the final freeing.

Drawback is that a skb->truesize error could lead to unfreeable sockets, or even worse, prematurely calling __sk_free() on a live socket.

Nice speedups on SMP. tbench for example, going from 2691 MB/s to 2711 MB/s on my 8 cpu dev machine, even if tbench was not really hitting sk_refcnt contention point. 5 % speedup on a UDP transmit workload (depends on number of flows), lowering TX completion cpu usage.

---
 include/net/sock.h    |    6 +++++-
 net/core/sock.c       |   29 +++++++++++++++++++++++++----
 net/ipv4/ip_output.c  |    1 -
 net/ipv6/ip6_output.c |    1 -
 4 files changed, 30 insertions(+), 7 deletions(-)
include/net/sock.h +5 -1
index 4bb1ff9..010e14a 100644
@@ -1217,9 +1217,13 @@ static inline int skb_copy_to_page(struct sock *sk, char __user *from,
 
 static inline void skb_set_owner_w(struct sk_buff *skb, struct sock *sk)
 {
-	sock_hold(sk);
 	skb->sk = sk;
 	skb->destructor = sock_wfree;
+	/*
+	 * We used to take a refcount on sk, but following operation
+	 * is enough to guarantee sk_free() wont free this sock until
+	 * all in-flight packets are completed
+	 */
 	atomic_add(skb->truesize, &sk->sk_wmem_alloc);
 }
 
net/core/sock.c +25 -4
index 58dec9d..ce0159a 100644
@@ -1005,7 +1005,7 @@ struct sock *sk_alloc(struct net *net, int family, gfp_t priority,
 }
 EXPORT_SYMBOL(sk_alloc);
 
-void sk_free(struct sock *sk)
+static void __sk_free(struct sock *sk)
 {
 	struct sk_filter *filter;
 
@@ -1028,6 +1028,17 @@ void sk_free(struct sock *sk)
 	put_net(sock_net(sk));
 	sk_prot_free(sk->sk_prot_creator, sk);
 }
+
+void sk_free(struct sock *sk)
+{
+	/*
+	 * We substract one from sk_wmem_alloc and can know if
+	 * some packets are still in some tx queue.
+	 * If not null, sock_wfree() will call __sk_free(sk) later
+	 */
+	if (atomic_dec_and_test(&sk->sk_wmem_alloc))
+		__sk_free(sk);
+}
 EXPORT_SYMBOL(sk_free);
 
 /*
@@ -1068,7 +1079,10 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority)
 		newsk->sk_backlog.head	= newsk->sk_backlog.tail = NULL;
 
 		atomic_set(&newsk->sk_rmem_alloc, 0);
-		atomic_set(&newsk->sk_wmem_alloc, 0);
+		/*
+		 * sk_wmem_alloc set to one (see sk_free() and sock_wfree())
+		 */
+		atomic_set(&newsk->sk_wmem_alloc, 1);
 		atomic_set(&newsk->sk_omem_alloc, 0);
 		skb_queue_head_init(&newsk->sk_receive_queue);
 		skb_queue_head_init(&newsk->sk_write_queue);
@@ -1172,12 +1186,18 @@ void __init sk_init(void)
 void sock_wfree(struct sk_buff *skb)
 {
 	struct sock *sk = skb->sk;
+	int res;
 
 	/* In case it might be waiting for more memory. */
-	atomic_sub(skb->truesize, &sk->sk_wmem_alloc);
+	res = atomic_sub_return(skb->truesize, &sk->sk_wmem_alloc);
 	if (!sock_flag(sk, SOCK_USE_WRITE_QUEUE))
 		sk->sk_write_space(sk);
-	sock_put(sk);
+	/*
+	 * if sk_wmem_alloc reached 0, we are last user and should
+	 * free this sock, as sk_free() call could not do it.
+	 */
+	if (res == 0)
+		__sk_free(sk);
 }
 EXPORT_SYMBOL(sock_wfree);
 
@@ -1816,6 +1836,7 @@ void sock_init_data(struct socket *sock, struct sock *sk)
 	sk->sk_stamp = ktime_set(-1L, 0);
 
 	atomic_set(&sk->sk_refcnt, 1);
+	atomic_set(&sk->sk_wmem_alloc, 1);
 	atomic_set(&sk->sk_drops, 0);
 }
 EXPORT_SYMBOL(sock_init_data);
net/ipv4/ip_output.c +0 -1
index 3d6167f..badbfde 100644
@@ -498,7 +498,6 @@ int ip_fragment(struct sk_buff *skb, int (*output)(struct sk_buff *))
 
 			BUG_ON(frag->sk);
 			if (skb->sk) {
-				sock_hold(skb->sk);
 				frag->sk = skb->sk;
 				frag->destructor = sock_wfree;
 				truesizes += frag->truesize;
net/ipv6/ip6_output.c +0 -1
index c8dc8e5..18b9630 100644
@@ -680,7 +680,6 @@ static int ip6_fragment(struct sk_buff *skb, int (*output)(struct sk_buff *))
 
 			BUG_ON(frag->sk);
 			if (skb->sk) {
-				sock_hold(skb->sk);
 				frag->sk = skb->sk;
 				frag->destructor = sock_wfree;
 				truesizes += frag->truesize;
maintainer
#12 lore raw

Re: [PATCH] net: No more expensive sock_hold()/sock_put() on each tx

From: Eric Dumazet <[email protected]>
Date: Thu, 04 Jun 2009 11:18:35 +0200

Quoted 2 lines

Next step would be to get rid of sk_callback_lock, I dont remember if we already discussed about that rwlock, after RCUification of sockets...

We discussed it, but the RCU'ification of core socket destruction is pretty expensive. Just do any simple connection rate based test, like lat_connection from lmbench, and the effects are clear.

maintainer
#13 lore raw

Re: [PATCH] net: No more expensive sock_hold()/sock_put() on each tx

In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Thu, 04 Jun 2009 11:18:35 +0200

Quoted 11 lines
net/core/sock.c +2 -1
@@ -1172,12 +1186,18 @@ void __init sk_init(void)
 void sock_wfree(struct sk_buff *skb)
 {
 	struct sock *sk = skb->sk;
+	int res;
 
 	/* In case it might be waiting for more memory. */
-	atomic_sub(skb->truesize, &sk->sk_wmem_alloc);
+	res = atomic_sub_return(skb->truesize, &sk->sk_wmem_alloc);
 	if (!sock_flag(sk, SOCK_USE_WRITE_QUEUE))
 		sk->sk_write_space(sk);

net/core/sock.c void __init sk_init(void) · line 1203

Quoted 10 lines
net/core/sock.c +6 -1
-	sock_put(sk);
+	/*
+	 * if sk_wmem_alloc reached 0, we are last user and should
+	 * free this sock, as sk_free() call could not do it.
+	 */
+	if (res == 0)
+		__sk_free(sk);
 }
 EXPORT_SYMBOL(sock_wfree);
 

Eric, I don't understand this part, please enlighten me :-)

Just because we've liberated all of the write buffer space, that does not mean that it's time to kill off the socket completely.

Right?

#14 lore raw

Re: [PATCH] net: No more expensive sock_hold()/sock_put() on each tx

Quoted 27 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Thu, 04 Jun 2009 11:18:35 +0200

Quoted 11 lines
net/core/sock.c +2 -1
@@ -1172,12 +1186,18 @@ void __init sk_init(void)
 void sock_wfree(struct sk_buff *skb)
 {
 	struct sock *sk = skb->sk;
+	int res;
 
 	/* In case it might be waiting for more memory. */
-	atomic_sub(skb->truesize, &sk->sk_wmem_alloc);
+	res = atomic_sub_return(skb->truesize, &sk->sk_wmem_alloc);
 	if (!sock_flag(sk, SOCK_USE_WRITE_QUEUE))
 		sk->sk_write_space(sk);

net/core/sock.c void __init sk_init(void) · line 1203

Quoted 10 lines
net/core/sock.c +6 -1
-	sock_put(sk);
+	/*
+	 * if sk_wmem_alloc reached 0, we are last user and should
+	 * free this sock, as sk_free() call could not do it.
+	 */
+	if (res == 0)
+		__sk_free(sk);
 }
 EXPORT_SYMBOL(sock_wfree);
 

Eric, I don't understand this part, please enlighten me :-)

Quoted 4 lines

Just because we've liberated all of the write buffer space, that does not mean that it's time to kill off the socket completely.

Right?

Remember we initialize this field to one.

If we freed all write buffer space, final value is one, not zero.

res == 0 only if we both freed all write buffer space, *and* socket was also refcounted to 0 (sk_free() then realized it could not yet call __sk_free())

So we cheat a litle bit, because of this offset of one, we might block a sender a litle bit earlier :)

Thank you

maintainer
#15 lore raw

Re: [PATCH] net: No more expensive sock_hold()/sock_put() on each tx

From: Eric Dumazet <[email protected]>
Date: Wed, 10 Jun 2009 10:30:49 +0200

Quoted 11 lines
Quoted 9 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Thu, 04 Jun 2009 11:18:35 +0200

Eric, I don't understand this part, please enlighten me :-)

Just because we've liberated all of the write buffer space, that does not mean that it's time to kill off the socket completely.

Right?

Quoted 8 lines

Remember we initialize this field to one.

If we freed all write buffer space, final value is one, not zero.

res == 0 only if we both freed all write buffer space, *and* socket was also refcounted to 0 (sk_free() then realized it could not yet call __sk_free())

So we cheat a litle bit, because of this offset of one, we might block a sender a litle bit earlier :)

Now I understand, thanks!

Patch applied.

maintainer
#16 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Thu, 04 Jun 2009 06:54:24 +0200

Quoted 6 lines

We also can avoid the sock_put()/sock_hold() pair for each tx packet, to only touch sk_wmem_alloc (with appropriate atomic_sub_return() in sock_wfree() and atomic_dec_test in sk_free

We could initialize sk->sk_wmem_alloc to one instead of 0, so that sock_wfree() could just synchronize itself with sk_free()

Excellent idea Eric.

Quoted 1 line

Patch will follow after some testing

I look forward to it :-)

maintainer
#17 lore raw
In reply to Rusty Russell

From: Rusty Russell <[email protected]>
Date: Thu, 4 Jun 2009 13:24:57 +0930

Quoted 7 lines · Eric Dumazet
Quoted 3 lines

Also, taking a reference on socket for each xmit packet in flight is very expensive, since it slows down receiver in __udp4_lib_lookup(). Several cpus are fighting for sk->refcnt cache line.

Now we have decent dynamic per-cpu, we can finally implement bigrefs. More obvious for device counts than sockets, but perhaps applicable here as well?

It might be very beneficial for longer lasting, active, connections, but for high connection rates it's going to be a lose in my estimation.

#18 lore raw
In reply to Rusty Russell
Quoted 2 lines · Rusty Russell

This patch adds skb_orphan to the start of dev_hard_start_xmit(): it can be premature in the NETDEV_TX_BUSY case, but that's uncommon.

Would it be possible to make the new skb_orphan() at the start of dev_hard_start_xmit() conditionally so that it is not executed for packets that are to be time stamped?

As discussed before
(http://article.gmane.org/gmane.linux.network/121378/), the skb->sk socket pointer is required for sending back the send time stamp from inside the device driver. Calling skb_orphan() unconditionally as in this patch would break the hardware time stamping of outgoing packets.

This should work without much overhead (skb_tx() expands to a lookup of the skb_shared_info):
if (!skb_tx(skb)->flags)

    skb_orphan(skb);

Instead of looking at the individual skb_shared_tx "hardware" and "software" bits I'm just checking the whole byte here because it is shorter and perhaps faster. Semantically the result is the same.

Signature · 6 lines
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
#19 lore raw
In reply to Rusty Russell
Quoted 2 lines · Rusty Russell

This patch adds skb_orphan to the start of dev_hard_start_xmit(): it can be premature in the NETDEV_TX_BUSY case, but that's uncommon.

Would it be possible to make the new skb_orphan() at the start of dev_hard_start_xmit() conditionally so that it is not executed for packets that are to be time stamped?

As discussed before
(http://article.gmane.org/gmane.linux.network/121378/), the skb->sk socket pointer is required for sending back the send time stamp from inside the device driver. Calling skb_orphan() unconditionally as in this patch would break the hardware time stamping of outgoing packets.

This should work without much overhead (skb_tx() expands to a lookup of the skb_shared_info):
if (!skb_tx(skb)->flags)

    skb_orphan(skb);

Instead of looking at the individual skb_shared_tx "hardware" and "software" bits I'm just checking the whole byte here because it is shorter and perhaps faster. Semantically the result is the same.

Signature · 6 lines
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
maintainer
#20 lore raw

From: Patrick Ohly <[email protected]>
Date: Mon, 01 Jun 2009 21:47:22 +0200

Quoted 8 lines · Rusty Russell
Quoted 2 lines

This patch adds skb_orphan to the start of dev_hard_start_xmit(): it can be premature in the NETDEV_TX_BUSY case, but that's uncommon.

Would it be possible to make the new skb_orphan() at the start of dev_hard_start_xmit() conditionally so that it is not executed for packets that are to be time stamped?

Quoted 5 lines

As discussed before
(http://article.gmane.org/gmane.linux.network/121378/), the skb->sk socket pointer is required for sending back the send time stamp from inside the device driver. Calling skb_orphan() unconditionally as in this patch would break the hardware time stamping of outgoing packets.

Indeed, we need to check that case, at a minimum.

And there are other potentially other problems. For example, I wonder how this interacts with the new TX MMAP af_packet support in net-next-2.6 :-/

#21 lore raw
Quoted 17 lines · David Miller

From: Patrick Ohly <[email protected]>
Date: Mon, 01 Jun 2009 21:47:22 +0200

Quoted 8 lines · Rusty Russell
Quoted 2 lines

This patch adds skb_orphan to the start of dev_hard_start_xmit(): it can be premature in the NETDEV_TX_BUSY case, but that's uncommon.

Would it be possible to make the new skb_orphan() at the start of dev_hard_start_xmit() conditionally so that it is not executed for packets that are to be time stamped?

Quoted 5 lines

As discussed before
(http://article.gmane.org/gmane.linux.network/121378/), the skb->sk socket pointer is required for sending back the send time stamp from inside the device driver. Calling skb_orphan() unconditionally as in this patch would break the hardware time stamping of outgoing packets.

Quoted 5 lines

Indeed, we need to check that case, at a minimum.

And there are other potentially other problems. For example, I wonder how this interacts with the new TX MMAP af_packet support in net-next-2.6 :-/

I think I'll do this in the driver for now, and let's revisit doing it generically later?

Thanks,
Rusty.

#22 lore raw
In reply to David Miller
Quoted 17 lines · David Miller

From: Patrick Ohly <[email protected]>
Date: Mon, 01 Jun 2009 21:47:22 +0200

Quoted 8 lines · Rusty Russell
Quoted 2 lines

This patch adds skb_orphan to the start of dev_hard_start_xmit(): it can be premature in the NETDEV_TX_BUSY case, but that's uncommon.

Would it be possible to make the new skb_orphan() at the start of dev_hard_start_xmit() conditionally so that it is not executed for packets that are to be time stamped?

Quoted 5 lines

As discussed before
(http://article.gmane.org/gmane.linux.network/121378/), the skb->sk socket pointer is required for sending back the send time stamp from inside the device driver. Calling skb_orphan() unconditionally as in this patch would break the hardware time stamping of outgoing packets.

Quoted 5 lines

Indeed, we need to check that case, at a minimum.

And there are other potentially other problems. For example, I wonder how this interacts with the new TX MMAP af_packet support in net-next-2.6 :-/

I think I'll do this in the driver for now, and let's revisit doing it generically later?

Thanks,
Rusty.

maintainer
#23 lore raw

From: Rusty Russell <[email protected]>
Date: Tue, 2 Jun 2009 23:38:29 +0930

Quoted 18 lines · David Miller
Quoted 17 lines

From: Patrick Ohly <[email protected]>
Date: Mon, 01 Jun 2009 21:47:22 +0200

Quoted 13 lines · Rusty Russell
Quoted 2 lines

This patch adds skb_orphan to the start of dev_hard_start_xmit(): it can be premature in the NETDEV_TX_BUSY case, but that's uncommon.

Would it be possible to make the new skb_orphan() at the start of dev_hard_start_xmit() conditionally so that it is not executed for packets that are to be time stamped?

As discussed before
(http://article.gmane.org/gmane.linux.network/121378/), the skb->sk socket pointer is required for sending back the send time stamp from inside the device driver. Calling skb_orphan() unconditionally as in this patch would break the hardware time stamping of outgoing packets.

Quoted 8 lines
Quoted 5 lines

Indeed, we need to check that case, at a minimum.

And there are other potentially other problems. For example, I wonder how this interacts with the new TX MMAP af_packet support in net-next-2.6 :-/

I think I'll do this in the driver for now, and let's revisit doing it generically later?

That might be the best course of action for the time being. This whole area is a rat's nest.

#24 lore raw
Quoted 6 lines · David Miller
Quoted 3 lines

I think I'll do this in the driver for now, and let's revisit doing it generically later?

That might be the best course of action for the time being. This whole area is a rat's nest.

Calling skb_orphan like this should be forbidden. Apart from the problems already raised, it is a sign that the driver is trying to paper over a more serious issue of not cleaning up skb's timely.

Yes skb_orphan will work for the cases where calling the skb destructor allows forward progress, but for the cases where you really need to the skb to be freed (e.g., iSCSI or Xen), this simply doesn't work.

So anytime someone tries to propose such a solution it is a sign that they have bigger problems.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#25 lore raw
In reply to David Miller
Quoted 6 lines · David Miller
Quoted 3 lines

I think I'll do this in the driver for now, and let's revisit doing it generically later?

That might be the best course of action for the time being. This whole area is a rat's nest.

Calling skb_orphan like this should be forbidden. Apart from the problems already raised, it is a sign that the driver is trying to paper over a more serious issue of not cleaning up skb's timely.

Yes skb_orphan will work for the cases where calling the skb destructor allows forward progress, but for the cases where you really need to the skb to be freed (e.g., iSCSI or Xen), this simply doesn't work.

So anytime someone tries to propose such a solution it is a sign that they have bigger problems.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
maintainer
#26 lore raw

From: Herbert Xu <[email protected]>
Date: Fri, 3 Jul 2009 15:55:30 +0800

Quoted 11 lines

Calling skb_orphan like this should be forbidden. Apart from the problems already raised, it is a sign that the driver is trying to paper over a more serious issue of not cleaning up skb's timely.

Yes skb_orphan will work for the cases where calling the skb destructor allows forward progress, but for the cases where you really need to the skb to be freed (e.g., iSCSI or Xen), this simply doesn't work.

So anytime someone tries to propose such a solution it is a sign that they have bigger problems.

Agreed, but alas we are foaming at the mouth until we have a truly usable alternative.

In particular the case of handling a device without usable TX completion event indications is still quite troublesome.

#27 lore raw
Quoted 3 lines · David Miller

In particular the case of handling a device without usable TX completion event indications is still quite troublesome.

Which particular devices do you have in mind?

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#28 lore raw
In reply to David Miller
Quoted 3 lines · David Miller

In particular the case of handling a device without usable TX completion event indications is still quite troublesome.

Which particular devices do you have in mind?

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
maintainer
#29 lore raw

From: Herbert Xu <[email protected]>
Date: Sat, 4 Jul 2009 11:08:30 +0800

Quoted 6 lines · David Miller
Quoted 3 lines

In particular the case of handling a device without usable TX completion event indications is still quite troublesome.

e
Which particular devices do you have in mind?

NIU

I basically can't defer interrupts because the chip supports per-TX-desc interrupt indications but it lacks an "all TX queue sent" event. So if, say, tell it to interrupt every 1/4 of the TX queue then up to 1/4 of the queue can have packets "stuck" in there if TX activity all of a sudden ceases.

The only thing I've come up with to be able to mitigate interrupts is to use an hrtimer of some sort. But that's going to be hard to get right, and who knows what kind of latencies will be introduced for TX completion packet freeing unless I am very carefull.

And finally this belongs in generic code, not in the NIU driver, whatever we come up with. Especially since my understanding is that this is similar to what Rusty needs.

#30 lore raw
Quoted 8 lines · David Miller

NIU

I basically can't defer interrupts because the chip supports per-TX-desc interrupt indications but it lacks an "all TX queue sent" event. So if, say, tell it to interrupt every 1/4 of the TX queue then up to 1/4 of the queue can have packets "stuck" in there if TX activity all of a sudden ceases.

Here's an idea: We let the sender decide whether we need to enable notification. This decision would be carried as a flag in the skb. For example, UDP would set this flag when its socket buffer is close to capacity. Routing would set this flag per NAPI run, etc.

Of course you'd ignore this flag completely if the qdisc queue is non-empty.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#31 lore raw
In reply to David Miller
Quoted 8 lines · David Miller

NIU

I basically can't defer interrupts because the chip supports per-TX-desc interrupt indications but it lacks an "all TX queue sent" event. So if, say, tell it to interrupt every 1/4 of the TX queue then up to 1/4 of the queue can have packets "stuck" in there if TX activity all of a sudden ceases.

Here's an idea: We let the sender decide whether we need to enable notification. This decision would be carried as a flag in the skb. For example, UDP would set this flag when its socket buffer is close to capacity. Routing would set this flag per NAPI run, etc.

Of course you'd ignore this flag completely if the qdisc queue is non-empty.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#32 lore raw
Quoted 5 lines · Herbert Xu

Here's an idea: We let the sender decide whether we need to enable notification. This decision would be carried as a flag in the skb. For example, UDP would set this flag when its socket buffer is close to capacity. Routing would set this flag per NAPI run, etc.

Actually it doesn't even matter for routing because only those that are charged by the skb's or the pages care and they're the only ones that would need to set this.

One potential problem is if the socket is constantly running close to capacity, but that should only happen if the device TX queue is also close to capacity which means that the qdisc queue should be non-empty.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#33 lore raw
In reply to Herbert Xu
Quoted 5 lines · Herbert Xu

Here's an idea: We let the sender decide whether we need to enable notification. This decision would be carried as a flag in the skb. For example, UDP would set this flag when its socket buffer is close to capacity. Routing would set this flag per NAPI run, etc.

Actually it doesn't even matter for routing because only those that are charged by the skb's or the pages care and they're the only ones that would need to set this.

One potential problem is if the socket is constantly running close to capacity, but that should only happen if the device TX queue is also close to capacity which means that the qdisc queue should be non-empty.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#34 lore raw
Quoted 5 lines · Herbert Xu

One potential problem is if the socket is constantly running close to capacity, but that should only happen if the device TX queue is also close to capacity which means that the qdisc queue should be non-empty.

Here's a another crazy idea:

Let's use dummy TX descriptors to generate an interrupt, either with or without transmitting an actual packet on the wire depending on the NIC.

xmit(skb)

if (TX queue contains no interrupting descriptor &&
qdisc is empty)
mark TX descriptor as interrupting

clean()

	do work
	if (TX queue contains no interrupting descriptor &&
	    TX queue non-empty &&
	    qdisc is empty)
		send dummy TX descriptor

This is based on the assumption that in the time it takes for the NIC to process one packet and interrupt us, we can generate more packets. Obviously if we can't then even if the NIC had a queue-empty interrupt capability it wouldn't help.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#35 lore raw
Quoted 6 lines · Herbert Xu

Here's a another crazy idea:

Let's use dummy TX descriptors to generate an interrupt, either with or without transmitting an actual packet on the wire depending on the NIC.

Here's an even crazier idea that doesn't use dummy descriptors.

xmit(skb)

if (TX queue contains no interrupting descriptor &&
qdisc is empty)
mark TX descriptor as interrupting

if (TX queue now contains an interrupting descriptor &&
qdisc len < 2)
stop queue

if (TX ring full)
stop queue

clean()

do work
wake queue as per usual

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#36 lore raw
In reply to Herbert Xu
Quoted 6 lines · Herbert Xu

Here's a another crazy idea:

Let's use dummy TX descriptors to generate an interrupt, either with or without transmitting an actual packet on the wire depending on the NIC.

Here's an even crazier idea that doesn't use dummy descriptors.

xmit(skb)

if (TX queue contains no interrupting descriptor &&
qdisc is empty)
mark TX descriptor as interrupting

if (TX queue now contains an interrupting descriptor &&
qdisc len < 2)
stop queue

if (TX ring full)
stop queue

clean()

do work
wake queue as per usual

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
maintainer
#37 lore raw

From: Herbert Xu <[email protected]>
Date: Sun, 5 Jul 2009 11:34:08 +0800

Quoted 12 lines

Here's an even crazier idea that doesn't use dummy descriptors.

xmit(skb)

if (TX queue contains no interrupting descriptor &&
qdisc is empty)
mark TX descriptor as interrupting

if (TX queue now contains an interrupting descriptor &&
qdisc len < 2)
stop queue

Quoted 7 lines

if (TX ring full)
stop queue

clean()

do work
wake queue as per usual

I'm pretty sure that for normal TCP and UDP workloads, this is just going to set the interrupt bit on the first packet that gets into the queue, and then not in the rest.

TCP just loops over packets in the send queue, and at initial state the qdisc will be empty.

It's very hard to get this to work as well as if we had a real queue empty interrupt status event.

Even if you get upstream status from the protocols saying "there's more packets coming" via some flag in the SKB, that only says what one client feeding the TX ring is about to do.

It says nothing about other threads of control which are about to start feeding packets to the same device.

#38 lore raw
Quoted 7 lines · David Miller

I'm pretty sure that for normal TCP and UDP workloads, this is just going to set the interrupt bit on the first packet that gets into the queue, and then not in the rest.

TCP just loops over packets in the send queue, and at initial state the qdisc will be empty.

The scheme I actually tried on the e1000e is not quite the same as what I had here. I'm basically just adding descriptors per usual.

However, I don't give them to the hardware immediately. Instead they're held back so that on average we have about three descriptors in the queue spaced equally that will cause interrupts. It seems to work fairly well for netperf-generated TCP/UDP traffic in terms of generating the right amount of interrupts (once every qlen/3 packets).

I haven't posted anything yet because I ran into a weird problem with the e1000e. It was generating a lot more interrupts than what I'm telling it to do. Even if I hard-code the interrupts at 0, qlen/3, 2qlen/2 I still get way more than qlen/3 interrupts for qlen packets.

This may be related to the fact that e1000e doesn't really have a way of turning the interrupts off for a given descriptor. Well actually it does but it also turns off status reporting which means that we will never know whether that entry has finished processing. So I've had to rely on using its TX interrupt delay mechanism as an approximation of turning interrupt notification off. Evidently that is not working in the way I intended it to.

I'm in the process of repeating the same experiment with cxgb3 which hopefully should let me turn interrupts off on descriptors while still reporting completion status.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#39 lore raw
In reply to David Miller
Quoted 7 lines · David Miller

I'm pretty sure that for normal TCP and UDP workloads, this is just going to set the interrupt bit on the first packet that gets into the queue, and then not in the rest.

TCP just loops over packets in the send queue, and at initial state the qdisc will be empty.

The scheme I actually tried on the e1000e is not quite the same as what I had here. I'm basically just adding descriptors per usual.

However, I don't give them to the hardware immediately. Instead they're held back so that on average we have about three descriptors in the queue spaced equally that will cause interrupts. It seems to work fairly well for netperf-generated TCP/UDP traffic in terms of generating the right amount of interrupts (once every qlen/3 packets).

I haven't posted anything yet because I ran into a weird problem with the e1000e. It was generating a lot more interrupts than what I'm telling it to do. Even if I hard-code the interrupts at 0, qlen/3, 2qlen/2 I still get way more than qlen/3 interrupts for qlen packets.

This may be related to the fact that e1000e doesn't really have a way of turning the interrupts off for a given descriptor. Well actually it does but it also turns off status reporting which means that we will never know whether that entry has finished processing. So I've had to rely on using its TX interrupt delay mechanism as an approximation of turning interrupt notification off. Evidently that is not working in the way I intended it to.

I'm in the process of repeating the same experiment with cxgb3 which hopefully should let me turn interrupts off on descriptors while still reporting completion status.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
maintainer
#40 lore raw

From: Herbert Xu <[email protected]>
Date: Wed, 19 Aug 2009 13:19:26 +1000

Quoted 3 lines

I'm in the process of repeating the same experiment with cxgb3 which hopefully should let me turn interrupts off on descriptors while still reporting completion status.

Ok, I look forward to seeing your work however it turns out.

Once I see what you've done, I'll give it a spin on niu.

maintainer
#41 lore raw
In reply to Herbert Xu

From: Herbert Xu <[email protected]>
Date: Wed, 19 Aug 2009 13:19:26 +1000

Quoted 3 lines

I'm in the process of repeating the same experiment with cxgb3 which hopefully should let me turn interrupts off on descriptors while still reporting completion status.

Ok, I look forward to seeing your work however it turns out.

Once I see what you've done, I'll give it a spin on niu.

maintainer
#42 lore raw
In reply to Herbert Xu

From: Herbert Xu <[email protected]>
Date: Sun, 5 Jul 2009 11:34:08 +0800

Quoted 12 lines

Here's an even crazier idea that doesn't use dummy descriptors.

xmit(skb)

if (TX queue contains no interrupting descriptor &&
qdisc is empty)
mark TX descriptor as interrupting

if (TX queue now contains an interrupting descriptor &&
qdisc len < 2)
stop queue

Quoted 7 lines

if (TX ring full)
stop queue

clean()

do work
wake queue as per usual

I'm pretty sure that for normal TCP and UDP workloads, this is just going to set the interrupt bit on the first packet that gets into the queue, and then not in the rest.

TCP just loops over packets in the send queue, and at initial state the qdisc will be empty.

It's very hard to get this to work as well as if we had a real queue empty interrupt status event.

Even if you get upstream status from the protocols saying "there's more packets coming" via some flag in the SKB, that only says what one client feeding the TX ring is about to do.

It says nothing about other threads of control which are about to start feeding packets to the same device.

maintainer
#43 lore raw
In reply to Herbert Xu

From: Herbert Xu <[email protected]>
Date: Sat, 4 Jul 2009 11:08:30 +0800

Quoted 6 lines · David Miller
Quoted 3 lines

In particular the case of handling a device without usable TX completion event indications is still quite troublesome.

e
Which particular devices do you have in mind?

NIU

I basically can't defer interrupts because the chip supports per-TX-desc interrupt indications but it lacks an "all TX queue sent" event. So if, say, tell it to interrupt every 1/4 of the TX queue then up to 1/4 of the queue can have packets "stuck" in there if TX activity all of a sudden ceases.

The only thing I've come up with to be able to mitigate interrupts is to use an hrtimer of some sort. But that's going to be hard to get right, and who knows what kind of latencies will be introduced for TX completion packet freeing unless I am very carefull.

And finally this belongs in generic code, not in the NIU driver, whatever we come up with. Especially since my understanding is that this is similar to what Rusty needs.

maintainer
#44 lore raw
In reply to Patrick Ohly

From: Patrick Ohly <[email protected]>
Date: Mon, 01 Jun 2009 21:47:22 +0200

Quoted 8 lines · Rusty Russell
Quoted 2 lines

This patch adds skb_orphan to the start of dev_hard_start_xmit(): it can be premature in the NETDEV_TX_BUSY case, but that's uncommon.

Would it be possible to make the new skb_orphan() at the start of dev_hard_start_xmit() conditionally so that it is not executed for packets that are to be time stamped?

Quoted 5 lines

As discussed before
(http://article.gmane.org/gmane.linux.network/121378/), the skb->sk socket pointer is required for sending back the send time stamp from inside the device driver. Calling skb_orphan() unconditionally as in this patch would break the hardware time stamping of outgoing packets.

Indeed, we need to check that case, at a minimum.

And there are other potentially other problems. For example, I wonder how this interacts with the new TX MMAP af_packet support in net-next-2.6 :-/