Skip to contentSkip to conversation

mlx4_en: moderate frequency of TX completions

PATCH In review Yevgeny Petrilin 3 participants 6 messages Feb 23, 2012

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 2/3 mlx4_en: moderate frequency of TX completions

No need to ask for completion for every packet being sent. So the method is to ask for a completion every 16 packets, or when the queue is about to be full.

---
 drivers/net/ethernet/mellanox/mlx4/en_tx.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_tx.c +5 -2
index ff32505..e452bba 100644
@@ -615,6 +615,7 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
 	int lso_header_size;
 	void *fragptr;
 	bool bounce = false;
+	int ring_busy;
 
 	if (!priv->port_up)
 		goto tx_drop;
@@ -638,8 +639,8 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
 		vlan_tag = vlan_tx_tag_get(skb);
 
 	/* Check available TXBBs And 2K spare for prefetch */
-	if (unlikely(((int)(ring->prod - ring->cons)) >
-		     ring->size - HEADROOM - MAX_DESC_TXBBS)) {
+	ring_busy = (int)(ring->prod - ring->cons) - (ring->size - HEADROOM - MAX_DESC_TXBBS);
+	if (unlikely(ring_busy > 0)) {
 		/* every full Tx ring stops queue */
 		netif_tx_stop_queue(netdev_get_tx_queue(dev, tx_ind));
 		ring->blocked = 1;
@@ -680,6 +681,8 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
 		!!vlan_tx_tag_present(skb);
 	tx_desc->ctrl.fence_size = (real_size / 16) & 0x3f;
 	tx_desc->ctrl.srcrb_flags = priv->ctrl_flags;
+	if (!(!(index & 0xf) || vlan_tag || (ring_busy > -MAX_DESC_TXBBS)))
+		tx_desc->ctrl.srcrb_flags &= cpu_to_be32(~MLX4_WQE_CTRL_CQ_UPDATE);
 	if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
 		tx_desc->ctrl.srcrb_flags |= cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM |
 							 MLX4_WQE_CTRL_TCP_UDP_CSUM);
Signature · 1 line
1.5.4.3
maintainer
#1 lore raw

From: Yevgeny Petrilin <[email protected]>
Date: Thu, 23 Feb 2012 15:34:27 +0200

Quoted 5 lines

No need to ask for completion for every packet being sent. So the method is to ask for a completion every 16 packets, or when the queue is about to be full.

You absolutely cannot do this, you must signal completion and free up TX queue packets in a finite amount of time.

This means that if you suddenly stop getting new packets to send you must still free up all the pending TX SKBs even if no more packets are given to the driver.

Does your hardware unconditionally give a TX completion interrupt when the TX queue empties completely? If not, then you cannot make the change contained in this patch.

#2 lore raw
Quoted 11 lines · David Miller

From: Yevgeny Petrilin <[email protected]>
Date: Thu, 23 Feb 2012 15:34:27 +0200

Quoted 5 lines

No need to ask for completion for every packet being sent. So the method is to ask for a completion every 16 packets, or when the queue is about to be full.

You absolutely cannot do this, you must signal completion and free up TX queue packets in a finite amount of time.

Really, sfc has been doing this forever.

Ben.

Quoted 11 lines

This means that if you suddenly stop getting new packets to send you must still free up all the pending TX SKBs even if no more packets are given to the driver.

Does your hardware unconditionally give a TX completion interrupt when the TX queue empties completely? If not, then you cannot make the change contained in this patch.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#3 lore raw
Quoted 8 lines · Ben Hutchings
Quoted 7 lines · David Miller

From: Yevgeny Petrilin <[email protected]>
Date: Thu, 23 Feb 2012 15:34:27 +0200

Quoted 4 lines

No need to ask for completion for every packet being sent. So the method is to ask for a completion every 16 packets, or when the queue is about to be full.

Quoted 16 lines
Quoted 4 lines
Quoted 1 line

You absolutely cannot do this, you must signal completion and free up TX queue packets in a finite amount of time.

Really, sfc has been doing this forever.

Ben.

Quoted 7 lines

This means that if you suddenly stop getting new packets to send you must still free up all the pending TX SKBs even if no more packets are given to the driver.

Does your hardware unconditionally give a TX completion interrupt when the TX queue empties completely? If not, then you cannot make the change contained in this patch.

And it's pretty obvious the hardware has to also send a completion when the queue is empty. I hope.

Ben.

Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
maintainer
#4 lore raw

From: Ben Hutchings <[email protected]>
Date: Thu, 23 Feb 2012 21:26:20 +0000

Quoted 2 lines

And it's pretty obvious the hardware has to also send a completion when the queue is empty. I hope.

Right and we know of several chips that don't.

maintainer
#5 lore raw
In reply to Ben Hutchings

From: Ben Hutchings <[email protected]>
Date: Thu, 23 Feb 2012 21:13:56 +0000

Quoted 8 lines · David Miller
Quoted 7 lines

From: Yevgeny Petrilin <[email protected]>
Date: Thu, 23 Feb 2012 15:34:27 +0200

Quoted 4 lines

No need to ask for completion for every packet being sent. So the method is to ask for a completion every 16 packets, or when the queue is about to be full.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

You absolutely cannot do this, you must signal completion and free up TX queue packets in a finite amount of time.

Really, sfc has been doing this forever.

You have to stop, because otherwise sockets can hang since those packets can hold a reference to the socket so they must be released in a finite amount of time regardless of network traffic.