Skip to contentSkip to conversation

ixgbe: Add support for multiple Tx queues for FCoE in 82599

PATCH Applied Jeff Kirsher 4 participants 9 messages Sep 4, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/3 ixgbe: Add support for multiple Tx queues for FCoE in 82599

From: Yi Zou <[email protected]>

This patch adds support for multiple transmit queues to the Fiber Channel over Ethernet (FCoE) feature found in 82599. Currently, FCoE has multiple Rx queues available, along with a redirection table, that helps distribute the I/O load across multiple CPUs based on the FC exchange ID. To make this the most effective, we need to provide the same layout of transmit queues to match receive.

Particularly, when Data Center Bridging (DCB) is enabled, the designated traffic class for FCoE can have dedicated queues for just FCoE traffic, while not affecting any other type of traffic flow.

---

 drivers/net/ixgbe/ixgbe_dcb_nl.c |   37 +++++++++++++++++++++++++++------
 drivers/net/ixgbe/ixgbe_main.c   |   43 ++++++++++++++++++++++++++++++--------
 2 files changed, 65 insertions(+), 15 deletions(-)
drivers/net/ixgbe/ixgbe_dcb_nl.c +31 -6
index 47d9dde..a6bc1ef 100644
@@ -36,6 +36,7 @@
 #define BIT_PFC		0x02
 #define BIT_PG_RX	0x04
 #define BIT_PG_TX	0x08
+#define BIT_APP_UPCHG	0x10
 #define BIT_RESETLINK   0x40
 #define BIT_LINKSPEED   0x80
 
@@ -348,8 +349,14 @@ static u8 ixgbe_dcbnl_set_all(struct net_device *netdev)
 		while (test_and_set_bit(__IXGBE_RESETTING, &adapter->state))
 			msleep(1);
 
-		if (netif_running(netdev))
-			ixgbe_down(adapter);
+		if (adapter->dcb_set_bitmap & BIT_APP_UPCHG) {
+			if (netif_running(netdev))
+				netdev->netdev_ops->ndo_stop(netdev);
+			ixgbe_clear_interrupt_scheme(adapter);
+		} else {
+			if (netif_running(netdev))
+				ixgbe_down(adapter);
+		}
 	}
 
 	ret = ixgbe_copy_dcb_cfg(&adapter->temp_dcb_cfg, &adapter->dcb_cfg,
@@ -373,8 +380,14 @@ static u8 ixgbe_dcbnl_set_all(struct net_device *netdev)
 	}
 
 	if (adapter->dcb_set_bitmap & BIT_RESETLINK) {
-		if (netif_running(netdev))
-			ixgbe_up(adapter);
+		if (adapter->dcb_set_bitmap & BIT_APP_UPCHG) {
+			ixgbe_init_interrupt_scheme(adapter);
+			if (netif_running(netdev))
+				netdev->netdev_ops->ndo_open(netdev);
+		} else {
+			if (netif_running(netdev))
+				ixgbe_up(adapter);
+		}
 		ret = DCB_HW_CHG_RST;
 	} else if (adapter->dcb_set_bitmap & BIT_PFC) {
 		if (adapter->hw.mac.type == ixgbe_mac_82598EB)
@@ -526,8 +539,20 @@ static u8 ixgbe_dcbnl_setapp(struct net_device *netdev,
 	switch (idtype) {
 	case DCB_APP_IDTYPE_ETHTYPE:
 #ifdef IXGBE_FCOE
-		if (id == ETH_P_FCOE)
-			rval = ixgbe_fcoe_setapp(netdev_priv(netdev), up);
+		if (id == ETH_P_FCOE) {
+			u8 tc;
+			struct ixgbe_adapter *adapter;
+
+			adapter = netdev_priv(netdev);
+			tc = adapter->fcoe.tc;
+			rval = ixgbe_fcoe_setapp(adapter, up);
+			if ((!rval) && (tc != adapter->fcoe.tc) &&
+			    (adapter->flags & IXGBE_FLAG_DCB_ENABLED) &&
+			    (adapter->flags & IXGBE_FLAG_FCOE_ENABLED)) {
+				adapter->dcb_set_bitmap |= BIT_APP_UPCHG;
+				adapter->dcb_set_bitmap |= BIT_RESETLINK;
+			}
+		}
 #endif
 		break;
 	case DCB_APP_IDTYPE_PORTNUM:
drivers/net/ixgbe/ixgbe_main.c +34 -9

81 lines not on this page · raw message

#1 lore raw

PATCH 2/3 ixgbe: Distribute transmission of FCoE traffic in 82599

From: Yi Zou <[email protected]>

This adds a simple selection of a FCoE tx queue based on the current cpu id to distribute transmission of FCoE traffic evenly among multiple FCoE transmit queues.

---

 drivers/net/ixgbe/ixgbe_main.c |   10 ++++++++--
 1 files changed, 8 insertions(+), 2 deletions(-)
drivers/net/ixgbe/ixgbe_main.c +8 -2
index 724754a..45bf8b9 100644
@@ -5168,9 +5168,15 @@ static netdev_tx_t ixgbe_xmit_frame(struct sk_buff *skb,
 	tx_ring = &adapter->tx_ring[r_idx];
 
 	if ((adapter->flags & IXGBE_FLAG_FCOE_ENABLED) &&
-	    (skb->protocol == htons(ETH_P_FCOE)))
+	    (skb->protocol == htons(ETH_P_FCOE))) {
 		tx_flags |= IXGBE_TX_FLAGS_FCOE;
-
+#ifdef IXGBE_FCOE
+		r_idx = smp_processor_id();
+		r_idx &= (adapter->ring_feature[RING_F_FCOE].indices - 1);
+		r_idx += adapter->ring_feature[RING_F_FCOE].mask;
+		tx_ring = &adapter->tx_ring[r_idx];
+#endif
+	}
 	/* four things can cause us to need a context descriptor */
 	if (skb_is_gso(skb) ||
 	    (skb->ip_summed == CHECKSUM_PARTIAL) ||
maintainer Applied
#2 lore raw

Re: [net-next PATCH 2/3] ixgbe: Distribute transmission of FCoE traffic in 82599

From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:56:10 -0700

Quoted 8 lines

From: Yi Zou <[email protected]>

This adds a simple selection of a FCoE tx queue based on the current cpu id to distribute transmission of FCoE traffic evenly among multiple FCoE transmit queues.

Applied.

Does it matter that arbitrary programs or other stacks could transmit ETH_P_FCOE traffic as well? Would that interfere with how this offload hardware works now that you're directing all ETH_P_FCOE traffic to FCOE rings?

#3 lore raw

Re: [net-next PATCH 2/3] ixgbe: Distribute transmission of FCoE traffic in 82599

Quoted 12 lines · David Miller

From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:56:10 -0700

Quoted 8 lines

From: Yi Zou <[email protected]>

This adds a simple selection of a FCoE tx queue based on the current cpu id to distribute transmission of FCoE traffic evenly among multiple FCoE transmit queues.

Quoted 6 lines

Applied.

Does it matter that arbitrary programs or other stacks could transmit ETH_P_FCOE traffic as well? Would that interfere with how this offload hardware works now that you're directing all ETH_P_FCOE traffic to FCOE rings?

If another stack uses the FCoE Ethertype, the filtering we use in the qdisc layer to filter FCoE frames would assume they belong in the FCoE flow ID in the driver. As long as the other stacks send standard FCoE frames, there won't be a problem. If a stack uses the Ethertype but doesn't follow the standard FCoE frame format, then I'd say that stack was in need of being fixed.

The FCoE offload on Tx in 82599 is basically just a segmenter, like TSO.

Cheers,
-PJ

#4 lore raw

RE: [net-next PATCH 2/3] ixgbe: Distribute transmission of FCoE traffic in 82599

Quoted 30 lines · David Miller
Quoted 7 lines

From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:56:10 -0700

Quoted 3 lines

From: Yi Zou <[email protected]>

This adds a simple selection of a FCoE tx queue based on the current cpu

id to

Quoted 1 line
Quoted 1 line

distribute transmission of FCoE traffic evenly among multiple FCoE

transmit

Quoted 11 lines
Quoted 4 lines

queues.

Applied.

Does it matter that arbitrary programs or other stacks could transmit ETH_P_FCOE traffic as well? Would that interfere with how this offload hardware works now that you're directing all ETH_P_FCOE traffic to FCOE rings?

If another stack uses the FCoE Ethertype, the filtering we use in the qdisc layer to filter FCoE frames would assume they belong in the FCoE flow ID in the driver. As long as the other stacks send standard FCoE frames, there won't be a problem. If a stack uses the Ethertype but doesn't follow the standard FCoE frame format, then I'd say that stack was in need of being fixed.

Quoted 4 lines

The FCoE offload on Tx in 82599 is basically just a segmenter, like TSO.

Cheers,
-PJ

Yes, any stack that wants to take advantage of 82599's FCoE offload capability has to fake its traffic as FCoE frames.

Thanks,
yi

#5 lore raw

PATCH 3/3 ixgbe: Add support for using FCoE DDP in 82599 as FCoE targets

In reply to Jeff Kirsher

From: Yi Zou <[email protected]>

The FCoE DDP in 82599 can be used for both FCoE initiator as well as FCoE target, depending on the indication of the exchange being the responder or originator in the F_CTL (frame control) field in the encapsulated Fiber Channel frame header (T10 Spec., FC-FS). For the initiator, OX_ID is used for FCoE DDP, where for the target RX_ID is used for FCoE DDP.

---

 drivers/net/ixgbe/ixgbe_fcoe.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)
drivers/net/ixgbe/ixgbe_fcoe.c +7 -1
index 0607cff..a3c9f99 100644
@@ -292,6 +292,7 @@ int ixgbe_fcoe_ddp(struct ixgbe_adapter *adapter,
 		   struct sk_buff *skb)
 {
 	u16 xid;
+	u32 fctl;
 	u32 sterr, fceofe, fcerr, fcstat;
 	int rc = -EINVAL;
 	struct ixgbe_fcoe *fcoe;
@@ -312,7 +313,12 @@ int ixgbe_fcoe_ddp(struct ixgbe_adapter *adapter,
 	skb_set_transport_header(skb, skb_network_offset(skb) +
 				 sizeof(struct fcoe_hdr));
 	fh = (struct fc_frame_header *)skb_transport_header(skb);
-	xid =  be16_to_cpu(fh->fh_ox_id);
+	fctl = ntoh24(fh->fh_f_ctl);
+	if (fctl & FC_FC_EX_CTX)
+		xid =  be16_to_cpu(fh->fh_ox_id);
+	else
+		xid =  be16_to_cpu(fh->fh_rx_id);
+
 	if (xid >= IXGBE_FCOE_DDP_MAX)
 		goto ddp_out;
 
maintainer Applied
#6 lore raw

Re: [net-next PATCH 3/3] ixgbe: Add support for using FCoE DDP in 82599 as FCoE targets

From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:56:31 -0700

Quoted 10 lines

From: Yi Zou <[email protected]>

The FCoE DDP in 82599 can be used for both FCoE initiator as well as FCoE target, depending on the indication of the exchange being the responder or originator in the F_CTL (frame control) field in the encapsulated Fiber Channel frame header (T10 Spec., FC-FS). For the initiator, OX_ID is used for FCoE DDP, where for the target RX_ID is used for FCoE DDP.

Applied.

maintainer Applied
#7 lore raw
In reply to Jeff Kirsher

From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:55:50 -0700

Quoted 9 lines

From: Yi Zou <[email protected]>

This patch adds support for multiple transmit queues to the Fiber Channel over Ethernet (FCoE) feature found in 82599. Currently, FCoE has multiple Rx queues available, along with a redirection table, that helps distribute the I/O load across multiple CPUs based on the FC exchange ID. To make this the most effective, we need to provide the same layout of transmit queues to match receive.

Quoted 6 lines

Particularly, when Data Center Bridging (DCB) is enabled, the designated traffic class for FCoE can have dedicated queues for just FCoE traffic, while not affecting any other type of traffic flow.

Applied.

So does this eat into the TX queues available to non-FCOE traffic or is this a shared setup?

#8 lore raw
Quoted 16 lines · David Miller

From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:55:50 -0700

Quoted 13 lines

From: Yi Zou <[email protected]>

This patch adds support for multiple transmit queues to the Fiber Channel over Ethernet (FCoE) feature found in 82599. Currently, FCoE has multiple Rx queues available, along with a redirection table, that helps distribute the I/O load across multiple CPUs based on the FC exchange ID. To make this the most effective, we need to provide the same layout of transmit queues to match receive.

Particularly, when Data Center Bridging (DCB) is enabled, the designated traffic class for FCoE can have dedicated queues for just FCoE traffic, while not affecting any other type of traffic flow.

Quoted 7 lines
Quoted 2 lines

Applied.

So does this eat into the TX queues available to non-FCOE traffic or is this a shared setup?

When DCB is enabled, then this is purely isolated to the FCoE traffic class. When DCB is enabled, all the queues in the hardware are hard-partitioned. Since our FCoE implementation sets a different MTU, and also disables packet split, to the FCoE queues, these are completely isolated.

Now when DCB is not enabled, 82599 operates in a kind of HBA-ish mode. FCoE and LAN will be running on the same queues, so yes, these queues would be eaten up from the LAN queues.

That all being said, 82599 has 128 Tx queues available. But we only enable 1 queue per online CPU in today's driver without FCoE, so we won't run out of queues (yet).

Cheers,
-PJ