ixgbe: Add support for multiple Tx queues for FCoE in 82599
Subsystems (2)
PATCH 1/3 ixgbe: Add support for multiple Tx queues for FCoE in 82599
From: Yi Zou <[email protected]>
This patch adds support for multiple transmit queues to the Fiber Channel over Ethernet (FCoE) feature found in 82599. Currently, FCoE has multiple Rx queues available, along with a redirection table, that helps distribute the I/O load across multiple CPUs based on the FC exchange ID. To make this the most effective, we need to provide the same layout of transmit queues to match receive.
Particularly, when Data Center Bridging (DCB) is enabled, the designated traffic class for FCoE can have dedicated queues for just FCoE traffic, while not affecting any other type of traffic flow.
---
drivers/net/ixgbe/ixgbe_dcb_nl.c | 37 +++++++++++++++++++++++++++------ drivers/net/ixgbe/ixgbe_main.c | 43 ++++++++++++++++++++++++++++++-------- 2 files changed, 65 insertions(+), 15 deletions(-)
drivers/net/ixgbe/ixgbe_dcb_nl.c +31 -6
@@ -36,6 +36,7 @@#define BIT_PFC 0x02#define BIT_PG_RX 0x04#define BIT_PG_TX 0x08 +#define BIT_APP_UPCHG 0x10#define BIT_RESETLINK 0x40#define BIT_LINKSPEED 0x80
@@ -348,8 +349,14 @@ static u8 ixgbe_dcbnl_set_all(struct net_device *netdev)while (test_and_set_bit (__IXGBE_RESETTING, &adapter->state))msleep (1 ); -if (netif_running (netdev)) -ixgbe_down (adapter); +if (adapter->dcb_set_bitmap &BIT_APP_UPCHG ) { +if (netif_running (netdev)) + netdev->netdev_ops->ndo_stop (netdev); +ixgbe_clear_interrupt_scheme (adapter); + }else { +if (netif_running (netdev)) +ixgbe_down (adapter); + } } ret = ixgbe_copy_dcb_cfg(&adapter->temp_dcb_cfg, &adapter->dcb_cfg,
@@ -373,8 +380,14 @@ static u8 ixgbe_dcbnl_set_all(struct net_device *netdev) }if (adapter->dcb_set_bitmap &BIT_RESETLINK ) { -if (netif_running (netdev)) -ixgbe_up (adapter); +if (adapter->dcb_set_bitmap &BIT_APP_UPCHG ) { +ixgbe_init_interrupt_scheme (adapter); +if (netif_running (netdev)) + netdev->netdev_ops->ndo_open (netdev); + }else { +if (netif_running (netdev)) +ixgbe_up (adapter); + } ret =DCB_HW_CHG_RST ; }else if (adapter->dcb_set_bitmap &BIT_PFC ) {if (adapter->hw.mac.type == ixgbe_mac_82598EB)
@@ -526,8 +539,20 @@ static u8 ixgbe_dcbnl_setapp(struct net_device *netdev,switch (idtype) {case DCB_APP_IDTYPE_ETHTYPE :#ifdef IXGBE_FCOE -if (id ==ETH_P_FCOE ) - rval =ixgbe_fcoe_setapp (netdev_priv (netdev), up); +if (id ==ETH_P_FCOE ) { +u8 tc; +struct ixgbe_adapter *adapter; + + adapter =netdev_priv (netdev); + tc = adapter->fcoe.tc; + rval =ixgbe_fcoe_setapp (adapter, up); +if ((!rval) && (tc != adapter->fcoe.tc) && + (adapter->flags &IXGBE_FLAG_DCB_ENABLED ) && + (adapter->flags &IXGBE_FLAG_FCOE_ENABLED )) { + adapter->dcb_set_bitmap |=BIT_APP_UPCHG ; + adapter->dcb_set_bitmap |=BIT_RESETLINK ; + } + }#endif break ;case DCB_APP_IDTYPE_PORTNUM :
drivers/net/ixgbe/ixgbe_main.c +34 -9
81 lines not on this page · raw message
PATCH 2/3 ixgbe: Distribute transmission of FCoE traffic in 82599
From: Yi Zou <[email protected]>
This adds a simple selection of a FCoE tx queue based on the current cpu id to distribute transmission of FCoE traffic evenly among multiple FCoE transmit queues.
---
drivers/net/ixgbe/ixgbe_main.c | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-)
drivers/net/ixgbe/ixgbe_main.c +8 -2
@@ -5168,9 +5168,15 @@ static netdev_tx_t ixgbe_xmit_frame(struct sk_buff *skb, tx_ring = &adapter->tx_ring[r_idx];if ((adapter->flags &IXGBE_FLAG_FCOE_ENABLED ) && - (skb->protocol ==htons (ETH_P_FCOE ))) + (skb->protocol ==htons (ETH_P_FCOE ))) { tx_flags |=IXGBE_TX_FLAGS_FCOE ; - +#ifdef IXGBE_FCOE + r_idx =smp_processor_id (); + r_idx &= (adapter->ring_feature[RING_F_FCOE ].indices -1 ); + r_idx += adapter->ring_feature[RING_F_FCOE ].mask; + tx_ring = &adapter->tx_ring[r_idx]; +#endif + }/* four things can cause us to need a context descriptor */ if (skb_is_gso(skb) || (skb->ip_summed == CHECKSUM_PARTIAL) ||
Re: [net-next PATCH 2/3] ixgbe: Distribute transmission of FCoE traffic in 82599
From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:56:10 -0700
Quoted 8 lines
From: Yi Zou <[email protected]>
This adds a simple selection of a FCoE tx queue based on the current cpu id to distribute transmission of FCoE traffic evenly among multiple FCoE transmit queues.
Signed-off-by Yi Zou <[email protected]>Signed-off-by Jeff Kirsher <[email protected]>
Applied.
Does it matter that arbitrary programs or other stacks could transmit ETH_P_FCOE traffic as well? Would that interfere with how this offload hardware works now that you're directing all ETH_P_FCOE traffic to FCOE rings?
Re: [net-next PATCH 2/3] ixgbe: Distribute transmission of FCoE traffic in 82599
Quoted 12 lines · David Miller
From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:56:10 -0700Quoted 8 lines
From: Yi Zou <[email protected]>
This adds a simple selection of a FCoE tx queue based on the current cpu id to distribute transmission of FCoE traffic evenly among multiple FCoE transmit queues.
Signed-off-by Yi Zou <[email protected]>Signed-off-by Jeff Kirsher <[email protected]>
Quoted 6 lines
Applied.
Does it matter that arbitrary programs or other stacks could transmit
ETH_P_FCOEtraffic as well? Would that interfere with how this offload hardware works now that you're directing allETH_P_FCOEtraffic to FCOE rings?
If another stack uses the FCoE Ethertype, the filtering we use in the qdisc layer to filter FCoE frames would assume they belong in the FCoE flow ID in the driver. As long as the other stacks send standard FCoE frames, there won't be a problem. If a stack uses the Ethertype but doesn't follow the standard FCoE frame format, then I'd say that stack was in need of being fixed.
The FCoE offload on Tx in 82599 is basically just a segmenter, like TSO.
Cheers,
-PJ
RE: [net-next PATCH 2/3] ixgbe: Distribute transmission of FCoE traffic in 82599
Quoted 30 lines · David Miller
Quoted 7 lines
From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:56:10 -0700Quoted 3 lines
From: Yi Zou <[email protected]>
This adds a simple selection of a FCoE tx queue based on the current cpu
id to
Quoted 1 line
Quoted 1 line
distribute transmission of FCoE traffic evenly among multiple FCoE
transmit
Quoted 11 lines
Quoted 4 lines
queues.
Signed-off-by Yi Zou <[email protected]>Signed-off-by Jeff Kirsher <[email protected]>Applied.
Does it matter that arbitrary programs or other stacks could transmit
ETH_P_FCOEtraffic as well? Would that interfere with how this offload hardware works now that you're directing allETH_P_FCOEtraffic to FCOE rings?If another stack uses the FCoE Ethertype, the filtering we use in the qdisc layer to filter FCoE frames would assume they belong in the FCoE flow ID in the driver. As long as the other stacks send standard FCoE frames, there won't be a problem. If a stack uses the Ethertype but doesn't follow the standard FCoE frame format, then I'd say that stack was in need of being fixed.
Quoted 4 lines
The FCoE offload on Tx in 82599 is basically just a segmenter, like TSO.
Cheers,
-PJ
Yes, any stack that wants to take advantage of 82599's FCoE offload capability has to fake its traffic as FCoE frames.
Thanks,
yi
PATCH 3/3 ixgbe: Add support for using FCoE DDP in 82599 as FCoE targets
In reply to Jeff KirsherFrom: Yi Zou <[email protected]>
The FCoE DDP in 82599 can be used for both FCoE initiator as well as FCoE target, depending on the indication of the exchange being the responder or originator in the F_CTL (frame control) field in the encapsulated Fiber Channel frame header (T10 Spec., FC-FS). For the initiator, OX_ID is used for FCoE DDP, where for the target RX_ID is used for FCoE DDP.
---
drivers/net/ixgbe/ixgbe_fcoe.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-)
drivers/net/ixgbe/ixgbe_fcoe.c +7 -1
@@ -292,6 +292,7 @@ int ixgbe_fcoe_ddp(struct ixgbe_adapter *adapter, struct sk_buff *skb) {u16 xid; +u32 fctl;u32 sterr, fceofe, fcerr, fcstat;int rc = -EINVAL ;struct ixgbe_fcoe *fcoe;
@@ -312,7 +313,12 @@ int ixgbe_fcoe_ddp(struct ixgbe_adapter *adapter,skb_set_transport_header (skb,skb_network_offset (skb) +sizeof (struct fcoe_hdr )); fh = (struct fc_frame_header *)skb_transport_header (skb); - xid =be16_to_cpu (fh->fh_ox_id); + fctl =ntoh24 (fh->fh_f_ctl); +if (fctl &FC_FC_EX_CTX ) + xid =be16_to_cpu (fh->fh_ox_id); +else + xid =be16_to_cpu (fh->fh_rx_id); +if (xid >=IXGBE_FCOE_DDP_MAX ) goto ddp_out;
Re: [net-next PATCH 3/3] ixgbe: Add support for using FCoE DDP in 82599 as FCoE targets
From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:56:31 -0700
Quoted 10 lines
From: Yi Zou <[email protected]>
The FCoE DDP in 82599 can be used for both FCoE initiator as well as FCoE target, depending on the indication of the exchange being the responder or originator in the
F_CTL(frame control) field in the encapsulated Fiber Channel frame header (T10 Spec., FC-FS). For the initiator,OX_IDis used for FCoE DDP, where for the targetRX_IDis used for FCoE DDP.Signed-off-by Yi Zou <[email protected]>Signed-off-by Jeff Kirsher <[email protected]>
Applied.
From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:55:50 -0700
Quoted 9 lines
From: Yi Zou <[email protected]>
This patch adds support for multiple transmit queues to the Fiber Channel over Ethernet (FCoE) feature found in 82599. Currently, FCoE has multiple Rx queues available, along with a redirection table, that helps distribute the I/O load across multiple CPUs based on the FC exchange ID. To make this the most effective, we need to provide the same layout of transmit queues to match receive.
Quoted 6 lines
Particularly, when Data Center Bridging (DCB) is enabled, the designated traffic class for FCoE can have dedicated queues for just FCoE traffic, while not affecting any other type of traffic flow.
Signed-off-by Yi Zou <[email protected]>Signed-off-by Jeff Kirsher <[email protected]>
Applied.
So does this eat into the TX queues available to non-FCOE traffic or is this a shared setup?
Quoted 16 lines · David Miller
From: Jeff Kirsher <[email protected]>
Date: Thu, 03 Sep 2009 17:55:50 -0700Quoted 13 lines
From: Yi Zou <[email protected]>
This patch adds support for multiple transmit queues to the Fiber Channel over Ethernet (FCoE) feature found in 82599. Currently, FCoE has multiple Rx queues available, along with a redirection table, that helps distribute the I/O load across multiple CPUs based on the FC exchange ID. To make this the most effective, we need to provide the same layout of transmit queues to match receive.
Particularly, when Data Center Bridging (DCB) is enabled, the designated traffic class for FCoE can have dedicated queues for just FCoE traffic, while not affecting any other type of traffic flow.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Yi Zou <[email protected]>Signed-off-by Jeff Kirsher <[email protected]>Applied.
So does this eat into the TX queues available to non-FCOE traffic or is this a shared setup?
When DCB is enabled, then this is purely isolated to the FCoE traffic class. When DCB is enabled, all the queues in the hardware are hard-partitioned. Since our FCoE implementation sets a different MTU, and also disables packet split, to the FCoE queues, these are completely isolated.
Now when DCB is not enabled, 82599 operates in a kind of HBA-ish mode. FCoE and LAN will be running on the same queues, so yes, these queues would be eaten up from the LAN queues.
That all being said, 82599 has 128 Tx queues available. But we only enable 1 queue per online CPU in today's driver without FCoE, so we won't run out of queues (yet).
Cheers,
-PJ