Skip to contentSkip to conversation

Add support for 4 queues VFs with 1 or 2 queue RSS on PF

PATCH New Alexander Duyck 3 participants 12 messages Sep 8, 2016 – Sep 10, 2016

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Add support for 4 queues VFs with 1 or 2 queue RSS on PF

This patch set addresses a few minor issues and allows us to enable 4 queue RSS on the VFs if the PF if configured for less than 4 queues. I found and fixed at least 2 buts with the first 2 patches in this set, and the third patch enables us to have 4 queues with RSS assuming that we have not exceeded 32 total VM pools and DCB is not enabled.

---

Alexander Duyck (3):
ixgbe: Allow setting multiple queues when SR-IOV is enabled
ixgbe: Limit reporting of redirection table if SR-IOV is enabled
ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF

 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |   10 +++++++---
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c     |    7 ++++---
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |   12 +++++++-----
 3 files changed, 18 insertions(+), 11 deletions(-)

--

#1 lore raw

PATCH 1/3 ixgbe: Allow setting multiple queues when SR-IOV is enabled

From: Alexander Duyck <[email protected]>

The maximum queue count reported was 1, however support for multiple queues with SR-IOV was added some time ago so we should report support for it to the user so that they can select multiple queues if they so desire.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +2 -2
index 730a99f0f002..2d872be336bb 100644
@@ -3060,8 +3060,8 @@ static unsigned int ixgbe_max_channels(struct ixgbe_adapter *adapter)
 		/* We only support one q_vector without MSI-X */
 		max_combined = 1;
 	} else if (adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) {
-		/* SR-IOV currently only allows one queue on the PF */
-		max_combined = 1;
+		/* Limit value based on the queue mask */
+		max_combined = adapter->ring_feature[RING_F_RSS].mask + 1;
 	} else if (tcs > 1) {
 		/* For DCB report channels per traffic class */
 		if (adapter->hw.mac.type == ixgbe_mac_82598EB) {
#2 lore raw

PATCH 1/3 ixgbe: Allow setting multiple queues when SR-IOV is enabled

Quoted 13 lines

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Wednesday, September 07, 2016 8:28 PM
To: intel-wired-lan at lists.osuosl.org
Cc: ruslan at purestorage.com; Jayakumar, Muthurajan <[email protected]>; Blevins, Christopher R <[email protected]>
Subject: [Intel-wired-lan] [next PATCH 1/3] ixgbe: Allow setting multiple queues when SR-IOV is enabled

From: Alexander Duyck <[email protected]>

Quoted 8 lines

The maximum queue count reported was 1, however support for multiple queues with SR-IOV was added some time ago so we should report support for it to the user so that they can select multiple queues if they so desire.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
#3 lore raw

PATCH 2/3 ixgbe: Limit reporting of redirection table if SR-IOV is enabled

In reply to Alexander Duyck

From: Alexander Duyck <[email protected]>

The hardware redirection table can support more queues then the PF currently has when SR-IOV is enabled. In order to account for this use the RSS mask to trim of the bits that are not used.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +5 -1
index 2d872be336bb..f49f80380aa5 100644
@@ -2947,9 +2947,13 @@ static u32 ixgbe_rss_indir_size(struct net_device *netdev)
 static void ixgbe_get_reta(struct ixgbe_adapter *adapter, u32 *indir)
 {
 	int i, reta_size = ixgbe_rss_indir_tbl_entries(adapter);
+	u16 rss_m = adapter->ring_feature[RING_F_RSS].mask;
+
+	if (adapter->flags & IXGBE_FLAG_SRIOV_ENABLED)
+		rss_m = adapter->ring_feature[RING_F_RSS].indices - 1;
 
 	for (i = 0; i < reta_size; i++)
-		indir[i] = adapter->rss_indir_tbl[i];
+		indir[i] = adapter->rss_indir_tbl[i] & rss_m;
 }
 
 static int ixgbe_get_rxfh(struct net_device *netdev, u32 *indir, u8 *key,
#4 lore raw

PATCH 2/3 ixgbe: Limit reporting of redirection table if SR-IOV is enabled

Quoted 13 lines

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Wednesday, September 07, 2016 8:28 PM
To: intel-wired-lan at lists.osuosl.org
Cc: ruslan at purestorage.com; Jayakumar, Muthurajan <[email protected]>; Blevins, Christopher R <[email protected]>
Subject: [Intel-wired-lan] [next PATCH 2/3] ixgbe: Limit reporting of redirection table if SR-IOV is enabled

From: Alexander Duyck <[email protected]>

Quoted 8 lines

The hardware redirection table can support more queues then the PF currently has when SR-IOV is enabled. In order to account for this use the RSS mask to trim of the bits that are not used.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
#5 lore raw

PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF

In reply to Alexander Duyck

From: Alexander Duyck <[email protected]>

Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.

The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c  |    7 ++++---
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c |   12 +++++++-----
 2 files changed, 11 insertions(+), 8 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4 -3
index bcdc88444ceb..15ab337fd7ad 100644
@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter)
 	vmdq_i = min_t(u16, IXGBE_MAX_VMDQ_INDICES, vmdq_i);
 
 	/* 64 pool mode with 2 queues per pool */
-	if ((vmdq_i > 32) || (rss_i < 4) || (vmdq_i > 16 && pools)) {
+	if ((vmdq_i > 32) || (vmdq_i > 16 && pools)) {
 		vmdq_m = IXGBE_82599_VMDQ_2Q_MASK;
 		rss_m = IXGBE_RSS_2Q_MASK;
 		rss_i = min_t(u16, rss_i, 2);
-	/* 32 pool mode with 4 queues per pool */
+	/* 32 pool mode with up to 4 queues per pool */
 	} else {
 		vmdq_m = IXGBE_82599_VMDQ_4Q_MASK;
 		rss_m = IXGBE_RSS_4Q_MASK;
-		rss_i = 4;
+		/* We can support 4, 2, or 1 queues */
+		rss_i = (rss_i > 3) ? 4 : (rss_i > 1) ? 2 : 1;
 	}
 
 #ifdef IXGBE_FCOE
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +7 -5
index 1c888588cecd..a244d9a67264 100644
@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter *adapter)
 			mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_8TC_8TQ;
 		else if (tcs > 1)
 			mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_4TC_4TQ;
-		else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+		else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+			 IXGBE_82599_VMDQ_4Q_MASK)
 			mtqc |= IXGBE_MTQC_32VF;
 		else
 			mtqc |= IXGBE_MTQC_64VF;
@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct ixgbe_adapter *adapter)
 	u32 reta_entries = ixgbe_rss_indir_tbl_entries(adapter);
 	u16 rss_i = adapter->ring_feature[RING_F_RSS].indices;
 
-	/* Program table for at least 2 queues w/ SR-IOV so that VFs can
+	/* Program table for at least 4 queues w/ SR-IOV so that VFs can
 	 * make full use of any rings they may have.  We will use the
 	 * PSRTYPE register to control how many rings we use within the PF.
 	 */
-	if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 2))
-		rss_i = 2;
+	if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 4))
+		rss_i = 4;
 
 	/* Fill out hash function seeds */
 	for (i = 0; i < 10; i++)
@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter *adapter)
 				mrqc = IXGBE_MRQC_VMDQRT8TCEN;	/* 8 TCs */
 			else if (tcs > 1)
 				mrqc = IXGBE_MRQC_VMDQRT4TCEN;	/* 4 TCs */
-			else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+			else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+				 IXGBE_82599_VMDQ_4Q_MASK)
 				mrqc = IXGBE_MRQC_VMDQRSS32EN;
 			else
 				mrqc = IXGBE_MRQC_VMDQRSS64EN;
#6 lore raw

PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF

Thank you very much for giving the feedback and creating new set of patches!

Ruslan

Quoted 83 lines · Alexander Duyck

From: Alexander Duyck <[email protected]>

Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.

The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.

---
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 7 ++++---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 12 +++++++-----

2 files changed, 11 insertions(+), 8 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4 -3
index bcdc88444ceb..15ab337fd7ad 100644
@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter)
 	vmdq_i = min_t(u16, IXGBE_MAX_VMDQ_INDICES, vmdq_i);
 
 	/* 64 pool mode with 2 queues per pool */
-	if ((vmdq_i > 32) || (rss_i < 4) || (vmdq_i > 16 && pools)) {
+	if ((vmdq_i > 32) || (vmdq_i > 16 && pools)) {
 		vmdq_m = IXGBE_82599_VMDQ_2Q_MASK;
 		rss_m = IXGBE_RSS_2Q_MASK;
 		rss_i = min_t(u16, rss_i, 2);
-	/* 32 pool mode with 4 queues per pool */
+	/* 32 pool mode with up to 4 queues per pool */
 	} else {
 		vmdq_m = IXGBE_82599_VMDQ_4Q_MASK;
 		rss_m = IXGBE_RSS_4Q_MASK;
-		rss_i = 4;
+		/* We can support 4, 2, or 1 queues */
+		rss_i = (rss_i > 3) ? 4 : (rss_i > 1) ? 2 : 1;
 	}
 
 #ifdef IXGBE_FCOE
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +7 -5
index 1c888588cecd..a244d9a67264 100644
@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter *adapter)
 			mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_8TC_8TQ;
 		else if (tcs > 1)
 			mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_4TC_4TQ;
-		else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+		else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+			 IXGBE_82599_VMDQ_4Q_MASK)
 			mtqc |= IXGBE_MTQC_32VF;
 		else
 			mtqc |= IXGBE_MTQC_64VF;
@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct ixgbe_adapter *adapter)
 	u32 reta_entries = ixgbe_rss_indir_tbl_entries(adapter);
 	u16 rss_i = adapter->ring_feature[RING_F_RSS].indices;
 
-	/* Program table for at least 2 queues w/ SR-IOV so that VFs can
+	/* Program table for at least 4 queues w/ SR-IOV so that VFs can
 	 * make full use of any rings they may have.  We will use the
 	 * PSRTYPE register to control how many rings we use within the PF.
 	 */
-	if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 2))
-		rss_i = 2;
+	if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 4))
+		rss_i = 4;
 
 	/* Fill out hash function seeds */
 	for (i = 0; i < 10; i++)
@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter *adapter)
 				mrqc = IXGBE_MRQC_VMDQRT8TCEN;	/* 8 TCs */
 			else if (tcs > 1)
 				mrqc = IXGBE_MRQC_VMDQRT4TCEN;	/* 4 TCs */
-			else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+			else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+				 IXGBE_82599_VMDQ_4Q_MASK)
 				mrqc = IXGBE_MRQC_VMDQRSS32EN;
 			else
 				mrqc = IXGBE_MRQC_VMDQRSS64EN;
#7 lore raw

PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF

I still have on more question. There is also ixgbe_setup_vfreta function in the code. Do we need to adjust rss_i there as well?

Thanks,
Ruslan

Quoted 76 lines · Ruslan Nikolaev

Thank you very much for giving the feedback and creating new set of patches!

Ruslan

Quoted 70 lines · Alexander Duyck

From: Alexander Duyck <[email protected]>

Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.

The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.

---
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 7 ++++---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 12 +++++++-----

2 files changed, 11 insertions(+), 8 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4 -3
index bcdc88444ceb..15ab337fd7ad 100644
@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter)
 	vmdq_i = min_t(u16, IXGBE_MAX_VMDQ_INDICES, vmdq_i);
 
 	/* 64 pool mode with 2 queues per pool */
-	if ((vmdq_i > 32) || (rss_i < 4) || (vmdq_i > 16 && pools)) {
+	if ((vmdq_i > 32) || (vmdq_i > 16 && pools)) {
 		vmdq_m = IXGBE_82599_VMDQ_2Q_MASK;
 		rss_m = IXGBE_RSS_2Q_MASK;
 		rss_i = min_t(u16, rss_i, 2);
-	/* 32 pool mode with 4 queues per pool */
+	/* 32 pool mode with up to 4 queues per pool */
 	} else {
 		vmdq_m = IXGBE_82599_VMDQ_4Q_MASK;
 		rss_m = IXGBE_RSS_4Q_MASK;
-		rss_i = 4;
+		/* We can support 4, 2, or 1 queues */
+		rss_i = (rss_i > 3) ? 4 : (rss_i > 1) ? 2 : 1;
 	}
 
 #ifdef IXGBE_FCOE
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +5 -4
index 1c888588cecd..a244d9a67264 100644
@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter *adapter)
 			mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_8TC_8TQ;
 		else if (tcs > 1)
 			mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_4TC_4TQ;
-		else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+		else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+			 IXGBE_82599_VMDQ_4Q_MASK)
 			mtqc |= IXGBE_MTQC_32VF;
 		else
 			mtqc |= IXGBE_MTQC_64VF;
@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct ixgbe_adapter *adapter)
 	u32 reta_entries = ixgbe_rss_indir_tbl_entries(adapter);
 	u16 rss_i = adapter->ring_feature[RING_F_RSS].indices;
 
-	/* Program table for at least 2 queues w/ SR-IOV so that VFs can
+	/* Program table for at least 4 queues w/ SR-IOV so that VFs can
 	 * make full use of any rings they may have.  We will use the
 	 * PSRTYPE register to control how many rings we use within the PF.
 	 */
-	if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 2))
-		rss_i = 2;
+	if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 4))
+		rss_i = 4;
 
Quoted 14 lines

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c static void ixgbe_setup_mrqc(struct ixgbe_adapter *adapter) · line 3552

Quoted 13 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +2 -1
 	/* Fill out hash function seeds */
 	for (i = 0; i < 10; i++)
@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter *adapter)
 				mrqc = IXGBE_MRQC_VMDQRT8TCEN;	/* 8 TCs */
 			else if (tcs > 1)
 				mrqc = IXGBE_MRQC_VMDQRT4TCEN;	/* 4 TCs */
-			else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+			else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+				 IXGBE_82599_VMDQ_4Q_MASK)
 				mrqc = IXGBE_MRQC_VMDQRSS32EN;
 			else
 				mrqc = IXGBE_MRQC_VMDQRSS64EN;

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20160908/07ad9b96/attachment-0001.html>

#8 lore raw

PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF

That shouldn't be needed since the VFRETA is actually per virtual pool so it isn't shared with the VFs. In theory we could support 3 queues on the X550 without any issues, but that would be a change of how things are currently handled so I figured I will leave it as it is for now.

- Alex

Quoted 112 lines · Ruslan Nikolaev

I still have on more question. There is also ixgbe_setup_vfreta function in the code. Do we need to adjust rss_i there as well?

Thanks,
Ruslan

On Sep 8, 2016, at 11:20 AM, Ruslan Nikolaev <[email protected]> wrote:

Thank you very much for giving the feedback and creating new set of patches!

Ruslan

On Sep 7, 2016, at 8:28 PM, Alexander Duyck <[email protected]> wrote:

From: Alexander Duyck <[email protected]>

Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.

The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.

---
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 7 ++++---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 12 +++++++-----

2 files changed, 11 insertions(+), 8 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4 -3
index bcdc88444ceb..15ab337fd7ad 100644
@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct
 ixgbe_adapter *adapter)
 vmdq_i = min_t(u16, IXGBE_MAX_VMDQ_INDICES, vmdq_i);
 
 /* 64 pool mode with 2 queues per pool */
- if ((vmdq_i > 32) || (rss_i < 4) || (vmdq_i > 16 && pools)) {
+ if ((vmdq_i > 32) || (vmdq_i > 16 && pools)) {
 vmdq_m = IXGBE_82599_VMDQ_2Q_MASK;
 rss_m = IXGBE_RSS_2Q_MASK;
 rss_i = min_t(u16, rss_i, 2);
- /* 32 pool mode with 4 queues per pool */
+ /* 32 pool mode with up to 4 queues per pool */
 } else {
 vmdq_m = IXGBE_82599_VMDQ_4Q_MASK;
 rss_m = IXGBE_RSS_4Q_MASK;
- rss_i = 4;
+ /* We can support 4, 2, or 1 queues */
+ rss_i = (rss_i > 3) ? 4 : (rss_i > 1) ? 2 : 1;
 }
 

#ifdef IXGBE_FCOE

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +2 -1
index 1c888588cecd..a244d9a67264 100644
@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter
 *adapter)
 mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_8TC_8TQ;
 else if (tcs > 1)
 mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_4TC_4TQ;
- else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+ else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+ IXGBE_82599_VMDQ_4Q_MASK)
 mtqc |= IXGBE_MTQC_32VF;

else
mtqc |= IXGBE_MTQC_64VF;

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +3 -3
@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct ixgbe_adapter
 *adapter)
 u32 reta_entries = ixgbe_rss_indir_tbl_entries(adapter);
 u16 rss_i = adapter->ring_feature[RING_F_RSS].indices;
 
- /* Program table for at least 2 queues w/ SR-IOV so that VFs can
+ /* Program table for at least 4 queues w/ SR-IOV so that VFs can
 * make full use of any rings they may have.  We will use the
 * PSRTYPE register to control how many rings we use within the PF.
 */
- if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 2))
- rss_i = 2;
+ if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 4))
+ rss_i = 4;
 
 /* Fill out hash function seeds */

for (i = 0; i < 10; i++)

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +0 -0
@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter
 *adapter)

mrqc = IXGBE_MRQC_VMDQRT8TCEN; /* 8 TCs */
else if (tcs > 1)
mrqc = IXGBE_MRQC_VMDQRT4TCEN; /* 4 TCs */
- else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+ else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+ IXGBE_82599_VMDQ_4Q_MASK)
mrqc = IXGBE_MRQC_VMDQRSS32EN;
else
mrqc = IXGBE_MRQC_VMDQRSS64EN;

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at lists.osuosl.org
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#9 lore raw

PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF

Sorry in advance if I have any misconception regarding how VFRETA works. But I was more thinking about rss_i being smaller than needed. For instance, rss_i is 1 (because RSS=1 for PF) but we use VFs with larger number of RX queues (RSS=2 or RSS=4 for VF). Should we program the table for VFs using value 2 (or 4) that we at least support 2 RX queues (or 4 RX queues if interrupts are shared)? I guess, it does not matter when we use just one RX queue.

Thanks,
Ruslan

Quoted 121 lines · Alexander Duyck

That shouldn't be needed since the VFRETA is actually per virtual pool so it isn't shared with the VFs. In theory we could support 3 queues on the X550 without any issues, but that would be a change of how things are currently handled so I figured I will leave it as it is for now.

- Alex

Quoted 112 lines · Ruslan Nikolaev

I still have on more question. There is also ixgbe_setup_vfreta function in the code. Do we need to adjust rss_i there as well?

Thanks,
Ruslan

On Sep 8, 2016, at 11:20 AM, Ruslan Nikolaev <[email protected]> wrote:

Thank you very much for giving the feedback and creating new set of patches!

Ruslan

On Sep 7, 2016, at 8:28 PM, Alexander Duyck <[email protected]> wrote:

From: Alexander Duyck <[email protected]>

Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.

The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.

---
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 7 ++++---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 12 +++++++-----

2 files changed, 11 insertions(+), 8 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4 -3
index bcdc88444ceb..15ab337fd7ad 100644
@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct
 ixgbe_adapter *adapter)
 vmdq_i = min_t(u16, IXGBE_MAX_VMDQ_INDICES, vmdq_i);
 
 /* 64 pool mode with 2 queues per pool */
- if ((vmdq_i > 32) || (rss_i < 4) || (vmdq_i > 16 && pools)) {
+ if ((vmdq_i > 32) || (vmdq_i > 16 && pools)) {
 vmdq_m = IXGBE_82599_VMDQ_2Q_MASK;
 rss_m = IXGBE_RSS_2Q_MASK;
 rss_i = min_t(u16, rss_i, 2);
- /* 32 pool mode with 4 queues per pool */
+ /* 32 pool mode with up to 4 queues per pool */
 } else {
 vmdq_m = IXGBE_82599_VMDQ_4Q_MASK;
 rss_m = IXGBE_RSS_4Q_MASK;
- rss_i = 4;
+ /* We can support 4, 2, or 1 queues */
+ rss_i = (rss_i > 3) ? 4 : (rss_i > 1) ? 2 : 1;
 }
 

#ifdef IXGBE_FCOE

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +2 -1
index 1c888588cecd..a244d9a67264 100644
@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter
 *adapter)
 mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_8TC_8TQ;
 else if (tcs > 1)
 mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_4TC_4TQ;
- else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+ else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+ IXGBE_82599_VMDQ_4Q_MASK)
 mtqc |= IXGBE_MTQC_32VF;

else
mtqc |= IXGBE_MTQC_64VF;

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +3 -3
@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct ixgbe_adapter
 *adapter)
 u32 reta_entries = ixgbe_rss_indir_tbl_entries(adapter);
 u16 rss_i = adapter->ring_feature[RING_F_RSS].indices;
 
- /* Program table for at least 2 queues w/ SR-IOV so that VFs can
+ /* Program table for at least 4 queues w/ SR-IOV so that VFs can
 * make full use of any rings they may have.  We will use the
 * PSRTYPE register to control how many rings we use within the PF.
 */
- if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 2))
- rss_i = 2;
+ if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 4))
+ rss_i = 4;
 
 /* Fill out hash function seeds */

for (i = 0; i < 10; i++)

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +0 -0
@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter
 *adapter)

mrqc = IXGBE_MRQC_VMDQRT8TCEN; /* 8 TCs */
else if (tcs > 1)
mrqc = IXGBE_MRQC_VMDQRT4TCEN; /* 4 TCs */
- else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+ else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+ IXGBE_82599_VMDQ_4Q_MASK)
mrqc = IXGBE_MRQC_VMDQRSS32EN;
else
mrqc = IXGBE_MRQC_VMDQRSS64EN;

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at lists.osuosl.org
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#10 lore raw

PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF

The VFRETA can be programmed as you like. Each VF has its own table so PF config will not impact the VFs. All it impacts is the local pool the PF is using for its own traffic. So if you write it to all zeros that will set it up for only one queue. As long as none of the VFs exceed the number of queues the VF has it shouldn't be a problem.

- Alex

Quoted 140 lines · Ruslan Nikolaev

Sorry in advance if I have any misconception regarding how VFRETA works. But I was more thinking about rss_i being smaller than needed. For instance, rss_i is 1 (because RSS=1 for PF) but we use VFs with larger number of RX queues (RSS=2 or RSS=4 for VF). Should we program the table for VFs using value 2 (or 4) that we at least support 2 RX queues (or 4 RX queues if interrupts are shared)? I guess, it does not matter when we use just one RX queue.

Thanks,
Ruslan

Quoted 9 lines · On Sep 9, 2016, at 8:32 AM, Alexander Duyck <[email protected]…

That shouldn't be needed since the VFRETA is actually per virtual pool so it isn't shared with the VFs. In theory we could support 3 queues on the X550 without any issues, but that would be a change of how things are currently handled so I figured I will leave it as it is for now.

- Alex

On Thu, Sep 8, 2016 at 6:14 PM, Ruslan Nikolaev <[email protected]

Quoted 1 line · <javascript:;>> wrote:
Quoted 1 line

I still have on more question. There is also ixgbe_setup_vfreta

function in

Quoted 6 lines
Quoted 6 lines

the code. Do we need to adjust rss_i there as well?

Thanks,
Ruslan

On Sep 8, 2016, at 11:20 AM, Ruslan Nikolaev <[email protected]

Quoted 2 lines · <javascript:;>> wrote:
Quoted 2 lines

Thank you very much for giving the feedback and creating new set of

patches!

Quoted 4 lines
Quoted 4 lines

Ruslan

On Sep 7, 2016, at 8:28 PM, Alexander Duyck <[email protected]

<javascript:;>>

Quoted 5 lines
Quoted 5 lines

wrote:

From: Alexander Duyck <[email protected] <javascript:;>>

Instead of limiting the VFs if we don't use 4 queues for RSS in the PF

we

Quoted 1 line
Quoted 1 line

can instead just limit the RSS queues used to a power of 2. By doing

this

Quoted 5 lines
Quoted 5 lines

we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.

The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues

in

Quoted 4 lines
Quoted 4 lines

order to be able to use the power of 2 masking provided by the psrtype register.

<javascript:;>>

Quoted 40 lines
Quoted 40 lines

---
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 7 ++++---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 12 +++++++-----

2 files changed, 11 insertions(+), 8 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4 -3
index bcdc88444ceb..15ab337fd7ad 100644
@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct
 ixgbe_adapter *adapter)
 vmdq_i = min_t(u16, IXGBE_MAX_VMDQ_INDICES, vmdq_i);
 
 /* 64 pool mode with 2 queues per pool */
- if ((vmdq_i > 32) || (rss_i < 4) || (vmdq_i > 16 && pools)) {
+ if ((vmdq_i > 32) || (vmdq_i > 16 && pools)) {
 vmdq_m = IXGBE_82599_VMDQ_2Q_MASK;
 rss_m = IXGBE_RSS_2Q_MASK;
 rss_i = min_t(u16, rss_i, 2);
- /* 32 pool mode with 4 queues per pool */
+ /* 32 pool mode with up to 4 queues per pool */
 } else {
 vmdq_m = IXGBE_82599_VMDQ_4Q_MASK;
 rss_m = IXGBE_RSS_4Q_MASK;
- rss_i = 4;
+ /* We can support 4, 2, or 1 queues */
+ rss_i = (rss_i > 3) ? 4 : (rss_i > 1) ? 2 : 1;
 }
 

#ifdef IXGBE_FCOE

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +0 -0
index 1c888588cecd..a244d9a67264 100644
@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter
 *adapter)
 mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_8TC_8TQ;
 else if (tcs > 1)
Quoted 8 lines

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c static void ixgbe_setup_reta(struct

Quoted 8 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +2 -1
 mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_4TC_4TQ;
- else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+ else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+ IXGBE_82599_VMDQ_4Q_MASK)
 mtqc |= IXGBE_MTQC_32VF;

else
mtqc |= IXGBE_MTQC_64VF;

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +0 -0
@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct

ixgbe_adapter

Quoted 31 lines
Quoted 31 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +3 -3
 *adapter)
 u32 reta_entries = ixgbe_rss_indir_tbl_entries(adapter);
 u16 rss_i = adapter->ring_feature[RING_F_RSS].indices;
 
- /* Program table for at least 2 queues w/ SR-IOV so that VFs can
+ /* Program table for at least 4 queues w/ SR-IOV so that VFs can
 * make full use of any rings they may have.  We will use the
 * PSRTYPE register to control how many rings we use within the PF.
 */
- if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 2))
- rss_i = 2;
+ if ((adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) && (rss_i < 4))
+ rss_i = 4;
 
 /* Fill out hash function seeds */

for (i = 0; i < 10; i++)

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +0 -0
@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter
 *adapter)

mrqc = IXGBE_MRQC_VMDQRT8TCEN; /* 8 TCs */
else if (tcs > 1)
mrqc = IXGBE_MRQC_VMDQRT4TCEN; /* 4 TCs */
- else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+ else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+ IXGBE_82599_VMDQ_4Q_MASK)
mrqc = IXGBE_MRQC_VMDQRSS32EN;
else
mrqc = IXGBE_MRQC_VMDQRSS64EN;

Quoted 11 lines
Quoted 5 lines
Quoted 5 lines

_______________________________________________
Intel-wired-lan mailing list Intel-wired-lan at lists.osuosl.org <javascript:;>
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan

_______________________________________________
Intel-wired-lan mailing list Intel-wired-lan at lists.osuosl.org <javascript:;>
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20160910/31b857d0/attachment-0001.html>

#11 lore raw

PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF

In reply to Alexander Duyck
Quoted 22 lines

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Wednesday, September 07, 2016 8:28 PM
To: intel-wired-lan at lists.osuosl.org
Cc: ruslan at purestorage.com; Jayakumar, Muthurajan <[email protected]>; Blevins, Christopher R <[email protected]>
Subject: [Intel-wired-lan] [next PATCH 3/3] ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF

From: Alexander Duyck <[email protected]>

Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.

The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.

Quoted 5 lines
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c  |    7 ++++---
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c |   12 +++++++-----
 2 files changed, 11 insertions(+), 8 deletions(-)