Add support for 4 queues VFs with 1 or 2 queue RSS on PF
Subsystems (2)
cover Add support for 4 queues VFs with 1 or 2 queue RSS on PF
This patch set addresses a few minor issues and allows us to enable 4 queue RSS on the VFs if the PF if configured for less than 4 queues. I found and fixed at least 2 buts with the first 2 patches in this set, and the third patch enables us to have 4 queues with RSS assuming that we have not exceeded 32 total VM pools and DCB is not enabled.
---
Alexander Duyck (3):
ixgbe: Allow setting multiple queues when SR-IOV is enabled
ixgbe: Limit reporting of redirection table if SR-IOV is enabled
ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 10 +++++++--- drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 7 ++++--- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 12 +++++++----- 3 files changed, 18 insertions(+), 11 deletions(-)
--
PATCH 1/3 ixgbe: Allow setting multiple queues when SR-IOV is enabled
From: Alexander Duyck <[email protected]>
The maximum queue count reported was 1, however support for multiple queues with SR-IOV was added some time ago so we should report support for it to the user so that they can select multiple queues if they so desire.
--- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +2 -2
@@ -3060,8 +3060,8 @@ static unsigned int ixgbe_max_channels(struct ixgbe_adapter *adapter)/* We only support one q_vector without MSI-X */ max_combined =1 ; }else if (adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) { -/* SR-IOV currently only allows one queue on the PF */ - max_combined =1 ; +/* Limit value based on the queue mask */ + max_combined = adapter->ring_feature[RING_F_RSS ].mask +1 ; }else if (tcs >1 ) {/* For DCB report channels per traffic class */ if (adapter->hw.mac.type == ixgbe_mac_82598EB) {
Bowers
PATCH 1/3 ixgbe: Allow setting multiple queues when SR-IOV is enabled
Quoted 13 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Wednesday, September 07, 2016 8:28 PM
To: intel-wired-lan at lists.osuosl.org
Cc: ruslan at purestorage.com; Jayakumar, Muthurajan <[email protected]>; Blevins, Christopher R <[email protected]>
Subject: [Intel-wired-lan] [next PATCH 1/3] ixgbe: Allow setting multiple queues when SR-IOV is enabledFrom: Alexander Duyck <[email protected]>
Quoted 8 lines
The maximum queue count reported was 1, however support for multiple queues with SR-IOV was added some time ago so we should report support for it to the user so that they can select multiple queues if they so desire.
Signed-off-by Alexander Duyck <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
PATCH 2/3 ixgbe: Limit reporting of redirection table if SR-IOV is enabled
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
The hardware redirection table can support more queues then the PF currently has when SR-IOV is enabled. In order to account for this use the RSS mask to trim of the bits that are not used.
--- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +5 -1
@@ -2947,9 +2947,13 @@ static u32 ixgbe_rss_indir_size(struct net_device *netdev)static void ixgbe_get_reta (struct ixgbe_adapter *adapter,u32 *indir) {int i, reta_size =ixgbe_rss_indir_tbl_entries (adapter); +u16 rss_m = adapter->ring_feature[RING_F_RSS ].mask; + +if (adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) + rss_m = adapter->ring_feature[RING_F_RSS ].indices -1 ;for (i =0 ; i < reta_size; i++) - indir[i] = adapter->rss_indir_tbl[i]; + indir[i] = adapter->rss_indir_tbl[i] & rss_m; } static int ixgbe_get_rxfh(struct net_device *netdev, u32 *indir, u8 *key,
Bowers
PATCH 2/3 ixgbe: Limit reporting of redirection table if SR-IOV is enabled
Quoted 13 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Wednesday, September 07, 2016 8:28 PM
To: intel-wired-lan at lists.osuosl.org
Cc: ruslan at purestorage.com; Jayakumar, Muthurajan <[email protected]>; Blevins, Christopher R <[email protected]>
Subject: [Intel-wired-lan] [next PATCH 2/3] ixgbe: Limit reporting of redirection table if SR-IOV is enabledFrom: Alexander Duyck <[email protected]>
Quoted 8 lines
The hardware redirection table can support more queues then the PF currently has when SR-IOV is enabled. In order to account for this use the RSS mask to trim of the bits that are not used.
Signed-off-by Alexander Duyck <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.
The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.
--- drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 7 ++++--- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 12 +++++++----- 2 files changed, 11 insertions(+), 8 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4 -3
@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter) vmdq_i =min_t (u16,IXGBE_MAX_VMDQ_INDICES , vmdq_i);/* 64 pool mode with 2 queues per pool */ -if ((vmdq_i >32 ) || (rss_i <4 ) || (vmdq_i >16 && pools)) { +if ((vmdq_i >32 ) || (vmdq_i >16 && pools)) { vmdq_m =IXGBE_82599_VMDQ_2Q_MASK ; rss_m =IXGBE_RSS_2Q_MASK ; rss_i =min_t (u16, rss_i,2 ); -/* 32 pool mode with 4 queues per pool */ +/* 32 pool mode with up to 4 queues per pool */ }else { vmdq_m =IXGBE_82599_VMDQ_4Q_MASK ; rss_m =IXGBE_RSS_4Q_MASK ; - rss_i =4 ; +/* We can support 4, 2, or 1 queues */ + rss_i = (rss_i >3 ) ?4 : (rss_i >1 ) ?2 :1 ; }#ifdef IXGBE_FCOE
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +7 -5
@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter *adapter) mtqc |=IXGBE_MTQC_RT_ENA |IXGBE_MTQC_8TC_8TQ ; else if (tcs > 1) mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_4TC_4TQ; - else if (adapter->ring_feature[RING_F_RSS].indices == 4) + else if (adapter->ring_feature[RING_F_VMDQ].mask == + IXGBE_82599_VMDQ_4Q_MASK) mtqc |= IXGBE_MTQC_32VF; else mtqc |= IXGBE_MTQC_64VF;
@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct ixgbe_adapter *adapter)u32 reta_entries =ixgbe_rss_indir_tbl_entries (adapter);u16 rss_i = adapter->ring_feature[RING_F_RSS ].indices; -/* Program table for at least 2 queues w/ SR-IOV so that VFs can +/* Program table for at least 4 queues w/ SR-IOV so that VFs can * make full use of any rings they may have. We will use the * PSRTYPE register to control how many rings we use within the PF. */ -if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <2 )) - rss_i =2 ; +if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <4 )) + rss_i =4 ;/* Fill out hash function seeds */ for (i =0 ; i <10 ; i++)
@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter *adapter) mrqc =IXGBE_MRQC_VMDQRT8TCEN ;/* 8 TCs */ else if (tcs > 1) mrqc =IXGBE_MRQC_VMDQRT4TCEN ;/* 4 TCs */ - else if (adapter->ring_feature[RING_F_RSS].indices == 4) + else if (adapter->ring_feature[RING_F_VMDQ].mask == +IXGBE_82599_VMDQ_4Q_MASK ) mrqc =IXGBE_MRQC_VMDQRSS32EN ; else mrqc =IXGBE_MRQC_VMDQRSS64EN ;
PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF
Thank you very much for giving the feedback and creating new set of patches!
Ruslan
Quoted 83 lines · Alexander Duyck
From: Alexander Duyck <[email protected]>
Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.
The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.
Signed-off-by Alexander Duyck <[email protected]>---
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c| 7 ++++---drivers/net/ethernet/intel/ixgbe/ixgbe_main.c| 12 +++++++-----2 files changed, 11 insertions(+), 8 deletions(-)drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4
-3@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter) vmdq_i =min_t (u16,IXGBE_MAX_VMDQ_INDICES , vmdq_i);/* 64 pool mode with 2 queues per pool */ -if ((vmdq_i >32 ) || (rss_i <4 ) || (vmdq_i >16 && pools)) { +if ((vmdq_i >32 ) || (vmdq_i >16 && pools)) { vmdq_m =IXGBE_82599_VMDQ_2Q_MASK ; rss_m =IXGBE_RSS_2Q_MASK ; rss_i =min_t (u16, rss_i,2 ); -/* 32 pool mode with 4 queues per pool */ +/* 32 pool mode with up to 4 queues per pool */ }else { vmdq_m =IXGBE_82599_VMDQ_4Q_MASK ; rss_m =IXGBE_RSS_4Q_MASK ; - rss_i =4 ; +/* We can support 4, 2, or 1 queues */ + rss_i = (rss_i >3 ) ?4 : (rss_i >1 ) ?2 :1 ; }#ifdef IXGBE_FCOE drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +7
-5@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter *adapter) mtqc |=IXGBE_MTQC_RT_ENA |IXGBE_MTQC_8TC_8TQ ; else if (tcs > 1) mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_4TC_4TQ; - else if (adapter->ring_feature[RING_F_RSS].indices == 4) + else if (adapter->ring_feature[RING_F_VMDQ].mask == + IXGBE_82599_VMDQ_4Q_MASK) mtqc |= IXGBE_MTQC_32VF; else mtqc |= IXGBE_MTQC_64VF;@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct ixgbe_adapter *adapter)u32 reta_entries =ixgbe_rss_indir_tbl_entries (adapter);u16 rss_i = adapter->ring_feature[RING_F_RSS ].indices; -/* Program table for at least 2 queues w/ SR-IOV so that VFs can +/* Program table for at least 4 queues w/ SR-IOV so that VFs can * make full use of any rings they may have. We will use the * PSRTYPE register to control how many rings we use within the PF. */ -if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <2 )) - rss_i =2 ; +if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <4 )) + rss_i =4 ;/* Fill out hash function seeds */ for (i =0 ; i <10 ; i++)@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter *adapter) mrqc =IXGBE_MRQC_VMDQRT8TCEN ;/* 8 TCs */ else if (tcs > 1) mrqc =IXGBE_MRQC_VMDQRT4TCEN ;/* 4 TCs */ - else if (adapter->ring_feature[RING_F_RSS].indices == 4) + else if (adapter->ring_feature[RING_F_VMDQ].mask == +IXGBE_82599_VMDQ_4Q_MASK ) mrqc =IXGBE_MRQC_VMDQRSS32EN ; else mrqc =IXGBE_MRQC_VMDQRSS64EN ;
PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF
I still have on more question. There is also ixgbe_setup_vfreta function in the code. Do we need to adjust rss_i there as well?
Thanks,
Ruslan
Quoted 76 lines · Ruslan Nikolaev
Thank you very much for giving the feedback and creating new set of patches!
Ruslan
Quoted 70 lines · Alexander Duyck
From: Alexander Duyck <[email protected]>
Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.
The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.
Signed-off-by Alexander Duyck <[email protected]>---
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c| 7 ++++---drivers/net/ethernet/intel/ixgbe/ixgbe_main.c| 12 +++++++-----2 files changed, 11 insertions(+), 8 deletions(-)drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4
-3@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter) vmdq_i =min_t (u16,IXGBE_MAX_VMDQ_INDICES , vmdq_i);/* 64 pool mode with 2 queues per pool */ -if ((vmdq_i >32 ) || (rss_i <4 ) || (vmdq_i >16 && pools)) { +if ((vmdq_i >32 ) || (vmdq_i >16 && pools)) { vmdq_m =IXGBE_82599_VMDQ_2Q_MASK ; rss_m =IXGBE_RSS_2Q_MASK ; rss_i =min_t (u16, rss_i,2 ); -/* 32 pool mode with 4 queues per pool */ +/* 32 pool mode with up to 4 queues per pool */ }else { vmdq_m =IXGBE_82599_VMDQ_4Q_MASK ; rss_m =IXGBE_RSS_4Q_MASK ; - rss_i =4 ; +/* We can support 4, 2, or 1 queues */ + rss_i = (rss_i >3 ) ?4 : (rss_i >1 ) ?2 :1 ; }#ifdef IXGBE_FCOE drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +5
-4@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter *adapter) mtqc |=IXGBE_MTQC_RT_ENA |IXGBE_MTQC_8TC_8TQ ; else if (tcs > 1) mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_4TC_4TQ; - else if (adapter->ring_feature[RING_F_RSS].indices == 4) + else if (adapter->ring_feature[RING_F_VMDQ].mask == + IXGBE_82599_VMDQ_4Q_MASK) mtqc |= IXGBE_MTQC_32VF; else mtqc |= IXGBE_MTQC_64VF;@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct ixgbe_adapter *adapter)u32 reta_entries =ixgbe_rss_indir_tbl_entries (adapter);u16 rss_i = adapter->ring_feature[RING_F_RSS ].indices; -/* Program table for at least 2 queues w/ SR-IOV so that VFs can +/* Program table for at least 4 queues w/ SR-IOV so that VFs can * make full use of any rings they may have. We will use the * PSRTYPE register to control how many rings we use within the PF. */ -if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <2 )) - rss_i =2 ; +if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <4 )) + rss_i =4 ;
Quoted 14 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c static void ixgbe_setup_mrqc(struct ixgbe_adapter *adapter) · line 3552
Quoted 13 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +2
-1/* Fill out hash function seeds */ for (i =0 ; i <10 ; i++)@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter *adapter) mrqc =IXGBE_MRQC_VMDQRT8TCEN ;/* 8 TCs */ else if (tcs > 1) mrqc =IXGBE_MRQC_VMDQRT4TCEN ;/* 4 TCs */ - else if (adapter->ring_feature[RING_F_RSS].indices == 4) + else if (adapter->ring_feature[RING_F_VMDQ].mask == +IXGBE_82599_VMDQ_4Q_MASK ) mrqc =IXGBE_MRQC_VMDQRSS32EN ; else mrqc =IXGBE_MRQC_VMDQRSS64EN ;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20160908/07ad9b96/attachment-0001.html>
PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF
That shouldn't be needed since the VFRETA is actually per virtual pool so it isn't shared with the VFs. In theory we could support 3 queues on the X550 without any issues, but that would be a change of how things are currently handled so I figured I will leave it as it is for now.
- Alex
Quoted 112 lines · Ruslan Nikolaev
I still have on more question. There is also
ixgbe_setup_vfretafunction in the code. Do we need to adjustrss_ithere as well?Thanks,
RuslanOn Sep 8, 2016, at 11:20 AM, Ruslan Nikolaev <[email protected]> wrote:
Thank you very much for giving the feedback and creating new set of patches!
Ruslan
On Sep 7, 2016, at 8:28 PM, Alexander Duyck <[email protected]> wrote:
From: Alexander Duyck <[email protected]>
Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.
The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.
Signed-off-by Alexander Duyck <[email protected]>---
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c| 7 ++++---drivers/net/ethernet/intel/ixgbe/ixgbe_main.c| 12 +++++++-----2 files changed, 11 insertions(+), 8 deletions(-)drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4
-3@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter) vmdq_i =min_t (u16,IXGBE_MAX_VMDQ_INDICES , vmdq_i);/* 64 pool mode with 2 queues per pool */ -if ((vmdq_i >32 ) || (rss_i <4 ) || (vmdq_i >16 && pools)) { +if ((vmdq_i >32 ) || (vmdq_i >16 && pools)) { vmdq_m =IXGBE_82599_VMDQ_2Q_MASK ; rss_m =IXGBE_RSS_2Q_MASK ; rss_i =min_t (u16, rss_i,2 ); -/* 32 pool mode with 4 queues per pool */ +/* 32 pool mode with up to 4 queues per pool */ }else { vmdq_m =IXGBE_82599_VMDQ_4Q_MASK ; rss_m =IXGBE_RSS_4Q_MASK ; - rss_i =4 ; +/* We can support 4, 2, or 1 queues */ + rss_i = (rss_i >3 ) ?4 : (rss_i >1 ) ?2 :1 ; }#ifdef
IXGBE_FCOEdrivers/net/ethernet/intel/ixgbe/ixgbe_main.c +2
-1@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter *adapter) mtqc |=IXGBE_MTQC_RT_ENA |IXGBE_MTQC_8TC_8TQ ; else if (tcs > 1) mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_4TC_4TQ; - else if (adapter->ring_feature[RING_F_RSS].indices == 4) + else if (adapter->ring_feature[RING_F_VMDQ].mask == + IXGBE_82599_VMDQ_4Q_MASK) mtqc |= IXGBE_MTQC_32VF;else
mtqc |=IXGBE_MTQC_64VF;drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +3
-3@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct ixgbe_adapter *adapter)u32 reta_entries =ixgbe_rss_indir_tbl_entries (adapter);u16 rss_i = adapter->ring_feature[RING_F_RSS ].indices; -/* Program table for at least 2 queues w/ SR-IOV so that VFs can +/* Program table for at least 4 queues w/ SR-IOV so that VFs can * make full use of any rings they may have. We will use the * PSRTYPE register to control how many rings we use within the PF. */ -if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <2 )) - rss_i =2 ; +if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <4 )) + rss_i =4 ;/* Fill out hash function seeds */ for (i = 0; i < 10; i++)
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +0
-0@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter *adapter)mrqc =
IXGBE_MRQC_VMDQRT8TCEN; /* 8 TCs */
else if (tcs > 1)
mrqc =IXGBE_MRQC_VMDQRT4TCEN; /* 4 TCs */
- else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+ else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+IXGBE_82599_VMDQ_4Q_MASK)
mrqc =IXGBE_MRQC_VMDQRSS32EN;
else
mrqc =IXGBE_MRQC_VMDQRSS64EN;_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at lists.osuosl.org
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan
PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF
Sorry in advance if I have any misconception regarding how VFRETA works. But I was more thinking about rss_i being smaller than needed. For instance, rss_i is 1 (because RSS=1 for PF) but we use VFs with larger number of RX queues (RSS=2 or RSS=4 for VF). Should we program the table for VFs using value 2 (or 4) that we at least support 2 RX queues (or 4 RX queues if interrupts are shared)? I guess, it does not matter when we use just one RX queue.
Thanks,
Ruslan
Quoted 121 lines · Alexander Duyck
That shouldn't be needed since the VFRETA is actually per virtual pool so it isn't shared with the VFs. In theory we could support 3 queues on the X550 without any issues, but that would be a change of how things are currently handled so I figured I will leave it as it is for now.
- Alex
Quoted 112 lines · Ruslan Nikolaev
I still have on more question. There is also
ixgbe_setup_vfretafunction in the code. Do we need to adjustrss_ithere as well?Thanks,
RuslanOn Sep 8, 2016, at 11:20 AM, Ruslan Nikolaev <[email protected]> wrote:
Thank you very much for giving the feedback and creating new set of patches!
Ruslan
On Sep 7, 2016, at 8:28 PM, Alexander Duyck <[email protected]> wrote:
From: Alexander Duyck <[email protected]>
Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.
The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.
Signed-off-by Alexander Duyck <[email protected]>---
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c| 7 ++++---drivers/net/ethernet/intel/ixgbe/ixgbe_main.c| 12 +++++++-----2 files changed, 11 insertions(+), 8 deletions(-)drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4
-3@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter) vmdq_i =min_t (u16,IXGBE_MAX_VMDQ_INDICES , vmdq_i);/* 64 pool mode with 2 queues per pool */ -if ((vmdq_i >32 ) || (rss_i <4 ) || (vmdq_i >16 && pools)) { +if ((vmdq_i >32 ) || (vmdq_i >16 && pools)) { vmdq_m =IXGBE_82599_VMDQ_2Q_MASK ; rss_m =IXGBE_RSS_2Q_MASK ; rss_i =min_t (u16, rss_i,2 ); -/* 32 pool mode with 4 queues per pool */ +/* 32 pool mode with up to 4 queues per pool */ }else { vmdq_m =IXGBE_82599_VMDQ_4Q_MASK ; rss_m =IXGBE_RSS_4Q_MASK ; - rss_i =4 ; +/* We can support 4, 2, or 1 queues */ + rss_i = (rss_i >3 ) ?4 : (rss_i >1 ) ?2 :1 ; }#ifdef
IXGBE_FCOEdrivers/net/ethernet/intel/ixgbe/ixgbe_main.c +2
-1@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter *adapter) mtqc |=IXGBE_MTQC_RT_ENA |IXGBE_MTQC_8TC_8TQ ; else if (tcs > 1) mtqc |= IXGBE_MTQC_RT_ENA | IXGBE_MTQC_4TC_4TQ; - else if (adapter->ring_feature[RING_F_RSS].indices == 4) + else if (adapter->ring_feature[RING_F_VMDQ].mask == + IXGBE_82599_VMDQ_4Q_MASK) mtqc |= IXGBE_MTQC_32VF;else
mtqc |=IXGBE_MTQC_64VF;drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +3
-3@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct ixgbe_adapter *adapter)u32 reta_entries =ixgbe_rss_indir_tbl_entries (adapter);u16 rss_i = adapter->ring_feature[RING_F_RSS ].indices; -/* Program table for at least 2 queues w/ SR-IOV so that VFs can +/* Program table for at least 4 queues w/ SR-IOV so that VFs can * make full use of any rings they may have. We will use the * PSRTYPE register to control how many rings we use within the PF. */ -if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <2 )) - rss_i =2 ; +if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <4 )) + rss_i =4 ;/* Fill out hash function seeds */ for (i = 0; i < 10; i++)
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +0
-0@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter *adapter)mrqc =
IXGBE_MRQC_VMDQRT8TCEN; /* 8 TCs */
else if (tcs > 1)
mrqc =IXGBE_MRQC_VMDQRT4TCEN; /* 4 TCs */
- else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+ else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+IXGBE_82599_VMDQ_4Q_MASK)
mrqc =IXGBE_MRQC_VMDQRSS32EN;
else
mrqc =IXGBE_MRQC_VMDQRSS64EN;_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at lists.osuosl.org
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan
PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF
The VFRETA can be programmed as you like. Each VF has its own table so PF config will not impact the VFs. All it impacts is the local pool the PF is using for its own traffic. So if you write it to all zeros that will set it up for only one queue. As long as none of the VFs exceed the number of queues the VF has it shouldn't be a problem.
- Alex
Quoted 140 lines · Ruslan Nikolaev
Sorry in advance if I have any misconception regarding how VFRETA works. But I was more thinking about
rss_ibeing smaller than needed. For instance,rss_iis 1 (because RSS=1 for PF) but we use VFs with larger number of RX queues (RSS=2 or RSS=4 for VF). Should we program the table for VFs using value 2 (or 4) that we at least support 2 RX queues (or 4 RX queues if interrupts are shared)? I guess, it does not matter when we use just one RX queue.Thanks,
RuslanQuoted 9 lines · On Sep 9, 2016, at 8:32 AM, Alexander Duyck <[email protected]…
That shouldn't be needed since the VFRETA is actually per virtual pool so it isn't shared with the VFs. In theory we could support 3 queues on the X550 without any issues, but that would be a change of how things are currently handled so I figured I will leave it as it is for now.
- Alex
On Thu, Sep 8, 2016 at 6:14 PM, Ruslan Nikolaev <[email protected]
Quoted 1 line · <javascript:;>> wrote:
Quoted 1 line
I still have on more question. There is also
ixgbe_setup_vfretafunction in
Quoted 6 lines
Quoted 6 lines
the code. Do we need to adjust
rss_ithere as well?Thanks,
RuslanOn Sep 8, 2016, at 11:20 AM, Ruslan Nikolaev <[email protected]
Quoted 2 lines · <javascript:;>> wrote:
Quoted 2 lines
Thank you very much for giving the feedback and creating new set of
patches!
Quoted 4 lines
Quoted 4 lines
Ruslan
On Sep 7, 2016, at 8:28 PM, Alexander Duyck <[email protected]
<javascript:;>>
Quoted 5 lines
Quoted 5 lines
wrote:
From: Alexander Duyck <[email protected] <javascript:;>>
Instead of limiting the VFs if we don't use 4 queues for RSS in the PF
we
Quoted 1 line
Quoted 1 line
can instead just limit the RSS queues used to a power of 2. By doing
this
Quoted 5 lines
Quoted 5 lines
we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.
The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues
in
Quoted 4 lines
Quoted 4 lines
order to be able to use the power of 2 masking provided by the psrtype register.
Signed-off-by Alexander Duyck <[email protected]<javascript:;>>
Quoted 40 lines
Quoted 40 lines
---
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c| 7 ++++---drivers/net/ethernet/intel/ixgbe/ixgbe_main.c| 12 +++++++-----2 files changed, 11 insertions(+), 8 deletions(-)drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +4
-3@@ -515,15 +515,16 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter) vmdq_i =min_t (u16,IXGBE_MAX_VMDQ_INDICES , vmdq_i);/* 64 pool mode with 2 queues per pool */ -if ((vmdq_i >32 ) || (rss_i <4 ) || (vmdq_i >16 && pools)) { +if ((vmdq_i >32 ) || (vmdq_i >16 && pools)) { vmdq_m =IXGBE_82599_VMDQ_2Q_MASK ; rss_m =IXGBE_RSS_2Q_MASK ; rss_i =min_t (u16, rss_i,2 ); -/* 32 pool mode with 4 queues per pool */ +/* 32 pool mode with up to 4 queues per pool */ }else { vmdq_m =IXGBE_82599_VMDQ_4Q_MASK ; rss_m =IXGBE_RSS_4Q_MASK ; - rss_i =4 ; +/* We can support 4, 2, or 1 queues */ + rss_i = (rss_i >3 ) ?4 : (rss_i >1 ) ?2 :1 ; }#ifdef
IXGBE_FCOEdrivers/net/ethernet/intel/ixgbe/ixgbe_main.c +0
-0@@ -3248,7 +3248,8 @@ static void ixgbe_setup_mtqc(struct ixgbe_adapter *adapter) mtqc |=IXGBE_MTQC_RT_ENA |IXGBE_MTQC_8TC_8TQ ; else if (tcs > 1)Quoted 8 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c static void ixgbe_setup_reta(struct
Quoted 8 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +2
-1mtqc |=IXGBE_MTQC_RT_ENA |IXGBE_MTQC_4TC_4TQ ; - else if (adapter->ring_feature[RING_F_RSS].indices == 4) + else if (adapter->ring_feature[RING_F_VMDQ].mask == + IXGBE_82599_VMDQ_4Q_MASK) mtqc |= IXGBE_MTQC_32VF;else
mtqc |=IXGBE_MTQC_64VF;drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +0
-0@@ -3475,12 +3476,12 @@ static void ixgbe_setup_reta(struct
ixgbe_adapterQuoted 31 lines
Quoted 31 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +3
-3*adapter)u32 reta_entries =ixgbe_rss_indir_tbl_entries (adapter);u16 rss_i = adapter->ring_feature[RING_F_RSS ].indices; -/* Program table for at least 2 queues w/ SR-IOV so that VFs can +/* Program table for at least 4 queues w/ SR-IOV so that VFs can * make full use of any rings they may have. We will use the * PSRTYPE register to control how many rings we use within the PF. */ -if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <2 )) - rss_i =2 ; +if ((adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && (rss_i <4 )) + rss_i =4 ;/* Fill out hash function seeds */ for (i = 0; i < 10; i++)
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +0
-0@@ -3544,7 +3545,8 @@ static void ixgbe_setup_mrqc(struct ixgbe_adapter *adapter)mrqc =
IXGBE_MRQC_VMDQRT8TCEN; /* 8 TCs */
else if (tcs > 1)
mrqc =IXGBE_MRQC_VMDQRT4TCEN; /* 4 TCs */
- else if (adapter->ring_feature[RING_F_RSS].indices == 4)
+ else if (adapter->ring_feature[RING_F_VMDQ].mask ==
+IXGBE_82599_VMDQ_4Q_MASK)
mrqc =IXGBE_MRQC_VMDQRSS32EN;
else
mrqc =IXGBE_MRQC_VMDQRSS64EN;
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
_______________________________________________
Intel-wired-lan mailing list Intel-wired-lan at lists.osuosl.org <javascript:;>
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan_______________________________________________
Intel-wired-lan mailing list Intel-wired-lan at lists.osuosl.org <javascript:;>
http://lists.osuosl.org/mailman/listinfo/intel-wired-lan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20160910/31b857d0/attachment-0001.html>
Bowers
PATCH 3/3 ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PF
In reply to Alexander DuyckQuoted 22 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Wednesday, September 07, 2016 8:28 PM
To: intel-wired-lan at lists.osuosl.org
Cc: ruslan at purestorage.com; Jayakumar, Muthurajan <[email protected]>; Blevins, Christopher R <[email protected]>
Subject: [Intel-wired-lan] [next PATCH 3/3] ixgbe: Support 4 queue RSS on VFs with 1 or 2 queue RSS on PFFrom: Alexander Duyck <[email protected]>
Instead of limiting the VFs if we don't use 4 queues for RSS in the PF we can instead just limit the RSS queues used to a power of 2. By doing this we can support use cases where VFs are using more queues than the PF is currently using and can support RSS if so desired.
The only limitation on this is that we cannot support 3 queues of RSS in the PF or VF. In either of these cases we should fall back to 2 queues in order to be able to use the power of 2 masking provided by the psrtype register.
Quoted 5 lines
Signed-off-by Alexander Duyck <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 7 ++++--- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 12 +++++++----- 2 files changed, 11 insertions(+), 8 deletions(-)