Skip to contentSkip to conversation

Configuring traffic classes via new hardware offload mechanism in tc/mqprio

RFC PATCH In review Amritha Nambiar 4 participants 18 messages Sep 7, 2017 – Sep 13, 2017

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v3 cover Configuring traffic classes via new hardware offload mechanism in tc/mqprio

The following series introduces a new hardware offload mode in tc/mqprio where the TCs, the queue configurations and bandwidth rate limits are offloaded to the hardware. The existing mqprio framework is extended to configure the queue counts and layout and also added support for rate limiting. This is achieved through new netlink attributes for the 'mode' option which takes values such as 'dcb' (default) and 'channel' and a 'shaper' option for QoS attributes such as bandwidth rate limits in hw mode 1. Legacy devices can fall back to the existing setup supporting hw mode 1 without these additional options where only the TCs are offloaded and then the 'mode' and 'shaper' options defaults to DCB support. The i40e driver enables the new mqprio hardware offload mechanism factoring the TCs, queue configuration and bandwidth rates by creating HW channel VSIs.

In this new mode, the priority to traffic class mapping and the user specified queue ranges are used to configure the traffic class when the 'mode' option is set to 'channel'. This is achieved by creating HW channels(VSI). A new channel is created for each of the traffic class configuration offloaded via mqprio framework except for the first TC (TC0) which is for the main VSI. TC0 for the main VSI is also reconfigured as per user provided queue parameters. Finally, bandwidth rate limits are set on these traffic classes through the shaper attribute by sending these rates in addition to the number of TCs and the queue configurations.

Example:

    # tc qdisc add dev eth0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1\
      queues 4 at 0 4 at 4 hw 1 mode channel shaper bw_rlimit\
      min_rate 1Gbit 2Gbit max_rate 4Gbit 5Gbit

To dump the bandwidth rates:

# tc qdisc show dev eth0

qdisc mqprio 804a: root tc 2 map 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0
queues:(0:3) (4:7)
mode:channel
shaper:bw_rlimit min_rate:1Gbit 2Gbit max_rate:4Gbit 5Gbit

---

Amritha Nambiar (6):
mqprio: Introduce new hardware offload mode and shaper in mqprio
i40e: Add macro for PF reset bit
i40e: Add infrastructure for queue channel support
i40e: Enable 'channel' mode in mqprio for TC configs
i40e: Refactor VF BW rate limiting
i40e: Add support setting TC max bandwidth rates

 drivers/net/ethernet/intel/i40e/i40e.h             |   44 +
 drivers/net/ethernet/intel/i40e/i40e_debugfs.c     |    3 
 drivers/net/ethernet/intel/i40e/i40e_ethtool.c     |    8 
 drivers/net/ethernet/intel/i40e/i40e_main.c        | 1463 +++++++++++++++++---
 drivers/net/ethernet/intel/i40e/i40e_txrx.h        |    2 
 drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c |   50 -
 include/net/pkt_cls.h                              |    9 
 include/uapi/linux/pkt_sched.h                     |   32 
 net/sched/sch_mqprio.c                             |  183 ++-
 9 files changed, 1551 insertions(+), 243 deletions(-)

--

#1 lore raw

RFC PATCH v3 1/6 mqprio: Introduce new hardware offload mode and shaper in mqprio

The offload types currently supported in mqprio are 0 (no offload) and 1 (offload only TCs) by setting these values for the 'hw' option. If offloads are supported by setting the 'hw' option to 1, the default offload mode is 'dcb' where only the TC values are offloaded to the device. This patch introduces a new hardware offload mode called 'channel' with 'hw' set to 1 in mqprio which makes full use of the mqprio options, the TCs, the queue configurations and the QoS parameters for the TCs. This is achieved through a new netlink attribute for the 'mode' option which takes values such as 'dcb' (default) and 'channel'. The 'channel' mode also supports QoS attributes for traffic class such as minimum and maximum values for bandwidth rate limits.

This patch enables configuring additional HW shaper attributes associated with a traffic class. Currently the shaper for bandwidth rate limiting is supported which takes options such as minimum and maximum bandwidth rates and are offloaded to the hardware in the 'channel' mode. The min and max limits for bandwidth rates are provided by the user along with the the TCs and the queue configurations when creating the mqprio qdisc. The interface can be extended to support new HW shapers in future through the 'shaper' attribute.

Introduces a new datastructure 'tc_mqprio_qopt_offload' for offloading mqprio queue options and use this to be shared between the kernel and device driver. This contains a copy of the exisiting datastructure for mqprio queue options. This new datastructure can be extended when adding new attributes for traffic class such as mode, shaper, shaper parameters (bandwidth rate limits). The existing datastructure for mqprio queue options will be shared between the kernel and userspace.

Example:

# tc qdisc add dev eth0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1\
  queues 4 at 0 4 at 4 hw 1 mode channel shaper bw_rlimit\
  min_rate 1Gbit 2Gbit max_rate 4Gbit 5Gbit

To dump the bandwidth rates:

# tc qdisc show dev eth0

qdisc mqprio 804a: root tc 2 map 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0
queues:(0:3) (4:7)
mode:channel
shaper:bw_rlimit min_rate:1Gbit 2Gbit max_rate:4Gbit 5Gbit

v3 : Removed supporting new offloads through value 2 for 'hw' option, introduced new netlink based options for offload mode and HW shaper. v2 : Jiri's changes accepted upstream removes the struct tc_to_netdev. Clean up the full offload related changes added to mqprio_init() and mqprio_destroy() to rebase on these changes.

---
 include/net/pkt_cls.h          |    9 ++
 include/uapi/linux/pkt_sched.h |   32 +++++++
 net/sched/sch_mqprio.c         |  183 ++++++++++++++++++++++++++++++++++++++--
 3 files changed, 215 insertions(+), 9 deletions(-)
include/net/pkt_cls.h +9 -0
index e80edd8..456017a 100644
@@ -546,6 +546,15 @@ struct tc_cls_bpf_offload {
 	u32 gen_flags;
 };
 
+struct tc_mqprio_qopt_offload {
+	/* struct tc_mqprio_qopt must always be the first element */
+	struct tc_mqprio_qopt qopt;
+	u16 mode;
+	u16 shaper;
+	u32 flags;
+	u64 min_rate[TC_QOPT_MAX_QUEUE];
+	u64 max_rate[TC_QOPT_MAX_QUEUE];
+};
 
 /* This structure holds cookie structure that is passed from user
  * to the kernel for actions and classifiers
include/uapi/linux/pkt_sched.h +32 -0
index 099bf55..e95b5c9 100644
@@ -625,6 +625,22 @@ enum {
 
 #define TC_MQPRIO_HW_OFFLOAD_MAX (__TC_MQPRIO_HW_OFFLOAD_MAX - 1)
 
+enum {
+	TC_MQPRIO_MODE_DCB,
+	TC_MQPRIO_MODE_CHANNEL,
+	__TC_MQPRIO_MODE_MAX
+};
+
+#define __TC_MQPRIO_MODE_MAX (__TC_MQPRIO_MODE_MAX - 1)
+
+enum {
+	TC_MQPRIO_SHAPER_DCB,
+	TC_MQPRIO_SHAPER_BW_RATE,	/* Add new shapers below */
+	__TC_MQPRIO_SHAPER_MAX
+};
+
+#define __TC_MQPRIO_SHAPER_MAX (__TC_MQPRIO_SHAPER_MAX - 1)
+
 struct tc_mqprio_qopt {
 	__u8	num_tc;
 	__u8	prio_tc_map[TC_QOPT_BITMASK + 1];
@@ -633,6 +649,22 @@ struct tc_mqprio_qopt {
 	__u16	offset[TC_QOPT_MAX_QUEUE];
 };
 
+#define TC_MQPRIO_F_MODE		0x1
+#define TC_MQPRIO_F_SHAPER		0x2
+#define TC_MQPRIO_F_MIN_RATE		0x4
+#define TC_MQPRIO_F_MAX_RATE		0x8
+
+enum {
+	TCA_MQPRIO_UNSPEC,
+	TCA_MQPRIO_MODE,
+	TCA_MQPRIO_SHAPER,
+	TCA_MQPRIO_MIN_RATE64,
+	TCA_MQPRIO_MAX_RATE64,
+	__TCA_MQPRIO_MAX,
+};
+
+#define TCA_MQPRIO_MAX (__TCA_MQPRIO_MAX - 1)
+
 /* SFB */
 
 enum {
net/sched/sch_mqprio.c +174 -9

248 lines not on this page · raw message

#2 lore raw

RFC PATCH v3 1/6 mqprio: Introduce new hardware offload mode and shaper in mqprio

Quoted 60 lines

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Amritha Nambiar
Sent: Thursday, September 7, 2017 4:00 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Cc: netdev at vger.kernel.org
Subject: [Intel-wired-lan] [RFC PATCH v3 1/6] mqprio: Introduce new hardware offload mode and shaper in mqprio

The offload types currently supported in mqprio are 0 (no offload) and 1 (offload only TCs) by setting these values for the 'hw' option. If offloads are supported by setting the 'hw' option to 1, the default offload mode is 'dcb' where only the TC values are offloaded to the device. This patch introduces a new hardware offload mode called 'channel' with 'hw' set to 1 in mqprio which makes full use of the mqprio options, the TCs, the queue configurations and the QoS parameters for the TCs. This is achieved through a new netlink attribute for the 'mode' option which takes values such as 'dcb' (default) and 'channel'.
The 'channel' mode also supports QoS attributes for traffic class such as minimum and maximum values for bandwidth rate limits.

This patch enables configuring additional HW shaper attributes associated with a traffic class. Currently the shaper for bandwidth rate limiting is supported which takes options such as minimum and maximum bandwidth rates and are offloaded to the hardware in the 'channel' mode. The min and max limits for bandwidth rates are provided by the user along with the the TCs and the queue configurations when creating the mqprio qdisc. The interface can be extended to support new HW shapers in future through the 'shaper'
attribute.

Introduces a new datastructure 'tc_mqprio_qopt_offload' for offloading mqprio queue options and use this to be shared between the kernel and device driver. This contains a copy of the exisiting datastructure for mqprio queue options. This new datastructure can be extended when adding new attributes for traffic class such as mode, shaper, shaper parameters (bandwidth rate limits). The existing datastructure for mqprio queue options will be shared between the kernel and userspace.

Example:

# tc qdisc add dev eth0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1\
  queues 4 at 0 4 at 4 hw 1 mode channel shaper bw_rlimit\
  min_rate 1Gbit 2Gbit max_rate 4Gbit 5Gbit

To dump the bandwidth rates:

# tc qdisc show dev eth0

qdisc mqprio 804a: root tc 2 map 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0
queues:(0:3) (4:7)
mode:channel
shaper:bw_rlimit min_rate:1Gbit 2Gbit max_rate:4Gbit 5Gbit

v3 : Removed supporting new offloads through value 2 for 'hw' option, introduced new netlink based options for offload mode and HW shaper. v2 : Jiri's changes accepted upstream removes the struct tc_to_netdev. Clean up the full offload related changes added to
mqprio_init() and mqprio_destroy() to rebase on these changes.

Quoted 7 lines
---
 include/net/pkt_cls.h          |    9 ++
 include/uapi/linux/pkt_sched.h |   32 +++++++
 net/sched/sch_mqprio.c         |  183
++++++++++++++++++++++++++++++++++++++--
 3 files changed, 215 insertions(+), 9 deletions(-)
#3 lore raw

RFC PATCH v3 2/6 i40e: Add macro for PF reset bit

In reply to Amritha Nambiar

Introduce a macro for the bit setting the PF reset flag and update its usages. This makes it easier to use this flag in functions to be introduced in future without encountering checkpatch issues related to alignment and line over 80 characters.

---
 drivers/net/ethernet/intel/i40e/i40e.h             |    2 ++
 drivers/net/ethernet/intel/i40e/i40e_debugfs.c     |    3 +--
 drivers/net/ethernet/intel/i40e/i40e_main.c        |    9 ++++-----
 drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c |    5 ++---
 4 files changed, 9 insertions(+), 10 deletions(-)
drivers/net/ethernet/intel/i40e/i40e.h +2 -0
index 18c453a..d414adc 100644
@@ -157,6 +157,8 @@ enum i40e_state_t {
 	__I40E_STATE_SIZE__,
 };
 
+#define I40E_PF_RESET_FLAG	BIT_ULL(__I40E_PF_RESET_REQUESTED)
+
 /* VSI state flags */
 enum i40e_vsi_state_t {
 	__I40E_VSI_DOWN,
drivers/net/ethernet/intel/i40e/i40e_debugfs.c +1 -2
index 6f2725f..2b8bbc8 100644
@@ -798,8 +798,7 @@ static ssize_t i40e_dbg_command_write(struct file *filp,
 		 */
 		if (!(pf->flags & I40E_FLAG_VEB_MODE_ENABLED)) {
 			pf->flags |= I40E_FLAG_VEB_MODE_ENABLED;
-			i40e_do_reset_safe(pf,
-					   BIT_ULL(__I40E_PF_RESET_REQUESTED));
+			i40e_do_reset_safe(pf, I40E_PF_RESET_FLAG);
 		}
 
 		vsi = i40e_vsi_setup(pf, I40E_VSI_VMDQ2, vsi_seid, 0);
drivers/net/ethernet/intel/i40e/i40e_main.c +4 -5
index 76b03f7..2fdb99f 100644
@@ -5753,7 +5753,7 @@ int i40e_vsi_open(struct i40e_vsi *vsi)
 err_setup_tx:
 	i40e_vsi_free_tx_resources(vsi);
 	if (vsi == pf->vsi[pf->lan_vsi])
-		i40e_do_reset(pf, BIT_ULL(__I40E_PF_RESET_REQUESTED), true);
+		i40e_do_reset(pf, I40E_PF_RESET_FLAG, true);
 
 	return err;
 }
@@ -5881,7 +5881,7 @@ void i40e_do_reset(struct i40e_pf *pf, u32 reset_flags, bool lock_acquired)
 		wr32(&pf->hw, I40E_GLGEN_RTRIG, val);
 		i40e_flush(&pf->hw);
 
-	} else if (reset_flags & BIT_ULL(__I40E_PF_RESET_REQUESTED)) {
+	} else if (reset_flags & I40E_PF_RESET_FLAG) {
 
 		/* Request a PF Reset
 		 *
@@ -9229,7 +9229,7 @@ static int i40e_set_features(struct net_device *netdev,
 	need_reset = i40e_set_ntuple(pf, features);
 
 	if (need_reset)
-		i40e_do_reset(pf, BIT_ULL(__I40E_PF_RESET_REQUESTED), true);
+		i40e_do_reset(pf, I40E_PF_RESET_FLAG, true);
 
 	return 0;
 }
@@ -9481,8 +9481,7 @@ static int i40e_ndo_bridge_setlink(struct net_device *dev,
 				pf->flags |= I40E_FLAG_VEB_MODE_ENABLED;
 			else
 				pf->flags &= ~I40E_FLAG_VEB_MODE_ENABLED;
-			i40e_do_reset(pf, BIT_ULL(__I40E_PF_RESET_REQUESTED),
-				      true);
+			i40e_do_reset(pf, I40E_PF_RESET_FLAG, true);
 			break;
 		}
 	}
drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c +2 -3
index 8bedc74c..ac88d96 100644
@@ -1424,8 +1424,7 @@ int i40e_pci_sriov_configure(struct pci_dev *pdev, int num_vfs)
 	if (num_vfs) {
 		if (!(pf->flags & I40E_FLAG_VEB_MODE_ENABLED)) {
 			pf->flags |= I40E_FLAG_VEB_MODE_ENABLED;
-			i40e_do_reset_safe(pf,
-					   BIT_ULL(__I40E_PF_RESET_REQUESTED));
+			i40e_do_reset_safe(pf, I40E_PF_RESET_FLAG);
 		}
 		return i40e_pci_sriov_enable(pdev, num_vfs);
 	}
@@ -1433,7 +1432,7 @@ int i40e_pci_sriov_configure(struct pci_dev *pdev, int num_vfs)
 	if (!pci_vfs_assigned(pf->pdev)) {
 		i40e_free_vfs(pf);
 		pf->flags &= ~I40E_FLAG_VEB_MODE_ENABLED;
-		i40e_do_reset_safe(pf, BIT_ULL(__I40E_PF_RESET_REQUESTED));
+		i40e_do_reset_safe(pf, I40E_PF_RESET_FLAG);
 	} else {
 		dev_warn(&pdev->dev, "Unable to free VFs because some are assigned to VMs.\n");
 		return -EINVAL;
#4 lore raw

RFC PATCH v3 2/6 i40e: Add macro for PF reset bit

Quoted 14 lines

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Amritha Nambiar
Sent: Thursday, September 7, 2017 4:00 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Cc: netdev at vger.kernel.org
Subject: [Intel-wired-lan] [RFC PATCH v3 2/6] i40e: Add macro for PF reset bit

Introduce a macro for the bit setting the PF reset flag and update its usages. This makes it easier to use this flag in functions to be introduced in future without encountering checkpatch issues related to alignment and line over 80 characters.

Quoted 7 lines
---
 drivers/net/ethernet/intel/i40e/i40e.h             |    2 ++
 drivers/net/ethernet/intel/i40e/i40e_debugfs.c     |    3 +--
 drivers/net/ethernet/intel/i40e/i40e_main.c        |    9 ++++-----
 drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c |    5 ++---
 4 files changed, 9 insertions(+), 10 deletions(-)
#5 lore raw

RFC PATCH v3 3/6 i40e: Add infrastructure for queue channel support

In reply to Amritha Nambiar

This patch sets up the infrastructure for offloading TCs and queue configurations to the hardware by creating HW channels(VSI). A new channel is created for each of the traffic class configuration offloaded via mqprio framework except for the first TC (TC0). TC0 for the main VSI is also reconfigured as per user provided queue parameters. Queue counts that are not power-of-2 are handled by reconfiguring RSS by reprogramming LUTs using the queue count value. This patch also handles configuring the TX rings for the channels, setting up the RX queue map for channel.

Also, the channels so created are removed and all the queue configuration is set to default when the qdisc is detached from the root of the device.

v3: Addressed Shannon's feedback removing unwanted code, variables, atomic operations (since setup_tc is protected with rtnl_lock held by stack) and other minor clean up.

---
 drivers/net/ethernet/intel/i40e/i40e.h      |   32 +
 drivers/net/ethernet/intel/i40e/i40e_main.c |  718 +++++++++++++++++++++++++++
 drivers/net/ethernet/intel/i40e/i40e_txrx.h |    2 
 3 files changed, 743 insertions(+), 9 deletions(-)
drivers/net/ethernet/intel/i40e/i40e.h +32 -0
index d414adc..2401931 100644
@@ -87,6 +87,7 @@
 #define I40E_AQ_LEN			256
 #define I40E_AQ_WORK_LIMIT		66 /* max number of VFs + a little */
 #define I40E_MAX_USER_PRIORITY		8
+#define I40E_MAX_QUEUES_PER_CH		64
 #define I40E_DEFAULT_TRAFFIC_CLASS	BIT(0)
 #define I40E_DEFAULT_MSG_ENABLE		4
 #define I40E_QUEUE_WAIT_RETRY_LIMIT	10
@@ -340,6 +341,23 @@ struct i40e_flex_pit {
 	u8 pit_index;
 };
 
+struct i40e_channel {
+	struct list_head list;
+	bool initialized;
+	u8 type;
+	u16 vsi_number; /* Assigned VSI number from AQ 'Add VSI' response */
+	u16 stat_counter_idx;
+	u16 base_queue;
+	u16 num_queue_pairs; /* Requested by user */
+	u16 seid;
+
+	u8 enabled_tc;
+	struct i40e_aqc_vsi_properties_data info;
+
+	/* track this channel belongs to which VSI */
+	struct i40e_vsi *parent_vsi;
+};
+
 /* struct that defines the Ethernet device */
 struct i40e_pf {
 	struct pci_dev *pdev;
@@ -456,6 +474,7 @@ struct i40e_pf {
 #define I40E_FLAG_CLIENT_RESET			BIT(26)
 #define I40E_FLAG_LINK_DOWN_ON_CLOSE_ENABLED	BIT(27)
 #define I40E_FLAG_SOURCE_PRUNING_DISABLED	BIT(28)
+#define I40E_FLAG_TC_MQPRIO			BIT(29)
 
 	struct i40e_client_instance *cinst;
 	bool stat_offsets_loaded;
@@ -536,6 +555,8 @@ struct i40e_pf {
 	u32 ioremap_len;
 	u32 fd_inv;
 	u16 phy_led_val;
+
+	u16 override_q_count;
 };
 
 /**
@@ -700,6 +721,15 @@ struct i40e_vsi {
 	bool current_isup;	/* Sync 'link up' logging */
 	enum i40e_aq_link_speed current_speed;	/* Sync link speed logging */
 
+	/* channel specific fields */
+	u16 cnt_q_avail;	/* num of queues available for channel usage */
+	u16 orig_rss_size;
+	u16 current_rss_size;
+
+	u16 next_base_queue;	/* next queue to be used for channel setup */
+
+	struct list_head ch_list;
+
 	void *priv;	/* client driver data reference. */
 
 	/* VSI specific handlers */
@@ -1007,4 +1037,6 @@ static inline bool i40e_enabled_xdp_vsi(struct i40e_vsi *vsi)
 {
 	return !!vsi->xdp_prog;
 }
+
+int i40e_create_queue_channel(struct i40e_vsi *vsi, struct i40e_channel *ch);
 #endif /* _I40E_H_ */
drivers/net/ethernet/intel/i40e/i40e_main.c +709 -9

769 lines not on this page · raw message

drivers/net/ethernet/intel/i40e/i40e_txrx.h +2 -0
index a4e3e66..81624dd 100644
@@ -424,6 +424,8 @@ struct i40e_ring {
 					 * i40e_clean_rx_ring_irq() is called
 					 * for this ring.
 					 */
+
+	struct i40e_channel *ch;
 } ____cacheline_internodealigned_in_smp;
 
 static inline bool ring_uses_build_skb(struct i40e_ring *ring)
#6 lore raw

RFC PATCH v3 3/6 i40e: Add infrastructure for queue channel support

Quoted 27 lines

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Amritha Nambiar
Sent: Thursday, September 7, 2017 4:00 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Cc: netdev at vger.kernel.org
Subject: [Intel-wired-lan] [RFC PATCH v3 3/6] i40e: Add infrastructure for queue channel support

This patch sets up the infrastructure for offloading TCs and queue configurations to the hardware by creating HW channels(VSI).
A new channel is created for each of the traffic class configuration offloaded via mqprio framework except for the first TC (TC0). TC0 for the main VSI is also reconfigured as per user provided queue parameters. Queue counts that are not power-of-2 are handled by reconfiguring RSS by reprogramming LUTs using the queue count value.
This patch also handles configuring the TX rings for the channels, setting up the RX queue map for channel.

Also, the channels so created are removed and all the queue configuration is set to default when the qdisc is detached from the root of the device.

v3: Addressed Shannon's feedback removing unwanted code, variables, atomic operations (since setup_tc is protected with rtnl_lock held by stack) and other minor clean up.

Quoted 8 lines
---
 drivers/net/ethernet/intel/i40e/i40e.h      |   32 +
 drivers/net/ethernet/intel/i40e/i40e_main.c |  718
+++++++++++++++++++++++++++
 drivers/net/ethernet/intel/i40e/i40e_txrx.h |    2
 3 files changed, 743 insertions(+), 9 deletions(-)
#7 lore raw

RFC PATCH v3 4/6 i40e: Enable 'channel' mode in mqprio for TC configs

In reply to Amritha Nambiar

The i40e driver is modified to enable the new mqprio hardware offload mode and factor the TCs and queue configuration by creating channel VSIs. In this mode, the priority to traffic class mapping and the user specified queue ranges are used to configure the traffic classes by setting the mode option to 'channel'.

Example:

# tc qdisc add dev eth0 root mqprio num_tc 4\
  map 0 0 0 0 1 2 2 3 queues 2 at 0 2 at 2 1 at 4 1 at 5\
  hw 1 mode channel
# tc qdisc show dev eth0

qdisc mqprio 8038: root tc 4 map 0 0 0 0 1 2 2 3 0 0 0 0 0 0 0 0

             queues:(0:1) (2:3) (4:4) (5:5)
             mode:channel
             shaper:dcb

The HW channels created are removed and all the queue configuration is set to default when the qdisc is detached from the root of the device.

# tc qdisc del dev eth0 root

This patch also disables setting up channels via ethtool (ethtool -L) when the TCs are configured using mqprio scheduler.

The patch also limits setting ethtool Rx flow hash indirection (ethtool -X eth0 equal N) to max queues configured via mqprio. The Rx flow hash indirection input through ethtool should be validated so that it is within in the queue range configured via tc/mqprio. The bound checking is achieved by reporting the current rss size to the kernel when queues are configured via mqprio.

Example:

# tc qdisc add dev eth0 root mqprio num_tc 4\
  map 0 0 0 1 0 2 3 0 queues 2 at 0 4 at 2 8 at 6 11 at 14\
  hw 1 mode channel
# ethtool -X eth0 equal 12
Cannot set RX flow hash configuration: Invalid argument

v3: Changes to incorporate new mqprio mode option. Minor clean up of setup_tc error handling based on Shannon's comments.
v2: Clean up __i40e_setup_tc() and i40e_setup_tc() to work with Jiri's changes to the ndo_setup_tc interface which now takes a type and the type_data for the offload. No need to disable ATR in MQPRIO mode.

---
 drivers/net/ethernet/intel/i40e/i40e.h         |    3 
 drivers/net/ethernet/intel/i40e/i40e_ethtool.c |    8 
 drivers/net/ethernet/intel/i40e/i40e_main.c    |  457 ++++++++++++++++++------
 3 files changed, 362 insertions(+), 106 deletions(-)
drivers/net/ethernet/intel/i40e/i40e.h +3 -0
index 2401931..ddb7292 100644
@@ -54,6 +54,7 @@
 #include <linux/clocksource.h>
 #include <linux/net_tstamp.h>
 #include <linux/ptp_clock_kernel.h>
+#include <net/pkt_cls.h>
 #include "i40e_type.h"
 #include "i40e_prototype.h"
 #include "i40e_client.h"
@@ -700,6 +701,7 @@ struct i40e_vsi {
 	enum i40e_vsi_type type;  /* VSI type, e.g., LAN, FCoE, etc */
 	s16 vf_id;		/* Virtual function ID for SRIOV VSIs */
 
+	struct tc_mqprio_qopt_offload mqprio_qopt; /* queue parameters */
 	struct i40e_tc_configuration tc_config;
 	struct i40e_aqc_vsi_properties_data info;
 
@@ -725,6 +727,7 @@ struct i40e_vsi {
 	u16 cnt_q_avail;	/* num of queues available for channel usage */
 	u16 orig_rss_size;
 	u16 current_rss_size;
+	bool reconfig_rss;
 
 	u16 next_base_queue;	/* next queue to be used for channel setup */
 
drivers/net/ethernet/intel/i40e/i40e_ethtool.c +7 -1
index b531aa3..3fa90a6 100644
@@ -2652,7 +2652,7 @@ static int i40e_get_rxnfc(struct net_device *netdev, struct ethtool_rxnfc *cmd,
 
 	switch (cmd->cmd) {
 	case ETHTOOL_GRXRINGS:
-		cmd->data = vsi->num_queue_pairs;
+		cmd->data = vsi->rss_size;
 		ret = 0;
 		break;
 	case ETHTOOL_GRXFH:
@@ -3897,6 +3897,12 @@ static int i40e_set_channels(struct net_device *dev,
 	if (vsi->type != I40E_VSI_MAIN)
 		return -EINVAL;
 
+	/* We do not support setting channels via ethtool when TCs are
+	 * configured through mqprio
+	 */
+	if (pf->flags & I40E_FLAG_TC_MQPRIO)
+		return -EINVAL;
+
 	/* verify they are not requesting separate vectors */
 	if (!count || ch->rx_count || ch->tx_count)
 		return -EINVAL;
drivers/net/ethernet/intel/i40e/i40e_main.c +352 -105

594 lines not on this page · raw message

#8 lore raw

RFC PATCH v3 4/6 i40e: Enable 'channel' mode in mqprio for TC configs

Quoted 58 lines

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Amritha Nambiar
Sent: Thursday, September 7, 2017 4:00 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Cc: netdev at vger.kernel.org
Subject: [Intel-wired-lan] [RFC PATCH v3 4/6] i40e: Enable 'channel' mode in mqprio for TC configs

The i40e driver is modified to enable the new mqprio hardware offload mode and factor the TCs and queue configuration by creating channel VSIs. In this mode, the priority to traffic class mapping and the user specified queue ranges are used to configure the traffic classes by setting the mode option to 'channel'.

Example:

# tc qdisc add dev eth0 root mqprio num_tc 4\
  map 0 0 0 0 1 2 2 3 queues 2 at 0 2 at 2 1 at 4 1 at 5\
  hw 1 mode channel
# tc qdisc show dev eth0

qdisc mqprio 8038: root tc 4 map 0 0 0 0 1 2 2 3 0 0 0 0 0 0 0 0

             queues:(0:1) (2:3) (4:4) (5:5)
             mode:channel
             shaper:dcb

The HW channels created are removed and all the queue configuration is set to default when the qdisc is detached from the root of the device.

# tc qdisc del dev eth0 root

This patch also disables setting up channels via ethtool (ethtool -L) when the TCs are configured using mqprio scheduler.

The patch also limits setting ethtool Rx flow hash indirection (ethtool -X eth0 equal N) to max queues configured via mqprio.
The Rx flow hash indirection input through ethtool should be validated so that it is within in the queue range configured via tc/mqprio. The bound checking is achieved by reporting the current rss size to the kernel when queues are configured via mqprio.

Example:

# tc qdisc add dev eth0 root mqprio num_tc 4\
  map 0 0 0 1 0 2 3 0 queues 2 at 0 4 at 2 8 at 6 11 at 14\
  hw 1 mode channel
# ethtool -X eth0 equal 12
Cannot set RX flow hash configuration: Invalid argument

v3: Changes to incorporate new mqprio mode option. Minor clean up of setup_tc error handling based on Shannon's comments.
v2: Clean up __i40e_setup_tc() and i40e_setup_tc() to work with Jiri's changes to the ndo_setup_tc interface which now takes a type and the type_data for the offload.
No need to disable ATR in MQPRIO mode.

Quoted 7 lines
---
 drivers/net/ethernet/intel/i40e/i40e.h         |    3
 drivers/net/ethernet/intel/i40e/i40e_ethtool.c |    8
 drivers/net/ethernet/intel/i40e/i40e_main.c    |  457
++++++++++++++++++------
 3 files changed, 362 insertions(+), 106 deletions(-)
#9 lore raw

RFC PATCH v3 5/6 i40e: Refactor VF BW rate limiting

In reply to Amritha Nambiar

This patch refactors the BW rate limiting for Tx traffic on the VF to be reused in the next patch for rate limiting Tx traffic for the VSIs on the PF as well.

v3: Minor fixes, clean up log messages based on Shannon's comments.

---
 drivers/net/ethernet/intel/i40e/i40e.h             |    5 ++
 drivers/net/ethernet/intel/i40e/i40e_main.c        |   64 ++++++++++++++++++++
 drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c |   45 +-------------
 3 files changed, 71 insertions(+), 43 deletions(-)
drivers/net/ethernet/intel/i40e/i40e.h +5 -0
index ddb7292..2ee0197 100644
@@ -128,6 +128,10 @@
 /* default to trying for four seconds */
 #define I40E_TRY_LINK_TIMEOUT	(4 * HZ)
 
+/* BW rate limiting */
+#define I40E_BW_CREDIT_DIVISOR		50 /* 50Mbps per BW credit */
+#define I40E_MAX_BW_INACTIVE_ACCUM	4  /* accumulate 4 credits max */
+
 /* driver state flags */
 enum i40e_state_t {
 	__I40E_TESTING,
@@ -1042,4 +1046,5 @@ static inline bool i40e_enabled_xdp_vsi(struct i40e_vsi *vsi)
 }
 
 int i40e_create_queue_channel(struct i40e_vsi *vsi, struct i40e_channel *ch);
+int i40e_set_bw_limit(struct i40e_vsi *vsi, u16 seid, u64 max_tx_rate);
 #endif /* _I40E_H_ */
drivers/net/ethernet/intel/i40e/i40e_main.c +64 -0

70 lines not on this page · raw message

drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c +2 -43

65 lines not on this page · raw message

#10 lore raw

RFC PATCH v3 5/6 i40e: Refactor VF BW rate limiting

Quoted 15 lines

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Amritha Nambiar
Sent: Thursday, September 7, 2017 4:00 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Cc: netdev at vger.kernel.org
Subject: [Intel-wired-lan] [RFC PATCH v3 5/6] i40e: Refactor VF BW rate limiting

This patch refactors the BW rate limiting for Tx traffic on the VF to be reused in the next patch for rate limiting Tx traffic for the VSIs on the PF as well.

v3: Minor fixes, clean up log messages based on Shannon's comments.

Quoted 7 lines
---
 drivers/net/ethernet/intel/i40e/i40e.h             |    5 ++
 drivers/net/ethernet/intel/i40e/i40e_main.c        |   64
++++++++++++++++++++
 drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c |   45 +-------------
 3 files changed, 71 insertions(+), 43 deletions(-)
#11 lore raw

RFC PATCH v3 6/6 i40e: Add support setting TC max bandwidth rates

In reply to Amritha Nambiar

This patch enables setting up maximum Tx rates for the traffic classes in i40e. The maximum rate is offloaded to the hardware through the mqprio framework by specifying the mode option as 'channel' and shaper option as 'bw_rlimit' and is configured for the VSI. Configuring minimum Tx rate limit is not supported in the device. The minimum usable value for Tx rate is 50Mbps.

Example:

# tc qdisc add dev eth0 root mqprio num_tc 2  map 0 0 0 0 1 1 1 1\
  queues 4 at 0 4 at 4 hw 1 mode channel shaper bw_rlimit\
  max_rate 4Gbit 5Gbit

To dump the bandwidth rates:

# tc qdisc show dev eth0

qdisc mqprio 804a: root tc 2 map 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0
queues:(0:3) (4:7)
mode:channel
shaper:bw_rlimit max_rate:4Gbit 5Gbit

---
 drivers/net/ethernet/intel/i40e/i40e.h      |    2 +
 drivers/net/ethernet/intel/i40e/i40e_main.c |  101 +++++++++++++++++++++++++--
 2 files changed, 94 insertions(+), 9 deletions(-)
drivers/net/ethernet/intel/i40e/i40e.h +2 -0
index 2ee0197..02cae34 100644
@@ -359,6 +359,8 @@ struct i40e_channel {
 	u8 enabled_tc;
 	struct i40e_aqc_vsi_properties_data info;
 
+	u64 max_tx_rate;
+
 	/* track this channel belongs to which VSI */
 	struct i40e_vsi *parent_vsi;
 };
drivers/net/ethernet/intel/i40e/i40e_main.c +92 -9

192 lines not on this page · raw message

#12 lore raw

RFC PATCH v3 6/6 i40e: Add support setting TC max bandwidth rates

Quoted 30 lines

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Amritha Nambiar
Sent: Thursday, September 7, 2017 4:01 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Cc: netdev at vger.kernel.org
Subject: [Intel-wired-lan] [RFC PATCH v3 6/6] i40e: Add support setting TC max bandwidth rates

This patch enables setting up maximum Tx rates for the traffic classes in i40e. The maximum rate is offloaded to the hardware through the mqprio framework by specifying the mode option as 'channel' and shaper option as 'bw_rlimit' and is configured for the VSI. Configuring minimum Tx rate limit is not supported in the device. The minimum usable value for Tx rate is 50Mbps.

Example:

# tc qdisc add dev eth0 root mqprio num_tc 2  map 0 0 0 0 1 1 1 1\
  queues 4 at 0 4 at 4 hw 1 mode channel shaper bw_rlimit\
  max_rate 4Gbit 5Gbit

To dump the bandwidth rates:

# tc qdisc show dev eth0

qdisc mqprio 804a: root tc 2 map 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0
queues:(0:3) (4:7)
mode:channel
shaper:bw_rlimit max_rate:4Gbit 5Gbit

Quoted 6 lines
---
 drivers/net/ethernet/intel/i40e/i40e.h      |    2 +
 drivers/net/ethernet/intel/i40e/i40e_main.c |  101
+++++++++++++++++++++++++--
 2 files changed, 94 insertions(+), 9 deletions(-)
#13 lore raw
In reply to Amritha Nambiar
Quoted 42 lines · Amritha Nambiar

The following series introduces a new hardware offload mode in tc/mqprio where the TCs, the queue configurations and bandwidth rate limits are offloaded to the hardware. The existing mqprio framework is extended to configure the queue counts and layout and also added support for rate limiting. This is achieved through new netlink attributes for the 'mode' option which takes values such as 'dcb' (default) and 'channel' and a 'shaper' option for QoS attributes such as bandwidth rate limits in hw mode 1. Legacy devices can fall back to the existing setup supporting hw mode 1 without these additional options where only the TCs are offloaded and then the 'mode' and 'shaper' options defaults to DCB support. The i40e driver enables the new mqprio hardware offload mechanism factoring the TCs, queue configuration and bandwidth rates by creating HW channel VSIs.

In this new mode, the priority to traffic class mapping and the user specified queue ranges are used to configure the traffic class when the 'mode' option is set to 'channel'. This is achieved by creating HW channels(VSI). A new channel is created for each of the traffic class configuration offloaded via mqprio framework except for the first TC (TC0) which is for the main VSI. TC0 for the main VSI is also reconfigured as per user provided queue parameters. Finally, bandwidth rate limits are set on these traffic classes through the shaper attribute by sending these rates in addition to the number of TCs and the queue configurations.

Example:

     # tc qdisc add dev eth0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1\
       queues 4@0 4@4 hw 1 mode channel shaper bw_rlimit\
       min_rate 1Gbit 2Gbit max_rate 4Gbit 5Gbit

To dump the bandwidth rates:

# tc qdisc show dev eth0

qdisc mqprio 804a: root tc 2 map 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0
queues:(0:3) (4:7)
mode:channel
shaper:bw_rlimit min_rate:1Gbit 2Gbit max_rate:4Gbit 5Gbit

---

Quoted 8 lines

Amritha Nambiar (6):
mqprio: Introduce new hardware offload mode and shaper in mqprio
i40e: Add macro for PF reset bit
i40e: Add infrastructure for queue channel support
i40e: Enable 'channel' mode in mqprio for TC configs
i40e: Refactor VF BW rate limiting
i40e: Add support setting TC max bandwidth rates

It would be nice to know what has changed since the last review, either summarized here or in the individual patch files. This helps in knowing how much attention should be given to this new set of patches, and encourages further review. I don't remember seeing any responses to my previous comments, and it looks like not all of them were acted upon.

sln

Quoted 18 lines
  drivers/net/ethernet/intel/i40e/i40e.h             |   44 +
  drivers/net/ethernet/intel/i40e/i40e_debugfs.c     |    3
  drivers/net/ethernet/intel/i40e/i40e_ethtool.c     |    8
  drivers/net/ethernet/intel/i40e/i40e_main.c        | 1463 +++++++++++++++++---
  drivers/net/ethernet/intel/i40e/i40e_txrx.h        |    2
  drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c |   50 -
  include/net/pkt_cls.h                              |    9
  include/uapi/linux/pkt_sched.h                     |   32
  net/sched/sch_mqprio.c                             |  183 ++-
  9 files changed, 1551 insertions(+), 243 deletions(-)

--
_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#14 lore raw
Quoted 52 lines · Shannon Nelson
Quoted 42 lines · Amritha Nambiar

The following series introduces a new hardware offload mode in tc/mqprio where the TCs, the queue configurations and bandwidth rate limits are offloaded to the hardware. The existing mqprio framework is extended to configure the queue counts and layout and also added support for rate limiting. This is achieved through new netlink attributes for the 'mode' option which takes values such as 'dcb' (default) and 'channel' and a 'shaper' option for QoS attributes such as bandwidth rate limits in hw mode 1. Legacy devices can fall back to the existing setup supporting hw mode 1 without these additional options where only the TCs are offloaded and then the 'mode' and 'shaper' options defaults to DCB support. The i40e driver enables the new mqprio hardware offload mechanism factoring the TCs, queue configuration and bandwidth rates by creating HW channel VSIs.

In this new mode, the priority to traffic class mapping and the user specified queue ranges are used to configure the traffic class when the 'mode' option is set to 'channel'. This is achieved by creating HW channels(VSI). A new channel is created for each of the traffic class configuration offloaded via mqprio framework except for the first TC (TC0) which is for the main VSI. TC0 for the main VSI is also reconfigured as per user provided queue parameters. Finally, bandwidth rate limits are set on these traffic classes through the shaper attribute by sending these rates in addition to the number of TCs and the queue configurations.

Example:

     # tc qdisc add dev eth0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1\
       queues 4 at 0 4 at 4 hw 1 mode channel shaper bw_rlimit\
       min_rate 1Gbit 2Gbit max_rate 4Gbit 5Gbit

To dump the bandwidth rates:

# tc qdisc show dev eth0

qdisc mqprio 804a: root tc 2 map 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0
queues:(0:3) (4:7)
mode:channel
shaper:bw_rlimit min_rate:1Gbit 2Gbit max_rate:4Gbit 5Gbit

---

Quoted 8 lines

Amritha Nambiar (6):
mqprio: Introduce new hardware offload mode and shaper in mqprio
i40e: Add macro for PF reset bit
i40e: Add infrastructure for queue channel support
i40e: Enable 'channel' mode in mqprio for TC configs
i40e: Refactor VF BW rate limiting
i40e: Add support setting TC max bandwidth rates

Quoted 5 lines

It would be nice to know what has changed since the last review, either summarized here or in the individual patch files. This helps in knowing how much attention should be given to this new set of patches, and encourages further review. I don't remember seeing any responses to my previous comments, and it looks like not all of them were acted upon.

For all those patch files that have changed since the last revision, I have captured all the new changes in the section titled "v3: " of each patch file. Also, I had replied to most of your comments that they'll be fixed in v3 and the one in the mqprio patch that error handling was already being done and these responses can be verified in patchwork. I missed to respond to your comment regarding supporting macvlan offloads through these channels, that will not be a part of this series and will be looked upon at a later time in a subsequent patch series.

Quoted 21 lines

sln

Quoted 18 lines
  drivers/net/ethernet/intel/i40e/i40e.h             |   44 +
  drivers/net/ethernet/intel/i40e/i40e_debugfs.c     |    3
  drivers/net/ethernet/intel/i40e/i40e_ethtool.c     |    8
  drivers/net/ethernet/intel/i40e/i40e_main.c        | 1463 +++++++++++++++++---
  drivers/net/ethernet/intel/i40e/i40e_txrx.h        |    2
  drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c |   50 -
  include/net/pkt_cls.h                              |    9
  include/uapi/linux/pkt_sched.h                     |   32
  net/sched/sch_mqprio.c                             |  183 ++-
  9 files changed, 1551 insertions(+), 243 deletions(-)

--
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

#15 lore raw
Quoted 1 line · Amritha

On 9/7/2017 9:45 AM, Shannon Nelson wrote:

[...]

Quoted 3 lines
Quoted 3 lines

It would be nice to know what has changed since the last review, either summarized here or in the individual patch files.

[...]

Quoted 4 lines

For all those patch files that have changed since the last revision, I have captured all the new changes in the section titled "v3: " of each patch file.

Mea culpa.

ugh.
sln

#16 lore raw
In reply to Amritha Nambiar
Quoted 8 lines · Amritha Nambiar

The following series introduces a new hardware offload mode in tc/mqprio where the TCs, the queue configurations and bandwidth rate limits are offloaded to the hardware. The existing mqprio framework is extended to configure the queue counts and layout and also added support for rate limiting. This is achieved through new netlink attributes for the 'mode' option which takes values such as 'dcb' (default) and 'channel' and a 'shaper' option for QoS attributes such as bandwidth rate limits in hw mode 1.

So "dcb" defines a default priorities to queue mapping?

Quoted 3 lines

Legacy devices can fall back to the existing setup supporting hw mode 1 without these additional options where only the TCs are offloaded and then the 'mode' and 'shaper' options defaults to DCB support.

That's the last part that confuses me, see below.

Quoted 3 lines

The i40e driver enables the new mqprio hardware offload mechanism factoring the TCs, queue configuration and bandwidth rates by creating HW channel VSIs.

I am really confused by what you call hw_mode 1, as I understand it there are really 3 different modes:

- legacy: you don't define any traffic class mapping, but you can still chain this scheduler with a match + action (like what Documentation/networking/multiqueue.txt) you can optionally also add "shaper" arguments, but there should not be any default DCB queue mapping either?

- dcb: a default mapping for traffic classes to queues is defined, optional "shaper" arguments

- channel: (maybe calling that "custom_tc_map" would be clearer?) where you express the exact traffic classes to queue mapping and optional "shaper" arguments

I think that's what you are doing, but I just got confused by the cover letter.

Quoted 12 lines

In this new mode, the priority to traffic class mapping and the user specified queue ranges are used to configure the traffic class when the 'mode' option is set to 'channel'. This is achieved by creating HW channels(VSI). A new channel is created for each of the traffic class configuration offloaded via mqprio framework except for the first TC (TC0) which is for the main VSI. TC0 for the main VSI is also reconfigured as per user provided queue parameters. Finally, bandwidth rate limits are set on these traffic classes through the shaper attribute by sending these rates in addition to the number of TCs and the queue configurations.

Quoted 3 lines

Example:

    # tc qdisc add dev eth0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1\
      queues 4 at 0 4 at 4 hw 1 mode channel shaper bw_rlimit\

Do you see a case where you can declare a different number of traffic classes say 4 and map them onto just 2 hardware queues? If not, it seems a tiny bit redundant to have to specify both the map and the queue mapping should be sufficient, right?

Quoted 10 lines

min_rate 1Gbit 2Gbit max_rate 4Gbit 5Gbit

To dump the bandwidth rates:

# tc qdisc show dev eth0

qdisc mqprio 804a: root tc 2 map 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0
queues:(0:3) (4:7)
mode:channel
shaper:bw_rlimit min_rate:1Gbit 2Gbit max_rate:4Gbit 5Gbit

I am not well versed into tc, but being able to specify "shaper" arguments has actually value outside of just the multiq scheduler and it could probably be an action on its own?

Quoted 25 lines

---

Amritha Nambiar (6):
mqprio: Introduce new hardware offload mode and shaper in mqprio
i40e: Add macro for PF reset bit
i40e: Add infrastructure for queue channel support
i40e: Enable 'channel' mode in mqprio for TC configs
i40e: Refactor VF BW rate limiting
i40e: Add support setting TC max bandwidth rates

 drivers/net/ethernet/intel/i40e/i40e.h             |   44 +
 drivers/net/ethernet/intel/i40e/i40e_debugfs.c     |    3 
 drivers/net/ethernet/intel/i40e/i40e_ethtool.c     |    8 
 drivers/net/ethernet/intel/i40e/i40e_main.c        | 1463 +++++++++++++++++---
 drivers/net/ethernet/intel/i40e/i40e_txrx.h        |    2 
 drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c |   50 -
 include/net/pkt_cls.h                              |    9 
 include/uapi/linux/pkt_sched.h                     |   32 
 net/sched/sch_mqprio.c                             |  183 ++-
 9 files changed, 1551 insertions(+), 243 deletions(-)

--

Signature · 1 line
Florian
#17 lore raw
Quoted 11 lines · Florian Fainelli
Quoted 8 lines · Amritha Nambiar

The following series introduces a new hardware offload mode in tc/mqprio where the TCs, the queue configurations and bandwidth rate limits are offloaded to the hardware. The existing mqprio framework is extended to configure the queue counts and layout and
also added support for rate limiting. This is achieved through new netlink attributes for the 'mode' option which takes values such
as 'dcb' (default) and 'channel' and a 'shaper' option for QoS attributes such as bandwidth rate limits in hw mode 1.

So "dcb" defines a default priorities to queue mapping?

In the default offload implementation, only the basic hw offload was supported factoring only the 'number of TCs' values. The rest of the queue configuration was being ignored. This is the legacy behavior with hw mode set to 1.
Example:

# tc qdisc add dev eth0 root mqprio num_tc 4  map 0 0 0 0 1 1 1 1 queues
4 at 0 4 at 4 hw 1
I just named this default behavior to 'dcb' while introducing a new
offload mechanism.
Quoted 7 lines
Quoted 5 lines

Legacy devices can fall back to the existing setup supporting hw mode 1 without these additional options where only the TCs are offloaded and then the 'mode' and 'shaper' options defaults to DCB
support.

That's the last part that confuses me, see below.

As I introduced new options for 'mode' and 'shaper', I set the defaults for these options to 'dcb' so existing offloaders can continue to work without supporting these new options. Patch 1 has a detailed description on how this is done.

Quoted 7 lines
Quoted 4 lines

The i40e driver enables the new mqprio hardware offload mechanism factoring the TCs, queue configuration and bandwidth rates by creating HW channel VSIs.

I am really confused by what you call hw_mode 1, as I understand it there are really 3 different modes:

There are actually 2 modes now with 'hw' option set to 1, legacy/dcb and channel.

Quoted 9 lines

- legacy: you don't define any traffic class mapping, but you can still chain this scheduler with a match + action (like what Documentation/networking/multiqueue.txt) you can optionally also add "shaper" arguments, but there should not be any default DCB queue mapping either?

- dcb: a default mapping for traffic classes to queues is defined, optional "shaper" arguments

The legacy mode now becomes the dcb mode. In this mode, although the TC values, the queue configurations, prio-tc-mapping are all offloaded to the device, the existing implementation in current drivers support only a basic hw offload factoring only the TC values.

Examples:

# ... num_tc 2  map 0 0 0 0 1 1 1 1 queues 4 at 0 4 at 4 hw 1
# ... num_tc 2  map 0 0 0 0 1 1 1 1 queues 4 at 0 4 at 4 hw 1 mode dcb
# ... num_tc 2  map 0 0 0 0 1 1 1 1 queues 4 at 0 4 at 4 hw 1 mode dcb\
  shaper dcb
Quoted 4 lines

- channel: (maybe calling that "custom_tc_map" would be clearer?) where you express the exact traffic classes to queue mapping and optional "shaper" arguments

In the channel mode, a full hw offload is supported, the TC values, the queue configurations and additionally QoS attributes (optional) are all used in the new implementation.

Examples:

# ... num_tc 2  map 0 0 0 0 1 1 1 1 queues 4 at 0 4 at 4 hw 1 mode channel
# ... num_tc 2  map 0 0 0 0 1 1 1 1 queues 4 at 0 4 at 4 hw 1 mode channel\
shaper bw_rlimit max_rate 4Gbit 5Gbit
Quoted 16 lines

I think that's what you are doing, but I just got confused by the cover letter.

Quoted 12 lines

In this new mode, the priority to traffic class mapping and the user specified queue ranges are used to configure the traffic class when the 'mode' option is set to 'channel'. This is achieved by creating HW channels(VSI). A new channel is created for each of the traffic class configuration offloaded via mqprio framework except for the first TC (TC0) which is for the main VSI. TC0 for the main VSI is also reconfigured as per user provided queue parameters. Finally, bandwidth rate limits are set on these traffic classes through the shaper attribute by sending these rates in addition to the number of TCs and the queue configurations.

Quoted 7 lines
Quoted 2 lines

Example: # tc qdisc add dev eth0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1\ queues 4 at 0 4 at 4 hw 1 mode channel shaper bw_rlimit\

Do you see a case where you can declare a different number of traffic classes say 4 and map them onto just 2 hardware queues? If not, it seems a tiny bit redundant to have to specify both the map and the queue mapping should be sufficient, right?

This will be subjected to validation of the user input and will be treated as invalid configuration. The 'map' specifies the mapping between user priorities and traffic classes while the queue mapping is the queue layout specifying the queue count and offsets.

Quoted 7 lines
Quoted 7 lines

min_rate 1Gbit 2Gbit max_rate 4Gbit 5Gbit

To dump the bandwidth rates:

# tc qdisc show dev eth0
Quoted 7 lines
Quoted 3 lines

qdisc mqprio 804a: root tc 2 map 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0
queues:(0:3) (4:7) mode:channel shaper:bw_rlimit min_rate:1Gbit
2Gbit max_rate:4Gbit 5Gbit

I am not well versed into tc, but being able to specify "shaper" arguments has actually value outside of just the multiq scheduler and it could probably be an action on its own?

The mqprio scheduler already supports configuring the traffic classes and enabling support for HW shapers was just a matter to extending it with new netlink based attributes.

Quoted 26 lines
Quoted 24 lines

---

Amritha Nambiar (6): mqprio: Introduce new hardware offload mode and shaper in mqprio i40e: Add macro for PF reset bit i40e: Add infrastructure for queue channel support i40e: Enable 'channel' mode in mqprio for TC configs i40e: Refactor VF BW rate limiting
i40e: Add support setting TC max bandwidth rates

drivers/net/ethernet/intel/i40e/i40e.h             |   44 + 
drivers/net/ethernet/intel/i40e/i40e_debugfs.c     |    3 
drivers/net/ethernet/intel/i40e/i40e_ethtool.c     |    8 
drivers/net/ethernet/intel/i40e/i40e_main.c        | 1463 
+++++++++++++++++--- drivers/net/ethernet/intel/i40e/i40e_txrx.h | 
2 drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c |   50 - 
include/net/pkt_cls.h                              |    9 
include/uapi/linux/pkt_sched.h                     |   32 
net/sched/sch_mqprio.c                             |  183 ++- 9 

files changed, 1551 insertions(+), 243 deletions(-)

--