Skip to contentSkip to conversation

net/mlx5: add hardware steering

PATCH Superseded Suanming Mou 3 participants 62 messages Feb 10, 2022 – Feb 24, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover net/mlx5: add hardware steering

As the new queue-based flow rules management mechanism is introduced to the rte_flow. A new steering mode PMD code is also developed to take advantage of the new API.

The HW steering prepares the required flow resources in advanced from the hints when create template table. That means the matcher and most of the actions have been prepared during flow table creation. Limited left actions will be constructed during flow creation if needed.

A flow postpone attribute bit describes if flow creation/destruction should be applied to the HW directly or not. An extra push function has also been prepared to force push all the enqueued flows to HW.

Once the flow has been applied to the HW, the pull function will be called to get the enqueued creation/destruction flows.

The asynchronous flow creation and destruction are handled by queue jobs. The queue job descriptor is currently introduced to convey the flow information and operation type between the flow management in in pull function.

Flow rule memory is allocated in PMD layer instead of allocating from HW layer. While destroying the flow, the flow rule memory can only be freed after the event received.

*** THIS PATCH SET DEPENDS ON THE NEW RTE_FLOW API ***

Suanming Mou (13):
net/mlx5: introduce hardware steering operation net/mlx5: introduce hardware steering enable routine net/mlx5: add port flow configuration net/mlx5: add pattern template management net/mlx5: add action template management net/mlx5: add table management
net/mlx5: add basic flow queue operation net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action

 drivers/net/mlx5/linux/mlx5_flow_os.h |    1 +
 drivers/net/mlx5/linux/mlx5_os.c      |   22 +-
 drivers/net/mlx5/meson.build          |    1 +
 drivers/net/mlx5/mlx5.c               |   50 +-
 drivers/net/mlx5/mlx5.h               |   63 +-
 drivers/net/mlx5/mlx5_devx.c          |   10 +
 drivers/net/mlx5/mlx5_flow.c          |  569 +++++-
 drivers/net/mlx5/mlx5_flow.h          |  276 +++
 drivers/net/mlx5/mlx5_flow_dv.c       |  175 +-
 drivers/net/mlx5/mlx5_flow_hw.c       | 2284 +++++++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow_verbs.c    |    7 +-
 drivers/net/mlx5/mlx5_rx.h            |    9 +-
 drivers/net/mlx5/mlx5_rxq.c           |   78 +-
 13 files changed, 3374 insertions(+), 171 deletions(-)
 create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/13 net/mlx5: introduce hardware steering operation

A new hardware based steering operation is going to be introduced for high insertion rate.

This commit adds the basic driver operation.

---
 drivers/net/mlx5/linux/mlx5_flow_os.h |  1 +
 drivers/net/mlx5/meson.build          |  1 +
 drivers/net/mlx5/mlx5_flow.c          |  1 +
 drivers/net/mlx5/mlx5_flow.h          |  1 +
 drivers/net/mlx5/mlx5_flow_hw.c       | 13 +++++++++++++
 5 files changed, 17 insertions(+)
 create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
drivers/net/mlx5/linux/mlx5_flow_os.h +1 -0
index 1926d26410..e28a9e0436 100644
@@ -9,6 +9,7 @@
 
 #ifdef HAVE_IBV_FLOW_DV_SUPPORT
 extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
 #endif
 
 /**
drivers/net/mlx5/meson.build +1 -0
index 7d12dccdd4..edd4f126b3 100644
@@ -16,6 +16,7 @@ sources = files(
         'mlx5_flow.c',
         'mlx5_flow_meter.c',
         'mlx5_flow_dv.c',
+        'mlx5_flow_hw.c',
         'mlx5_flow_aso.c',
         'mlx5_flow_flex.c',
         'mlx5_mac.c',
drivers/net/mlx5/mlx5_flow.c +1 -0
index d7cb1eb89b..21d17aca44 100644
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = {
 	[MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	[MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops,
+	[MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops,
 #endif
 	[MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops,
 	[MLX5_FLOW_TYPE_MAX] = &mlx5_flow_null_drv_ops
drivers/net/mlx5/mlx5_flow.h +1 -0
index 7fec79afb3..26f5d97a7d 100644
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {
 	MLX5_FLOW_TYPE_MIN,
 	MLX5_FLOW_TYPE_DV,
 	MLX5_FLOW_TYPE_VERBS,
+	MLX5_FLOW_TYPE_HW,
 	MLX5_FLOW_TYPE_MAX,
 };
 
drivers/net/mlx5/mlx5_flow_hw.c new file +13 -0
new file mode 100644
index 0000000000..33875c7d08
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#include <rte_flow.h>
+
+#include "mlx5_flow.h"
+
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+
+#endif
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/13 net/mlx5: introduce hardware steering enable routine

In reply to Suanming Mou

As the new hardware steering operation will be implemented under the new rte_flow_q APIs. This is not compatible with the existing rte_flow PMD's Direct Rules flow operation routine.

This commit introduces an extra dv_flow_en = 2 to specify the new flow operation initialize routine.

---
 drivers/net/mlx5/linux/mlx5_os.c | 4 ++++
 drivers/net/mlx5/mlx5.c          | 2 +-
 drivers/net/mlx5/mlx5.h          | 3 ++-
 3 files changed, 7 insertions(+), 2 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +4 -0
index aecdc5a68a..52e52a4ad7 100644
@@ -295,6 +295,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
 	err = mlx5_alloc_table_hash_list(priv);
 	if (err)
 		goto error;
+	if (priv->config.dv_flow_en == 2)
+		return 0;
 	/* The resources below are only valid with DV support. */
 #ifdef HAVE_IBV_FLOW_DV_SUPPORT
 	/* Init port id action list. */
@@ -1712,6 +1714,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 	priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
 	if (!priv->drop_queue.hrxq)
 		goto error;
+	if (priv->config.dv_flow_en == 2)
+		return eth_dev;
 	/* Port representor shares the same max priority with pf port. */
 	if (!priv->sh->flow_priority_check_flag) {
 		/* Supported Verbs flow priority number detection. */
drivers/net/mlx5/mlx5.c +1 -1
index 67eda41a60..a4826a583b 100644
@@ -1933,7 +1933,7 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
 	} else if (strcmp(MLX5_DV_ESW_EN, key) == 0) {
 		config->dv_esw_en = !!tmp;
 	} else if (strcmp(MLX5_DV_FLOW_EN, key) == 0) {
-		config->dv_flow_en = !!tmp;
+		config->dv_flow_en = tmp;
 	} else if (strcmp(MLX5_DV_XMETA_EN, key) == 0) {
 		if (tmp != MLX5_XMETA_MODE_LEGACY &&
 		    tmp != MLX5_XMETA_MODE_META16 &&
drivers/net/mlx5/mlx5.h +2 -1
index 737ad6895c..f3b991e549 100644
@@ -259,7 +259,8 @@ struct mlx5_dev_config {
 	unsigned int l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
 	unsigned int vf_nl_en:1; /* Enable Netlink requests in VF mode. */
 	unsigned int dv_esw_en:1; /* Enable E-Switch DV flow. */
-	unsigned int dv_flow_en:1; /* Enable DV flow. */
+	/* Enable DV flow. 1 means SW steering, 2 means HW steering. */
+	unsigned int dv_flow_en:2;
 	unsigned int dv_xmeta_en:2; /* Enable extensive flow metadata. */
 	unsigned int lacp_by_user:1;
 	/* Enable user to manage LACP traffic. */
Signature · 1 line
2.25.1
#3 lore raw

PATCH 3/13 net/mlx5: add port flow configuration

In reply to Suanming Mou

The hardware steering is backend to support rte_flow_q API in mlx5 PMD. The port configuration function creates the queues and needed flow management resources.

The PMD layer configuration function allocates the queues' context and per-queue job descriptor. The job descriptor size is equal to the queue size, and the job descriptors will be popped from LIFO to convey the flow information during flow insertion/destruction. So when polling the result, the flow information will be extracted from the descriptor and then the job descriptor will be push back to the LIFO.

The commit creates the flow port queue and the job descriptors.

---
 drivers/net/mlx5/mlx5.c         |   3 +
 drivers/net/mlx5/mlx5.h         |  26 ++++++-
 drivers/net/mlx5/mlx5_flow.c    |  37 +++++++++
 drivers/net/mlx5/mlx5_flow.h    |   9 +++
 drivers/net/mlx5/mlx5_flow_hw.c | 132 ++++++++++++++++++++++++++++++++
 5 files changed, 206 insertions(+), 1 deletion(-)
drivers/net/mlx5/mlx5.c +3 -0
index a4826a583b..f1933fd253 100644
@@ -1567,6 +1567,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)
 	/* Free the eCPRI flex parser resource. */
 	mlx5_flex_parser_ecpri_release(dev);
 	mlx5_flex_item_port_cleanup(dev);
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+	flow_hw_resource_release(dev);
+#endif
 	if (priv->rxq_privs != NULL) {
 		/* XXX race condition if mlx5_rx_burst() is still running. */
 		rte_delay_us_sleep(1000);
drivers/net/mlx5/mlx5.h +25 -1
index f3b991e549..31a13ca69a 100644
@@ -33,7 +33,7 @@
 #include "mlx5_utils.h"
 #include "mlx5_os.h"
 #include "mlx5_autoconf.h"
-
+#include "mlx5dr.h"
 
 #define MLX5_SH(dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
 
@@ -324,6 +324,26 @@ struct mlx5_lb_ctx {
 	uint16_t refcnt; /* Reference count for representors. */
 };
 
+/* HW steering queue job descriptor type. */
+enum {
+	MLX5_HW_Q_JOB_TYPE_CREATE, /* Flow create job type. */
+	MLX5_HW_Q_JOB_TYPE_DESTROY, /* Flow destroy job type. */
+};
+
+/* HW steering flow management job descriptor. */
+struct mlx5_hw_q_job {
+	uint32_t type; /* Job type. */
+	struct rte_flow *flow; /* Flow attached to the job. */
+	void *user_data; /* Job user data. */
+};
+
+/* HW steering job descriptor LIFO header . */
+struct mlx5_hw_q {
+	uint32_t job_idx; /* Free job index. */
+	uint32_t size; /* LIFO size. */
+	struct mlx5_hw_q_job **job; /* LIFO pointer. */
+} __rte_cache_aligned;
+
 #define MLX5_COUNTERS_PER_POOL 512
 #define MLX5_MAX_PENDING_QUERIES 4
 #define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1480,6 +1500,10 @@ struct mlx5_priv {
 	struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM];
 	/* Flex items have been created on the port. */
 	uint32_t flex_item_map; /* Map of allocated flex item elements. */
+	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
+	uint32_t nb_queue; /* HW steering queue number. */
+	/* HW steering queue polling mechanism job descriptor LIFO. */
+	struct mlx5_hw_q *hw_q;
 };
 
 #define PORT_ID(priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.c +37 -0
index 21d17aca44..5ff96642b4 100644
@@ -805,6 +805,12 @@ static int
 mlx5_flow_flex_item_release(struct rte_eth_dev *dev,
 			    const struct rte_flow_item_flex_handle *handle,
 			    struct rte_flow_error *error);
+static int
+mlx5_flow_port_configure(struct rte_eth_dev *dev,
+			 const struct rte_flow_port_attr *port_attr,
+			 uint16_t nb_queue,
+			 const struct rte_flow_queue_attr *queue_attr[],
+			 struct rte_flow_error *err);
 
 static const struct rte_flow_ops mlx5_flow_ops = {
 	.validate = mlx5_flow_validate,
@@ -826,6 +832,7 @@ static const struct rte_flow_ops mlx5_flow_ops = {
 	.get_restore_info = mlx5_flow_tunnel_get_restore_info,
 	.flex_item_create = mlx5_flow_flex_item_create,
 	.flex_item_release = mlx5_flow_flex_item_release,
+	.configure = mlx5_flow_port_configure,
 };
 
 /* Tunnel information. */
@@ -7814,6 +7821,36 @@ mlx5_counter_query(struct rte_eth_dev *dev, uint32_t cnt,
 	return -ENOTSUP;
 }
 
+/**
+ * Configure port HWS resources.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] port_attr
+ *   Port configuration attributes.
+ * @param[in] nb_queue
+ *   Number of queue.
+ * @param[in] queue_attr
+ *   Array that holds attributes for each flow queue.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_port_configure(struct rte_eth_dev *dev,
+			 const struct rte_flow_port_attr *port_attr,
+			 uint16_t nb_queue,
+			 const struct rte_flow_queue_attr *queue_attr[],
+			 struct rte_flow_error *err)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->configure(dev, port_attr, nb_queue, queue_attr, err);
+}
+
 /**
  * Allocate a new memory for the counter values wrapped by all the needed
  * management.
drivers/net/mlx5/mlx5_flow.h +9 -0
index 26f5d97a7d..731478ff05 100644
@@ -1257,6 +1257,12 @@ typedef int (*mlx5_flow_item_update_t)
 			 const struct rte_flow_item_flex_handle *handle,
 			 const struct rte_flow_item_flex_conf *conf,
 			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_port_configure_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_port_attr *port_attr,
+			 uint16_t nb_queue,
+			 const struct rte_flow_queue_attr *queue_attr[],
+			 struct rte_flow_error *err);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1295,6 +1301,7 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_item_create_t item_create;
 	mlx5_flow_item_release_t item_release;
 	mlx5_flow_item_update_t item_update;
+	mlx5_flow_port_configure_t configure;
 };
 
 /* mlx5_flow.c */
@@ -1767,4 +1774,6 @@ const struct mlx5_flow_tunnel *
 mlx5_get_tof(const struct rte_flow_item *items,
 	     const struct rte_flow_action *actions,
 	     enum mlx5_tof_rule_type *rule_type);
+void
+flow_hw_resource_release(struct rte_eth_dev *dev);
 #endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_hw.c +132 -0

142 lines not on this page · raw message

Signature · 1 line
2.25.1
#4 lore raw

PATCH 4/13 net/mlx5: add pattern template management

In reply to Suanming Mou

The pattern template defines flows that have the same matching fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be (eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while the values for each rule will be different.

Due to the pattern template can be used in different domains, the items will only be cached in pattern template create stage, while the template is binded to a dedicated table, the HW criteria will be created and saved to the table. And different tables may create the same criteria and will not shared between each other in order to have better performance.

This commit adds pattern template management.

---
 drivers/net/mlx5/mlx5.h         |  2 +
 drivers/net/mlx5/mlx5_flow.c    | 64 +++++++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    | 20 ++++++++
 drivers/net/mlx5/mlx5_flow_hw.c | 82 +++++++++++++++++++++++++++++++++
 4 files changed, 168 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
index 31a13ca69a..96048ad0ea 100644
@@ -1500,6 +1500,8 @@ struct mlx5_priv {
 	struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM];
 	/* Flex items have been created on the port. */
 	uint32_t flex_item_map; /* Map of allocated flex item elements. */
+	/* Item template list. */
+	LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
 	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +64 -0

82 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +20 -0
index 731478ff05..88102f0991 100644
@@ -1015,6 +1015,15 @@ struct rte_flow {
 	uint32_t geneve_tlv_option; /**< Holds Geneve TLV option id. > */
 } __rte_packed;
 
+/* Flow item template struct. */
+struct rte_flow_pattern_template {
+	LIST_ENTRY(rte_flow_pattern_template) next;
+	/* Template attributes. */
+	struct rte_flow_pattern_template_attr attr;
+	struct mlx5dr_match_template *mt; /* mlx5 match template. */
+	uint32_t refcnt;  /* Reference counter. */
+};
+
 /*
  * Define list of valid combinations of RX Hash fields
  * (see enum ibv_rx_hash_fields).
@@ -1263,6 +1272,15 @@ typedef int (*mlx5_flow_port_configure_t)
 			 uint16_t nb_queue,
 			 const struct rte_flow_queue_attr *queue_attr[],
 			 struct rte_flow_error *err);
+typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_pattern_template_attr *attr,
+			 const struct rte_flow_item items[],
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_pattern_template_destroy_t)
+			(struct rte_eth_dev *dev,
+			 struct rte_flow_pattern_template *template,
+			 struct rte_flow_error *error);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1302,6 +1320,8 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_item_release_t item_release;
 	mlx5_flow_item_update_t item_update;
 	mlx5_flow_port_configure_t configure;
+	mlx5_flow_pattern_template_create_t pattern_template_create;
+	mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
 };
 
 /* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +82 -0

103 lines not on this page · raw message

Signature · 1 line
2.25.1
#5 lore raw

PATCH 5/13 net/mlx5: add action template management

In reply to Suanming Mou

The action template holds a list of action types that will be used together on the same rule. The template's actions instances will be created only when the template bind to the dedicated group. And the created actions will be saved to each individual group in order for best performance. The actions will not be shared with each other.

This commit adds the action template management which caches the flow action template.

---
 drivers/net/mlx5/mlx5.h         |   2 +
 drivers/net/mlx5/mlx5_flow.c    |  66 ++++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    |  27 ++++++++
 drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++
 4 files changed, 211 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
index 96048ad0ea..80dc72175d 100644
@@ -1502,6 +1502,8 @@ struct mlx5_priv {
 	uint32_t flex_item_map; /* Map of allocated flex item elements. */
 	/* Item template list. */
 	LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
+	/* Action template list. */
+	LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;
 	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +66 -0

84 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +27 -0
index 88102f0991..f5ababb32f 100644
@@ -1024,6 +1024,21 @@ struct rte_flow_pattern_template {
 	uint32_t refcnt;  /* Reference counter. */
 };
 
+/* Flow action template attribute. */
+struct rte_flow_actions_template_attr {
+	int32_t reserve;
+};
+
+/* Flow action template struct. */
+struct rte_flow_actions_template {
+	LIST_ENTRY(rte_flow_actions_template) next;
+	/* Template attributes. */
+	struct rte_flow_actions_template_attr attr;
+	struct rte_flow_action *actions; /* Cached flow actions. */
+	struct rte_flow_action *masks; /* Cached action masks.*/
+	uint32_t refcnt; /* Reference counter. */
+};
+
 /*
  * Define list of valid combinations of RX Hash fields
  * (see enum ibv_rx_hash_fields).
@@ -1281,6 +1296,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t)
 			(struct rte_eth_dev *dev,
 			 struct rte_flow_pattern_template *template,
 			 struct rte_flow_error *error);
+typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_actions_template_attr *attr,
+			 const struct rte_flow_action actions[],
+			 const struct rte_flow_action masks[],
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_actions_template_destroy_t)
+			(struct rte_eth_dev *dev,
+			 struct rte_flow_actions_template *template,
+			 struct rte_flow_error *error);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1322,6 +1347,8 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_port_configure_t configure;
 	mlx5_flow_pattern_template_create_t pattern_template_create;
 	mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
+	mlx5_flow_actions_template_create_t actions_template_create;
+	mlx5_flow_actions_template_destroy_t actions_template_destroy;
 };
 
 /* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +116 -0

140 lines not on this page · raw message

Signature · 1 line
2.25.1
#6 lore raw

PATCH 6/13 net/mlx5: add table management

In reply to Suanming Mou

Flow table is a group of flows with the same matching criteria and the same actions defined for them. The table defines rules that have the same matching fields but with different matching values. For example, matching on 5 tuple, the table will be (IPv4 source + IPv4 dest + s_port + d_port + next_proto) while the values for each rule will be different.

The templates' relevant matching criteria and action instances will be created in the table creation and saved in the table. As table attributes indicate the supported flow number, the flow memory will also be allocated at the same time.

This commit adds the table management functions.

---
 drivers/net/mlx5/mlx5.c         |  45 ++-
 drivers/net/mlx5/mlx5.h         |  21 +-
 drivers/net/mlx5/mlx5_flow.c    |  81 +++++
 drivers/net/mlx5/mlx5_flow.h    |  72 +++++
 drivers/net/mlx5/mlx5_flow_hw.c | 522 ++++++++++++++++++++++++++++++++
 5 files changed, 735 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.c +41 -4
index f1933fd253..51f3d9bf99 100644
@@ -1354,12 +1354,46 @@ void
 mlx5_free_table_hash_list(struct mlx5_priv *priv)
 {
 	struct mlx5_dev_ctx_shared *sh = priv->sh;
-
-	if (!sh->flow_tbls)
+	struct mlx5_hlist **tbls = (priv->config.dv_flow_en == 2) ?
+				   &sh->groups : &sh->flow_tbls;
+	if (*tbls == NULL)
 		return;
-	mlx5_hlist_destroy(sh->flow_tbls);
-	sh->flow_tbls = NULL;
+	mlx5_hlist_destroy(*tbls);
+	*tbls = NULL;
+}
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/**
+ * Allocate HW steering group hash list.
+ *
+ * @param[in] priv
+ *   Pointer to the private device data structure.
+ */
+static int
+mlx5_alloc_hw_group_hash_list(struct mlx5_priv *priv)
+{
+	int err = 0;
+	struct mlx5_dev_ctx_shared *sh = priv->sh;
+	char s[MLX5_NAME_SIZE];
+
+	MLX5_ASSERT(sh);
+	snprintf(s, sizeof(s), "%s_flow_groups", priv->sh->ibdev_name);
+	sh->groups = mlx5_hlist_create
+			(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
+			 false, true, sh,
+			 flow_hw_grp_create_cb,
+			 flow_hw_grp_match_cb,
+			 flow_hw_grp_remove_cb,
+			 flow_hw_grp_clone_cb,
+			 flow_hw_grp_clone_free_cb);
+	if (!sh->groups) {
+		DRV_LOG(ERR, "flow groups with hash creation failed.");
+		err = ENOMEM;
+	}
+	return err;
 }
+#endif
+
 
 /**
  * Initialize flow table hash list and create the root tables entry
@@ -1375,11 +1409,14 @@ int
 mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused)
 {
 	int err = 0;
+
 	/* Tables are only used in DV and DR modes. */
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	struct mlx5_dev_ctx_shared *sh = priv->sh;
 	char s[MLX5_NAME_SIZE];
 
+	if (priv->config.dv_flow_en == 2)
+		return mlx5_alloc_hw_group_hash_list(priv);
 	MLX5_ASSERT(sh);
 	snprintf(s, sizeof(s), "%s_flow_table", priv->sh->ibdev_name);
 	sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
drivers/net/mlx5/mlx5.h +19 -2
index 80dc72175d..a4bc8d1fb7 100644
@@ -62,7 +62,9 @@ enum mlx5_ipool_index {
 	MLX5_IPOOL_PUSH_VLAN, /* Pool for push vlan resource. */
 	MLX5_IPOOL_TAG, /* Pool for tag resource. */
 	MLX5_IPOOL_PORT_ID, /* Pool for port id resource. */
-	MLX5_IPOOL_JUMP, /* Pool for jump resource. */
+	MLX5_IPOOL_JUMP, /* Pool for SWS jump resource. */
+	/* Pool for HWS group. Jump action will be created internally. */
+	MLX5_IPOOL_HW_GRP = MLX5_IPOOL_JUMP,
 	MLX5_IPOOL_SAMPLE, /* Pool for sample resource. */
 	MLX5_IPOOL_DEST_ARRAY, /* Pool for destination array resource. */
 	MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -106,6 +108,13 @@ enum mlx5_delay_drop_mode {
 	MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */
 };
 
+/* The HWS action type root/non-root. */
+enum mlx5_hw_action_flag_type {
+	MLX5_HW_ACTION_FLAG_ROOT, /* Root action. */
+	MLX5_HW_ACTION_FLAG_NONE_ROOT, /* Non-root ation. */
+	MLX5_HW_ACTION_FLAG_MAX, /* Maximum action flag. */
+};
+
 /* Hlist and list callback context. */
 struct mlx5_flow_cb_ctx {
 	struct rte_eth_dev *dev;
@@ -1203,7 +1212,10 @@ struct mlx5_dev_ctx_shared {
 	rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX];
 	/* UAR same-page access control required in 32bit implementations. */
 #endif
-	struct mlx5_hlist *flow_tbls;
+	union {
+		struct mlx5_hlist *flow_tbls; /* SWS flow table. */
+		struct mlx5_hlist *groups; /* HWS flow group. */
+	};
 	struct mlx5_flow_tunnel_hub *tunnel_hub;
 	/* Direct Rules tables for FDB, NIC TX+RX */
 	void *dr_drop_action; /* Pointer to DR drop action, any domain. */
@@ -1508,6 +1520,11 @@ struct mlx5_priv {
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
 	struct mlx5_hw_q *hw_q;
+	/* HW steering rte flow table list header. */
+	LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl;
+	/* HW steering global drop action. */
+	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
+				     [MLX5DR_TABLE_TYPE_MAX];
 };
 
 #define PORT_ID(priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.c +81 -0

99 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +72 -0

96 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_hw.c +522 -0

571 lines not on this page · raw message

Signature · 1 line
2.25.1
#7 lore raw

PATCH 7/13 net/mlx5: add basic flow queue operation

In reply to Suanming Mou

The HW steering uses queue-based flow rules management mechanism. The matcher and part of the actions have been prepared during flow table creation. Some left actions will be constructed during flow creation if needed.

A flow postpone attribute bit decribles if flow creation/destruction should be applied to the HW directly. An extra drain function has also been prepared to force push all the cached flows to the HW.

Once the flow has been applied to the HW, the pull function will be called to get the enqueued creation/destruction flows.

The DR rule flow memory is represented in PMD layer instead of allocating from HW steering layer. While destroying the flow, the flow rule memory can only be freed after the CQE received.

The HW queue job descriptor is currently introduced to convey the flow information and operation type between the flow insertion/destruction in the pull function.

This commit adds the basic flow queue operation for:
rte_flow_q_flow_create(); rte_flow_q_flow_destroy(); rte_flow_q_push();
rte_flow_q_pull();

---
 drivers/net/mlx5/mlx5.h         |   2 +-
 drivers/net/mlx5/mlx5_flow.c    | 158 ++++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    |  39 +++++
 drivers/net/mlx5/mlx5_flow_hw.c | 278 +++++++++++++++++++++++++++++++-
 4 files changed, 475 insertions(+), 2 deletions(-)
drivers/net/mlx5/mlx5.h +1 -1
index a4bc8d1fb7..ec4eb7ee94 100644
@@ -342,7 +342,7 @@ enum {
 /* HW steering flow management job descriptor. */
 struct mlx5_hw_q_job {
 	uint32_t type; /* Job type. */
-	struct rte_flow *flow; /* Flow attached to the job. */
+	struct rte_flow_hw *flow; /* Flow attached to the job. */
 	void *user_data; /* Job user data. */
 };
 
drivers/net/mlx5/mlx5_flow.c +158 -0

176 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +39 -0
index 02eb03e4cf..40eb8d79aa 100644
@@ -1015,6 +1015,13 @@ struct rte_flow {
 	uint32_t geneve_tlv_option; /**< Holds Geneve TLV option id. > */
 } __rte_packed;
 
+/* HWS flow struct. */
+struct rte_flow_hw {
+	uint32_t idx; /* Flow index from indexed pool. */
+	struct rte_flow_template_table *table; /* The table flow allcated from. */
+	struct mlx5dr_rule rule; /* HWS layer data struct. */
+} __rte_packed;
+
 /* Flow item template struct. */
 struct rte_flow_pattern_template {
 	LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1366,6 +1373,32 @@ typedef int (*mlx5_flow_table_destroy_t)
 			(struct rte_eth_dev *dev,
 			 struct rte_flow_template_table *table,
 			 struct rte_flow_error *error);
+typedef struct rte_flow *(*mlx5_flow_q_flow_create_t)
+			(struct rte_eth_dev *dev,
+			 uint32_t queue,
+			 const struct rte_flow_q_ops_attr *attr,
+			 struct rte_flow_template_table *table,
+			 const struct rte_flow_item items[],
+			 uint8_t pattern_template_index,
+			 const struct rte_flow_action actions[],
+			 uint8_t action_template_index,
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_q_flow_destroy_t)
+			(struct rte_eth_dev *dev,
+			 uint32_t queue,
+			 const struct rte_flow_q_ops_attr *attr,
+			 struct rte_flow *flow,
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_q_pull_t)
+			(struct rte_eth_dev *dev,
+			 uint32_t queue,
+			 struct rte_flow_q_op_res res[],
+			 uint16_t n_res,
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_q_push_t)
+			(struct rte_eth_dev *dev,
+			 uint32_t queue,
+			 struct rte_flow_error *error);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1411,6 +1444,10 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_actions_template_destroy_t actions_template_destroy;
 	mlx5_flow_table_create_t template_table_create;
 	mlx5_flow_table_destroy_t template_table_destroy;
+	mlx5_flow_q_flow_create_t q_flow_create;
+	mlx5_flow_q_flow_destroy_t q_flow_destroy;
+	mlx5_flow_q_pull_t q_pull;
+	mlx5_flow_q_push_t q_push;
 };
 
 /* mlx5_flow.c */
@@ -1581,6 +1618,8 @@ mlx5_translate_tunnel_etypes(uint64_t pattern_flags)
 	return 0;
 }
 
+int flow_hw_q_flow_flush(struct rte_eth_dev *dev,
+			 struct rte_flow_error *error);
 int mlx5_flow_group_to_table(struct rte_eth_dev *dev,
 			     const struct mlx5_flow_tunnel *tunnel,
 			     uint32_t group, uint32_t *table,
drivers/net/mlx5/mlx5_flow_hw.c +277 -1

302 lines not on this page · raw message

Signature · 1 line
2.25.1
#8 lore raw

PATCH 8/13 net/mlx5: add flow flush function

In reply to Suanming Mou

In case port restarting, all created flows should be flushed. This commit adds the flow flush helper function.

---
 drivers/net/mlx5/mlx5_flow.c    |   8 +++
 drivers/net/mlx5/mlx5_flow_hw.c | 117 ++++++++++++++++++++++++++++++++
 2 files changed, 125 insertions(+)
drivers/net/mlx5/mlx5_flow.c +8 -0
index b48a3af0fb..9ac96ac979 100644
@@ -6991,6 +6991,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,
 	uint32_t num_flushed = 0, fidx = 1;
 	struct rte_flow *flow;
 
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+	if (priv->config.dv_flow_en == 2 &&
+	    type == MLX5_FLOW_TYPE_GEN) {
+		flow_hw_q_flow_flush(dev, NULL);
+		return;
+	}
+#endif
+
 	MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {
 		flow_list_destroy(dev, type, fidx);
 		num_flushed++;
drivers/net/mlx5/mlx5_flow_hw.c +117 -0

123 lines not on this page · raw message

Signature · 1 line
2.25.1
#9 lore raw

PATCH 9/13 net/mlx5: add flow jump action

In reply to Suanming Mou

Jump action connects different level of flow tables as a complete data flow.
A new action construct data struct is also added in this commit to help handle the dynamic actions.

---
 drivers/net/mlx5/mlx5.h         |   1 +
 drivers/net/mlx5/mlx5_flow.h    |  25 ++-
 drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++---
 3 files changed, 275 insertions(+), 21 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
index ec4eb7ee94..0bc9897101 100644
@@ -1525,6 +1525,7 @@ struct mlx5_priv {
 	/* HW steering global drop action. */
 	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
 				     [MLX5DR_TABLE_TYPE_MAX];
+	struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
 };
 
 #define PORT_ID(priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.h +24 -1
index 40eb8d79aa..a1ab9173d9 100644
@@ -1018,10 +1018,25 @@ struct rte_flow {
 /* HWS flow struct. */
 struct rte_flow_hw {
 	uint32_t idx; /* Flow index from indexed pool. */
+	uint32_t fate_type; /* Fate action type. */
+	union {
+		/* Jump action. */
+		struct mlx5_hw_jump_action *jump;
+	};
 	struct rte_flow_template_table *table; /* The table flow allcated from. */
 	struct mlx5dr_rule rule; /* HWS layer data struct. */
 } __rte_packed;
 
+/* rte flow action translate to DR action struct. */
+struct mlx5_action_construct_data {
+	LIST_ENTRY(mlx5_action_construct_data) next;
+	/* Ensure the action types are matched. */
+	int type;
+	uint32_t idx;  /* Data index. */
+	uint16_t action_src; /* rte_flow_action src offset. */
+	uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+};
+
 /* Flow item template struct. */
 struct rte_flow_pattern_template {
 	LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1054,9 +1069,17 @@ struct mlx5_hw_jump_action {
 	struct mlx5dr_action *hws_action;
 };
 
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
 /* DR action set struct. */
 struct mlx5_hw_actions {
-	struct mlx5dr_action *drop; /* Drop action. */
+	/* Dynamic action list. */
+	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
+	struct mlx5_hw_jump_action *jump; /* Jump action. */
+	uint32_t acts_num:4; /* Total action number. */
+	/* Translated DR action array from action template. */
+	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
 };
 
 /* mlx5 action template struct. */
drivers/net/mlx5/mlx5_flow_hw.c +250 -20

399 lines not on this page · raw message

Signature · 1 line
2.25.1
#10 lore raw

PATCH 10/13 net/mlx5: add queue and RSS action

In reply to Suanming Mou

This commit adds the queue and RSS action. Similar to the jump action, dynamic ones will be added to the action construct list.

Due to the queue and RSS action in template should not be destroyed during port restart, the actions are created with standalone indirect table as indirect action does. When port stops, detaches the indirect table from action, when port starts, attaches the indirect table back to the action.

One more change is made to accelerate the action creation. Currently the mlx5_hrxq_get() function returns the object index instead of object pointer. This introduced an extra converting the index to the object by calling mlx5_ipool_get() in most of the case. And that extra converting hurts multi-thread performance since mlx5_ipool_get() uses the global lock inside. As the hash Rx queue object itself also contains the index, returns the object directly will achieve better performance without the global lock.

---
 drivers/net/mlx5/linux/mlx5_os.c   |  18 ++--
 drivers/net/mlx5/mlx5.h            |   4 +
 drivers/net/mlx5/mlx5_devx.c       |  10 ++
 drivers/net/mlx5/mlx5_flow.c       |  38 +++-----
 drivers/net/mlx5/mlx5_flow.h       |   7 ++
 drivers/net/mlx5/mlx5_flow_dv.c    | 150 ++++++++++++++---------------
 drivers/net/mlx5/mlx5_flow_hw.c    | 101 +++++++++++++++++++
 drivers/net/mlx5/mlx5_flow_verbs.c |   7 +-
 drivers/net/mlx5/mlx5_rx.h         |   9 +-
 drivers/net/mlx5/mlx5_rxq.c        |  78 +++++++++------
 10 files changed, 271 insertions(+), 151 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +9 -9
index 52e52a4ad7..8f0b15aad0 100644
@@ -1714,6 +1714,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 	priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
 	if (!priv->drop_queue.hrxq)
 		goto error;
+	priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
+				       mlx5_hrxq_create_cb,
+				       mlx5_hrxq_match_cb,
+				       mlx5_hrxq_remove_cb,
+				       mlx5_hrxq_clone_cb,
+				       mlx5_hrxq_clone_free_cb);
+	if (!priv->hrxqs)
+		goto error;
+	rte_rwlock_init(&priv->ind_tbls_lock);
 	if (priv->config.dv_flow_en == 2)
 		return eth_dev;
 	/* Port representor shares the same max priority with pf port. */
@@ -1744,15 +1753,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 			err = ENOTSUP;
 			goto error;
 	}
-	priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
-				       mlx5_hrxq_create_cb,
-				       mlx5_hrxq_match_cb,
-				       mlx5_hrxq_remove_cb,
-				       mlx5_hrxq_clone_cb,
-				       mlx5_hrxq_clone_free_cb);
-	if (!priv->hrxqs)
-		goto error;
-	rte_rwlock_init(&priv->ind_tbls_lock);
 	/* Query availability of metadata reg_c's. */
 	if (!priv->sh->metadata_regc_check_flag) {
 		err = mlx5_flow_discover_mreg_c(eth_dev);
drivers/net/mlx5/mlx5.h +4 -0
index 0bc9897101..6fb82bf1f3 100644
@@ -1286,6 +1286,7 @@ struct mlx5_flow_rss_desc {
 	uint64_t hash_fields; /* Verbs Hash fields. */
 	uint8_t key[MLX5_RSS_HASH_KEY_LEN]; /**< RSS hash key. */
 	uint32_t key_len; /**< RSS hash key len. */
+	uint32_t hws_flags; /**< HW steering action. */
 	uint32_t tunnel; /**< Queue in tunnel. */
 	uint32_t shared_rss; /**< Shared RSS index. */
 	struct mlx5_ind_table_obj *ind_tbl;
@@ -1347,6 +1348,7 @@ struct mlx5_hrxq {
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	void *action; /* DV QP action pointer. */
 #endif
+	uint32_t hws_flags; /* Hw steering flags. */
 	uint64_t hash_fields; /* Verbs Hash fields. */
 	uint32_t rss_key_len; /* Hash key length in bytes. */
 	uint32_t idx; /* Hash Rx queue index. */
@@ -1477,6 +1479,8 @@ struct mlx5_priv {
 	LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. */
 	/* Indirection tables. */
 	LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls;
+	/* Standalone indirect tables. */
+	LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;
 	/* Pointer to next element. */
 	rte_rwlock_t ind_tbls_lock;
 	uint32_t refcnt; /**< Reference counter. */
drivers/net/mlx5/mlx5_devx.c +10 -0
index 91243f684f..af131bcd1b 100644
@@ -807,6 +807,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
 		goto error;
 	}
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	if (hrxq->hws_flags) {
+		hrxq->action = mlx5dr_action_create_dest_tir
+			(priv->dr_ctx,
+			 (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags);
+		if (!hrxq->action)
+			goto error;
+		return 0;
+	}
 	if (mlx5_flow_os_create_flow_action_dest_devx_tir(hrxq->tir,
 							  &hrxq->action)) {
 		rte_errno = errno;
@@ -1042,6 +1050,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)
 		DRV_LOG(ERR, "Cannot create drop RX queue");
 		return ret;
 	}
+	if (priv->config.dv_flow_en == 2)
+		return 0;
 	/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */
 	ret = mlx5_devx_ind_table_new(dev, 0, hrxq->ind_table);
 	if (ret != 0) {
drivers/net/mlx5/mlx5_flow.c +12 -26
index 9ac96ac979..9cad84ebc6 100644
@@ -9302,14 +9302,10 @@ int
 mlx5_action_handle_attach(struct rte_eth_dev *dev)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct mlx5_indexed_pool *ipool =
-			priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
-	struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
 	int ret = 0;
-	uint32_t idx;
+	struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
 
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		const char *message;
 		uint32_t queue_idx;
 
@@ -9325,9 +9321,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
 	}
 	if (ret != 0)
 		return ret;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		ret = mlx5_ind_table_obj_attach(dev, ind_tbl);
 		if (ret != 0) {
 			DRV_LOG(ERR, "Port %u could not attach "
@@ -9336,13 +9330,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
 			goto error;
 		}
 	}
+
 	return 0;
 error:
-	shared_rss_last = shared_rss;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
-		if (shared_rss == shared_rss_last)
+	ind_tbl_last = ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+		if (ind_tbl == ind_tbl_last)
 			break;
 		if (mlx5_ind_table_obj_detach(dev, ind_tbl) != 0)
 			DRV_LOG(CRIT, "Port %u could not detach "
@@ -9365,15 +9358,10 @@ int
 mlx5_action_handle_detach(struct rte_eth_dev *dev)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct mlx5_indexed_pool *ipool =
-			priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
-	struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
 	int ret = 0;
-	uint32_t idx;
-
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+	struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
 
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		ret = mlx5_ind_table_obj_detach(dev, ind_tbl);
 		if (ret != 0) {
 			DRV_LOG(ERR, "Port %u could not detach "
@@ -9384,11 +9372,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev)
 	}
 	return 0;
 error:
-	shared_rss_last = shared_rss;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
-		if (shared_rss == shared_rss_last)
+	ind_tbl_last = ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+		if (ind_tbl == ind_tbl_last)
 			break;
 		if (mlx5_ind_table_obj_attach(dev, ind_tbl) != 0)
 			DRV_LOG(CRIT, "Port %u could not attach "
drivers/net/mlx5/mlx5_flow.h +7 -0
index a1ab9173d9..33094c8c07 100644
@@ -1022,6 +1022,7 @@ struct rte_flow_hw {
 	union {
 		/* Jump action. */
 		struct mlx5_hw_jump_action *jump;
+		struct mlx5_hrxq *hrxq; /* TIR action. */
 	};
 	struct rte_flow_template_table *table; /* The table flow allcated from. */
 	struct mlx5dr_rule rule; /* HWS layer data struct. */
@@ -1077,6 +1078,7 @@ struct mlx5_hw_actions {
 	/* Dynamic action list. */
 	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
+	struct mlx5_hrxq *tir; /* TIR action. */
 	uint32_t acts_num:4; /* Total action number. */
 	/* Translated DR action array from action template. */
 	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
@@ -1907,6 +1909,11 @@ int flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx,
 int
 flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
 		    struct rte_flow_error *error);
+void flow_dv_hashfields_set(uint64_t item_flags,
+			    struct mlx5_flow_rss_desc *rss_desc,
+			    uint64_t *hash_fields);
+void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+					uint64_t *hash_field);
 
 struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
 void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
drivers/net/mlx5/mlx5_flow_dv.c +70 -80

336 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_hw.c +101 -0

149 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_verbs.c +2 -5
index 90ccb9aaff..f08aa7a770 100644
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			MLX5_ASSERT(priv->drop_queue.hrxq);
 			hrxq = priv->drop_queue.hrxq;
 		} else {
-			uint32_t hrxq_idx;
 			struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;
 
 			MLX5_ASSERT(rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			rss_desc->tunnel = !!(handle->layers &
 					      MLX5_FLOW_LAYER_TUNNEL);
 			rss_desc->shared_rss = 0;
-			hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
-			hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
-					      hrxq_idx);
+			hrxq = mlx5_hrxq_get(dev, rss_desc);
 			if (!hrxq) {
 				rte_flow_error_set
 					(error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 					 "cannot get hash queue");
 				goto error;
 			}
-			handle->rix_hrxq = hrxq_idx;
+			handle->rix_hrxq = hrxq->idx;
 		}
 		MLX5_ASSERT(hrxq);
 		handle->drv_flow = mlx5_glue->create_flow
drivers/net/mlx5/mlx5_rx.h +7 -2
index cb5d51340d..468772ee27 100644
@@ -225,9 +225,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);
 struct mlx5_ind_table_obj *mlx5_ind_table_obj_get(struct rte_eth_dev *dev,
 						  const uint16_t *queues,
 						  uint32_t queues_n);
+struct mlx5_ind_table_obj *mlx5_ind_table_obj_new(struct rte_eth_dev *dev,
+						  const uint16_t *queues,
+						  uint32_t queues_n,
+						  bool standalone,
+						  bool ref_qs);
 int mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
 			       struct mlx5_ind_table_obj *ind_tbl,
-			       bool standalone,
 			       bool deref_rxqs);
 int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
 			     struct mlx5_ind_table_obj *ind_tbl,
@@ -250,8 +254,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx,
 					   void *cb_ctx __rte_unused);
 void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,
 			     struct mlx5_list_entry *entry);
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
 		       struct mlx5_flow_rss_desc *rss_desc);
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq);
 int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hxrq_idx);
 uint32_t mlx5_hrxq_verify(struct rte_eth_dev *dev);
 enum mlx5_rxq_type mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t idx);
drivers/net/mlx5/mlx5_rxq.c +49 -29

195 lines not on this page · raw message

Signature · 1 line
2.25.1
#11 lore raw

PATCH 11/13 net/mlx5: add mark action

In reply to Suanming Mou

The mark action is covered by tag action internally. While it is added the HW will add a tag to the packet. The mark value can be set as fixed or dynamic as the action mask indicates.

---
 drivers/net/mlx5/mlx5.h         |  3 ++
 drivers/net/mlx5/mlx5_flow.h    |  1 +
 drivers/net/mlx5/mlx5_flow_hw.c | 87 ++++++++++++++++++++++++++++++---
 3 files changed, 85 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.h +3 -0
index 6fb82bf1f3..c78dc3c431 100644
@@ -1529,6 +1529,9 @@ struct mlx5_priv {
 	/* HW steering global drop action. */
 	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
 				     [MLX5DR_TABLE_TYPE_MAX];
+	/* HW steering global drop action. */
+	struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX]
+				    [MLX5DR_TABLE_TYPE_MAX];
 	struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
 };
 
drivers/net/mlx5/mlx5_flow.h +1 -0
index 33094c8c07..8e65486a1f 100644
@@ -1080,6 +1080,7 @@ struct mlx5_hw_actions {
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
 	struct mlx5_hrxq *tir; /* TIR action. */
 	uint32_t acts_num:4; /* Total action number. */
+	uint32_t mark:1; /* Indicate the mark action. */
 	/* Translated DR action array from action template. */
 	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
 };
drivers/net/mlx5/mlx5_flow_hw.c +81 -6

135 lines not on this page · raw message

Signature · 1 line
2.25.1
#12 lore raw

PATCH 12/13 net/mlx5: add indirect action

In reply to Suanming Mou

HW steering can support indirect action as well. With indirect action, the flow can be created with more flexible shared RSS action selection. This will can save the action template with different RSS actions.

This commit adds the flow queue operation callback for:
rte_flow_q_action_handle_create(); rte_flow_q_action_handle_destroy(); rte_flow_q_action_handle_update();

---
 drivers/net/mlx5/mlx5_flow.c    | 116 +++++++++
 drivers/net/mlx5/mlx5_flow.h    |  56 +++++
 drivers/net/mlx5/mlx5_flow_dv.c |  21 +-
 drivers/net/mlx5/mlx5_flow_hw.c | 402 +++++++++++++++++++++++++++++++-
 4 files changed, 582 insertions(+), 13 deletions(-)
drivers/net/mlx5/mlx5_flow.c +116 -0
index 9cad84ebc6..46950044e0 100644
@@ -873,6 +873,26 @@ mlx5_flow_q_push(struct rte_eth_dev *dev,
 		 uint32_t queue,
 		 struct rte_flow_error *error);
 
+static struct rte_flow_action_handle *
+mlx5_flow_q_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_q_ops_attr *attr,
+				 const struct rte_flow_indir_action_conf *conf,
+				 const struct rte_flow_action *action,
+				 struct rte_flow_error *error);
+
+static int
+mlx5_flow_q_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_q_ops_attr *attr,
+				 struct rte_flow_action_handle *handle,
+				 const void *update,
+				 struct rte_flow_error *error);
+
+static int
+mlx5_flow_q_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+				  const struct rte_flow_q_ops_attr *attr,
+				  struct rte_flow_action_handle *handle,
+				  struct rte_flow_error *error);
+
 static const struct rte_flow_ops mlx5_flow_ops = {
 	.validate = mlx5_flow_validate,
 	.create = mlx5_flow_create,
@@ -904,6 +924,9 @@ static const struct rte_flow_ops mlx5_flow_ops = {
 	.q_flow_destroy = mlx5_flow_q_flow_destroy,
 	.q_pull = mlx5_flow_q_pull,
 	.q_push = mlx5_flow_q_push,
+	.q_action_handle_create = mlx5_flow_q_action_handle_create,
+	.q_action_handle_update = mlx5_flow_q_action_handle_update,
+	.q_action_handle_destroy = mlx5_flow_q_action_handle_destroy,
 };
 
 /* Tunnel information. */
@@ -8228,6 +8251,99 @@ mlx5_flow_q_push(struct rte_eth_dev *dev,
 	return fops->q_push(dev, queue, error);
 }
 
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] conf
+ *   Indirect action configuration.
+ * @param[in] action
+ *   rte_flow action detail.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+mlx5_flow_q_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_q_ops_attr *attr,
+				 const struct rte_flow_indir_action_conf *conf,
+				 const struct rte_flow_action *action,
+				 struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->q_action_create(dev, queue, attr, conf, action, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] handle
+ *   Action handle to be updated.
+ * @param[in] update
+ *   Update value.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_q_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_q_ops_attr *attr,
+				 struct rte_flow_action_handle *handle,
+				 const void *update,
+				 struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->q_action_update(dev, queue, attr, handle, update, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] handle
+ *   Action handle to be destroyed.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_q_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+				  const struct rte_flow_q_ops_attr *attr,
+				  struct rte_flow_action_handle *handle,
+				  struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->q_action_destroy(dev, queue, attr, handle, error);
+}
+
 /**
  * Allocate a new memory for the counter values wrapped by all the needed
  * management.
drivers/net/mlx5/mlx5_flow.h +56 -0

96 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_dv.c +12 -9
index c3d9d30dba..ca8ae4214b 100644
@@ -13837,9 +13837,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,
  * @return
  *   Valid hash RX queue index, otherwise 0.
  */
-static uint32_t
-__flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
-				 const uint64_t hash_fields)
+uint32_t
+flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+			       const uint64_t hash_fields)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_shared_action_rss *shared_rss =
@@ -13967,7 +13967,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			struct mlx5_hrxq *hrxq = NULL;
 			uint32_t hrxq_idx;
 
-			hrxq_idx = __flow_dv_action_rss_hrxq_lookup(dev,
+			hrxq_idx = flow_dv_action_rss_hrxq_lookup(dev,
 						rss_desc->shared_rss,
 						dev_flow->hash_fields);
 			if (hrxq_idx)
@@ -14691,6 +14691,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
 			   struct mlx5_shared_action_rss *shared_rss,
 			   struct rte_flow_error *error)
 {
+	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_flow_rss_desc rss_desc = { 0 };
 	size_t i;
 	int err;
@@ -14711,6 +14712,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
 	/* Set non-zero value to indicate a shared RSS. */
 	rss_desc.shared_rss = action_idx;
 	rss_desc.ind_tbl = shared_rss->ind_tbl;
+	if (priv->config.dv_flow_en == 2)
+		rss_desc.hws_flags = MLX5DR_ACTION_FLAG_HWS_RX;
 	for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
 		struct mlx5_hrxq *hrxq;
 		uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14902,7 +14905,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
  *   A valid shared action handle in case of success, NULL otherwise and
  *   rte_errno is set.
  */
-static struct rte_flow_action_handle *
+struct rte_flow_action_handle *
 flow_dv_action_create(struct rte_eth_dev *dev,
 		      const struct rte_flow_indir_action_conf *conf,
 		      const struct rte_flow_action *action,
@@ -14972,7 +14975,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_destroy(struct rte_eth_dev *dev,
 		       struct rte_flow_action_handle *handle,
 		       struct rte_flow_error *error)
@@ -15181,7 +15184,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_update(struct rte_eth_dev *dev,
 			struct rte_flow_action_handle *handle,
 			const void *update,
@@ -15895,7 +15898,7 @@ flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx,
 				  "counters are not available");
 }
 
-static int
+int
 flow_dv_action_query(struct rte_eth_dev *dev,
 		     const struct rte_flow_action_handle *handle, void *data,
 		     struct rte_flow_error *error)
@@ -17584,7 +17587,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_validate(struct rte_eth_dev *dev,
 			const struct rte_flow_indir_action_conf *conf,
 			const struct rte_flow_action *action,
drivers/net/mlx5/mlx5_flow_hw.c +398 -4

481 lines not on this page · raw message

Signature · 1 line
2.25.1
#13 lore raw

PATCH 13/13 net/mlx5: add header reformat action

In reply to Suanming Mou

HW steering header reformat action can work under bulk mode. In this case, when create the table, bulk size of header reformat actions will be allocated in low level. Afterwards, when create flow, just simply specify the action index in the bulk and the encapsulation data to the action will be enough.

---
 drivers/net/mlx5/mlx5.h         |   1 +
 drivers/net/mlx5/mlx5_flow.h    |  21 +++
 drivers/net/mlx5/mlx5_flow_dv.c |   4 +-
 drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++-
 4 files changed, 251 insertions(+), 3 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
index c78dc3c431..e10f55bf8c 100644
@@ -344,6 +344,7 @@ struct mlx5_hw_q_job {
 	uint32_t type; /* Job type. */
 	struct rte_flow_hw *flow; /* Flow attached to the job. */
 	void *user_data; /* Job user data. */
+	uint8_t *encap_data; /* Encap data. */
 };
 
 /* HW steering job descriptor LIFO header . */
drivers/net/mlx5/mlx5_flow.h +21 -0
index 097e5bf587..16fb6e643b 100644
@@ -1038,6 +1038,14 @@ struct mlx5_action_construct_data {
 	uint16_t action_src; /* rte_flow_action src offset. */
 	uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
 	union {
+		struct {
+			/* encap src(item) offset. */
+			uint16_t src;
+			/* encap dst data offset. */
+			uint16_t dst;
+			/* encap data len. */
+			uint16_t len;
+		} encap;
 		struct {
 			uint64_t types; /* RSS hash types. */
 			uint32_t level; /* RSS level. */
@@ -1079,6 +1087,13 @@ struct mlx5_hw_jump_action {
 	struct mlx5dr_action *hws_action;
 };
 
+/* Encap decap action struct. */
+struct mlx5_hw_encap_decap_action {
+	struct mlx5dr_action *action; /* Action object. */
+	size_t data_size; /* Action metadata size. */
+	uint8_t data[]; /* Action data. */
+};
+
 /* The maximum actions support in the flow. */
 #define MLX5_HW_MAX_ACTS 16
 
@@ -1088,6 +1103,9 @@ struct mlx5_hw_actions {
 	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
 	struct mlx5_hrxq *tir; /* TIR action. */
+	/* Encap/Decap action. */
+	struct mlx5_hw_encap_decap_action *encap_decap;
+	uint16_t encap_decap_pos; /* Encap/Decap action position. */
 	uint32_t acts_num:4; /* Total action number. */
 	uint32_t mark:1; /* Indicate the mark action. */
 	/* Translated DR action array from action template. */
@@ -2021,4 +2039,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,
 			 const struct rte_flow_action_handle *handle,
 			 void *data,
 			 struct rte_flow_error *error);
+size_t flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type);
+int flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
+			   size_t *size, struct rte_flow_error *error);
 #endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_dv.c +2 -2
index ca8ae4214b..377ed6c1db 100644
@@ -4024,7 +4024,7 @@ flow_dv_push_vlan_action_resource_register
  * @return
  *   sizeof struct item_type, 0 if void or irrelevant.
  */
-static size_t
+size_t
 flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
 {
 	size_t retval;
@@ -4090,7 +4090,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
  * @return
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
-static int
+int
 flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
 			   size_t *size, struct rte_flow_error *error)
 {
drivers/net/mlx5/mlx5_flow_hw.c +227 -1

299 lines not on this page · raw message

Signature · 1 line
2.25.1
#14 lore raw

v2 cover net/mlx5: add hardware steering

In reply to Suanming Mou

The Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:

- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly

However, there are still some disadvantages:

- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects

- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion

- security, exposing the low-level steering entries directly to the userspace may cause security risks

A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.

In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.

*** THIS PATCH SET DEPENDS ON THE NEW RTE_FLOW ASYNC API ***

---

v2:
- New HW steering low-level abstract code added.
- commit message improvement.
- add protection for rte_flow and rte_flow_async callbacks.
- rebase to rte_flow_async v9.
- fix some rte_flow error not filled bugs.

Suanming Mou (14):
net/mlx5: introduce hardware steering operation net/mlx5: add HW steering low-level abstract code net/mlx5: introduce hardware steering enable routine net/mlx5: add port flow configuration net/mlx5: add pattern template management net/mlx5: add action template management net/mlx5: add table management
net/mlx5: add basic flow queue operation net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action

 doc/guides/nics/mlx5.rst                |   19 +-
 doc/guides/rel_notes/release_22_03.rst  |    6 +
 drivers/net/mlx5/linux/mlx5_flow_os.h   |    1 +
 drivers/net/mlx5/linux/mlx5_os.c        |   22 +-
 drivers/net/mlx5/meson.build            |    2 +
 drivers/net/mlx5/mlx5.c                 |   55 +-
 drivers/net/mlx5/mlx5.h                 |   66 +-
 drivers/net/mlx5/mlx5_devx.c            |   10 +
 drivers/net/mlx5/mlx5_dr.c              |  383 ++++
 drivers/net/mlx5/mlx5_dr.h              |  456 +++++
 drivers/net/mlx5/mlx5_flow.c            |  721 ++++++-
 drivers/net/mlx5/mlx5_flow.h            |  287 +++
 drivers/net/mlx5/mlx5_flow_dv.c         |  186 +-
 drivers/net/mlx5/mlx5_flow_hw.c         | 2335 +++++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow_verbs.c      |    7 +-
 drivers/net/mlx5/mlx5_rx.h              |    9 +-
 drivers/net/mlx5/mlx5_rxq.c             |   85 +-
 drivers/net/mlx5/windows/mlx5_flow_os.h |    1 +
 18 files changed, 4471 insertions(+), 180 deletions(-)
 create mode 100644 drivers/net/mlx5/mlx5_dr.c
 create mode 100644 drivers/net/mlx5/mlx5_dr.h
 create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
Signature · 1 line
2.25.1
#15 lore raw

PATCH v2 1/14 net/mlx5: introduce hardware steering operation

The Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:

- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly

However, there are still some disadvantages:

- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects

- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion

- security, exposing the low-level steering entries directly to the userspace may cause security risks

A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.

In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.

This commit adds the basic driver operation.

---
 drivers/net/mlx5/linux/mlx5_flow_os.h   |  1 +
 drivers/net/mlx5/meson.build            |  1 +
 drivers/net/mlx5/mlx5_flow.c            |  1 +
 drivers/net/mlx5/mlx5_flow.h            |  1 +
 drivers/net/mlx5/mlx5_flow_hw.c         | 13 +++++++++++++
 drivers/net/mlx5/windows/mlx5_flow_os.h |  1 +
 6 files changed, 18 insertions(+)
 create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
drivers/net/mlx5/linux/mlx5_flow_os.h +1 -0
index 1926d26410..e28a9e0436 100644
@@ -9,6 +9,7 @@
 
 #ifdef HAVE_IBV_FLOW_DV_SUPPORT
 extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
 #endif
 
 /**
drivers/net/mlx5/meson.build +1 -0
index 2f6d8cbb3d..39a2b8c523 100644
@@ -16,6 +16,7 @@ sources = files(
         'mlx5_flow.c',
         'mlx5_flow_meter.c',
         'mlx5_flow_dv.c',
+        'mlx5_flow_hw.c',
         'mlx5_flow_aso.c',
         'mlx5_flow_flex.c',
         'mlx5_mac.c',
drivers/net/mlx5/mlx5_flow.c +1 -0
index 5b5b00cc3e..de203c7b33 100644
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = {
 	[MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	[MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops,
+	[MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops,
 #endif
 	[MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops,
 	[MLX5_FLOW_TYPE_MAX] = &mlx5_flow_null_drv_ops
drivers/net/mlx5/mlx5_flow.h +1 -0
index 583e8b7321..5e88a23f22 100644
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {
 	MLX5_FLOW_TYPE_MIN,
 	MLX5_FLOW_TYPE_DV,
 	MLX5_FLOW_TYPE_VERBS,
+	MLX5_FLOW_TYPE_HW,
 	MLX5_FLOW_TYPE_MAX,
 };
 
drivers/net/mlx5/mlx5_flow_hw.c new file +13 -0
new file mode 100644
index 0000000000..729d5914a8
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2022 NVIDIA Corporation & Affiliates
+ */
+
+#include <rte_flow.h>
+
+#include "mlx5_flow.h"
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+
+#endif
drivers/net/mlx5/windows/mlx5_flow_os.h +1 -0
index dfcb012334..52013b06a0 100644
@@ -10,6 +10,7 @@
 
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
 #endif
 
 /**
Signature · 1 line
2.25.1
#16 lore raw

PATCH v2 2/14 net/mlx5: add HW steering low-level abstract code

In reply to Suanming Mou

The HW steering low-level implementation will be added later in another patch series. To avoid the linkage issues the abstract stub replacement is provided currently.

---
 drivers/net/mlx5/meson.build |   1 +
 drivers/net/mlx5/mlx5_dr.c   | 383 +++++++++++++++++++++++++++++
 drivers/net/mlx5/mlx5_dr.h   | 456 +++++++++++++++++++++++++++++++++++
 3 files changed, 840 insertions(+)
 create mode 100644 drivers/net/mlx5/mlx5_dr.c
 create mode 100644 drivers/net/mlx5/mlx5_dr.h
drivers/net/mlx5/meson.build +1 -0
index 39a2b8c523..393b2c97ac 100644
@@ -14,6 +14,7 @@ sources = files(
         'mlx5.c',
         'mlx5_ethdev.c',
         'mlx5_flow.c',
+	'mlx5_dr.c',
         'mlx5_flow_meter.c',
         'mlx5_flow_dv.c',
         'mlx5_flow_hw.c',
drivers/net/mlx5/mlx5_dr.c new file +383 -0

383 lines not on this page · raw message

drivers/net/mlx5/mlx5_dr.h new file +456 -0

456 lines not on this page · raw message

Signature · 1 line
2.25.1
#17 lore raw

PATCH v2 3/14 net/mlx5: introduce hardware steering enable routine

In reply to Suanming Mou

The new hardware steering engine relies on using dedicated steering WQEs instead of direct writing to the low-level steering table entries directly. In the first introduce implementation the hardware steering engine supports the new queue based Flow API, the existing synchronous non-queue based Flow API is not supported.

A new dv_flow_en value 2 is added to manage mlx5 PMD steering engine:

dv_flow_en rte_flow API rte_flow_async API
------------------------------------------------
0 support not support
1 support not support
2 not support support

This commit introduces the extra dv_flow_en = 2 to specify the new flow initialize and manage operation routine.

---
 doc/guides/nics/mlx5.rst         | 13 ++++++++++---
 drivers/net/mlx5/linux/mlx5_os.c |  4 ++++
 drivers/net/mlx5/mlx5.c          |  7 ++++++-
 drivers/net/mlx5/mlx5.h          |  3 ++-
 drivers/net/mlx5/mlx5_flow.c     | 22 ++++++++++++++++++++++
 5 files changed, 44 insertions(+), 5 deletions(-)
doc/guides/nics/mlx5.rst +10 -3
index c3cc0c0f41..c3f7d61e7d 100644
@@ -1042,10 +1042,17 @@ Driver options
 
 - ``dv_flow_en`` parameter [int]
 
-  A nonzero value enables the DV flow steering assuming it is supported
-  by the driver (RDMA Core library version is rdma-core-24.0 or higher).
+  Value 0 means legacy Verbs flow offloading.
 
-  Enabled by default if supported.
+  Value 1 enables the DV flow steering assuming it is supported by the
+  driver (RDMA Core library version is rdma-core-24.0 or higher).
+
+  Value 2 enables the WQE based hardware steering. In this mode only
+  the queue-based rte_flow_q flow management is supported.
+
+  Configured by default to 1 DV flow steering if the driver(RDMA CORE library)
+  supported. Otherwise, the value will be 0 which indicates legacy Verbs flow
+  offloading.
 
 - ``dv_esw_en`` parameter [int]
 
drivers/net/mlx5/linux/mlx5_os.c +4 -0
index ecf823da56..0faf26f5b8 100644
@@ -482,6 +482,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
 	err = mlx5_alloc_table_hash_list(priv);
 	if (err)
 		goto error;
+	if (priv->sh->config.dv_flow_en == 2)
+		return 0;
 	/* The resources below are only valid with DV support. */
 #ifdef HAVE_IBV_FLOW_DV_SUPPORT
 	/* Init port id action list. */
@@ -1519,6 +1521,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 	priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
 	if (!priv->drop_queue.hrxq)
 		goto error;
+	if (priv->sh->config.dv_flow_en == 2)
+		return eth_dev;
 	/* Port representor shares the same max priority with pf port. */
 	if (!priv->sh->flow_priority_check_flag) {
 		/* Supported Verbs flow priority number detection. */
drivers/net/mlx5/mlx5.c +6 -1
index 9f65a8f901..f49d30c05c 100644
@@ -1199,7 +1199,12 @@ mlx5_dev_args_check_handler(const char *key, const char *val, void *opaque)
 	} else if (strcmp(MLX5_DV_ESW_EN, key) == 0) {
 		config->dv_esw_en = !!tmp;
 	} else if (strcmp(MLX5_DV_FLOW_EN, key) == 0) {
-		config->dv_flow_en = !!tmp;
+		if (tmp > 2) {
+			DRV_LOG(ERR, "Invalid %s parameter.", key);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		config->dv_flow_en = tmp;
 	} else if (strcmp(MLX5_DV_XMETA_EN, key) == 0) {
 		if (tmp != MLX5_XMETA_MODE_LEGACY &&
 		    tmp != MLX5_XMETA_MODE_META16 &&
drivers/net/mlx5/mlx5.h +2 -1
index 35ea3fb47c..42e725be63 100644
@@ -287,7 +287,8 @@ struct mlx5_sh_config {
 	int tx_skew; /* Tx scheduling skew between WQE and data on wire. */
 	uint32_t reclaim_mode:2; /* Memory reclaim mode. */
 	uint32_t dv_esw_en:1; /* Enable E-Switch DV flow. */
-	uint32_t dv_flow_en:1; /* Enable DV flow. */
+	/* Enable DV flow. 1 means SW steering, 2 means HW steering. */
+	unsigned int dv_flow_en:2;
 	uint32_t dv_xmeta_en:2; /* Enable extensive flow metadata. */
 	uint32_t dv_miss_info:1; /* Restore packet after partial hw miss. */
 	uint32_t l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
drivers/net/mlx5/mlx5_flow.c +22 -0
index de203c7b33..0f384cc13f 100644
@@ -6838,6 +6838,15 @@ mlx5_flow_create(struct rte_eth_dev *dev,
 		 const struct rte_flow_action actions[],
 		 struct rte_flow_error *error)
 {
+	struct mlx5_priv *priv = dev->data->dev_private;
+
+	if (priv->sh->config.dv_flow_en == 2) {
+		rte_flow_error_set(error, ENOTSUP,
+			  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+			  NULL,
+			  "Flow non-Q creation not supported");
+		return NULL;
+	}
 	/*
 	 * If the device is not started yet, it is not allowed to created a
 	 * flow from application. PMD default flows and traffic control flows
@@ -7334,6 +7343,13 @@ mlx5_flow_destroy(struct rte_eth_dev *dev,
 		  struct rte_flow *flow,
 		  struct rte_flow_error *error __rte_unused)
 {
+	struct mlx5_priv *priv = dev->data->dev_private;
+
+	if (priv->sh->config.dv_flow_en == 2)
+		return rte_flow_error_set(error, ENOTSUP,
+			  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+			  NULL,
+			  "Flow non-Q destruction not supported");
 	flow_list_destroy(dev, MLX5_FLOW_TYPE_GEN,
 				(uintptr_t)(void *)flow);
 	return 0;
@@ -7431,7 +7447,13 @@ mlx5_flow_query(struct rte_eth_dev *dev,
 		struct rte_flow_error *error)
 {
 	int ret;
+	struct mlx5_priv *priv = dev->data->dev_private;
 
+	if (priv->sh->config.dv_flow_en == 2)
+		return rte_flow_error_set(error, ENOTSUP,
+			  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+			  NULL,
+			  "Flow non-Q query not supported");
 	ret = flow_drv_query(dev, (uintptr_t)(void *)flow, actions, data,
 			     error);
 	if (ret < 0)
Signature · 1 line
2.25.1
#18 lore raw

PATCH v2 4/14 net/mlx5: add port flow configuration

In reply to Suanming Mou

The hardware steering is backend to support rte_flow_async API in mlx5 PMD. The port configuration function creates the queues and needed flow management resources.

The PMD layer configuration function allocates the queues' context and per-queue job descriptor pool. The job descriptor pool size is equal to the queue size, and the job descriptors will be popped from pool with LIFO strategy to convey the flow information during flow insertion/destruction. Then, while polling the queued operation result, the flow information will be extracted from the job descriptor and the descriptor will be pushed back to the LIFO pool.

The commit creates the flow port queues and the job descriptor pools.

---
 drivers/net/mlx5/mlx5.c         |   3 +
 drivers/net/mlx5/mlx5.h         |  30 +++++-
 drivers/net/mlx5/mlx5_flow.c    |  86 +++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    |  15 +++
 drivers/net/mlx5/mlx5_flow_hw.c | 159 ++++++++++++++++++++++++++++++++
 5 files changed, 292 insertions(+), 1 deletion(-)
drivers/net/mlx5/mlx5.c +3 -0
index f49d30c05c..0079aa83c1 100644
@@ -1820,6 +1820,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)
 	/* Free the eCPRI flex parser resource. */
 	mlx5_flex_parser_ecpri_release(dev);
 	mlx5_flex_item_port_cleanup(dev);
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	flow_hw_resource_release(dev);
+#endif
 	if (priv->rxq_privs != NULL) {
 		/* XXX race condition if mlx5_rx_burst() is still running. */
 		rte_delay_us_sleep(1000);
drivers/net/mlx5/mlx5.h +29 -1
index 42e725be63..5a6c4937a5 100644
@@ -33,7 +33,9 @@
 #include "mlx5_utils.h"
 #include "mlx5_os.h"
 #include "mlx5_autoconf.h"
-
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+#include "mlx5_dr.h"
+#endif
 
 #define MLX5_SH(dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
 
@@ -320,6 +322,26 @@ struct mlx5_lb_ctx {
 	uint16_t refcnt; /* Reference count for representors. */
 };
 
+/* HW steering queue job descriptor type. */
+enum {
+	MLX5_HW_Q_JOB_TYPE_CREATE, /* Flow create job type. */
+	MLX5_HW_Q_JOB_TYPE_DESTROY, /* Flow destroy job type. */
+};
+
+/* HW steering flow management job descriptor. */
+struct mlx5_hw_q_job {
+	uint32_t type; /* Job type. */
+	struct rte_flow *flow; /* Flow attached to the job. */
+	void *user_data; /* Job user data. */
+};
+
+/* HW steering job descriptor LIFO pool. */
+struct mlx5_hw_q {
+	uint32_t job_idx; /* Free job index. */
+	uint32_t size; /* LIFO size. */
+	struct mlx5_hw_q_job **job; /* LIFO header. */
+} __rte_cache_aligned;
+
 #define MLX5_COUNTERS_PER_POOL 512
 #define MLX5_MAX_PENDING_QUERIES 4
 #define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1476,6 +1498,12 @@ struct mlx5_priv {
 	struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM];
 	/* Flex items have been created on the port. */
 	uint32_t flex_item_map; /* Map of allocated flex item elements. */
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
+	uint32_t nb_queue; /* HW steering queue number. */
+	/* HW steering queue polling mechanism job descriptor LIFO. */
+	struct mlx5_hw_q *hw_q;
+#endif
 };
 
 #define PORT_ID(priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.c +86 -0

110 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +15 -0
index 5e88a23f22..1c7dee8b29 100644
@@ -1257,6 +1257,17 @@ typedef int (*mlx5_flow_item_update_t)
 			 const struct rte_flow_item_flex_handle *handle,
 			 const struct rte_flow_item_flex_conf *conf,
 			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_info_get_t)
+			(struct rte_eth_dev *dev,
+			 struct rte_flow_port_info *port_info,
+			 struct rte_flow_queue_info *queue_info,
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_port_configure_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_port_attr *port_attr,
+			 uint16_t nb_queue,
+			 const struct rte_flow_queue_attr *queue_attr[],
+			 struct rte_flow_error *err);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1295,6 +1306,8 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_item_create_t item_create;
 	mlx5_flow_item_release_t item_release;
 	mlx5_flow_item_update_t item_update;
+	mlx5_flow_info_get_t info_get;
+	mlx5_flow_port_configure_t configure;
 };
 
 /* mlx5_flow.c */
@@ -1767,4 +1780,6 @@ const struct mlx5_flow_tunnel *
 mlx5_get_tof(const struct rte_flow_item *items,
 	     const struct rte_flow_action *actions,
 	     enum mlx5_tof_rule_type *rule_type);
+void
+flow_hw_resource_release(struct rte_eth_dev *dev);
 #endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_hw.c +159 -0

169 lines not on this page · raw message

Signature · 1 line
2.25.1
#19 lore raw

PATCH v2 5/14 net/mlx5: add pattern template management

In reply to Suanming Mou

The pattern template defines flows that have the same matching fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be (eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while the values for each rule will be different.

Due to the pattern template can be used in different domains, the items will only be cached in pattern template create stage, while the template is binded to a dedicated table, the HW criteria will be created and saved to the table. The pattern templates can be used by multiple tables. But different tables create the same criteria and will not share the matcher between each other in order to have better performance.

This commit adds pattern template management.

---
 drivers/net/mlx5/mlx5.h         |  2 +
 drivers/net/mlx5/mlx5_flow.c    | 76 +++++++++++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    | 24 +++++++++
 drivers/net/mlx5/mlx5_flow_hw.c | 86 +++++++++++++++++++++++++++++++++
 4 files changed, 188 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
index 5a6c4937a5..44694daf1b 100644
@@ -1499,6 +1499,8 @@ struct mlx5_priv {
 	/* Flex items have been created on the port. */
 	uint32_t flex_item_map; /* Map of allocated flex item elements. */
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	/* Item template list. */
+	LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
 	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +76 -0

94 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +24 -0
index 1c7dee8b29..12dbb4655b 100644
@@ -1015,6 +1015,19 @@ struct rte_flow {
 	uint32_t geneve_tlv_option; /**< Holds Geneve TLV option id. > */
 } __rte_packed;
 
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+/* Flow item template struct. */
+struct rte_flow_pattern_template {
+	LIST_ENTRY(rte_flow_pattern_template) next;
+	/* Template attributes. */
+	struct rte_flow_pattern_template_attr attr;
+	struct mlx5dr_match_template *mt; /* mlx5 match template. */
+	uint32_t refcnt;  /* Reference counter. */
+};
+
+#endif
+
 /*
  * Define list of valid combinations of RX Hash fields
  * (see enum ibv_rx_hash_fields).
@@ -1268,6 +1281,15 @@ typedef int (*mlx5_flow_port_configure_t)
 			 uint16_t nb_queue,
 			 const struct rte_flow_queue_attr *queue_attr[],
 			 struct rte_flow_error *err);
+typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_pattern_template_attr *attr,
+			 const struct rte_flow_item items[],
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_pattern_template_destroy_t)
+			(struct rte_eth_dev *dev,
+			 struct rte_flow_pattern_template *template,
+			 struct rte_flow_error *error);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1308,6 +1330,8 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_item_update_t item_update;
 	mlx5_flow_info_get_t info_get;
 	mlx5_flow_port_configure_t configure;
+	mlx5_flow_pattern_template_create_t pattern_template_create;
+	mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
 };
 
 /* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +86 -0

107 lines not on this page · raw message

Signature · 1 line
2.25.1
#20 lore raw

PATCH v2 6/14 net/mlx5: add action template management

In reply to Suanming Mou

The action template holds a list of action types that will be used together on the same rule. The template's actions instances will be created only when the template bind to the dedicated group. And the created actions will be saved to each individual group in order for best performance. The actions in a group will not be shared with each other unless shared actions are specified.

This commit adds the action template management which stores the flow action template.

---
 drivers/net/mlx5/mlx5.h         |   2 +
 drivers/net/mlx5/mlx5_flow.c    |  78 +++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    |  22 ++++++
 drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++
 4 files changed, 218 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
index 44694daf1b..8e36805e54 100644
@@ -1501,6 +1501,8 @@ struct mlx5_priv {
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	/* Item template list. */
 	LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
+	/* Action template list. */
+	LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;
 	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +78 -0

96 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +22 -0
index 12dbb4655b..7e2d6bf4eb 100644
@@ -1026,6 +1026,16 @@ struct rte_flow_pattern_template {
 	uint32_t refcnt;  /* Reference counter. */
 };
 
+/* Flow action template struct. */
+struct rte_flow_actions_template {
+	LIST_ENTRY(rte_flow_actions_template) next;
+	/* Template attributes. */
+	struct rte_flow_actions_template_attr attr;
+	struct rte_flow_action *actions; /* Cached flow actions. */
+	struct rte_flow_action *masks; /* Cached action masks.*/
+	uint32_t refcnt; /* Reference counter. */
+};
+
 #endif
 
 /*
@@ -1290,6 +1300,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t)
 			(struct rte_eth_dev *dev,
 			 struct rte_flow_pattern_template *template,
 			 struct rte_flow_error *error);
+typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_actions_template_attr *attr,
+			 const struct rte_flow_action actions[],
+			 const struct rte_flow_action masks[],
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_actions_template_destroy_t)
+			(struct rte_eth_dev *dev,
+			 struct rte_flow_actions_template *template,
+			 struct rte_flow_error *error);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1332,6 +1352,8 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_port_configure_t configure;
 	mlx5_flow_pattern_template_create_t pattern_template_create;
 	mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
+	mlx5_flow_actions_template_create_t actions_template_create;
+	mlx5_flow_actions_template_destroy_t actions_template_destroy;
 };
 
 /* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +116 -0

140 lines not on this page · raw message

Signature · 1 line
2.25.1
#21 lore raw

PATCH v2 7/14 net/mlx5: add table management

In reply to Suanming Mou

Flow table is a group of flows with the same matching criteria and the same actions defined for them. The table defines rules that have the same matching fields but with different matching values. For example, matching on 5 tuple, the table will be (IPv4 source + IPv4 dest + s_port + d_port + next_proto) while the values for each rule will be different.

The templates' relevant matching criteria and action instances will be created in the table creation and saved in the table. As table attributes indicate the supported flow number, the flow memory will also be allocated at the same time.

This commit adds the table management functions.

---
 drivers/net/mlx5/mlx5.c         |  45 ++-
 drivers/net/mlx5/mlx5.h         |  21 +-
 drivers/net/mlx5/mlx5_flow.c    |  93 ++++++
 drivers/net/mlx5/mlx5_flow.h    |  73 +++++
 drivers/net/mlx5/mlx5_flow_hw.c | 525 ++++++++++++++++++++++++++++++++
 5 files changed, 751 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.c +41 -4
index 0079aa83c1..7611fdd62b 100644
@@ -1607,12 +1607,46 @@ void
 mlx5_free_table_hash_list(struct mlx5_priv *priv)
 {
 	struct mlx5_dev_ctx_shared *sh = priv->sh;
-
-	if (!sh->flow_tbls)
+	struct mlx5_hlist **tbls = (priv->sh->config.dv_flow_en == 2) ?
+				   &sh->groups : &sh->flow_tbls;
+	if (*tbls == NULL)
 		return;
-	mlx5_hlist_destroy(sh->flow_tbls);
-	sh->flow_tbls = NULL;
+	mlx5_hlist_destroy(*tbls);
+	*tbls = NULL;
+}
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/**
+ * Allocate HW steering group hash list.
+ *
+ * @param[in] priv
+ *   Pointer to the private device data structure.
+ */
+static int
+mlx5_alloc_hw_group_hash_list(struct mlx5_priv *priv)
+{
+	int err = 0;
+	struct mlx5_dev_ctx_shared *sh = priv->sh;
+	char s[MLX5_NAME_SIZE];
+
+	MLX5_ASSERT(sh);
+	snprintf(s, sizeof(s), "%s_flow_groups", priv->sh->ibdev_name);
+	sh->groups = mlx5_hlist_create
+			(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
+			 false, true, sh,
+			 flow_hw_grp_create_cb,
+			 flow_hw_grp_match_cb,
+			 flow_hw_grp_remove_cb,
+			 flow_hw_grp_clone_cb,
+			 flow_hw_grp_clone_free_cb);
+	if (!sh->groups) {
+		DRV_LOG(ERR, "flow groups with hash creation failed.");
+		err = ENOMEM;
+	}
+	return err;
 }
+#endif
+
 
 /**
  * Initialize flow table hash list and create the root tables entry
@@ -1628,11 +1662,14 @@ int
 mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused)
 {
 	int err = 0;
+
 	/* Tables are only used in DV and DR modes. */
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	struct mlx5_dev_ctx_shared *sh = priv->sh;
 	char s[MLX5_NAME_SIZE];
 
+	if (priv->sh->config.dv_flow_en == 2)
+		return mlx5_alloc_hw_group_hash_list(priv);
 	MLX5_ASSERT(sh);
 	snprintf(s, sizeof(s), "%s_flow_table", priv->sh->ibdev_name);
 	sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
drivers/net/mlx5/mlx5.h +19 -2
index 8e36805e54..1cc63e1d9f 100644
@@ -64,7 +64,9 @@ enum mlx5_ipool_index {
 	MLX5_IPOOL_PUSH_VLAN, /* Pool for push vlan resource. */
 	MLX5_IPOOL_TAG, /* Pool for tag resource. */
 	MLX5_IPOOL_PORT_ID, /* Pool for port id resource. */
-	MLX5_IPOOL_JUMP, /* Pool for jump resource. */
+	MLX5_IPOOL_JUMP, /* Pool for SWS jump resource. */
+	/* Pool for HWS group. Jump action will be created internally. */
+	MLX5_IPOOL_HW_GRP = MLX5_IPOOL_JUMP,
 	MLX5_IPOOL_SAMPLE, /* Pool for sample resource. */
 	MLX5_IPOOL_DEST_ARRAY, /* Pool for destination array resource. */
 	MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -108,6 +110,13 @@ enum mlx5_delay_drop_mode {
 	MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */
 };
 
+/* The HWS action type root/non-root. */
+enum mlx5_hw_action_flag_type {
+	MLX5_HW_ACTION_FLAG_ROOT, /* Root action. */
+	MLX5_HW_ACTION_FLAG_NONE_ROOT, /* Non-root ation. */
+	MLX5_HW_ACTION_FLAG_MAX, /* Maximum action flag. */
+};
+
 /* Hlist and list callback context. */
 struct mlx5_flow_cb_ctx {
 	struct rte_eth_dev *dev;
@@ -1201,7 +1210,10 @@ struct mlx5_dev_ctx_shared {
 	rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX];
 	/* UAR same-page access control required in 32bit implementations. */
 #endif
-	struct mlx5_hlist *flow_tbls;
+	union {
+		struct mlx5_hlist *flow_tbls; /* SWS flow table. */
+		struct mlx5_hlist *groups; /* HWS flow group. */
+	};
 	struct mlx5_flow_tunnel_hub *tunnel_hub;
 	/* Direct Rules tables for FDB, NIC TX+RX */
 	void *dr_drop_action; /* Pointer to DR drop action, any domain. */
@@ -1507,6 +1519,11 @@ struct mlx5_priv {
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
 	struct mlx5_hw_q *hw_q;
+	/* HW steering rte flow table list header. */
+	LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl;
+	/* HW steering global drop action. */
+	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
+				     [MLX5DR_TABLE_TYPE_MAX];
 #endif
 };
 
drivers/net/mlx5/mlx5_flow.c +93 -0

111 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +73 -0

97 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_hw.c +525 -0

574 lines not on this page · raw message

Signature · 1 line
2.25.1
#22 lore raw

PATCH v2 8/14 net/mlx5: add basic flow queue operation

In reply to Suanming Mou

The HW steering uses async queue-based flow rules management mechanism. The matcher and part of the actions have been prepared during flow table creation. Some remaining actions will be constructed during flow creation if needed.

A flow postpone attribute bit describes if flow management should be applied to the HW directly. An extra push function is provided to force push all the cached flows to the HW.

Once the flow has been applied to the HW, the pull function will be called to get the queued creation/destruction flows.

The DR rule flow memory is represented in PMD layer instead of allocating from HW steering layer. While destroying the flow, the flow rule memory can only be freed after the CQE received.

The HW queue job descriptor is currently introduced to convey the flow information and operation type between the flow insertion/destruction in the pull function.

This commit adds the basic flow queue operation for:
rte_flow_async_create(); rte_flow_async_destroy();
rte_flow_push();
rte_flow_pull();

---
 doc/guides/nics/mlx5.rst               |   6 +
 doc/guides/rel_notes/release_22_03.rst |   6 +
 drivers/net/mlx5/mlx5.h                |   2 +-
 drivers/net/mlx5/mlx5_flow.c           | 188 ++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h           |  41 ++++
 drivers/net/mlx5/mlx5_flow_hw.c        | 292 ++++++++++++++++++++++++-
 6 files changed, 533 insertions(+), 2 deletions(-)
doc/guides/nics/mlx5.rst +6 -0
index c3f7d61e7d..a1b3161c4a 100644
@@ -511,6 +511,12 @@ Limitations
     from the reference "Clock Queue" completions,
     the scheduled send timestamps should not be specified with non-zero MSB.
 
+  - HW steering:
+
+    - WQE based high scaling and safer flow insertion/destruction.
+    - Set ``dv_flow_en`` to 2 in order to enable HW steering.
+    - Async queue-based ``rte_flow_q`` APIs supported only.
+
 Statistics
 ----------
 
doc/guides/rel_notes/release_22_03.rst +6 -0
index da186315a5..5c39380320 100644
@@ -156,6 +156,12 @@ New Features
 
   * Added LED OEM support.
 
+* **Updated Mellanox mlx5 driver.**
+
+  Updated the Mellanox mlx5 driver with new features and improvements, including:
+
+  * Added WQE based hardware steering support with ``rte_flow_q`` API.
+
 * **Added an API for private user data in asymmetric crypto session.**
 
   An API was added to get/set an asymmetric crypto session's user data.
drivers/net/mlx5/mlx5.h +1 -1
index 1cc63e1d9f..17aa4b683b 100644
@@ -340,7 +340,7 @@ enum {
 /* HW steering flow management job descriptor. */
 struct mlx5_hw_q_job {
 	uint32_t type; /* Job type. */
-	struct rte_flow *flow; /* Flow attached to the job. */
+	struct rte_flow_hw *flow; /* Flow attached to the job. */
 	void *user_data; /* Job user data. */
 };
 
drivers/net/mlx5/mlx5_flow.c +188 -0

206 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +41 -0

65 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_hw.c +291 -1

316 lines not on this page · raw message

Signature · 1 line
2.25.1
#23 lore raw

PATCH v2 9/14 net/mlx5: add flow flush function

In reply to Suanming Mou

In case port is being stopped, all created flows should be flushed. This commit adds the flow flush helper function.

---
 drivers/net/mlx5/mlx5_flow.c    |   8 ++
 drivers/net/mlx5/mlx5_flow_hw.c | 129 ++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+)
drivers/net/mlx5/mlx5_flow.c +8 -0
index c6224e4f06..ff1c9f0067 100644
@@ -7013,6 +7013,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,
 	uint32_t num_flushed = 0, fidx = 1;
 	struct rte_flow *flow;
 
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+	if (priv->sh->config.dv_flow_en == 2 &&
+	    type == MLX5_FLOW_TYPE_GEN) {
+		flow_hw_q_flow_flush(dev, NULL);
+		return;
+	}
+#endif
+
 	MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {
 		flow_list_destroy(dev, type, fidx);
 		num_flushed++;
drivers/net/mlx5/mlx5_flow_hw.c +129 -0

141 lines not on this page · raw message

Signature · 1 line
2.25.1
#24 lore raw

PATCH v2 10/14 net/mlx5: add flow jump action

In reply to Suanming Mou

Jump action connects different level of flow tables and allows packet handling in the chain of flows.

A new action construct data struct is also added in this commit to help to handle not only the dynamic jump action but also for the other generic dynamic actions. The actions with empty mask configuration means dynamic action, and the dedicated action will be created with the flow action configuration during flow creation. In that dynamic action case, the action will be appended to the table template's action list during table creation. When creating the flows, traverse the action list and pick the dynamic action configuration details from flow actions as the action construct data struct describes, then create the dedicated dynamic actions.

This commit adds the jump action and the generic dynamic action construct mechanism.

---
 drivers/net/mlx5/mlx5.h         |   1 +
 drivers/net/mlx5/mlx5_flow.h    |  25 ++-
 drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++---
 3 files changed, 275 insertions(+), 21 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
index 17aa4b683b..bb0d07d870 100644
@@ -1524,6 +1524,7 @@ struct mlx5_priv {
 	/* HW steering global drop action. */
 	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
 				     [MLX5DR_TABLE_TYPE_MAX];
+	struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
 #endif
 };
 
drivers/net/mlx5/mlx5_flow.h +24 -1
index a0cc4028fa..675a5bfee5 100644
@@ -1020,10 +1020,25 @@ struct rte_flow {
 /* HWS flow struct. */
 struct rte_flow_hw {
 	uint32_t idx; /* Flow index from indexed pool. */
+	uint32_t fate_type; /* Fate action type. */
+	union {
+		/* Jump action. */
+		struct mlx5_hw_jump_action *jump;
+	};
 	struct rte_flow_template_table *table; /* The table flow allcated from. */
 	struct mlx5dr_rule rule; /* HWS layer data struct. */
 } __rte_packed;
 
+/* rte flow action translate to DR action struct. */
+struct mlx5_action_construct_data {
+	LIST_ENTRY(mlx5_action_construct_data) next;
+	/* Ensure the action types are matched. */
+	int type;
+	uint32_t idx;  /* Data index. */
+	uint16_t action_src; /* rte_flow_action src offset. */
+	uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+};
+
 /* Flow item template struct. */
 struct rte_flow_pattern_template {
 	LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1051,9 +1066,17 @@ struct mlx5_hw_jump_action {
 	struct mlx5dr_action *hws_action;
 };
 
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
 /* DR action set struct. */
 struct mlx5_hw_actions {
-	struct mlx5dr_action *drop; /* Drop action. */
+	/* Dynamic action list. */
+	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
+	struct mlx5_hw_jump_action *jump; /* Jump action. */
+	uint32_t acts_num:4; /* Total action number. */
+	/* Translated DR action array from action template. */
+	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
 };
 
 /* mlx5 action template struct. */
drivers/net/mlx5/mlx5_flow_hw.c +250 -20

399 lines not on this page · raw message

Signature · 1 line
2.25.1
#25 lore raw

PATCH v2 11/14 net/mlx5: add queue and RSS action

In reply to Suanming Mou

This commit adds the queue and RSS action. Similar to the jump action, dynamic ones will be added to the action construct list.

Due to the queue and RSS action in template should not be destroyed during port restart, the actions are created with standalone indirect table as indirect action does. When port stops, detaches the indirect table from action, when port starts, attaches the indirect table back to the action.

One more change is made to accelerate the action creation. Currently the mlx5_hrxq_get() function returns the object index instead of object pointer. This introduced an extra converting the index to the object by calling mlx5_ipool_get() in most of the case. And that extra converting hurts multi-thread performance since mlx5_ipool_get() uses the global lock inside. As the hash Rx queue object itself also contains the index, returns the object directly will achieve better performance without the global lock.

---
 drivers/net/mlx5/linux/mlx5_os.c   |  18 ++--
 drivers/net/mlx5/mlx5.h            |   4 +
 drivers/net/mlx5/mlx5_devx.c       |  10 ++
 drivers/net/mlx5/mlx5_flow.c       |  38 +++----
 drivers/net/mlx5/mlx5_flow.h       |   7 ++
 drivers/net/mlx5/mlx5_flow_dv.c    | 161 ++++++++++++++---------------
 drivers/net/mlx5/mlx5_flow_hw.c    | 101 ++++++++++++++++++
 drivers/net/mlx5/mlx5_flow_verbs.c |   7 +-
 drivers/net/mlx5/mlx5_rx.h         |   9 +-
 drivers/net/mlx5/mlx5_rxq.c        |  85 +++++++++------
 10 files changed, 283 insertions(+), 157 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +9 -9
index 0faf26f5b8..2e1606a733 100644
@@ -1521,6 +1521,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 	priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
 	if (!priv->drop_queue.hrxq)
 		goto error;
+	priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
+				       mlx5_hrxq_create_cb,
+				       mlx5_hrxq_match_cb,
+				       mlx5_hrxq_remove_cb,
+				       mlx5_hrxq_clone_cb,
+				       mlx5_hrxq_clone_free_cb);
+	if (!priv->hrxqs)
+		goto error;
+	rte_rwlock_init(&priv->ind_tbls_lock);
 	if (priv->sh->config.dv_flow_en == 2)
 		return eth_dev;
 	/* Port representor shares the same max priority with pf port. */
@@ -1545,15 +1554,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 			err = ENOTSUP;
 			goto error;
 	}
-	priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
-				       mlx5_hrxq_create_cb,
-				       mlx5_hrxq_match_cb,
-				       mlx5_hrxq_remove_cb,
-				       mlx5_hrxq_clone_cb,
-				       mlx5_hrxq_clone_free_cb);
-	if (!priv->hrxqs)
-		goto error;
-	rte_rwlock_init(&priv->ind_tbls_lock);
 	/* Query availability of metadata reg_c's. */
 	if (!priv->sh->metadata_regc_check_flag) {
 		err = mlx5_flow_discover_mreg_c(eth_dev);
drivers/net/mlx5/mlx5.h +4 -0
index bb0d07d870..43a913fcc5 100644
@@ -1284,6 +1284,7 @@ struct mlx5_flow_rss_desc {
 	uint64_t hash_fields; /* Verbs Hash fields. */
 	uint8_t key[MLX5_RSS_HASH_KEY_LEN]; /**< RSS hash key. */
 	uint32_t key_len; /**< RSS hash key len. */
+	uint32_t hws_flags; /**< HW steering action. */
 	uint32_t tunnel; /**< Queue in tunnel. */
 	uint32_t shared_rss; /**< Shared RSS index. */
 	struct mlx5_ind_table_obj *ind_tbl;
@@ -1345,6 +1346,7 @@ struct mlx5_hrxq {
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	void *action; /* DV QP action pointer. */
 #endif
+	uint32_t hws_flags; /* Hw steering flags. */
 	uint64_t hash_fields; /* Verbs Hash fields. */
 	uint32_t rss_key_len; /* Hash key length in bytes. */
 	uint32_t idx; /* Hash Rx queue index. */
@@ -1475,6 +1477,8 @@ struct mlx5_priv {
 	LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. */
 	/* Indirection tables. */
 	LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls;
+	/* Standalone indirect tables. */
+	LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;
 	/* Pointer to next element. */
 	rte_rwlock_t ind_tbls_lock;
 	uint32_t refcnt; /**< Reference counter. */
drivers/net/mlx5/mlx5_devx.c +10 -0
index e57787cfec..cafd7ead0d 100644
@@ -807,6 +807,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
 		goto error;
 	}
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	if (hrxq->hws_flags) {
+		hrxq->action = mlx5dr_action_create_dest_tir
+			(priv->dr_ctx,
+			 (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags);
+		if (!hrxq->action)
+			goto error;
+		return 0;
+	}
 	if (mlx5_flow_os_create_flow_action_dest_devx_tir(hrxq->tir,
 							  &hrxq->action)) {
 		rte_errno = errno;
@@ -1042,6 +1050,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)
 		DRV_LOG(ERR, "Cannot create drop RX queue");
 		return ret;
 	}
+	if (priv->sh->config.dv_flow_en == 2)
+		return 0;
 	/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */
 	ret = mlx5_devx_ind_table_new(dev, 0, hrxq->ind_table);
 	if (ret != 0) {
drivers/net/mlx5/mlx5_flow.c +12 -26
index ff1c9f0067..0b3134764d 100644
@@ -9438,14 +9438,10 @@ int
 mlx5_action_handle_attach(struct rte_eth_dev *dev)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct mlx5_indexed_pool *ipool =
-			priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
-	struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
 	int ret = 0;
-	uint32_t idx;
+	struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
 
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		const char *message;
 		uint32_t queue_idx;
 
@@ -9461,9 +9457,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
 	}
 	if (ret != 0)
 		return ret;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		ret = mlx5_ind_table_obj_attach(dev, ind_tbl);
 		if (ret != 0) {
 			DRV_LOG(ERR, "Port %u could not attach "
@@ -9472,13 +9466,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
 			goto error;
 		}
 	}
+
 	return 0;
 error:
-	shared_rss_last = shared_rss;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
-		if (shared_rss == shared_rss_last)
+	ind_tbl_last = ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+		if (ind_tbl == ind_tbl_last)
 			break;
 		if (mlx5_ind_table_obj_detach(dev, ind_tbl) != 0)
 			DRV_LOG(CRIT, "Port %u could not detach "
@@ -9501,15 +9494,10 @@ int
 mlx5_action_handle_detach(struct rte_eth_dev *dev)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct mlx5_indexed_pool *ipool =
-			priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
-	struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
 	int ret = 0;
-	uint32_t idx;
-
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+	struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
 
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		ret = mlx5_ind_table_obj_detach(dev, ind_tbl);
 		if (ret != 0) {
 			DRV_LOG(ERR, "Port %u could not detach "
@@ -9520,11 +9508,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev)
 	}
 	return 0;
 error:
-	shared_rss_last = shared_rss;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
-		if (shared_rss == shared_rss_last)
+	ind_tbl_last = ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+		if (ind_tbl == ind_tbl_last)
 			break;
 		if (mlx5_ind_table_obj_attach(dev, ind_tbl) != 0)
 			DRV_LOG(CRIT, "Port %u could not attach "
drivers/net/mlx5/mlx5_flow.h +7 -0
index 675a5bfee5..aa1709e7cb 100644
@@ -1024,6 +1024,7 @@ struct rte_flow_hw {
 	union {
 		/* Jump action. */
 		struct mlx5_hw_jump_action *jump;
+		struct mlx5_hrxq *hrxq; /* TIR action. */
 	};
 	struct rte_flow_template_table *table; /* The table flow allcated from. */
 	struct mlx5dr_rule rule; /* HWS layer data struct. */
@@ -1074,6 +1075,7 @@ struct mlx5_hw_actions {
 	/* Dynamic action list. */
 	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
+	struct mlx5_hrxq *tir; /* TIR action. */
 	uint32_t acts_num:4; /* Total action number. */
 	/* Translated DR action array from action template. */
 	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
@@ -1915,6 +1917,11 @@ int flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx,
 int
 flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
 		    struct rte_flow_error *error);
+void flow_dv_hashfields_set(uint64_t item_flags,
+			    struct mlx5_flow_rss_desc *rss_desc,
+			    uint64_t *hash_fields);
+void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+					uint64_t *hash_field);
 
 struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
 void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
drivers/net/mlx5/mlx5_flow_dv.c +76 -85

371 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_hw.c +101 -0

149 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_verbs.c +2 -5
index 90ccb9aaff..f08aa7a770 100644
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			MLX5_ASSERT(priv->drop_queue.hrxq);
 			hrxq = priv->drop_queue.hrxq;
 		} else {
-			uint32_t hrxq_idx;
 			struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;
 
 			MLX5_ASSERT(rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			rss_desc->tunnel = !!(handle->layers &
 					      MLX5_FLOW_LAYER_TUNNEL);
 			rss_desc->shared_rss = 0;
-			hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
-			hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
-					      hrxq_idx);
+			hrxq = mlx5_hrxq_get(dev, rss_desc);
 			if (!hrxq) {
 				rte_flow_error_set
 					(error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 					 "cannot get hash queue");
 				goto error;
 			}
-			handle->rix_hrxq = hrxq_idx;
+			handle->rix_hrxq = hrxq->idx;
 		}
 		MLX5_ASSERT(hrxq);
 		handle->drv_flow = mlx5_glue->create_flow
drivers/net/mlx5/mlx5_rx.h +7 -2
index 7e417819f7..221f4ded97 100644
@@ -225,9 +225,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);
 struct mlx5_ind_table_obj *mlx5_ind_table_obj_get(struct rte_eth_dev *dev,
 						  const uint16_t *queues,
 						  uint32_t queues_n);
+struct mlx5_ind_table_obj *mlx5_ind_table_obj_new(struct rte_eth_dev *dev,
+						  const uint16_t *queues,
+						  uint32_t queues_n,
+						  bool standalone,
+						  bool ref_qs);
 int mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
 			       struct mlx5_ind_table_obj *ind_tbl,
-			       bool standalone,
 			       bool deref_rxqs);
 int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
 			     struct mlx5_ind_table_obj *ind_tbl,
@@ -250,8 +254,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx,
 					   void *cb_ctx __rte_unused);
 void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,
 			     struct mlx5_list_entry *entry);
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
 		       struct mlx5_flow_rss_desc *rss_desc);
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq);
 int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hxrq_idx);
 uint32_t mlx5_hrxq_verify(struct rte_eth_dev *dev);
 enum mlx5_rxq_type mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t idx);
drivers/net/mlx5/mlx5_rxq.c +55 -30

209 lines not on this page · raw message

Signature · 1 line
2.25.1
#26 lore raw

PATCH v2 12/14 net/mlx5: add mark action

In reply to Suanming Mou

The mark action is covered by tag action internally. While it is added the HW will add a tag to the packet. The mark value can be set as fixed or dynamic as the action mask indicates.

---
 drivers/net/mlx5/mlx5.h         |  2 +
 drivers/net/mlx5/mlx5_flow.h    |  1 +
 drivers/net/mlx5/mlx5_flow_hw.c | 66 ++++++++++++++++++++++++++++++---
 3 files changed, 63 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.h +2 -0
index 43a913fcc5..e78eb5e380 100644
@@ -1528,6 +1528,8 @@ struct mlx5_priv {
 	/* HW steering global drop action. */
 	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
 				     [MLX5DR_TABLE_TYPE_MAX];
+	/* HW steering global drop action. */
+	struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX];
 	struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
 #endif
 };
drivers/net/mlx5/mlx5_flow.h +1 -0
index aa1709e7cb..ec759c1aa4 100644
@@ -1077,6 +1077,7 @@ struct mlx5_hw_actions {
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
 	struct mlx5_hrxq *tir; /* TIR action. */
 	uint32_t acts_num:4; /* Total action number. */
+	uint32_t mark:1; /* Indicate the mark action. */
 	/* Translated DR action array from action template. */
 	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
 };
drivers/net/mlx5/mlx5_flow_hw.c +60 -6

115 lines not on this page · raw message

Signature · 1 line
2.25.1
#27 lore raw

PATCH v2 13/14 net/mlx5: add indirect action

In reply to Suanming Mou

HW steering can support indirect action as well. With indirect action, the flow can be created with more flexible shared RSS action selection. This will can save the action template with different RSS actions.

This commit adds the flow queue operation callback for:
rte_flow_async_action_handle_create(); rte_flow_async_action_handle_destroy(); rte_flow_async_action_handle_update();

---
 drivers/net/mlx5/mlx5_flow.c    | 131 ++++++++++
 drivers/net/mlx5/mlx5_flow.h    |  59 +++++
 drivers/net/mlx5/mlx5_flow_dv.c |  21 +-
 drivers/net/mlx5/mlx5_flow_hw.c | 414 +++++++++++++++++++++++++++++++-
 4 files changed, 612 insertions(+), 13 deletions(-)
drivers/net/mlx5/mlx5_flow.c +131 -0
index 0b3134764d..bb4d2f6bae 100644
@@ -879,6 +879,29 @@ mlx5_flow_push(struct rte_eth_dev *dev,
 	       uint32_t queue,
 	       struct rte_flow_error *error);
 
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_op_attr *attr,
+				 const struct rte_flow_indir_action_conf *conf,
+				 const struct rte_flow_action *action,
+				 void *user_data,
+				 struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_op_attr *attr,
+				 struct rte_flow_action_handle *handle,
+				 const void *update,
+				 void *user_data,
+				 struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+				  const struct rte_flow_op_attr *attr,
+				  struct rte_flow_action_handle *handle,
+				  void *user_data,
+				  struct rte_flow_error *error);
+
 static const struct rte_flow_ops mlx5_flow_ops = {
 	.validate = mlx5_flow_validate,
 	.create = mlx5_flow_create,
@@ -911,6 +934,9 @@ static const struct rte_flow_ops mlx5_flow_ops = {
 	.async_destroy = mlx5_flow_async_flow_destroy,
 	.pull = mlx5_flow_pull,
 	.push = mlx5_flow_push,
+	.async_action_handle_create = mlx5_flow_async_action_handle_create,
+	.async_action_handle_update = mlx5_flow_async_action_handle_update,
+	.async_action_handle_destroy = mlx5_flow_async_action_handle_destroy,
 };
 
 /* Tunnel information. */
@@ -8364,6 +8390,111 @@ mlx5_flow_push(struct rte_eth_dev *dev,
 	return fops->push(dev, queue, error);
 }
 
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] conf
+ *   Indirect action configuration.
+ * @param[in] action
+ *   rte_flow action detail.
+ * @param[in] user_data
+ *   Pointer to the user_data.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_op_attr *attr,
+				 const struct rte_flow_indir_action_conf *conf,
+				 const struct rte_flow_action *action,
+				 void *user_data,
+				 struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->async_action_create(dev, queue, attr, conf, action,
+					 user_data, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] handle
+ *   Action handle to be updated.
+ * @param[in] update
+ *   Update value.
+ * @param[in] user_data
+ *   Pointer to the user_data.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+				     const struct rte_flow_op_attr *attr,
+				     struct rte_flow_action_handle *handle,
+				     const void *update,
+				     void *user_data,
+				     struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->async_action_update(dev, queue, attr, handle,
+					 update, user_data, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] handle
+ *   Action handle to be destroyed.
+ * @param[in] user_data
+ *   Pointer to the user_data.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+				      const struct rte_flow_op_attr *attr,
+				      struct rte_flow_action_handle *handle,
+				      void *user_data,
+				      struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->async_action_destroy(dev, queue, attr, handle,
+					  user_data, error);
+}
+
 /**
  * Allocate a new memory for the counter values wrapped by all the needed
  * management.
drivers/net/mlx5/mlx5_flow.h +59 -0

99 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_dv.c +12 -9
index 15d8535e6e..b49b88a13f 100644
@@ -13835,9 +13835,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,
  * @return
  *   Valid hash RX queue index, otherwise 0.
  */
-static uint32_t
-__flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
-				 const uint64_t hash_fields)
+uint32_t
+flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+			       const uint64_t hash_fields)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_shared_action_rss *shared_rss =
@@ -13965,7 +13965,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			struct mlx5_hrxq *hrxq = NULL;
 			uint32_t hrxq_idx;
 
-			hrxq_idx = __flow_dv_action_rss_hrxq_lookup(dev,
+			hrxq_idx = flow_dv_action_rss_hrxq_lookup(dev,
 						rss_desc->shared_rss,
 						dev_flow->hash_fields);
 			if (hrxq_idx)
@@ -14689,6 +14689,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
 			   struct mlx5_shared_action_rss *shared_rss,
 			   struct rte_flow_error *error)
 {
+	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_flow_rss_desc rss_desc = { 0 };
 	size_t i;
 	int err;
@@ -14709,6 +14710,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
 	/* Set non-zero value to indicate a shared RSS. */
 	rss_desc.shared_rss = action_idx;
 	rss_desc.ind_tbl = shared_rss->ind_tbl;
+	if (priv->sh->config.dv_flow_en == 2)
+		rss_desc.hws_flags = MLX5DR_ACTION_FLAG_HWS_RX;
 	for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
 		struct mlx5_hrxq *hrxq;
 		uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14900,7 +14903,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
  *   A valid shared action handle in case of success, NULL otherwise and
  *   rte_errno is set.
  */
-static struct rte_flow_action_handle *
+struct rte_flow_action_handle *
 flow_dv_action_create(struct rte_eth_dev *dev,
 		      const struct rte_flow_indir_action_conf *conf,
 		      const struct rte_flow_action *action,
@@ -14970,7 +14973,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_destroy(struct rte_eth_dev *dev,
 		       struct rte_flow_action_handle *handle,
 		       struct rte_flow_error *error)
@@ -15180,7 +15183,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_update(struct rte_eth_dev *dev,
 			struct rte_flow_action_handle *handle,
 			const void *update,
@@ -15894,7 +15897,7 @@ flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx,
 				  "counters are not available");
 }
 
-static int
+int
 flow_dv_action_query(struct rte_eth_dev *dev,
 		     const struct rte_flow_action_handle *handle, void *data,
 		     struct rte_flow_error *error)
@@ -17584,7 +17587,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_validate(struct rte_eth_dev *dev,
 			const struct rte_flow_indir_action_conf *conf,
 			const struct rte_flow_action *action,
drivers/net/mlx5/mlx5_flow_hw.c +410 -4

493 lines not on this page · raw message

Signature · 1 line
2.25.1
#28 lore raw

PATCH v2 14/14 net/mlx5: add header reformat action

In reply to Suanming Mou

HW steering header reformat action can work under bulk mode. In this case, when create the table, bulk size of header reformat actions will be allocated in low level. Afterwards, when create flow, just simply specify the action index in the bulk and the encapsulation data to the action will be enough.

---
 drivers/net/mlx5/mlx5.h         |   1 +
 drivers/net/mlx5/mlx5_flow.h    |  21 +++
 drivers/net/mlx5/mlx5_flow_dv.c |   4 +-
 drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++-
 4 files changed, 251 insertions(+), 3 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
index e78eb5e380..bb7067d62d 100644
@@ -342,6 +342,7 @@ struct mlx5_hw_q_job {
 	uint32_t type; /* Job type. */
 	struct rte_flow_hw *flow; /* Flow attached to the job. */
 	void *user_data; /* Job user data. */
+	uint8_t *encap_data; /* Encap data. */
 };
 
 /* HW steering job descriptor LIFO pool. */
drivers/net/mlx5/mlx5_flow.h +21 -0
index 9ac6745597..ca9011958e 100644
@@ -1040,6 +1040,14 @@ struct mlx5_action_construct_data {
 	uint16_t action_src; /* rte_flow_action src offset. */
 	uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
 	union {
+		struct {
+			/* encap src(item) offset. */
+			uint16_t src;
+			/* encap dst data offset. */
+			uint16_t dst;
+			/* encap data len. */
+			uint16_t len;
+		} encap;
 		struct {
 			uint64_t types; /* RSS hash types. */
 			uint32_t level; /* RSS level. */
@@ -1076,6 +1084,13 @@ struct mlx5_hw_jump_action {
 	struct mlx5dr_action *hws_action;
 };
 
+/* Encap decap action struct. */
+struct mlx5_hw_encap_decap_action {
+	struct mlx5dr_action *action; /* Action object. */
+	size_t data_size; /* Action metadata size. */
+	uint8_t data[]; /* Action data. */
+};
+
 /* The maximum actions support in the flow. */
 #define MLX5_HW_MAX_ACTS 16
 
@@ -1085,6 +1100,9 @@ struct mlx5_hw_actions {
 	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
 	struct mlx5_hrxq *tir; /* TIR action. */
+	/* Encap/Decap action. */
+	struct mlx5_hw_encap_decap_action *encap_decap;
+	uint16_t encap_decap_pos; /* Encap/Decap action position. */
 	uint32_t acts_num:4; /* Total action number. */
 	uint32_t mark:1; /* Indicate the mark action. */
 	/* Translated DR action array from action template. */
@@ -2032,4 +2050,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,
 			 const struct rte_flow_action_handle *handle,
 			 void *data,
 			 struct rte_flow_error *error);
+size_t flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type);
+int flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
+			   size_t *size, struct rte_flow_error *error);
 #endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_dv.c +2 -2
index b49b88a13f..48ea079dd5 100644
@@ -4026,7 +4026,7 @@ flow_dv_push_vlan_action_resource_register
  * @return
  *   sizeof struct item_type, 0 if void or irrelevant.
  */
-static size_t
+size_t
 flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
 {
 	size_t retval;
@@ -4092,7 +4092,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
  * @return
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
-static int
+int
 flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
 			   size_t *size, struct rte_flow_error *error)
 {
drivers/net/mlx5/mlx5_flow_hw.c +227 -1

299 lines not on this page · raw message

Signature · 1 line
2.25.1
#29 lore raw

v3 cover net/mlx5: add hardware steering

In reply to Suanming Mou

The Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:

- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly

However, there are still some disadvantages:

- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects

- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion

- security, exposing the low-level steering entries directly to the userspace may cause security risks

A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.

In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.

*** THIS PATCH SET DEPENDS ON THE NEW RTE_FLOW ASYNC API ***

---

v3:
- rebase to the latest version.

v2:
- New HW steering low-level abstract code added.
- commit message improvement.
- add protection for rte_flow and rte_flow_async callbacks.
- rebase to rte_flow_async v9.
- fix some rte_flow error not filled bugs.

Suanming Mou (14):
net/mlx5: introduce hardware steering operation net/mlx5: add HW steering low-level abstract code net/mlx5: introduce hardware steering enable routine net/mlx5: add port flow configuration net/mlx5: add pattern template management net/mlx5: add action template management net/mlx5: add table management
net/mlx5: add basic flow queue operation net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action

 doc/guides/nics/mlx5.rst                |   19 +-
 doc/guides/rel_notes/release_22_03.rst  |    6 +
 drivers/net/mlx5/linux/mlx5_flow_os.h   |    1 +
 drivers/net/mlx5/linux/mlx5_os.c        |   22 +-
 drivers/net/mlx5/meson.build            |    2 +
 drivers/net/mlx5/mlx5.c                 |   55 +-
 drivers/net/mlx5/mlx5.h                 |   66 +-
 drivers/net/mlx5/mlx5_devx.c            |   10 +
 drivers/net/mlx5/mlx5_dr.c              |  383 ++++
 drivers/net/mlx5/mlx5_dr.h              |  456 +++++
 drivers/net/mlx5/mlx5_flow.c            |  721 ++++++-
 drivers/net/mlx5/mlx5_flow.h            |  287 +++
 drivers/net/mlx5/mlx5_flow_dv.c         |  186 +-
 drivers/net/mlx5/mlx5_flow_hw.c         | 2335 +++++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow_verbs.c      |    7 +-
 drivers/net/mlx5/mlx5_rx.h              |    9 +-
 drivers/net/mlx5/mlx5_rxq.c             |   85 +-
 drivers/net/mlx5/windows/mlx5_flow_os.h |    1 +
 18 files changed, 4471 insertions(+), 180 deletions(-)
 create mode 100644 drivers/net/mlx5/mlx5_dr.c
 create mode 100644 drivers/net/mlx5/mlx5_dr.h
 create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
Signature · 1 line
2.25.1
#30 lore raw

PATCH v3 1/14 net/mlx5: introduce hardware steering operation

The Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:

- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly

However, there are still some disadvantages:

- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects

- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion

- security, exposing the low-level steering entries directly to the userspace may cause security risks

A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.

In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.

This commit adds the basic driver operation.

---
 drivers/net/mlx5/linux/mlx5_flow_os.h   |  1 +
 drivers/net/mlx5/meson.build            |  1 +
 drivers/net/mlx5/mlx5_flow.c            |  1 +
 drivers/net/mlx5/mlx5_flow.h            |  1 +
 drivers/net/mlx5/mlx5_flow_hw.c         | 13 +++++++++++++
 drivers/net/mlx5/windows/mlx5_flow_os.h |  1 +
 6 files changed, 18 insertions(+)
 create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
drivers/net/mlx5/linux/mlx5_flow_os.h +1 -0
index 1926d26410..e28a9e0436 100644
@@ -9,6 +9,7 @@
 
 #ifdef HAVE_IBV_FLOW_DV_SUPPORT
 extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
 #endif
 
 /**
drivers/net/mlx5/meson.build +1 -0
index 2f6d8cbb3d..39a2b8c523 100644
@@ -16,6 +16,7 @@ sources = files(
         'mlx5_flow.c',
         'mlx5_flow_meter.c',
         'mlx5_flow_dv.c',
+        'mlx5_flow_hw.c',
         'mlx5_flow_aso.c',
         'mlx5_flow_flex.c',
         'mlx5_mac.c',
drivers/net/mlx5/mlx5_flow.c +1 -0
index a87ac8e6d7..b289f13fc0 100644
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = {
 	[MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	[MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops,
+	[MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops,
 #endif
 	[MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops,
 	[MLX5_FLOW_TYPE_MAX] = &mlx5_flow_null_drv_ops
drivers/net/mlx5/mlx5_flow.h +1 -0
index a20773eeb2..b70ef0c1b8 100644
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {
 	MLX5_FLOW_TYPE_MIN,
 	MLX5_FLOW_TYPE_DV,
 	MLX5_FLOW_TYPE_VERBS,
+	MLX5_FLOW_TYPE_HW,
 	MLX5_FLOW_TYPE_MAX,
 };
 
drivers/net/mlx5/mlx5_flow_hw.c new file +13 -0
new file mode 100644
index 0000000000..729d5914a8
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2022 NVIDIA Corporation & Affiliates
+ */
+
+#include <rte_flow.h>
+
+#include "mlx5_flow.h"
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+
+#endif
drivers/net/mlx5/windows/mlx5_flow_os.h +1 -0
index dfcb012334..52013b06a0 100644
@@ -10,6 +10,7 @@
 
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
 #endif
 
 /**
Signature · 1 line
2.25.1
#31 lore raw

PATCH v3 2/14 net/mlx5: add HW steering low-level abstract code

In reply to Suanming Mou

The HW steering low-level implementation will be added later in another patch series. To avoid the linkage issues the abstract stub replacement is provided currently.

---
 drivers/net/mlx5/meson.build |   1 +
 drivers/net/mlx5/mlx5_dr.c   | 383 +++++++++++++++++++++++++++++
 drivers/net/mlx5/mlx5_dr.h   | 456 +++++++++++++++++++++++++++++++++++
 3 files changed, 840 insertions(+)
 create mode 100644 drivers/net/mlx5/mlx5_dr.c
 create mode 100644 drivers/net/mlx5/mlx5_dr.h
drivers/net/mlx5/meson.build +1 -0
index 39a2b8c523..393b2c97ac 100644
@@ -14,6 +14,7 @@ sources = files(
         'mlx5.c',
         'mlx5_ethdev.c',
         'mlx5_flow.c',
+	'mlx5_dr.c',
         'mlx5_flow_meter.c',
         'mlx5_flow_dv.c',
         'mlx5_flow_hw.c',
drivers/net/mlx5/mlx5_dr.c new file +383 -0

383 lines not on this page · raw message

drivers/net/mlx5/mlx5_dr.h new file +456 -0

456 lines not on this page · raw message

Signature · 1 line
2.25.1
#32 lore raw

PATCH v3 3/14 net/mlx5: introduce hardware steering enable routine

In reply to Suanming Mou

The new hardware steering engine relies on using dedicated steering WQEs instead of direct writing to the low-level steering table entries directly. In the first introduce implementation the hardware steering engine supports the new queue based Flow API, the existing synchronous non-queue based Flow API is not supported.

A new dv_flow_en value 2 is added to manage mlx5 PMD steering engine:

dv_flow_en rte_flow API rte_flow_async API
------------------------------------------------
0 support not support
1 support not support
2 not support support

This commit introduces the extra dv_flow_en = 2 to specify the new flow initialize and manage operation routine.

---
 doc/guides/nics/mlx5.rst         | 13 ++++++++++---
 drivers/net/mlx5/linux/mlx5_os.c |  4 ++++
 drivers/net/mlx5/mlx5.c          |  7 ++++++-
 drivers/net/mlx5/mlx5.h          |  3 ++-
 drivers/net/mlx5/mlx5_flow.c     | 22 ++++++++++++++++++++++
 5 files changed, 44 insertions(+), 5 deletions(-)
doc/guides/nics/mlx5.rst +10 -3
index 8956cd1dd8..0e0169c8bb 100644
@@ -942,10 +942,17 @@ for an additional list of options shared with other mlx5 drivers.
 
 - ``dv_flow_en`` parameter [int]
 
-  A nonzero value enables the DV flow steering assuming it is supported
-  by the driver (RDMA Core library version is rdma-core-24.0 or higher).
+  Value 0 means legacy Verbs flow offloading.
 
-  Enabled by default if supported.
+  Value 1 enables the DV flow steering assuming it is supported by the
+  driver (RDMA Core library version is rdma-core-24.0 or higher).
+
+  Value 2 enables the WQE based hardware steering. In this mode only
+  the queue-based rte_flow_q flow management is supported.
+
+  Configured by default to 1 DV flow steering if the driver(RDMA CORE library)
+  supported. Otherwise, the value will be 0 which indicates legacy Verbs flow
+  offloading.
 
 - ``dv_esw_en`` parameter [int]
 
drivers/net/mlx5/linux/mlx5_os.c +4 -0
index ecf823da56..0faf26f5b8 100644
@@ -482,6 +482,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
 	err = mlx5_alloc_table_hash_list(priv);
 	if (err)
 		goto error;
+	if (priv->sh->config.dv_flow_en == 2)
+		return 0;
 	/* The resources below are only valid with DV support. */
 #ifdef HAVE_IBV_FLOW_DV_SUPPORT
 	/* Init port id action list. */
@@ -1519,6 +1521,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 	priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
 	if (!priv->drop_queue.hrxq)
 		goto error;
+	if (priv->sh->config.dv_flow_en == 2)
+		return eth_dev;
 	/* Port representor shares the same max priority with pf port. */
 	if (!priv->sh->flow_priority_check_flag) {
 		/* Supported Verbs flow priority number detection. */
drivers/net/mlx5/mlx5.c +6 -1
index 9f65a8f901..f49d30c05c 100644
@@ -1199,7 +1199,12 @@ mlx5_dev_args_check_handler(const char *key, const char *val, void *opaque)
 	} else if (strcmp(MLX5_DV_ESW_EN, key) == 0) {
 		config->dv_esw_en = !!tmp;
 	} else if (strcmp(MLX5_DV_FLOW_EN, key) == 0) {
-		config->dv_flow_en = !!tmp;
+		if (tmp > 2) {
+			DRV_LOG(ERR, "Invalid %s parameter.", key);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		config->dv_flow_en = tmp;
 	} else if (strcmp(MLX5_DV_XMETA_EN, key) == 0) {
 		if (tmp != MLX5_XMETA_MODE_LEGACY &&
 		    tmp != MLX5_XMETA_MODE_META16 &&
drivers/net/mlx5/mlx5.h +2 -1
index 0f465d0e9e..b2259fc1fb 100644
@@ -287,7 +287,8 @@ struct mlx5_sh_config {
 	int tx_skew; /* Tx scheduling skew between WQE and data on wire. */
 	uint32_t reclaim_mode:2; /* Memory reclaim mode. */
 	uint32_t dv_esw_en:1; /* Enable E-Switch DV flow. */
-	uint32_t dv_flow_en:1; /* Enable DV flow. */
+	/* Enable DV flow. 1 means SW steering, 2 means HW steering. */
+	unsigned int dv_flow_en:2;
 	uint32_t dv_xmeta_en:2; /* Enable extensive flow metadata. */
 	uint32_t dv_miss_info:1; /* Restore packet after partial hw miss. */
 	uint32_t l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
drivers/net/mlx5/mlx5_flow.c +22 -0
index b289f13fc0..cdb40c0756 100644
@@ -6840,6 +6840,15 @@ mlx5_flow_create(struct rte_eth_dev *dev,
 		 const struct rte_flow_action actions[],
 		 struct rte_flow_error *error)
 {
+	struct mlx5_priv *priv = dev->data->dev_private;
+
+	if (priv->sh->config.dv_flow_en == 2) {
+		rte_flow_error_set(error, ENOTSUP,
+			  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+			  NULL,
+			  "Flow non-Q creation not supported");
+		return NULL;
+	}
 	/*
 	 * If the device is not started yet, it is not allowed to created a
 	 * flow from application. PMD default flows and traffic control flows
@@ -7336,6 +7345,13 @@ mlx5_flow_destroy(struct rte_eth_dev *dev,
 		  struct rte_flow *flow,
 		  struct rte_flow_error *error __rte_unused)
 {
+	struct mlx5_priv *priv = dev->data->dev_private;
+
+	if (priv->sh->config.dv_flow_en == 2)
+		return rte_flow_error_set(error, ENOTSUP,
+			  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+			  NULL,
+			  "Flow non-Q destruction not supported");
 	flow_list_destroy(dev, MLX5_FLOW_TYPE_GEN,
 				(uintptr_t)(void *)flow);
 	return 0;
@@ -7433,7 +7449,13 @@ mlx5_flow_query(struct rte_eth_dev *dev,
 		struct rte_flow_error *error)
 {
 	int ret;
+	struct mlx5_priv *priv = dev->data->dev_private;
 
+	if (priv->sh->config.dv_flow_en == 2)
+		return rte_flow_error_set(error, ENOTSUP,
+			  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+			  NULL,
+			  "Flow non-Q query not supported");
 	ret = flow_drv_query(dev, (uintptr_t)(void *)flow, actions, data,
 			     error);
 	if (ret < 0)
Signature · 1 line
2.25.1
#33 lore raw

PATCH v3 4/14 net/mlx5: add port flow configuration

In reply to Suanming Mou

The hardware steering is backend to support rte_flow_async API in mlx5 PMD. The port configuration function creates the queues and needed flow management resources.

The PMD layer configuration function allocates the queues' context and per-queue job descriptor pool. The job descriptor pool size is equal to the queue size, and the job descriptors will be popped from pool with LIFO strategy to convey the flow information during flow insertion/destruction. Then, while polling the queued operation result, the flow information will be extracted from the job descriptor and the descriptor will be pushed back to the LIFO pool.

The commit creates the flow port queues and the job descriptor pools.

---
 drivers/net/mlx5/mlx5.c         |   3 +
 drivers/net/mlx5/mlx5.h         |  30 +++++-
 drivers/net/mlx5/mlx5_flow.c    |  86 +++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    |  15 +++
 drivers/net/mlx5/mlx5_flow_hw.c | 159 ++++++++++++++++++++++++++++++++
 5 files changed, 292 insertions(+), 1 deletion(-)
drivers/net/mlx5/mlx5.c +3 -0
index f49d30c05c..0079aa83c1 100644
@@ -1820,6 +1820,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)
 	/* Free the eCPRI flex parser resource. */
 	mlx5_flex_parser_ecpri_release(dev);
 	mlx5_flex_item_port_cleanup(dev);
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	flow_hw_resource_release(dev);
+#endif
 	if (priv->rxq_privs != NULL) {
 		/* XXX race condition if mlx5_rx_burst() is still running. */
 		rte_delay_us_sleep(1000);
drivers/net/mlx5/mlx5.h +29 -1
index b2259fc1fb..f0edf7f559 100644
@@ -33,7 +33,9 @@
 #include "mlx5_utils.h"
 #include "mlx5_os.h"
 #include "mlx5_autoconf.h"
-
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+#include "mlx5_dr.h"
+#endif
 
 #define MLX5_SH(dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
 
@@ -320,6 +322,26 @@ struct mlx5_lb_ctx {
 	uint16_t refcnt; /* Reference count for representors. */
 };
 
+/* HW steering queue job descriptor type. */
+enum {
+	MLX5_HW_Q_JOB_TYPE_CREATE, /* Flow create job type. */
+	MLX5_HW_Q_JOB_TYPE_DESTROY, /* Flow destroy job type. */
+};
+
+/* HW steering flow management job descriptor. */
+struct mlx5_hw_q_job {
+	uint32_t type; /* Job type. */
+	struct rte_flow *flow; /* Flow attached to the job. */
+	void *user_data; /* Job user data. */
+};
+
+/* HW steering job descriptor LIFO pool. */
+struct mlx5_hw_q {
+	uint32_t job_idx; /* Free job index. */
+	uint32_t size; /* LIFO size. */
+	struct mlx5_hw_q_job **job; /* LIFO header. */
+} __rte_cache_aligned;
+
 #define MLX5_COUNTERS_PER_POOL 512
 #define MLX5_MAX_PENDING_QUERIES 4
 #define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1479,6 +1501,12 @@ struct mlx5_priv {
 	struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM];
 	/* Flex items have been created on the port. */
 	uint32_t flex_item_map; /* Map of allocated flex item elements. */
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
+	uint32_t nb_queue; /* HW steering queue number. */
+	/* HW steering queue polling mechanism job descriptor LIFO. */
+	struct mlx5_hw_q *hw_q;
+#endif
 };
 
 #define PORT_ID(priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.c +86 -0

110 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +15 -0
index b70ef0c1b8..9f0dc4bde7 100644
@@ -1257,6 +1257,17 @@ typedef int (*mlx5_flow_item_update_t)
 			 const struct rte_flow_item_flex_handle *handle,
 			 const struct rte_flow_item_flex_conf *conf,
 			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_info_get_t)
+			(struct rte_eth_dev *dev,
+			 struct rte_flow_port_info *port_info,
+			 struct rte_flow_queue_info *queue_info,
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_port_configure_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_port_attr *port_attr,
+			 uint16_t nb_queue,
+			 const struct rte_flow_queue_attr *queue_attr[],
+			 struct rte_flow_error *err);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1295,6 +1306,8 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_item_create_t item_create;
 	mlx5_flow_item_release_t item_release;
 	mlx5_flow_item_update_t item_update;
+	mlx5_flow_info_get_t info_get;
+	mlx5_flow_port_configure_t configure;
 };
 
 /* mlx5_flow.c */
@@ -1762,4 +1775,6 @@ const struct mlx5_flow_tunnel *
 mlx5_get_tof(const struct rte_flow_item *items,
 	     const struct rte_flow_action *actions,
 	     enum mlx5_tof_rule_type *rule_type);
+void
+flow_hw_resource_release(struct rte_eth_dev *dev);
 #endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_hw.c +159 -0

169 lines not on this page · raw message

Signature · 1 line
2.25.1
#34 lore raw

PATCH v3 5/14 net/mlx5: add pattern template management

In reply to Suanming Mou

The pattern template defines flows that have the same matching fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be (eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while the values for each rule will be different.

Due to the pattern template can be used in different domains, the items will only be cached in pattern template create stage, while the template is binded to a dedicated table, the HW criteria will be created and saved to the table. The pattern templates can be used by multiple tables. But different tables create the same criteria and will not share the matcher between each other in order to have better performance.

This commit adds pattern template management.

---
 drivers/net/mlx5/mlx5.h         |  2 +
 drivers/net/mlx5/mlx5_flow.c    | 76 +++++++++++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    | 24 +++++++++
 drivers/net/mlx5/mlx5_flow_hw.c | 86 +++++++++++++++++++++++++++++++++
 4 files changed, 188 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
index f0edf7f559..e36adde708 100644
@@ -1502,6 +1502,8 @@ struct mlx5_priv {
 	/* Flex items have been created on the port. */
 	uint32_t flex_item_map; /* Map of allocated flex item elements. */
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	/* Item template list. */
+	LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
 	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +76 -0

94 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +24 -0
index 9f0dc4bde7..49027d6a3a 100644
@@ -1015,6 +1015,19 @@ struct rte_flow {
 	uint32_t geneve_tlv_option; /**< Holds Geneve TLV option id. > */
 } __rte_packed;
 
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+/* Flow item template struct. */
+struct rte_flow_pattern_template {
+	LIST_ENTRY(rte_flow_pattern_template) next;
+	/* Template attributes. */
+	struct rte_flow_pattern_template_attr attr;
+	struct mlx5dr_match_template *mt; /* mlx5 match template. */
+	uint32_t refcnt;  /* Reference counter. */
+};
+
+#endif
+
 /*
  * Define list of valid combinations of RX Hash fields
  * (see enum ibv_rx_hash_fields).
@@ -1268,6 +1281,15 @@ typedef int (*mlx5_flow_port_configure_t)
 			 uint16_t nb_queue,
 			 const struct rte_flow_queue_attr *queue_attr[],
 			 struct rte_flow_error *err);
+typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_pattern_template_attr *attr,
+			 const struct rte_flow_item items[],
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_pattern_template_destroy_t)
+			(struct rte_eth_dev *dev,
+			 struct rte_flow_pattern_template *template,
+			 struct rte_flow_error *error);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1308,6 +1330,8 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_item_update_t item_update;
 	mlx5_flow_info_get_t info_get;
 	mlx5_flow_port_configure_t configure;
+	mlx5_flow_pattern_template_create_t pattern_template_create;
+	mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
 };
 
 /* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +86 -0

107 lines not on this page · raw message

Signature · 1 line
2.25.1
#35 lore raw

PATCH v3 6/14 net/mlx5: add action template management

In reply to Suanming Mou

The action template holds a list of action types that will be used together on the same rule. The template's actions instances will be created only when the template bind to the dedicated group. And the created actions will be saved to each individual group in order for best performance. The actions in a group will not be shared with each other unless shared actions are specified.

This commit adds the action template management which stores the flow action template.

---
 drivers/net/mlx5/mlx5.h         |   2 +
 drivers/net/mlx5/mlx5_flow.c    |  78 +++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    |  22 ++++++
 drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++
 4 files changed, 218 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
index e36adde708..156b8613d8 100644
@@ -1504,6 +1504,8 @@ struct mlx5_priv {
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	/* Item template list. */
 	LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
+	/* Action template list. */
+	LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;
 	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +78 -0

96 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +22 -0
index 49027d6a3a..9a643fe0a8 100644
@@ -1026,6 +1026,16 @@ struct rte_flow_pattern_template {
 	uint32_t refcnt;  /* Reference counter. */
 };
 
+/* Flow action template struct. */
+struct rte_flow_actions_template {
+	LIST_ENTRY(rte_flow_actions_template) next;
+	/* Template attributes. */
+	struct rte_flow_actions_template_attr attr;
+	struct rte_flow_action *actions; /* Cached flow actions. */
+	struct rte_flow_action *masks; /* Cached action masks.*/
+	uint32_t refcnt; /* Reference counter. */
+};
+
 #endif
 
 /*
@@ -1290,6 +1300,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t)
 			(struct rte_eth_dev *dev,
 			 struct rte_flow_pattern_template *template,
 			 struct rte_flow_error *error);
+typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_actions_template_attr *attr,
+			 const struct rte_flow_action actions[],
+			 const struct rte_flow_action masks[],
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_actions_template_destroy_t)
+			(struct rte_eth_dev *dev,
+			 struct rte_flow_actions_template *template,
+			 struct rte_flow_error *error);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1332,6 +1352,8 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_port_configure_t configure;
 	mlx5_flow_pattern_template_create_t pattern_template_create;
 	mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
+	mlx5_flow_actions_template_create_t actions_template_create;
+	mlx5_flow_actions_template_destroy_t actions_template_destroy;
 };
 
 /* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +116 -0

140 lines not on this page · raw message

Signature · 1 line
2.25.1
#36 lore raw

PATCH v3 7/14 net/mlx5: add table management

In reply to Suanming Mou

Flow table is a group of flows with the same matching criteria and the same actions defined for them. The table defines rules that have the same matching fields but with different matching values. For example, matching on 5 tuple, the table will be (IPv4 source + IPv4 dest + s_port + d_port + next_proto) while the values for each rule will be different.

The templates' relevant matching criteria and action instances will be created in the table creation and saved in the table. As table attributes indicate the supported flow number, the flow memory will also be allocated at the same time.

This commit adds the table management functions.

---
 drivers/net/mlx5/mlx5.c         |  45 ++-
 drivers/net/mlx5/mlx5.h         |  21 +-
 drivers/net/mlx5/mlx5_flow.c    |  93 ++++++
 drivers/net/mlx5/mlx5_flow.h    |  73 +++++
 drivers/net/mlx5/mlx5_flow_hw.c | 525 ++++++++++++++++++++++++++++++++
 5 files changed, 751 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.c +41 -4
index 0079aa83c1..7611fdd62b 100644
@@ -1607,12 +1607,46 @@ void
 mlx5_free_table_hash_list(struct mlx5_priv *priv)
 {
 	struct mlx5_dev_ctx_shared *sh = priv->sh;
-
-	if (!sh->flow_tbls)
+	struct mlx5_hlist **tbls = (priv->sh->config.dv_flow_en == 2) ?
+				   &sh->groups : &sh->flow_tbls;
+	if (*tbls == NULL)
 		return;
-	mlx5_hlist_destroy(sh->flow_tbls);
-	sh->flow_tbls = NULL;
+	mlx5_hlist_destroy(*tbls);
+	*tbls = NULL;
+}
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/**
+ * Allocate HW steering group hash list.
+ *
+ * @param[in] priv
+ *   Pointer to the private device data structure.
+ */
+static int
+mlx5_alloc_hw_group_hash_list(struct mlx5_priv *priv)
+{
+	int err = 0;
+	struct mlx5_dev_ctx_shared *sh = priv->sh;
+	char s[MLX5_NAME_SIZE];
+
+	MLX5_ASSERT(sh);
+	snprintf(s, sizeof(s), "%s_flow_groups", priv->sh->ibdev_name);
+	sh->groups = mlx5_hlist_create
+			(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
+			 false, true, sh,
+			 flow_hw_grp_create_cb,
+			 flow_hw_grp_match_cb,
+			 flow_hw_grp_remove_cb,
+			 flow_hw_grp_clone_cb,
+			 flow_hw_grp_clone_free_cb);
+	if (!sh->groups) {
+		DRV_LOG(ERR, "flow groups with hash creation failed.");
+		err = ENOMEM;
+	}
+	return err;
 }
+#endif
+
 
 /**
  * Initialize flow table hash list and create the root tables entry
@@ -1628,11 +1662,14 @@ int
 mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused)
 {
 	int err = 0;
+
 	/* Tables are only used in DV and DR modes. */
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	struct mlx5_dev_ctx_shared *sh = priv->sh;
 	char s[MLX5_NAME_SIZE];
 
+	if (priv->sh->config.dv_flow_en == 2)
+		return mlx5_alloc_hw_group_hash_list(priv);
 	MLX5_ASSERT(sh);
 	snprintf(s, sizeof(s), "%s_flow_table", priv->sh->ibdev_name);
 	sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
drivers/net/mlx5/mlx5.h +19 -2
index 156b8613d8..f523173ad5 100644
@@ -64,7 +64,9 @@ enum mlx5_ipool_index {
 	MLX5_IPOOL_PUSH_VLAN, /* Pool for push vlan resource. */
 	MLX5_IPOOL_TAG, /* Pool for tag resource. */
 	MLX5_IPOOL_PORT_ID, /* Pool for port id resource. */
-	MLX5_IPOOL_JUMP, /* Pool for jump resource. */
+	MLX5_IPOOL_JUMP, /* Pool for SWS jump resource. */
+	/* Pool for HWS group. Jump action will be created internally. */
+	MLX5_IPOOL_HW_GRP = MLX5_IPOOL_JUMP,
 	MLX5_IPOOL_SAMPLE, /* Pool for sample resource. */
 	MLX5_IPOOL_DEST_ARRAY, /* Pool for destination array resource. */
 	MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -108,6 +110,13 @@ enum mlx5_delay_drop_mode {
 	MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */
 };
 
+/* The HWS action type root/non-root. */
+enum mlx5_hw_action_flag_type {
+	MLX5_HW_ACTION_FLAG_ROOT, /* Root action. */
+	MLX5_HW_ACTION_FLAG_NONE_ROOT, /* Non-root ation. */
+	MLX5_HW_ACTION_FLAG_MAX, /* Maximum action flag. */
+};
+
 /* Hlist and list callback context. */
 struct mlx5_flow_cb_ctx {
 	struct rte_eth_dev *dev;
@@ -1204,7 +1213,10 @@ struct mlx5_dev_ctx_shared {
 	rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX];
 	/* UAR same-page access control required in 32bit implementations. */
 #endif
-	struct mlx5_hlist *flow_tbls;
+	union {
+		struct mlx5_hlist *flow_tbls; /* SWS flow table. */
+		struct mlx5_hlist *groups; /* HWS flow group. */
+	};
 	struct mlx5_flow_tunnel_hub *tunnel_hub;
 	/* Direct Rules tables for FDB, NIC TX+RX */
 	void *dr_drop_action; /* Pointer to DR drop action, any domain. */
@@ -1510,6 +1522,11 @@ struct mlx5_priv {
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
 	struct mlx5_hw_q *hw_q;
+	/* HW steering rte flow table list header. */
+	LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl;
+	/* HW steering global drop action. */
+	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
+				     [MLX5DR_TABLE_TYPE_MAX];
 #endif
 };
 
drivers/net/mlx5/mlx5_flow.c +93 -0

111 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +73 -0

97 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_hw.c +525 -0

574 lines not on this page · raw message

Signature · 1 line
2.25.1
#37 lore raw

PATCH v3 8/14 net/mlx5: add basic flow queue operation

In reply to Suanming Mou

The HW steering uses async queue-based flow rules management mechanism. The matcher and part of the actions have been prepared during flow table creation. Some remaining actions will be constructed during flow creation if needed.

A flow postpone attribute bit describes if flow management should be applied to the HW directly. An extra push function is provided to force push all the cached flows to the HW.

Once the flow has been applied to the HW, the pull function will be called to get the queued creation/destruction flows.

The DR rule flow memory is represented in PMD layer instead of allocating from HW steering layer. While destroying the flow, the flow rule memory can only be freed after the CQE received.

The HW queue job descriptor is currently introduced to convey the flow information and operation type between the flow insertion/destruction in the pull function.

This commit adds the basic flow queue operation for:
rte_flow_async_create(); rte_flow_async_destroy();
rte_flow_push();
rte_flow_pull();

---
 doc/guides/nics/mlx5.rst               |   6 +
 doc/guides/rel_notes/release_22_03.rst |   6 +
 drivers/net/mlx5/mlx5.h                |   2 +-
 drivers/net/mlx5/mlx5_flow.c           | 188 ++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h           |  41 ++++
 drivers/net/mlx5/mlx5_flow_hw.c        | 292 ++++++++++++++++++++++++-
 6 files changed, 533 insertions(+), 2 deletions(-)
doc/guides/nics/mlx5.rst +6 -0
index 0e0169c8bb..7b04e9bac5 100644
@@ -487,6 +487,12 @@ Limitations
     from the reference "Clock Queue" completions,
     the scheduled send timestamps should not be specified with non-zero MSB.
 
+  - HW steering:
+
+    - WQE based high scaling and safer flow insertion/destruction.
+    - Set ``dv_flow_en`` to 2 in order to enable HW steering.
+    - Async queue-based ``rte_flow_q`` APIs supported only.
+
 Statistics
 ----------
 
doc/guides/rel_notes/release_22_03.rst +6 -0
index 512c727e81..375df6ba74 100644
@@ -162,6 +162,12 @@ New Features
 
   * Added LED OEM support.
 
+* **Updated Mellanox mlx5 driver.**
+
+  Updated the Mellanox mlx5 driver with new features and improvements, including:
+
+  * Added WQE based hardware steering support with ``rte_flow_q`` API.
+
 * **Added an API for private user data in asymmetric crypto session.**
 
   An API was added to get/set an asymmetric crypto session's user data.
drivers/net/mlx5/mlx5.h +1 -1
index f523173ad5..d94e98db77 100644
@@ -340,7 +340,7 @@ enum {
 /* HW steering flow management job descriptor. */
 struct mlx5_hw_q_job {
 	uint32_t type; /* Job type. */
-	struct rte_flow *flow; /* Flow attached to the job. */
+	struct rte_flow_hw *flow; /* Flow attached to the job. */
 	void *user_data; /* Job user data. */
 };
 
drivers/net/mlx5/mlx5_flow.c +188 -0

206 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +41 -0

65 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_hw.c +291 -1

316 lines not on this page · raw message

Signature · 1 line
2.25.1
#38 lore raw

PATCH v3 9/14 net/mlx5: add flow flush function

In reply to Suanming Mou

In case port is being stopped, all created flows should be flushed. This commit adds the flow flush helper function.

---
 drivers/net/mlx5/mlx5_flow.c    |   8 ++
 drivers/net/mlx5/mlx5_flow_hw.c | 129 ++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+)
drivers/net/mlx5/mlx5_flow.c +8 -0
index ad131c1b22..1672939200 100644
@@ -7015,6 +7015,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,
 	uint32_t num_flushed = 0, fidx = 1;
 	struct rte_flow *flow;
 
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+	if (priv->sh->config.dv_flow_en == 2 &&
+	    type == MLX5_FLOW_TYPE_GEN) {
+		flow_hw_q_flow_flush(dev, NULL);
+		return;
+	}
+#endif
+
 	MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {
 		flow_list_destroy(dev, type, fidx);
 		num_flushed++;
drivers/net/mlx5/mlx5_flow_hw.c +129 -0

141 lines not on this page · raw message

Signature · 1 line
2.25.1
#39 lore raw

PATCH v3 10/14 net/mlx5: add flow jump action

In reply to Suanming Mou

Jump action connects different level of flow tables and allows packet handling in the chain of flows.

A new action construct data struct is also added in this commit to help to handle not only the dynamic jump action but also for the other generic dynamic actions. The actions with empty mask configuration means dynamic action, and the dedicated action will be created with the flow action configuration during flow creation. In that dynamic action case, the action will be appended to the table template's action list during table creation. When creating the flows, traverse the action list and pick the dynamic action configuration details from flow actions as the action construct data struct describes, then create the dedicated dynamic actions.

This commit adds the jump action and the generic dynamic action construct mechanism.

---
 drivers/net/mlx5/mlx5.h         |   1 +
 drivers/net/mlx5/mlx5_flow.h    |  25 ++-
 drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++---
 3 files changed, 275 insertions(+), 21 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
index d94e98db77..f3732958a2 100644
@@ -1527,6 +1527,7 @@ struct mlx5_priv {
 	/* HW steering global drop action. */
 	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
 				     [MLX5DR_TABLE_TYPE_MAX];
+	struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
 #endif
 };
 
drivers/net/mlx5/mlx5_flow.h +24 -1
index 3add4c4a81..963dbd7806 100644
@@ -1020,10 +1020,25 @@ struct rte_flow {
 /* HWS flow struct. */
 struct rte_flow_hw {
 	uint32_t idx; /* Flow index from indexed pool. */
+	uint32_t fate_type; /* Fate action type. */
+	union {
+		/* Jump action. */
+		struct mlx5_hw_jump_action *jump;
+	};
 	struct rte_flow_template_table *table; /* The table flow allcated from. */
 	struct mlx5dr_rule rule; /* HWS layer data struct. */
 } __rte_packed;
 
+/* rte flow action translate to DR action struct. */
+struct mlx5_action_construct_data {
+	LIST_ENTRY(mlx5_action_construct_data) next;
+	/* Ensure the action types are matched. */
+	int type;
+	uint32_t idx;  /* Data index. */
+	uint16_t action_src; /* rte_flow_action src offset. */
+	uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+};
+
 /* Flow item template struct. */
 struct rte_flow_pattern_template {
 	LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1051,9 +1066,17 @@ struct mlx5_hw_jump_action {
 	struct mlx5dr_action *hws_action;
 };
 
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
 /* DR action set struct. */
 struct mlx5_hw_actions {
-	struct mlx5dr_action *drop; /* Drop action. */
+	/* Dynamic action list. */
+	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
+	struct mlx5_hw_jump_action *jump; /* Jump action. */
+	uint32_t acts_num:4; /* Total action number. */
+	/* Translated DR action array from action template. */
+	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
 };
 
 /* mlx5 action template struct. */
drivers/net/mlx5/mlx5_flow_hw.c +250 -20

399 lines not on this page · raw message

Signature · 1 line
2.25.1
#40 lore raw

PATCH v3 11/14 net/mlx5: add queue and RSS action

In reply to Suanming Mou

This commit adds the queue and RSS action. Similar to the jump action, dynamic ones will be added to the action construct list.

Due to the queue and RSS action in template should not be destroyed during port restart, the actions are created with standalone indirect table as indirect action does. When port stops, detaches the indirect table from action, when port starts, attaches the indirect table back to the action.

One more change is made to accelerate the action creation. Currently the mlx5_hrxq_get() function returns the object index instead of object pointer. This introduced an extra converting the index to the object by calling mlx5_ipool_get() in most of the case. And that extra converting hurts multi-thread performance since mlx5_ipool_get() uses the global lock inside. As the hash Rx queue object itself also contains the index, returns the object directly will achieve better performance without the global lock.

---
 drivers/net/mlx5/linux/mlx5_os.c   |  18 ++--
 drivers/net/mlx5/mlx5.h            |   4 +
 drivers/net/mlx5/mlx5_devx.c       |  10 ++
 drivers/net/mlx5/mlx5_flow.c       |  38 +++----
 drivers/net/mlx5/mlx5_flow.h       |   7 ++
 drivers/net/mlx5/mlx5_flow_dv.c    | 161 ++++++++++++++---------------
 drivers/net/mlx5/mlx5_flow_hw.c    | 101 ++++++++++++++++++
 drivers/net/mlx5/mlx5_flow_verbs.c |   7 +-
 drivers/net/mlx5/mlx5_rx.h         |   9 +-
 drivers/net/mlx5/mlx5_rxq.c        |  85 +++++++++------
 10 files changed, 283 insertions(+), 157 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +9 -9
index 0faf26f5b8..2e1606a733 100644
@@ -1521,6 +1521,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 	priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
 	if (!priv->drop_queue.hrxq)
 		goto error;
+	priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
+				       mlx5_hrxq_create_cb,
+				       mlx5_hrxq_match_cb,
+				       mlx5_hrxq_remove_cb,
+				       mlx5_hrxq_clone_cb,
+				       mlx5_hrxq_clone_free_cb);
+	if (!priv->hrxqs)
+		goto error;
+	rte_rwlock_init(&priv->ind_tbls_lock);
 	if (priv->sh->config.dv_flow_en == 2)
 		return eth_dev;
 	/* Port representor shares the same max priority with pf port. */
@@ -1545,15 +1554,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 			err = ENOTSUP;
 			goto error;
 	}
-	priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
-				       mlx5_hrxq_create_cb,
-				       mlx5_hrxq_match_cb,
-				       mlx5_hrxq_remove_cb,
-				       mlx5_hrxq_clone_cb,
-				       mlx5_hrxq_clone_free_cb);
-	if (!priv->hrxqs)
-		goto error;
-	rte_rwlock_init(&priv->ind_tbls_lock);
 	/* Query availability of metadata reg_c's. */
 	if (!priv->sh->metadata_regc_check_flag) {
 		err = mlx5_flow_discover_mreg_c(eth_dev);
drivers/net/mlx5/mlx5.h +4 -0
index f3732958a2..f60a40d669 100644
@@ -1287,6 +1287,7 @@ struct mlx5_flow_rss_desc {
 	uint64_t hash_fields; /* Verbs Hash fields. */
 	uint8_t key[MLX5_RSS_HASH_KEY_LEN]; /**< RSS hash key. */
 	uint32_t key_len; /**< RSS hash key len. */
+	uint32_t hws_flags; /**< HW steering action. */
 	uint32_t tunnel; /**< Queue in tunnel. */
 	uint32_t shared_rss; /**< Shared RSS index. */
 	struct mlx5_ind_table_obj *ind_tbl;
@@ -1348,6 +1349,7 @@ struct mlx5_hrxq {
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	void *action; /* DV QP action pointer. */
 #endif
+	uint32_t hws_flags; /* Hw steering flags. */
 	uint64_t hash_fields; /* Verbs Hash fields. */
 	uint32_t rss_key_len; /* Hash key length in bytes. */
 	uint32_t idx; /* Hash Rx queue index. */
@@ -1478,6 +1480,8 @@ struct mlx5_priv {
 	LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. */
 	/* Indirection tables. */
 	LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls;
+	/* Standalone indirect tables. */
+	LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;
 	/* Pointer to next element. */
 	rte_rwlock_t ind_tbls_lock;
 	uint32_t refcnt; /**< Reference counter. */
drivers/net/mlx5/mlx5_devx.c +10 -0
index a9b8c2a1b7..8d151fa4ab 100644
@@ -807,6 +807,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
 		goto error;
 	}
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	if (hrxq->hws_flags) {
+		hrxq->action = mlx5dr_action_create_dest_tir
+			(priv->dr_ctx,
+			 (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags);
+		if (!hrxq->action)
+			goto error;
+		return 0;
+	}
 	if (mlx5_flow_os_create_flow_action_dest_devx_tir(hrxq->tir,
 							  &hrxq->action)) {
 		rte_errno = errno;
@@ -1042,6 +1050,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)
 		DRV_LOG(ERR, "Cannot create drop RX queue");
 		return ret;
 	}
+	if (priv->sh->config.dv_flow_en == 2)
+		return 0;
 	/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */
 	ret = mlx5_devx_ind_table_new(dev, 0, hrxq->ind_table);
 	if (ret != 0) {
drivers/net/mlx5/mlx5_flow.c +12 -26
index 1672939200..cbd8408e30 100644
@@ -9439,14 +9439,10 @@ int
 mlx5_action_handle_attach(struct rte_eth_dev *dev)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct mlx5_indexed_pool *ipool =
-			priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
-	struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
 	int ret = 0;
-	uint32_t idx;
+	struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
 
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		const char *message;
 		uint32_t queue_idx;
 
@@ -9462,9 +9458,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
 	}
 	if (ret != 0)
 		return ret;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		ret = mlx5_ind_table_obj_attach(dev, ind_tbl);
 		if (ret != 0) {
 			DRV_LOG(ERR, "Port %u could not attach "
@@ -9473,13 +9467,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
 			goto error;
 		}
 	}
+
 	return 0;
 error:
-	shared_rss_last = shared_rss;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
-		if (shared_rss == shared_rss_last)
+	ind_tbl_last = ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+		if (ind_tbl == ind_tbl_last)
 			break;
 		if (mlx5_ind_table_obj_detach(dev, ind_tbl) != 0)
 			DRV_LOG(CRIT, "Port %u could not detach "
@@ -9502,15 +9495,10 @@ int
 mlx5_action_handle_detach(struct rte_eth_dev *dev)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct mlx5_indexed_pool *ipool =
-			priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
-	struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
 	int ret = 0;
-	uint32_t idx;
-
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+	struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
 
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		ret = mlx5_ind_table_obj_detach(dev, ind_tbl);
 		if (ret != 0) {
 			DRV_LOG(ERR, "Port %u could not detach "
@@ -9521,11 +9509,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev)
 	}
 	return 0;
 error:
-	shared_rss_last = shared_rss;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
-		if (shared_rss == shared_rss_last)
+	ind_tbl_last = ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+		if (ind_tbl == ind_tbl_last)
 			break;
 		if (mlx5_ind_table_obj_attach(dev, ind_tbl) != 0)
 			DRV_LOG(CRIT, "Port %u could not attach "
drivers/net/mlx5/mlx5_flow.h +7 -0
index 963dbd7806..70e6cf633f 100644
@@ -1024,6 +1024,7 @@ struct rte_flow_hw {
 	union {
 		/* Jump action. */
 		struct mlx5_hw_jump_action *jump;
+		struct mlx5_hrxq *hrxq; /* TIR action. */
 	};
 	struct rte_flow_template_table *table; /* The table flow allcated from. */
 	struct mlx5dr_rule rule; /* HWS layer data struct. */
@@ -1074,6 +1075,7 @@ struct mlx5_hw_actions {
 	/* Dynamic action list. */
 	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
+	struct mlx5_hrxq *tir; /* TIR action. */
 	uint32_t acts_num:4; /* Total action number. */
 	/* Translated DR action array from action template. */
 	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
@@ -1910,6 +1912,11 @@ struct mlx5_list_entry *flow_dv_dest_array_clone_cb(void *tool_ctx,
 				   struct mlx5_list_entry *entry, void *cb_ctx);
 void flow_dv_dest_array_clone_free_cb(void *tool_ctx,
 				      struct mlx5_list_entry *entry);
+void flow_dv_hashfields_set(uint64_t item_flags,
+			    struct mlx5_flow_rss_desc *rss_desc,
+			    uint64_t *hash_fields);
+void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+					uint64_t *hash_field);
 
 struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
 void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
drivers/net/mlx5/mlx5_flow_dv.c +76 -85

371 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_hw.c +101 -0

149 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_verbs.c +2 -5
index 90ccb9aaff..f08aa7a770 100644
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			MLX5_ASSERT(priv->drop_queue.hrxq);
 			hrxq = priv->drop_queue.hrxq;
 		} else {
-			uint32_t hrxq_idx;
 			struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;
 
 			MLX5_ASSERT(rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			rss_desc->tunnel = !!(handle->layers &
 					      MLX5_FLOW_LAYER_TUNNEL);
 			rss_desc->shared_rss = 0;
-			hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
-			hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
-					      hrxq_idx);
+			hrxq = mlx5_hrxq_get(dev, rss_desc);
 			if (!hrxq) {
 				rte_flow_error_set
 					(error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 					 "cannot get hash queue");
 				goto error;
 			}
-			handle->rix_hrxq = hrxq_idx;
+			handle->rix_hrxq = hrxq->idx;
 		}
 		MLX5_ASSERT(hrxq);
 		handle->drv_flow = mlx5_glue->create_flow
drivers/net/mlx5/mlx5_rx.h +7 -2
index 38335fd744..295dba063b 100644
@@ -224,9 +224,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);
 struct mlx5_ind_table_obj *mlx5_ind_table_obj_get(struct rte_eth_dev *dev,
 						  const uint16_t *queues,
 						  uint32_t queues_n);
+struct mlx5_ind_table_obj *mlx5_ind_table_obj_new(struct rte_eth_dev *dev,
+						  const uint16_t *queues,
+						  uint32_t queues_n,
+						  bool standalone,
+						  bool ref_qs);
 int mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
 			       struct mlx5_ind_table_obj *ind_tbl,
-			       bool standalone,
 			       bool deref_rxqs);
 int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
 			     struct mlx5_ind_table_obj *ind_tbl,
@@ -249,8 +253,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx,
 					   void *cb_ctx __rte_unused);
 void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,
 			     struct mlx5_list_entry *entry);
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
 		       struct mlx5_flow_rss_desc *rss_desc);
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq);
 int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hxrq_idx);
 uint32_t mlx5_hrxq_verify(struct rte_eth_dev *dev);
 enum mlx5_rxq_type mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t idx);
drivers/net/mlx5/mlx5_rxq.c +55 -30

209 lines not on this page · raw message

Signature · 1 line
2.25.1
#41 lore raw

PATCH v3 12/14 net/mlx5: add mark action

In reply to Suanming Mou

The mark action is covered by tag action internally. While it is added the HW will add a tag to the packet. The mark value can be set as fixed or dynamic as the action mask indicates.

---
 drivers/net/mlx5/mlx5.h         |  2 +
 drivers/net/mlx5/mlx5_flow.h    |  1 +
 drivers/net/mlx5/mlx5_flow_hw.c | 66 ++++++++++++++++++++++++++++++---
 3 files changed, 63 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.h +2 -0
index f60a40d669..a0f9fbe365 100644
@@ -1531,6 +1531,8 @@ struct mlx5_priv {
 	/* HW steering global drop action. */
 	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
 				     [MLX5DR_TABLE_TYPE_MAX];
+	/* HW steering global drop action. */
+	struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX];
 	struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
 #endif
 };
drivers/net/mlx5/mlx5_flow.h +1 -0
index 70e6cf633f..c83e73c793 100644
@@ -1077,6 +1077,7 @@ struct mlx5_hw_actions {
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
 	struct mlx5_hrxq *tir; /* TIR action. */
 	uint32_t acts_num:4; /* Total action number. */
+	uint32_t mark:1; /* Indicate the mark action. */
 	/* Translated DR action array from action template. */
 	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
 };
drivers/net/mlx5/mlx5_flow_hw.c +60 -6

115 lines not on this page · raw message

Signature · 1 line
2.25.1
#42 lore raw

PATCH v3 13/14 net/mlx5: add indirect action

In reply to Suanming Mou

HW steering can support indirect action as well. With indirect action, the flow can be created with more flexible shared RSS action selection. This will can save the action template with different RSS actions.

This commit adds the flow queue operation callback for:
rte_flow_async_action_handle_create(); rte_flow_async_action_handle_destroy(); rte_flow_async_action_handle_update();

---
 drivers/net/mlx5/mlx5_flow.c    | 131 ++++++++++
 drivers/net/mlx5/mlx5_flow.h    |  59 +++++
 drivers/net/mlx5/mlx5_flow_dv.c |  21 +-
 drivers/net/mlx5/mlx5_flow_hw.c | 414 +++++++++++++++++++++++++++++++-
 4 files changed, 612 insertions(+), 13 deletions(-)
drivers/net/mlx5/mlx5_flow.c +131 -0
index cbd8408e30..5a4e000c12 100644
@@ -879,6 +879,29 @@ mlx5_flow_push(struct rte_eth_dev *dev,
 	       uint32_t queue,
 	       struct rte_flow_error *error);
 
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_op_attr *attr,
+				 const struct rte_flow_indir_action_conf *conf,
+				 const struct rte_flow_action *action,
+				 void *user_data,
+				 struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_op_attr *attr,
+				 struct rte_flow_action_handle *handle,
+				 const void *update,
+				 void *user_data,
+				 struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+				  const struct rte_flow_op_attr *attr,
+				  struct rte_flow_action_handle *handle,
+				  void *user_data,
+				  struct rte_flow_error *error);
+
 static const struct rte_flow_ops mlx5_flow_ops = {
 	.validate = mlx5_flow_validate,
 	.create = mlx5_flow_create,
@@ -911,6 +934,9 @@ static const struct rte_flow_ops mlx5_flow_ops = {
 	.async_destroy = mlx5_flow_async_flow_destroy,
 	.pull = mlx5_flow_pull,
 	.push = mlx5_flow_push,
+	.async_action_handle_create = mlx5_flow_async_action_handle_create,
+	.async_action_handle_update = mlx5_flow_async_action_handle_update,
+	.async_action_handle_destroy = mlx5_flow_async_action_handle_destroy,
 };
 
 /* Tunnel information. */
@@ -8367,6 +8393,111 @@ mlx5_flow_push(struct rte_eth_dev *dev,
 	return fops->push(dev, queue, error);
 }
 
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] conf
+ *   Indirect action configuration.
+ * @param[in] action
+ *   rte_flow action detail.
+ * @param[in] user_data
+ *   Pointer to the user_data.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_op_attr *attr,
+				 const struct rte_flow_indir_action_conf *conf,
+				 const struct rte_flow_action *action,
+				 void *user_data,
+				 struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->async_action_create(dev, queue, attr, conf, action,
+					 user_data, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] handle
+ *   Action handle to be updated.
+ * @param[in] update
+ *   Update value.
+ * @param[in] user_data
+ *   Pointer to the user_data.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+				     const struct rte_flow_op_attr *attr,
+				     struct rte_flow_action_handle *handle,
+				     const void *update,
+				     void *user_data,
+				     struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->async_action_update(dev, queue, attr, handle,
+					 update, user_data, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] handle
+ *   Action handle to be destroyed.
+ * @param[in] user_data
+ *   Pointer to the user_data.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+				      const struct rte_flow_op_attr *attr,
+				      struct rte_flow_action_handle *handle,
+				      void *user_data,
+				      struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->async_action_destroy(dev, queue, attr, handle,
+					  user_data, error);
+}
+
 /**
  * Allocate a new memory for the counter values wrapped by all the needed
  * management.
drivers/net/mlx5/mlx5_flow.h +59 -0

99 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_dv.c +12 -9
index d48726cf05..5f85100324 100644
@@ -13845,9 +13845,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,
  * @return
  *   Valid hash RX queue index, otherwise 0.
  */
-static uint32_t
-__flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
-				 const uint64_t hash_fields)
+uint32_t
+flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+			       const uint64_t hash_fields)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_shared_action_rss *shared_rss =
@@ -13975,7 +13975,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			struct mlx5_hrxq *hrxq = NULL;
 			uint32_t hrxq_idx;
 
-			hrxq_idx = __flow_dv_action_rss_hrxq_lookup(dev,
+			hrxq_idx = flow_dv_action_rss_hrxq_lookup(dev,
 						rss_desc->shared_rss,
 						dev_flow->hash_fields);
 			if (hrxq_idx)
@@ -14699,6 +14699,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
 			   struct mlx5_shared_action_rss *shared_rss,
 			   struct rte_flow_error *error)
 {
+	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_flow_rss_desc rss_desc = { 0 };
 	size_t i;
 	int err;
@@ -14719,6 +14720,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
 	/* Set non-zero value to indicate a shared RSS. */
 	rss_desc.shared_rss = action_idx;
 	rss_desc.ind_tbl = shared_rss->ind_tbl;
+	if (priv->sh->config.dv_flow_en == 2)
+		rss_desc.hws_flags = MLX5DR_ACTION_FLAG_HWS_RX;
 	for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
 		struct mlx5_hrxq *hrxq;
 		uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14910,7 +14913,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
  *   A valid shared action handle in case of success, NULL otherwise and
  *   rte_errno is set.
  */
-static struct rte_flow_action_handle *
+struct rte_flow_action_handle *
 flow_dv_action_create(struct rte_eth_dev *dev,
 		      const struct rte_flow_indir_action_conf *conf,
 		      const struct rte_flow_action *action,
@@ -14980,7 +14983,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_destroy(struct rte_eth_dev *dev,
 		       struct rte_flow_action_handle *handle,
 		       struct rte_flow_error *error)
@@ -15190,7 +15193,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_update(struct rte_eth_dev *dev,
 			struct rte_flow_action_handle *handle,
 			const void *update,
@@ -15862,7 +15865,7 @@ flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
 				  "counters are not available");
 }
 
-static int
+int
 flow_dv_action_query(struct rte_eth_dev *dev,
 		     const struct rte_flow_action_handle *handle, void *data,
 		     struct rte_flow_error *error)
@@ -17555,7 +17558,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_validate(struct rte_eth_dev *dev,
 			const struct rte_flow_indir_action_conf *conf,
 			const struct rte_flow_action *action,
drivers/net/mlx5/mlx5_flow_hw.c +410 -4

493 lines not on this page · raw message

Signature · 1 line
2.25.1
#43 lore raw

PATCH v3 14/14 net/mlx5: add header reformat action

In reply to Suanming Mou

HW steering header reformat action can work under bulk mode. In this case, when create the table, bulk size of header reformat actions will be allocated in low level. Afterwards, when create flow, just simply specify the action index in the bulk and the encapsulation data to the action will be enough.

---
 drivers/net/mlx5/mlx5.h         |   1 +
 drivers/net/mlx5/mlx5_flow.h    |  21 +++
 drivers/net/mlx5/mlx5_flow_dv.c |   4 +-
 drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++-
 4 files changed, 251 insertions(+), 3 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
index a0f9fbe365..bd69aa2334 100644
@@ -342,6 +342,7 @@ struct mlx5_hw_q_job {
 	uint32_t type; /* Job type. */
 	struct rte_flow_hw *flow; /* Flow attached to the job. */
 	void *user_data; /* Job user data. */
+	uint8_t *encap_data; /* Encap data. */
 };
 
 /* HW steering job descriptor LIFO pool. */
drivers/net/mlx5/mlx5_flow.h +21 -0
index 4c224bbf52..09f0d7a75d 100644
@@ -1040,6 +1040,14 @@ struct mlx5_action_construct_data {
 	uint16_t action_src; /* rte_flow_action src offset. */
 	uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
 	union {
+		struct {
+			/* encap src(item) offset. */
+			uint16_t src;
+			/* encap dst data offset. */
+			uint16_t dst;
+			/* encap data len. */
+			uint16_t len;
+		} encap;
 		struct {
 			uint64_t types; /* RSS hash types. */
 			uint32_t level; /* RSS level. */
@@ -1076,6 +1084,13 @@ struct mlx5_hw_jump_action {
 	struct mlx5dr_action *hws_action;
 };
 
+/* Encap decap action struct. */
+struct mlx5_hw_encap_decap_action {
+	struct mlx5dr_action *action; /* Action object. */
+	size_t data_size; /* Action metadata size. */
+	uint8_t data[]; /* Action data. */
+};
+
 /* The maximum actions support in the flow. */
 #define MLX5_HW_MAX_ACTS 16
 
@@ -1085,6 +1100,9 @@ struct mlx5_hw_actions {
 	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
 	struct mlx5_hrxq *tir; /* TIR action. */
+	/* Encap/Decap action. */
+	struct mlx5_hw_encap_decap_action *encap_decap;
+	uint16_t encap_decap_pos; /* Encap/Decap action position. */
 	uint32_t acts_num:4; /* Total action number. */
 	uint32_t mark:1; /* Indicate the mark action. */
 	/* Translated DR action array from action template. */
@@ -2027,4 +2045,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,
 			 const struct rte_flow_action_handle *handle,
 			 void *data,
 			 struct rte_flow_error *error);
+size_t flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type);
+int flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
+			   size_t *size, struct rte_flow_error *error);
 #endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_dv.c +2 -2
index 5f85100324..7a012f7bb9 100644
@@ -4034,7 +4034,7 @@ flow_dv_push_vlan_action_resource_register
  * @return
  *   sizeof struct item_type, 0 if void or irrelevant.
  */
-static size_t
+size_t
 flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
 {
 	size_t retval;
@@ -4100,7 +4100,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
  * @return
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
-static int
+int
 flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
 			   size_t *size, struct rte_flow_error *error)
 {
drivers/net/mlx5/mlx5_flow_hw.c +227 -1

299 lines not on this page · raw message

Signature · 1 line
2.25.1
#44 lore raw

v4 cover net/mlx5: add hardware steering

In reply to Suanming Mou

The Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:

- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly

However, there are still some disadvantages:

- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects

- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion

- security, exposing the low-level steering entries directly to the userspace may cause security risks

A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.

In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.

---

v4:
v3:
- rebase to the latest version.

v2:
- New HW steering low-level abstract code added.
- commit message improvement.
- add protection for rte_flow and rte_flow_async callbacks.
- rebase to rte_flow_async v9.
- fix some rte_flow error not filled bugs.

Suanming Mou (14):
net/mlx5: introduce hardware steering operation net/mlx5: add HW steering low-level abstract code net/mlx5: introduce hardware steering enable routine net/mlx5: add port flow configuration net/mlx5: add pattern template management net/mlx5: add action template management net/mlx5: add table management
net/mlx5: add basic flow queue operation net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action

 doc/guides/nics/mlx5.rst                |   19 +-
 doc/guides/rel_notes/release_22_03.rst  |    1 +
 drivers/net/mlx5/linux/mlx5_flow_os.h   |    1 +
 drivers/net/mlx5/linux/mlx5_os.c        |   22 +-
 drivers/net/mlx5/meson.build            |    2 +
 drivers/net/mlx5/mlx5.c                 |   55 +-
 drivers/net/mlx5/mlx5.h                 |   66 +-
 drivers/net/mlx5/mlx5_devx.c            |   10 +
 drivers/net/mlx5/mlx5_dr.c              |  383 ++++
 drivers/net/mlx5/mlx5_dr.h              |  456 +++++
 drivers/net/mlx5/mlx5_flow.c            |  721 ++++++-
 drivers/net/mlx5/mlx5_flow.h            |  287 +++
 drivers/net/mlx5/mlx5_flow_dv.c         |  186 +-
 drivers/net/mlx5/mlx5_flow_hw.c         | 2337 +++++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow_verbs.c      |    7 +-
 drivers/net/mlx5/mlx5_rx.h              |    9 +-
 drivers/net/mlx5/mlx5_rxq.c             |   85 +-
 drivers/net/mlx5/windows/mlx5_flow_os.h |    1 +
 18 files changed, 4468 insertions(+), 180 deletions(-)
 create mode 100644 drivers/net/mlx5/mlx5_dr.c
 create mode 100644 drivers/net/mlx5/mlx5_dr.h
 create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
Signature · 1 line
2.25.1
#45 lore raw

PATCH v4 1/14 net/mlx5: introduce hardware steering operation

The Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:

- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly

However, there are still some disadvantages:

- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects

- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion

- security, exposing the low-level steering entries directly to the userspace may cause security risks

A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.

In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.

This commit adds the basic driver operation.

---
 drivers/net/mlx5/linux/mlx5_flow_os.h   |  1 +
 drivers/net/mlx5/meson.build            |  1 +
 drivers/net/mlx5/mlx5_flow.c            |  1 +
 drivers/net/mlx5/mlx5_flow.h            |  1 +
 drivers/net/mlx5/mlx5_flow_hw.c         | 13 +++++++++++++
 drivers/net/mlx5/windows/mlx5_flow_os.h |  1 +
 6 files changed, 18 insertions(+)
 create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
drivers/net/mlx5/linux/mlx5_flow_os.h +1 -0
index 1926d26410..e28a9e0436 100644
@@ -9,6 +9,7 @@
 
 #ifdef HAVE_IBV_FLOW_DV_SUPPORT
 extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
 #endif
 
 /**
drivers/net/mlx5/meson.build +1 -0
index 2f6d8cbb3d..39a2b8c523 100644
@@ -16,6 +16,7 @@ sources = files(
         'mlx5_flow.c',
         'mlx5_flow_meter.c',
         'mlx5_flow_dv.c',
+        'mlx5_flow_hw.c',
         'mlx5_flow_aso.c',
         'mlx5_flow_flex.c',
         'mlx5_mac.c',
drivers/net/mlx5/mlx5_flow.c +1 -0
index 96f3402418..aec14ea39d 100644
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = {
 	[MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	[MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops,
+	[MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops,
 #endif
 	[MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops,
 	[MLX5_FLOW_TYPE_MAX] = &mlx5_flow_null_drv_ops
drivers/net/mlx5/mlx5_flow.h +1 -0
index a20773eeb2..b70ef0c1b8 100644
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {
 	MLX5_FLOW_TYPE_MIN,
 	MLX5_FLOW_TYPE_DV,
 	MLX5_FLOW_TYPE_VERBS,
+	MLX5_FLOW_TYPE_HW,
 	MLX5_FLOW_TYPE_MAX,
 };
 
drivers/net/mlx5/mlx5_flow_hw.c new file +13 -0
new file mode 100644
index 0000000000..729d5914a8
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2022 NVIDIA Corporation & Affiliates
+ */
+
+#include <rte_flow.h>
+
+#include "mlx5_flow.h"
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
+
+#endif
drivers/net/mlx5/windows/mlx5_flow_os.h +1 -0
index dfcb012334..52013b06a0 100644
@@ -10,6 +10,7 @@
 
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops;
+extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;
 #endif
 
 /**
Signature · 1 line
2.25.1
#46 lore raw

PATCH v4 2/14 net/mlx5: add HW steering low-level abstract code

In reply to Suanming Mou

The HW steering low-level implementation will be added later in another patch series. To avoid the linkage issues the abstract stub replacement is provided currently.

---
 drivers/net/mlx5/meson.build |   1 +
 drivers/net/mlx5/mlx5_dr.c   | 383 +++++++++++++++++++++++++++++
 drivers/net/mlx5/mlx5_dr.h   | 456 +++++++++++++++++++++++++++++++++++
 3 files changed, 840 insertions(+)
 create mode 100644 drivers/net/mlx5/mlx5_dr.c
 create mode 100644 drivers/net/mlx5/mlx5_dr.h
drivers/net/mlx5/meson.build +1 -0
index 39a2b8c523..393b2c97ac 100644
@@ -14,6 +14,7 @@ sources = files(
         'mlx5.c',
         'mlx5_ethdev.c',
         'mlx5_flow.c',
+	'mlx5_dr.c',
         'mlx5_flow_meter.c',
         'mlx5_flow_dv.c',
         'mlx5_flow_hw.c',
drivers/net/mlx5/mlx5_dr.c new file +383 -0

383 lines not on this page · raw message

drivers/net/mlx5/mlx5_dr.h new file +456 -0

456 lines not on this page · raw message

Signature · 1 line
2.25.1
#47 lore raw

Re: [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code

Quoted 19 lines · Suanming Mou

The HW steering low-level implementation will be added later in another patch series. To avoid the linkage issues the abstract stub replacement is provided currently.

---
  drivers/net/mlx5/meson.build |   1 +
  drivers/net/mlx5/mlx5_dr.c   | 383 +++++++++++++++++++++++++++++
  drivers/net/mlx5/mlx5_dr.h   | 456 +++++++++++++++++++++++++++++++++++
  3 files changed, 840 insertions(+)

create mode 100644 drivers/net/mlx5/mlx5_dr.c
create mode 100644 drivers/net/mlx5/mlx5_dr.h

drivers/net/mlx5/meson.build +0 -0
index 39a2b8c523..393b2c97ac 100644

drivers/net/mlx5/meson.build sources = files( · line 20

Quoted 7 lines
drivers/net/mlx5/meson.build +1 -0
@@ -14,6 +14,7 @@ sources = files(
          'mlx5.c',
          'mlx5_ethdev.c',
          'mlx5_flow.c',
+	'mlx5_dr.c',
          'mlx5_flow_meter.c',
          'mlx5_flow_dv.c',
          'mlx5_flow_hw.c',

gives syntax warning [1], will fix while merging.

[1]
./devtools/check-meson.py
Error parsing drivers/net/mlx5/meson.build:16, got some tabulation
Error: Incorrect indent at drivers/net/mlx5/meson.build:17

#48 lore raw

RE: [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code

Hi,

Quoted 11 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, February 25, 2022 6:58 AM
To: Suanming Mou <[email protected]>; Slava Ovsiienko <[email protected]>; Matan Azrad <[email protected]>
Cc: Raslan Darawsheh <[email protected]>; Ori Kam <[email protected]>; [email protected]
Subject: Re: [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code

On 2/24/2022 1:40 PM, Suanming Mou wrote:

[snip]

Quoted 10 lines

drivers/net/mlx5/meson.build sources = files( · line 20

Quoted 8 lines
drivers/net/mlx5/meson.build +1 -0
@@ -14,6 +14,7 @@ sources = files(
          'mlx5.c',
          'mlx5_ethdev.c',
          'mlx5_flow.c',
+	'mlx5_dr.c',
          'mlx5_flow_meter.c',
          'mlx5_flow_dv.c',
          'mlx5_flow_hw.c',

gives syntax warning [1], will fix while merging.

Just checked, yes, seems vim auto indent placed tab instead of space here. Thank you for the fixing.

Quoted 6 lines

[1]
./devtools/check-meson.py
Error parsing drivers/net/mlx5/meson.build:16, got some tabulation
Error: Incorrect indent at drivers/net/mlx5/meson.build:17

#49 lore raw

PATCH v4 3/14 net/mlx5: introduce hardware steering enable routine

In reply to Suanming Mou

The new hardware steering engine relies on using dedicated steering WQEs instead of direct writing to the low-level steering table entries directly. In the first introduce implementation the hardware steering engine supports the new queue based Flow API, the existing synchronous non-queue based Flow API is not supported.

A new dv_flow_en value 2 is added to manage mlx5 PMD steering engine:

dv_flow_en rte_flow API rte_flow_async API
------------------------------------------------
0 support not support
1 support not support
2 not support support

This commit introduces the extra dv_flow_en = 2 to specify the new flow initialize and manage operation routine.

---
 doc/guides/nics/mlx5.rst         | 13 ++++++++++---
 drivers/net/mlx5/linux/mlx5_os.c |  4 ++++
 drivers/net/mlx5/mlx5.c          |  7 ++++++-
 drivers/net/mlx5/mlx5.h          |  3 ++-
 drivers/net/mlx5/mlx5_flow.c     | 22 ++++++++++++++++++++++
 5 files changed, 44 insertions(+), 5 deletions(-)
doc/guides/nics/mlx5.rst +10 -3
index 34be031360..92127675a3 100644
@@ -943,10 +943,17 @@ for an additional list of options shared with other mlx5 drivers.
 
 - ``dv_flow_en`` parameter [int]
 
-  A nonzero value enables the DV flow steering assuming it is supported
-  by the driver (RDMA Core library version is rdma-core-24.0 or higher).
+  Value 0 means legacy Verbs flow offloading.
 
-  Enabled by default if supported.
+  Value 1 enables the DV flow steering assuming it is supported by the
+  driver (RDMA Core library version is rdma-core-24.0 or higher).
+
+  Value 2 enables the WQE based hardware steering. In this mode only
+  the queue-based rte_flow_q flow management is supported.
+
+  Configured by default to 1 DV flow steering if the driver(RDMA CORE library)
+  supported. Otherwise, the value will be 0 which indicates legacy Verbs flow
+  offloading.
 
 - ``dv_esw_en`` parameter [int]
 
drivers/net/mlx5/linux/mlx5_os.c +4 -0
index 058c140fe1..5b91e057b2 100644
@@ -482,6 +482,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
 	err = mlx5_alloc_table_hash_list(priv);
 	if (err)
 		goto error;
+	if (priv->sh->config.dv_flow_en == 2)
+		return 0;
 	/* The resources below are only valid with DV support. */
 #ifdef HAVE_IBV_FLOW_DV_SUPPORT
 	/* Init port id action list. */
@@ -1535,6 +1537,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 	priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
 	if (!priv->drop_queue.hrxq)
 		goto error;
+	if (priv->sh->config.dv_flow_en == 2)
+		return eth_dev;
 	/* Port representor shares the same max priority with pf port. */
 	if (!priv->sh->flow_priority_check_flag) {
 		/* Supported Verbs flow priority number detection. */
drivers/net/mlx5/mlx5.c +6 -1
index 9760f52b46..baa4ae75f3 100644
@@ -1199,7 +1199,12 @@ mlx5_dev_args_check_handler(const char *key, const char *val, void *opaque)
 	} else if (strcmp(MLX5_DV_ESW_EN, key) == 0) {
 		config->dv_esw_en = !!tmp;
 	} else if (strcmp(MLX5_DV_FLOW_EN, key) == 0) {
-		config->dv_flow_en = !!tmp;
+		if (tmp > 2) {
+			DRV_LOG(ERR, "Invalid %s parameter.", key);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		config->dv_flow_en = tmp;
 	} else if (strcmp(MLX5_DV_XMETA_EN, key) == 0) {
 		if (tmp != MLX5_XMETA_MODE_LEGACY &&
 		    tmp != MLX5_XMETA_MODE_META16 &&
drivers/net/mlx5/mlx5.h +2 -1
index fa27f65a36..d36d679625 100644
@@ -287,7 +287,8 @@ struct mlx5_sh_config {
 	int tx_skew; /* Tx scheduling skew between WQE and data on wire. */
 	uint32_t reclaim_mode:2; /* Memory reclaim mode. */
 	uint32_t dv_esw_en:1; /* Enable E-Switch DV flow. */
-	uint32_t dv_flow_en:1; /* Enable DV flow. */
+	/* Enable DV flow. 1 means SW steering, 2 means HW steering. */
+	unsigned int dv_flow_en:2;
 	uint32_t dv_xmeta_en:2; /* Enable extensive flow metadata. */
 	uint32_t dv_miss_info:1; /* Restore packet after partial hw miss. */
 	uint32_t l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
drivers/net/mlx5/mlx5_flow.c +22 -0
index aec14ea39d..ef7f2ceaee 100644
@@ -6849,6 +6849,15 @@ mlx5_flow_create(struct rte_eth_dev *dev,
 		 const struct rte_flow_action actions[],
 		 struct rte_flow_error *error)
 {
+	struct mlx5_priv *priv = dev->data->dev_private;
+
+	if (priv->sh->config.dv_flow_en == 2) {
+		rte_flow_error_set(error, ENOTSUP,
+			  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+			  NULL,
+			  "Flow non-Q creation not supported");
+		return NULL;
+	}
 	/*
 	 * If the device is not started yet, it is not allowed to created a
 	 * flow from application. PMD default flows and traffic control flows
@@ -7345,6 +7354,13 @@ mlx5_flow_destroy(struct rte_eth_dev *dev,
 		  struct rte_flow *flow,
 		  struct rte_flow_error *error __rte_unused)
 {
+	struct mlx5_priv *priv = dev->data->dev_private;
+
+	if (priv->sh->config.dv_flow_en == 2)
+		return rte_flow_error_set(error, ENOTSUP,
+			  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+			  NULL,
+			  "Flow non-Q destruction not supported");
 	flow_list_destroy(dev, MLX5_FLOW_TYPE_GEN,
 				(uintptr_t)(void *)flow);
 	return 0;
@@ -7442,7 +7458,13 @@ mlx5_flow_query(struct rte_eth_dev *dev,
 		struct rte_flow_error *error)
 {
 	int ret;
+	struct mlx5_priv *priv = dev->data->dev_private;
 
+	if (priv->sh->config.dv_flow_en == 2)
+		return rte_flow_error_set(error, ENOTSUP,
+			  RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
+			  NULL,
+			  "Flow non-Q query not supported");
 	ret = flow_drv_query(dev, (uintptr_t)(void *)flow, actions, data,
 			     error);
 	if (ret < 0)
Signature · 1 line
2.25.1
#50 lore raw

PATCH v4 4/14 net/mlx5: add port flow configuration

In reply to Suanming Mou

The hardware steering is backend to support rte_flow_async API in mlx5 PMD. The port configuration function creates the queues and needed flow management resources.

The PMD layer configuration function allocates the queues' context and per-queue job descriptor pool. The job descriptor pool size is equal to the queue size, and the job descriptors will be popped from pool with LIFO strategy to convey the flow information during flow insertion/destruction. Then, while polling the queued operation result, the flow information will be extracted from the job descriptor and the descriptor will be pushed back to the LIFO pool.

The commit creates the flow port queues and the job descriptor pools.

---
 drivers/net/mlx5/mlx5.c         |   3 +
 drivers/net/mlx5/mlx5.h         |  30 +++++-
 drivers/net/mlx5/mlx5_flow.c    |  86 +++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    |  15 +++
 drivers/net/mlx5/mlx5_flow_hw.c | 159 ++++++++++++++++++++++++++++++++
 5 files changed, 292 insertions(+), 1 deletion(-)
drivers/net/mlx5/mlx5.c +3 -0
index baa4ae75f3..24bbfee519 100644
@@ -1820,6 +1820,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)
 	/* Free the eCPRI flex parser resource. */
 	mlx5_flex_parser_ecpri_release(dev);
 	mlx5_flex_item_port_cleanup(dev);
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	flow_hw_resource_release(dev);
+#endif
 	if (priv->rxq_privs != NULL) {
 		/* XXX race condition if mlx5_rx_burst() is still running. */
 		rte_delay_us_sleep(1000);
drivers/net/mlx5/mlx5.h +29 -1
index d36d679625..86c84d1510 100644
@@ -33,7 +33,9 @@
 #include "mlx5_utils.h"
 #include "mlx5_os.h"
 #include "mlx5_autoconf.h"
-
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+#include "mlx5_dr.h"
+#endif
 
 #define MLX5_SH(dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
 
@@ -320,6 +322,26 @@ struct mlx5_lb_ctx {
 	uint16_t refcnt; /* Reference count for representors. */
 };
 
+/* HW steering queue job descriptor type. */
+enum {
+	MLX5_HW_Q_JOB_TYPE_CREATE, /* Flow create job type. */
+	MLX5_HW_Q_JOB_TYPE_DESTROY, /* Flow destroy job type. */
+};
+
+/* HW steering flow management job descriptor. */
+struct mlx5_hw_q_job {
+	uint32_t type; /* Job type. */
+	struct rte_flow *flow; /* Flow attached to the job. */
+	void *user_data; /* Job user data. */
+};
+
+/* HW steering job descriptor LIFO pool. */
+struct mlx5_hw_q {
+	uint32_t job_idx; /* Free job index. */
+	uint32_t size; /* LIFO size. */
+	struct mlx5_hw_q_job **job; /* LIFO header. */
+} __rte_cache_aligned;
+
 #define MLX5_COUNTERS_PER_POOL 512
 #define MLX5_MAX_PENDING_QUERIES 4
 #define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1480,6 +1502,12 @@ struct mlx5_priv {
 	struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM];
 	/* Flex items have been created on the port. */
 	uint32_t flex_item_map; /* Map of allocated flex item elements. */
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
+	uint32_t nb_queue; /* HW steering queue number. */
+	/* HW steering queue polling mechanism job descriptor LIFO. */
+	struct mlx5_hw_q *hw_q;
+#endif
 };
 
 #define PORT_ID(priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.c +86 -0

110 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +15 -0
index b70ef0c1b8..9f0dc4bde7 100644
@@ -1257,6 +1257,17 @@ typedef int (*mlx5_flow_item_update_t)
 			 const struct rte_flow_item_flex_handle *handle,
 			 const struct rte_flow_item_flex_conf *conf,
 			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_info_get_t)
+			(struct rte_eth_dev *dev,
+			 struct rte_flow_port_info *port_info,
+			 struct rte_flow_queue_info *queue_info,
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_port_configure_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_port_attr *port_attr,
+			 uint16_t nb_queue,
+			 const struct rte_flow_queue_attr *queue_attr[],
+			 struct rte_flow_error *err);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1295,6 +1306,8 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_item_create_t item_create;
 	mlx5_flow_item_release_t item_release;
 	mlx5_flow_item_update_t item_update;
+	mlx5_flow_info_get_t info_get;
+	mlx5_flow_port_configure_t configure;
 };
 
 /* mlx5_flow.c */
@@ -1762,4 +1775,6 @@ const struct mlx5_flow_tunnel *
 mlx5_get_tof(const struct rte_flow_item *items,
 	     const struct rte_flow_action *actions,
 	     enum mlx5_tof_rule_type *rule_type);
+void
+flow_hw_resource_release(struct rte_eth_dev *dev);
 #endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_hw.c +159 -0

169 lines not on this page · raw message

Signature · 1 line
2.25.1
#51 lore raw

PATCH v4 5/14 net/mlx5: add pattern template management

In reply to Suanming Mou

The pattern template defines flows that have the same matching fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be (eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while the values for each rule will be different.

Due to the pattern template can be used in different domains, the items will only be cached in pattern template create stage, while the template is binded to a dedicated table, the HW criteria will be created and saved to the table. The pattern templates can be used by multiple tables. But different tables create the same criteria and will not share the matcher between each other in order to have better performance.

This commit adds pattern template management.

---
 drivers/net/mlx5/mlx5.h         |  2 +
 drivers/net/mlx5/mlx5_flow.c    | 76 +++++++++++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    | 24 +++++++++
 drivers/net/mlx5/mlx5_flow_hw.c | 86 +++++++++++++++++++++++++++++++++
 4 files changed, 188 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
index 86c84d1510..7b458c410d 100644
@@ -1503,6 +1503,8 @@ struct mlx5_priv {
 	/* Flex items have been created on the port. */
 	uint32_t flex_item_map; /* Map of allocated flex item elements. */
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	/* Item template list. */
+	LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
 	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +76 -0

94 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +24 -0
index 9f0dc4bde7..49027d6a3a 100644
@@ -1015,6 +1015,19 @@ struct rte_flow {
 	uint32_t geneve_tlv_option; /**< Holds Geneve TLV option id. > */
 } __rte_packed;
 
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+
+/* Flow item template struct. */
+struct rte_flow_pattern_template {
+	LIST_ENTRY(rte_flow_pattern_template) next;
+	/* Template attributes. */
+	struct rte_flow_pattern_template_attr attr;
+	struct mlx5dr_match_template *mt; /* mlx5 match template. */
+	uint32_t refcnt;  /* Reference counter. */
+};
+
+#endif
+
 /*
  * Define list of valid combinations of RX Hash fields
  * (see enum ibv_rx_hash_fields).
@@ -1268,6 +1281,15 @@ typedef int (*mlx5_flow_port_configure_t)
 			 uint16_t nb_queue,
 			 const struct rte_flow_queue_attr *queue_attr[],
 			 struct rte_flow_error *err);
+typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_pattern_template_attr *attr,
+			 const struct rte_flow_item items[],
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_pattern_template_destroy_t)
+			(struct rte_eth_dev *dev,
+			 struct rte_flow_pattern_template *template,
+			 struct rte_flow_error *error);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1308,6 +1330,8 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_item_update_t item_update;
 	mlx5_flow_info_get_t info_get;
 	mlx5_flow_port_configure_t configure;
+	mlx5_flow_pattern_template_create_t pattern_template_create;
+	mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
 };
 
 /* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +86 -0

107 lines not on this page · raw message

Signature · 1 line
2.25.1
#52 lore raw

PATCH v4 6/14 net/mlx5: add action template management

In reply to Suanming Mou

The action template holds a list of action types that will be used together on the same rule. The template's actions instances will be created only when the template bind to the dedicated group. And the created actions will be saved to each individual group in order for best performance. The actions in a group will not be shared with each other unless shared actions are specified.

This commit adds the action template management which stores the flow action template.

---
 drivers/net/mlx5/mlx5.h         |   2 +
 drivers/net/mlx5/mlx5_flow.c    |  78 +++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h    |  22 ++++++
 drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++
 4 files changed, 218 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
index 7b458c410d..1722d38aad 100644
@@ -1505,6 +1505,8 @@ struct mlx5_priv {
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	/* Item template list. */
 	LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;
+	/* Action template list. */
+	LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;
 	struct mlx5dr_context *dr_ctx; /**< HW steering DR context. */
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +78 -0

96 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +22 -0
index 49027d6a3a..9a643fe0a8 100644
@@ -1026,6 +1026,16 @@ struct rte_flow_pattern_template {
 	uint32_t refcnt;  /* Reference counter. */
 };
 
+/* Flow action template struct. */
+struct rte_flow_actions_template {
+	LIST_ENTRY(rte_flow_actions_template) next;
+	/* Template attributes. */
+	struct rte_flow_actions_template_attr attr;
+	struct rte_flow_action *actions; /* Cached flow actions. */
+	struct rte_flow_action *masks; /* Cached action masks.*/
+	uint32_t refcnt; /* Reference counter. */
+};
+
 #endif
 
 /*
@@ -1290,6 +1300,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t)
 			(struct rte_eth_dev *dev,
 			 struct rte_flow_pattern_template *template,
 			 struct rte_flow_error *error);
+typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t)
+			(struct rte_eth_dev *dev,
+			 const struct rte_flow_actions_template_attr *attr,
+			 const struct rte_flow_action actions[],
+			 const struct rte_flow_action masks[],
+			 struct rte_flow_error *error);
+typedef int (*mlx5_flow_actions_template_destroy_t)
+			(struct rte_eth_dev *dev,
+			 struct rte_flow_actions_template *template,
+			 struct rte_flow_error *error);
 
 struct mlx5_flow_driver_ops {
 	mlx5_flow_validate_t validate;
@@ -1332,6 +1352,8 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_port_configure_t configure;
 	mlx5_flow_pattern_template_create_t pattern_template_create;
 	mlx5_flow_pattern_template_destroy_t pattern_template_destroy;
+	mlx5_flow_actions_template_create_t actions_template_create;
+	mlx5_flow_actions_template_destroy_t actions_template_destroy;
 };
 
 /* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +116 -0

140 lines not on this page · raw message

Signature · 1 line
2.25.1
#53 lore raw

PATCH v4 7/14 net/mlx5: add table management

In reply to Suanming Mou

Flow table is a group of flows with the same matching criteria and the same actions defined for them. The table defines rules that have the same matching fields but with different matching values. For example, matching on 5 tuple, the table will be (IPv4 source + IPv4 dest + s_port + d_port + next_proto) while the values for each rule will be different.

The templates' relevant matching criteria and action instances will be created in the table creation and saved in the table. As table attributes indicate the supported flow number, the flow memory will also be allocated at the same time.

This commit adds the table management functions.

---
 drivers/net/mlx5/mlx5.c         |  45 ++-
 drivers/net/mlx5/mlx5.h         |  21 +-
 drivers/net/mlx5/mlx5_flow.c    |  93 ++++++
 drivers/net/mlx5/mlx5_flow.h    |  73 +++++
 drivers/net/mlx5/mlx5_flow_hw.c | 527 ++++++++++++++++++++++++++++++++
 5 files changed, 753 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.c +41 -4
index 24bbfee519..44e3e76737 100644
@@ -1607,12 +1607,46 @@ void
 mlx5_free_table_hash_list(struct mlx5_priv *priv)
 {
 	struct mlx5_dev_ctx_shared *sh = priv->sh;
-
-	if (!sh->flow_tbls)
+	struct mlx5_hlist **tbls = (priv->sh->config.dv_flow_en == 2) ?
+				   &sh->groups : &sh->flow_tbls;
+	if (*tbls == NULL)
 		return;
-	mlx5_hlist_destroy(sh->flow_tbls);
-	sh->flow_tbls = NULL;
+	mlx5_hlist_destroy(*tbls);
+	*tbls = NULL;
+}
+
+#if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+/**
+ * Allocate HW steering group hash list.
+ *
+ * @param[in] priv
+ *   Pointer to the private device data structure.
+ */
+static int
+mlx5_alloc_hw_group_hash_list(struct mlx5_priv *priv)
+{
+	int err = 0;
+	struct mlx5_dev_ctx_shared *sh = priv->sh;
+	char s[MLX5_NAME_SIZE];
+
+	MLX5_ASSERT(sh);
+	snprintf(s, sizeof(s), "%s_flow_groups", priv->sh->ibdev_name);
+	sh->groups = mlx5_hlist_create
+			(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
+			 false, true, sh,
+			 flow_hw_grp_create_cb,
+			 flow_hw_grp_match_cb,
+			 flow_hw_grp_remove_cb,
+			 flow_hw_grp_clone_cb,
+			 flow_hw_grp_clone_free_cb);
+	if (!sh->groups) {
+		DRV_LOG(ERR, "flow groups with hash creation failed.");
+		err = ENOMEM;
+	}
+	return err;
 }
+#endif
+
 
 /**
  * Initialize flow table hash list and create the root tables entry
@@ -1628,11 +1662,14 @@ int
 mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused)
 {
 	int err = 0;
+
 	/* Tables are only used in DV and DR modes. */
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	struct mlx5_dev_ctx_shared *sh = priv->sh;
 	char s[MLX5_NAME_SIZE];
 
+	if (priv->sh->config.dv_flow_en == 2)
+		return mlx5_alloc_hw_group_hash_list(priv);
 	MLX5_ASSERT(sh);
 	snprintf(s, sizeof(s), "%s_flow_table", priv->sh->ibdev_name);
 	sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
drivers/net/mlx5/mlx5.h +19 -2
index 1722d38aad..4eedc8859f 100644
@@ -64,7 +64,9 @@ enum mlx5_ipool_index {
 	MLX5_IPOOL_PUSH_VLAN, /* Pool for push vlan resource. */
 	MLX5_IPOOL_TAG, /* Pool for tag resource. */
 	MLX5_IPOOL_PORT_ID, /* Pool for port id resource. */
-	MLX5_IPOOL_JUMP, /* Pool for jump resource. */
+	MLX5_IPOOL_JUMP, /* Pool for SWS jump resource. */
+	/* Pool for HWS group. Jump action will be created internally. */
+	MLX5_IPOOL_HW_GRP = MLX5_IPOOL_JUMP,
 	MLX5_IPOOL_SAMPLE, /* Pool for sample resource. */
 	MLX5_IPOOL_DEST_ARRAY, /* Pool for destination array resource. */
 	MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -108,6 +110,13 @@ enum mlx5_delay_drop_mode {
 	MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */
 };
 
+/* The HWS action type root/non-root. */
+enum mlx5_hw_action_flag_type {
+	MLX5_HW_ACTION_FLAG_ROOT, /* Root action. */
+	MLX5_HW_ACTION_FLAG_NONE_ROOT, /* Non-root ation. */
+	MLX5_HW_ACTION_FLAG_MAX, /* Maximum action flag. */
+};
+
 /* Hlist and list callback context. */
 struct mlx5_flow_cb_ctx {
 	struct rte_eth_dev *dev;
@@ -1204,7 +1213,10 @@ struct mlx5_dev_ctx_shared {
 	rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX];
 	/* UAR same-page access control required in 32bit implementations. */
 #endif
-	struct mlx5_hlist *flow_tbls;
+	union {
+		struct mlx5_hlist *flow_tbls; /* SWS flow table. */
+		struct mlx5_hlist *groups; /* HWS flow group. */
+	};
 	struct mlx5_flow_tunnel_hub *tunnel_hub;
 	/* Direct Rules tables for FDB, NIC TX+RX */
 	void *dr_drop_action; /* Pointer to DR drop action, any domain. */
@@ -1511,6 +1523,11 @@ struct mlx5_priv {
 	uint32_t nb_queue; /* HW steering queue number. */
 	/* HW steering queue polling mechanism job descriptor LIFO. */
 	struct mlx5_hw_q *hw_q;
+	/* HW steering rte flow table list header. */
+	LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl;
+	/* HW steering global drop action. */
+	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
+				     [MLX5DR_TABLE_TYPE_MAX];
 #endif
 };
 
drivers/net/mlx5/mlx5_flow.c +93 -0

111 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +73 -0

97 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_hw.c +527 -0

576 lines not on this page · raw message

Signature · 1 line
2.25.1
#54 lore raw

PATCH v4 8/14 net/mlx5: add basic flow queue operation

In reply to Suanming Mou

The HW steering uses async queue-based flow rules management mechanism. The matcher and part of the actions have been prepared during flow table creation. Some remaining actions will be constructed during flow creation if needed.

A flow postpone attribute bit describes if flow management should be applied to the HW directly. An extra push function is provided to force push all the cached flows to the HW.

Once the flow has been applied to the HW, the pull function will be called to get the queued creation/destruction flows.

The DR rule flow memory is represented in PMD layer instead of allocating from HW steering layer. While destroying the flow, the flow rule memory can only be freed after the CQE received.

The HW queue job descriptor is currently introduced to convey the flow information and operation type between the flow insertion/destruction in the pull function.

This commit adds the basic flow queue operation for:
rte_flow_async_create(); rte_flow_async_destroy();
rte_flow_push();
rte_flow_pull();

---
 doc/guides/nics/mlx5.rst               |   6 +
 doc/guides/rel_notes/release_22_03.rst |   1 +
 drivers/net/mlx5/mlx5.h                |   2 +-
 drivers/net/mlx5/mlx5_flow.c           | 188 ++++++++++++++++
 drivers/net/mlx5/mlx5_flow.h           |  41 ++++
 drivers/net/mlx5/mlx5_flow_hw.c        | 292 ++++++++++++++++++++++++-
 6 files changed, 528 insertions(+), 2 deletions(-)
doc/guides/nics/mlx5.rst +6 -0
index 92127675a3..a5b3298f0c 100644
@@ -488,6 +488,12 @@ Limitations
     from the reference "Clock Queue" completions,
     the scheduled send timestamps should not be specified with non-zero MSB.
 
+  - HW steering:
+
+    - WQE based high scaling and safer flow insertion/destruction.
+    - Set ``dv_flow_en`` to 2 in order to enable HW steering.
+    - Async queue-based ``rte_flow_q`` APIs supported only.
+
 Statistics
 ----------
 
doc/guides/rel_notes/release_22_03.rst +1 -0
index 3b4a6814f3..0c39011b24 100644
@@ -164,6 +164,7 @@ New Features
 
   * Support ConnectX-7 capability to schedule traffic sending on timestamp
   * Support steering for external Rx queue created outside the PMD.
+  * Added WQE based hardware steering support with ``rte_flow_async`` API.
 
 * **Updated Wangxun ngbe driver.**
 
drivers/net/mlx5/mlx5.h +1 -1
index 4eedc8859f..6f7ec2f067 100644
@@ -340,7 +340,7 @@ enum {
 /* HW steering flow management job descriptor. */
 struct mlx5_hw_q_job {
 	uint32_t type; /* Job type. */
-	struct rte_flow *flow; /* Flow attached to the job. */
+	struct rte_flow_hw *flow; /* Flow attached to the job. */
 	void *user_data; /* Job user data. */
 };
 
drivers/net/mlx5/mlx5_flow.c +188 -0

206 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +41 -0

65 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_hw.c +291 -1

316 lines not on this page · raw message

Signature · 1 line
2.25.1
#55 lore raw

PATCH v4 9/14 net/mlx5: add flow flush function

In reply to Suanming Mou

In case port is being stopped, all created flows should be flushed. This commit adds the flow flush helper function.

---
 drivers/net/mlx5/mlx5_flow.c    |   8 ++
 drivers/net/mlx5/mlx5_flow_hw.c | 129 ++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+)
drivers/net/mlx5/mlx5_flow.c +8 -0
index 1702d60fc0..5af71585fc 100644
@@ -7024,6 +7024,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,
 	uint32_t num_flushed = 0, fidx = 1;
 	struct rte_flow *flow;
 
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+	if (priv->sh->config.dv_flow_en == 2 &&
+	    type == MLX5_FLOW_TYPE_GEN) {
+		flow_hw_q_flow_flush(dev, NULL);
+		return;
+	}
+#endif
+
 	MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {
 		flow_list_destroy(dev, type, fidx);
 		num_flushed++;
drivers/net/mlx5/mlx5_flow_hw.c +129 -0

141 lines not on this page · raw message

Signature · 1 line
2.25.1
#56 lore raw

PATCH v4 10/14 net/mlx5: add flow jump action

In reply to Suanming Mou

Jump action connects different level of flow tables and allows packet handling in the chain of flows.

A new action construct data struct is also added in this commit to help to handle not only the dynamic jump action but also for the other generic dynamic actions. The actions with empty mask configuration means dynamic action, and the dedicated action will be created with the flow action configuration during flow creation. In that dynamic action case, the action will be appended to the table template's action list during table creation. When creating the flows, traverse the action list and pick the dynamic action configuration details from flow actions as the action construct data struct describes, then create the dedicated dynamic actions.

This commit adds the jump action and the generic dynamic action construct mechanism.

---
 drivers/net/mlx5/mlx5.h         |   1 +
 drivers/net/mlx5/mlx5_flow.h    |  25 ++-
 drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++---
 3 files changed, 275 insertions(+), 21 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
index 6f7ec2f067..3d4e9cf8ee 100644
@@ -1528,6 +1528,7 @@ struct mlx5_priv {
 	/* HW steering global drop action. */
 	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
 				     [MLX5DR_TABLE_TYPE_MAX];
+	struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
 #endif
 };
 
drivers/net/mlx5/mlx5_flow.h +24 -1
index 3add4c4a81..963dbd7806 100644
@@ -1020,10 +1020,25 @@ struct rte_flow {
 /* HWS flow struct. */
 struct rte_flow_hw {
 	uint32_t idx; /* Flow index from indexed pool. */
+	uint32_t fate_type; /* Fate action type. */
+	union {
+		/* Jump action. */
+		struct mlx5_hw_jump_action *jump;
+	};
 	struct rte_flow_template_table *table; /* The table flow allcated from. */
 	struct mlx5dr_rule rule; /* HWS layer data struct. */
 } __rte_packed;
 
+/* rte flow action translate to DR action struct. */
+struct mlx5_action_construct_data {
+	LIST_ENTRY(mlx5_action_construct_data) next;
+	/* Ensure the action types are matched. */
+	int type;
+	uint32_t idx;  /* Data index. */
+	uint16_t action_src; /* rte_flow_action src offset. */
+	uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
+};
+
 /* Flow item template struct. */
 struct rte_flow_pattern_template {
 	LIST_ENTRY(rte_flow_pattern_template) next;
@@ -1051,9 +1066,17 @@ struct mlx5_hw_jump_action {
 	struct mlx5dr_action *hws_action;
 };
 
+/* The maximum actions support in the flow. */
+#define MLX5_HW_MAX_ACTS 16
+
 /* DR action set struct. */
 struct mlx5_hw_actions {
-	struct mlx5dr_action *drop; /* Drop action. */
+	/* Dynamic action list. */
+	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
+	struct mlx5_hw_jump_action *jump; /* Jump action. */
+	uint32_t acts_num:4; /* Total action number. */
+	/* Translated DR action array from action template. */
+	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
 };
 
 /* mlx5 action template struct. */
drivers/net/mlx5/mlx5_flow_hw.c +250 -20

399 lines not on this page · raw message

Signature · 1 line
2.25.1
#57 lore raw

PATCH v4 11/14 net/mlx5: add queue and RSS action

In reply to Suanming Mou

This commit adds the queue and RSS action. Similar to the jump action, dynamic ones will be added to the action construct list.

Due to the queue and RSS action in template should not be destroyed during port restart, the actions are created with standalone indirect table as indirect action does. When port stops, detaches the indirect table from action, when port starts, attaches the indirect table back to the action.

One more change is made to accelerate the action creation. Currently the mlx5_hrxq_get() function returns the object index instead of object pointer. This introduced an extra converting the index to the object by calling mlx5_ipool_get() in most of the case. And that extra converting hurts multi-thread performance since mlx5_ipool_get() uses the global lock inside. As the hash Rx queue object itself also contains the index, returns the object directly will achieve better performance without the global lock.

---
 drivers/net/mlx5/linux/mlx5_os.c   |  18 ++--
 drivers/net/mlx5/mlx5.h            |   4 +
 drivers/net/mlx5/mlx5_devx.c       |  10 ++
 drivers/net/mlx5/mlx5_flow.c       |  38 +++----
 drivers/net/mlx5/mlx5_flow.h       |   7 ++
 drivers/net/mlx5/mlx5_flow_dv.c    | 161 ++++++++++++++---------------
 drivers/net/mlx5/mlx5_flow_hw.c    | 101 ++++++++++++++++++
 drivers/net/mlx5/mlx5_flow_verbs.c |   7 +-
 drivers/net/mlx5/mlx5_rx.h         |   9 +-
 drivers/net/mlx5/mlx5_rxq.c        |  85 +++++++++------
 10 files changed, 283 insertions(+), 157 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +9 -9
index 5b91e057b2..c9e8017ff7 100644
@@ -1537,6 +1537,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 	priv->drop_queue.hrxq = mlx5_drop_action_create(eth_dev);
 	if (!priv->drop_queue.hrxq)
 		goto error;
+	priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
+				       mlx5_hrxq_create_cb,
+				       mlx5_hrxq_match_cb,
+				       mlx5_hrxq_remove_cb,
+				       mlx5_hrxq_clone_cb,
+				       mlx5_hrxq_clone_free_cb);
+	if (!priv->hrxqs)
+		goto error;
+	rte_rwlock_init(&priv->ind_tbls_lock);
 	if (priv->sh->config.dv_flow_en == 2)
 		return eth_dev;
 	/* Port representor shares the same max priority with pf port. */
@@ -1561,15 +1570,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 			err = ENOTSUP;
 			goto error;
 	}
-	priv->hrxqs = mlx5_list_create("hrxq", eth_dev, true,
-				       mlx5_hrxq_create_cb,
-				       mlx5_hrxq_match_cb,
-				       mlx5_hrxq_remove_cb,
-				       mlx5_hrxq_clone_cb,
-				       mlx5_hrxq_clone_free_cb);
-	if (!priv->hrxqs)
-		goto error;
-	rte_rwlock_init(&priv->ind_tbls_lock);
 	/* Query availability of metadata reg_c's. */
 	if (!priv->sh->metadata_regc_check_flag) {
 		err = mlx5_flow_discover_mreg_c(eth_dev);
drivers/net/mlx5/mlx5.h +4 -0
index 3d4e9cf8ee..fe422f7fd1 100644
@@ -1287,6 +1287,7 @@ struct mlx5_flow_rss_desc {
 	uint64_t hash_fields; /* Verbs Hash fields. */
 	uint8_t key[MLX5_RSS_HASH_KEY_LEN]; /**< RSS hash key. */
 	uint32_t key_len; /**< RSS hash key len. */
+	uint32_t hws_flags; /**< HW steering action. */
 	uint32_t tunnel; /**< Queue in tunnel. */
 	uint32_t shared_rss; /**< Shared RSS index. */
 	struct mlx5_ind_table_obj *ind_tbl;
@@ -1348,6 +1349,7 @@ struct mlx5_hrxq {
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
 	void *action; /* DV QP action pointer. */
 #endif
+	uint32_t hws_flags; /* Hw steering flags. */
 	uint64_t hash_fields; /* Verbs Hash fields. */
 	uint32_t rss_key_len; /* Hash key length in bytes. */
 	uint32_t idx; /* Hash Rx queue index. */
@@ -1479,6 +1481,8 @@ struct mlx5_priv {
 	LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. */
 	/* Indirection tables. */
 	LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls;
+	/* Standalone indirect tables. */
+	LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;
 	/* Pointer to next element. */
 	rte_rwlock_t ind_tbls_lock;
 	uint32_t refcnt; /**< Reference counter. */
drivers/net/mlx5/mlx5_devx.c +10 -0
index 8aa68d9658..af106bda50 100644
@@ -815,6 +815,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
 		goto error;
 	}
 #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H)
+	if (hrxq->hws_flags) {
+		hrxq->action = mlx5dr_action_create_dest_tir
+			(priv->dr_ctx,
+			 (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags);
+		if (!hrxq->action)
+			goto error;
+		return 0;
+	}
 	if (mlx5_flow_os_create_flow_action_dest_devx_tir(hrxq->tir,
 							  &hrxq->action)) {
 		rte_errno = errno;
@@ -1050,6 +1058,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)
 		DRV_LOG(ERR, "Cannot create drop RX queue");
 		return ret;
 	}
+	if (priv->sh->config.dv_flow_en == 2)
+		return 0;
 	/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */
 	ret = mlx5_devx_ind_table_new(dev, 0, hrxq->ind_table);
 	if (ret != 0) {
drivers/net/mlx5/mlx5_flow.c +12 -26
index 5af71585fc..7c3e4ec47a 100644
@@ -9448,14 +9448,10 @@ int
 mlx5_action_handle_attach(struct rte_eth_dev *dev)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct mlx5_indexed_pool *ipool =
-			priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
-	struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
 	int ret = 0;
-	uint32_t idx;
+	struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
 
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		const char *message;
 		uint32_t queue_idx;
 
@@ -9471,9 +9467,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
 	}
 	if (ret != 0)
 		return ret;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		ret = mlx5_ind_table_obj_attach(dev, ind_tbl);
 		if (ret != 0) {
 			DRV_LOG(ERR, "Port %u could not attach "
@@ -9482,13 +9476,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev)
 			goto error;
 		}
 	}
+
 	return 0;
 error:
-	shared_rss_last = shared_rss;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
-		if (shared_rss == shared_rss_last)
+	ind_tbl_last = ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+		if (ind_tbl == ind_tbl_last)
 			break;
 		if (mlx5_ind_table_obj_detach(dev, ind_tbl) != 0)
 			DRV_LOG(CRIT, "Port %u could not detach "
@@ -9511,15 +9504,10 @@ int
 mlx5_action_handle_detach(struct rte_eth_dev *dev)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct mlx5_indexed_pool *ipool =
-			priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS];
-	struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;
 	int ret = 0;
-	uint32_t idx;
-
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
+	struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last;
 
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
 		ret = mlx5_ind_table_obj_detach(dev, ind_tbl);
 		if (ret != 0) {
 			DRV_LOG(ERR, "Port %u could not detach "
@@ -9530,11 +9518,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev)
 	}
 	return 0;
 error:
-	shared_rss_last = shared_rss;
-	ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) {
-		struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl;
-
-		if (shared_rss == shared_rss_last)
+	ind_tbl_last = ind_tbl;
+	LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {
+		if (ind_tbl == ind_tbl_last)
 			break;
 		if (mlx5_ind_table_obj_attach(dev, ind_tbl) != 0)
 			DRV_LOG(CRIT, "Port %u could not attach "
drivers/net/mlx5/mlx5_flow.h +7 -0
index 963dbd7806..70e6cf633f 100644
@@ -1024,6 +1024,7 @@ struct rte_flow_hw {
 	union {
 		/* Jump action. */
 		struct mlx5_hw_jump_action *jump;
+		struct mlx5_hrxq *hrxq; /* TIR action. */
 	};
 	struct rte_flow_template_table *table; /* The table flow allcated from. */
 	struct mlx5dr_rule rule; /* HWS layer data struct. */
@@ -1074,6 +1075,7 @@ struct mlx5_hw_actions {
 	/* Dynamic action list. */
 	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
+	struct mlx5_hrxq *tir; /* TIR action. */
 	uint32_t acts_num:4; /* Total action number. */
 	/* Translated DR action array from action template. */
 	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
@@ -1910,6 +1912,11 @@ struct mlx5_list_entry *flow_dv_dest_array_clone_cb(void *tool_ctx,
 				   struct mlx5_list_entry *entry, void *cb_ctx);
 void flow_dv_dest_array_clone_free_cb(void *tool_ctx,
 				      struct mlx5_list_entry *entry);
+void flow_dv_hashfields_set(uint64_t item_flags,
+			    struct mlx5_flow_rss_desc *rss_desc,
+			    uint64_t *hash_fields);
+void flow_dv_action_rss_l34_hash_adjust(uint64_t rss_types,
+					uint64_t *hash_field);
 
 struct mlx5_list_entry *flow_hw_grp_create_cb(void *tool_ctx, void *cb_ctx);
 void flow_hw_grp_remove_cb(void *tool_ctx, struct mlx5_list_entry *entry);
drivers/net/mlx5/mlx5_flow_dv.c +76 -85

371 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_hw.c +101 -0

149 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_verbs.c +2 -5
index 90ccb9aaff..f08aa7a770 100644
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			MLX5_ASSERT(priv->drop_queue.hrxq);
 			hrxq = priv->drop_queue.hrxq;
 		} else {
-			uint32_t hrxq_idx;
 			struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;
 
 			MLX5_ASSERT(rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			rss_desc->tunnel = !!(handle->layers &
 					      MLX5_FLOW_LAYER_TUNNEL);
 			rss_desc->shared_rss = 0;
-			hrxq_idx = mlx5_hrxq_get(dev, rss_desc);
-			hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ],
-					      hrxq_idx);
+			hrxq = mlx5_hrxq_get(dev, rss_desc);
 			if (!hrxq) {
 				rte_flow_error_set
 					(error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 					 "cannot get hash queue");
 				goto error;
 			}
-			handle->rix_hrxq = hrxq_idx;
+			handle->rix_hrxq = hrxq->idx;
 		}
 		MLX5_ASSERT(hrxq);
 		handle->drv_flow = mlx5_glue->create_flow
drivers/net/mlx5/mlx5_rx.h +7 -2
index 29652a8c9f..acebe3348c 100644
@@ -231,9 +231,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);
 struct mlx5_ind_table_obj *mlx5_ind_table_obj_get(struct rte_eth_dev *dev,
 						  const uint16_t *queues,
 						  uint32_t queues_n);
+struct mlx5_ind_table_obj *mlx5_ind_table_obj_new(struct rte_eth_dev *dev,
+						  const uint16_t *queues,
+						  uint32_t queues_n,
+						  bool standalone,
+						  bool ref_qs);
 int mlx5_ind_table_obj_release(struct rte_eth_dev *dev,
 			       struct mlx5_ind_table_obj *ind_tbl,
-			       bool standalone,
 			       bool deref_rxqs);
 int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev,
 			     struct mlx5_ind_table_obj *ind_tbl,
@@ -256,8 +260,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx,
 					   void *cb_ctx __rte_unused);
 void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,
 			     struct mlx5_list_entry *entry);
-uint32_t mlx5_hrxq_get(struct rte_eth_dev *dev,
+struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev,
 		       struct mlx5_flow_rss_desc *rss_desc);
+int mlx5_hrxq_obj_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq);
 int mlx5_hrxq_release(struct rte_eth_dev *dev, uint32_t hxrq_idx);
 uint32_t mlx5_hrxq_verify(struct rte_eth_dev *dev);
 bool mlx5_rxq_is_hairpin(struct rte_eth_dev *dev, uint16_t idx);
drivers/net/mlx5/mlx5_rxq.c +55 -30

209 lines not on this page · raw message

Signature · 1 line
2.25.1
#58 lore raw

PATCH v4 12/14 net/mlx5: add mark action

In reply to Suanming Mou

The mark action is covered by tag action internally. While it is added the HW will add a tag to the packet. The mark value can be set as fixed or dynamic as the action mask indicates.

---
 drivers/net/mlx5/mlx5.h         |  2 +
 drivers/net/mlx5/mlx5_flow.h    |  1 +
 drivers/net/mlx5/mlx5_flow_hw.c | 66 ++++++++++++++++++++++++++++++---
 3 files changed, 63 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.h +2 -0
index fe422f7fd1..df0257ff23 100644
@@ -1532,6 +1532,8 @@ struct mlx5_priv {
 	/* HW steering global drop action. */
 	struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX]
 				     [MLX5DR_TABLE_TYPE_MAX];
+	/* HW steering global drop action. */
+	struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX];
 	struct mlx5_indexed_pool *acts_ipool; /* Action data indexed pool. */
 #endif
 };
drivers/net/mlx5/mlx5_flow.h +1 -0
index 70e6cf633f..c83e73c793 100644
@@ -1077,6 +1077,7 @@ struct mlx5_hw_actions {
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
 	struct mlx5_hrxq *tir; /* TIR action. */
 	uint32_t acts_num:4; /* Total action number. */
+	uint32_t mark:1; /* Indicate the mark action. */
 	/* Translated DR action array from action template. */
 	struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS];
 };
drivers/net/mlx5/mlx5_flow_hw.c +60 -6

115 lines not on this page · raw message

Signature · 1 line
2.25.1
#59 lore raw

PATCH v4 13/14 net/mlx5: add indirect action

In reply to Suanming Mou

HW steering can support indirect action as well. With indirect action, the flow can be created with more flexible shared RSS action selection. This will can save the action template with different RSS actions.

This commit adds the flow queue operation callback for:
rte_flow_async_action_handle_create(); rte_flow_async_action_handle_destroy(); rte_flow_async_action_handle_update();

---
 drivers/net/mlx5/mlx5_flow.c    | 131 ++++++++++
 drivers/net/mlx5/mlx5_flow.h    |  59 +++++
 drivers/net/mlx5/mlx5_flow_dv.c |  21 +-
 drivers/net/mlx5/mlx5_flow_hw.c | 414 +++++++++++++++++++++++++++++++-
 4 files changed, 612 insertions(+), 13 deletions(-)
drivers/net/mlx5/mlx5_flow.c +131 -0
index 7c3e4ec47a..3875160708 100644
@@ -879,6 +879,29 @@ mlx5_flow_push(struct rte_eth_dev *dev,
 	       uint32_t queue,
 	       struct rte_flow_error *error);
 
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_op_attr *attr,
+				 const struct rte_flow_indir_action_conf *conf,
+				 const struct rte_flow_action *action,
+				 void *user_data,
+				 struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_op_attr *attr,
+				 struct rte_flow_action_handle *handle,
+				 const void *update,
+				 void *user_data,
+				 struct rte_flow_error *error);
+
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+				  const struct rte_flow_op_attr *attr,
+				  struct rte_flow_action_handle *handle,
+				  void *user_data,
+				  struct rte_flow_error *error);
+
 static const struct rte_flow_ops mlx5_flow_ops = {
 	.validate = mlx5_flow_validate,
 	.create = mlx5_flow_create,
@@ -911,6 +934,9 @@ static const struct rte_flow_ops mlx5_flow_ops = {
 	.async_destroy = mlx5_flow_async_flow_destroy,
 	.pull = mlx5_flow_pull,
 	.push = mlx5_flow_push,
+	.async_action_handle_create = mlx5_flow_async_action_handle_create,
+	.async_action_handle_update = mlx5_flow_async_action_handle_update,
+	.async_action_handle_destroy = mlx5_flow_async_action_handle_destroy,
 };
 
 /* Tunnel information. */
@@ -8376,6 +8402,111 @@ mlx5_flow_push(struct rte_eth_dev *dev,
 	return fops->push(dev, queue, error);
 }
 
+/**
+ * Create shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] conf
+ *   Indirect action configuration.
+ * @param[in] action
+ *   rte_flow action detail.
+ * @param[in] user_data
+ *   Pointer to the user_data.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   Action handle on success, NULL otherwise and rte_errno is set.
+ */
+static struct rte_flow_action_handle *
+mlx5_flow_async_action_handle_create(struct rte_eth_dev *dev, uint32_t queue,
+				 const struct rte_flow_op_attr *attr,
+				 const struct rte_flow_indir_action_conf *conf,
+				 const struct rte_flow_action *action,
+				 void *user_data,
+				 struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->async_action_create(dev, queue, attr, conf, action,
+					 user_data, error);
+}
+
+/**
+ * Update shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] handle
+ *   Action handle to be updated.
+ * @param[in] update
+ *   Update value.
+ * @param[in] user_data
+ *   Pointer to the user_data.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_update(struct rte_eth_dev *dev, uint32_t queue,
+				     const struct rte_flow_op_attr *attr,
+				     struct rte_flow_action_handle *handle,
+				     const void *update,
+				     void *user_data,
+				     struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->async_action_update(dev, queue, attr, handle,
+					 update, user_data, error);
+}
+
+/**
+ * Destroy shared action.
+ *
+ * @param[in] dev
+ *   Pointer to the rte_eth_dev structure.
+ * @param[in] queue
+ *   Which queue to be used..
+ * @param[in] attr
+ *   Operation attribute.
+ * @param[in] handle
+ *   Action handle to be destroyed.
+ * @param[in] user_data
+ *   Pointer to the user_data.
+ * @param[out] error
+ *   Pointer to error structure.
+ *
+ * @return
+ *   0 on success, negative value otherwise and rte_errno is set.
+ */
+static int
+mlx5_flow_async_action_handle_destroy(struct rte_eth_dev *dev, uint32_t queue,
+				      const struct rte_flow_op_attr *attr,
+				      struct rte_flow_action_handle *handle,
+				      void *user_data,
+				      struct rte_flow_error *error)
+{
+	const struct mlx5_flow_driver_ops *fops =
+			flow_get_drv_ops(MLX5_FLOW_TYPE_HW);
+
+	return fops->async_action_destroy(dev, queue, attr, handle,
+					  user_data, error);
+}
+
 /**
  * Allocate a new memory for the counter values wrapped by all the needed
  * management.
drivers/net/mlx5/mlx5_flow.h +59 -0

99 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_dv.c +12 -9
index 2ffdfad61f..5686986f5c 100644
@@ -13841,9 +13841,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,
  * @return
  *   Valid hash RX queue index, otherwise 0.
  */
-static uint32_t
-__flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
-				 const uint64_t hash_fields)
+uint32_t
+flow_dv_action_rss_hrxq_lookup(struct rte_eth_dev *dev, uint32_t idx,
+			       const uint64_t hash_fields)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_shared_action_rss *shared_rss =
@@ -13971,7 +13971,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,
 			struct mlx5_hrxq *hrxq = NULL;
 			uint32_t hrxq_idx;
 
-			hrxq_idx = __flow_dv_action_rss_hrxq_lookup(dev,
+			hrxq_idx = flow_dv_action_rss_hrxq_lookup(dev,
 						rss_desc->shared_rss,
 						dev_flow->hash_fields);
 			if (hrxq_idx)
@@ -14695,6 +14695,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
 			   struct mlx5_shared_action_rss *shared_rss,
 			   struct rte_flow_error *error)
 {
+	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_flow_rss_desc rss_desc = { 0 };
 	size_t i;
 	int err;
@@ -14715,6 +14716,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,
 	/* Set non-zero value to indicate a shared RSS. */
 	rss_desc.shared_rss = action_idx;
 	rss_desc.ind_tbl = shared_rss->ind_tbl;
+	if (priv->sh->config.dv_flow_en == 2)
+		rss_desc.hws_flags = MLX5DR_ACTION_FLAG_HWS_RX;
 	for (i = 0; i < MLX5_RSS_HASH_FIELDS_LEN; i++) {
 		struct mlx5_hrxq *hrxq;
 		uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14906,7 +14909,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,
  *   A valid shared action handle in case of success, NULL otherwise and
  *   rte_errno is set.
  */
-static struct rte_flow_action_handle *
+struct rte_flow_action_handle *
 flow_dv_action_create(struct rte_eth_dev *dev,
 		      const struct rte_flow_indir_action_conf *conf,
 		      const struct rte_flow_action *action,
@@ -14976,7 +14979,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_destroy(struct rte_eth_dev *dev,
 		       struct rte_flow_action_handle *handle,
 		       struct rte_flow_error *error)
@@ -15186,7 +15189,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_update(struct rte_eth_dev *dev,
 			struct rte_flow_action_handle *handle,
 			const void *update,
@@ -15858,7 +15861,7 @@ flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
 				  "counters are not available");
 }
 
-static int
+int
 flow_dv_action_query(struct rte_eth_dev *dev,
 		     const struct rte_flow_action_handle *handle, void *data,
 		     struct rte_flow_error *error)
@@ -17551,7 +17554,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)
  * @return
  *   0 on success, otherwise negative errno value.
  */
-static int
+int
 flow_dv_action_validate(struct rte_eth_dev *dev,
 			const struct rte_flow_indir_action_conf *conf,
 			const struct rte_flow_action *action,
drivers/net/mlx5/mlx5_flow_hw.c +410 -4

493 lines not on this page · raw message

Signature · 1 line
2.25.1
#60 lore raw

PATCH v4 14/14 net/mlx5: add header reformat action

In reply to Suanming Mou

HW steering header reformat action can work under bulk mode. In this case, when create the table, bulk size of header reformat actions will be allocated in low level. Afterwards, when create flow, just simply specify the action index in the bulk and the encapsulation data to the action will be enough.

---
 drivers/net/mlx5/mlx5.h         |   1 +
 drivers/net/mlx5/mlx5_flow.h    |  21 +++
 drivers/net/mlx5/mlx5_flow_dv.c |   4 +-
 drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++-
 4 files changed, 251 insertions(+), 3 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
index df0257ff23..0f825396a2 100644
@@ -342,6 +342,7 @@ struct mlx5_hw_q_job {
 	uint32_t type; /* Job type. */
 	struct rte_flow_hw *flow; /* Flow attached to the job. */
 	void *user_data; /* Job user data. */
+	uint8_t *encap_data; /* Encap data. */
 };
 
 /* HW steering job descriptor LIFO pool. */
drivers/net/mlx5/mlx5_flow.h +21 -0
index 4c224bbf52..09f0d7a75d 100644
@@ -1040,6 +1040,14 @@ struct mlx5_action_construct_data {
 	uint16_t action_src; /* rte_flow_action src offset. */
 	uint16_t action_dst; /* mlx5dr_rule_action dst offset. */
 	union {
+		struct {
+			/* encap src(item) offset. */
+			uint16_t src;
+			/* encap dst data offset. */
+			uint16_t dst;
+			/* encap data len. */
+			uint16_t len;
+		} encap;
 		struct {
 			uint64_t types; /* RSS hash types. */
 			uint32_t level; /* RSS level. */
@@ -1076,6 +1084,13 @@ struct mlx5_hw_jump_action {
 	struct mlx5dr_action *hws_action;
 };
 
+/* Encap decap action struct. */
+struct mlx5_hw_encap_decap_action {
+	struct mlx5dr_action *action; /* Action object. */
+	size_t data_size; /* Action metadata size. */
+	uint8_t data[]; /* Action data. */
+};
+
 /* The maximum actions support in the flow. */
 #define MLX5_HW_MAX_ACTS 16
 
@@ -1085,6 +1100,9 @@ struct mlx5_hw_actions {
 	LIST_HEAD(act_list, mlx5_action_construct_data) act_list;
 	struct mlx5_hw_jump_action *jump; /* Jump action. */
 	struct mlx5_hrxq *tir; /* TIR action. */
+	/* Encap/Decap action. */
+	struct mlx5_hw_encap_decap_action *encap_decap;
+	uint16_t encap_decap_pos; /* Encap/Decap action position. */
 	uint32_t acts_num:4; /* Total action number. */
 	uint32_t mark:1; /* Indicate the mark action. */
 	/* Translated DR action array from action template. */
@@ -2027,4 +2045,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,
 			 const struct rte_flow_action_handle *handle,
 			 void *data,
 			 struct rte_flow_error *error);
+size_t flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type);
+int flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
+			   size_t *size, struct rte_flow_error *error);
 #endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_dv.c +2 -2
index 5686986f5c..313dc64604 100644
@@ -4034,7 +4034,7 @@ flow_dv_push_vlan_action_resource_register
  * @return
  *   sizeof struct item_type, 0 if void or irrelevant.
  */
-static size_t
+size_t
 flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
 {
 	size_t retval;
@@ -4100,7 +4100,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)
  * @return
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
-static int
+int
 flow_dv_convert_encap_data(const struct rte_flow_item *items, uint8_t *buf,
 			   size_t *size, struct rte_flow_error *error)
 {
drivers/net/mlx5/mlx5_flow_hw.c +227 -1

299 lines not on this page · raw message

Signature · 1 line
2.25.1
#61 lore raw
In reply to Suanming Mou

Hi,

Quoted 88 lines

-----Original Message-----
From: Suanming Mou <[email protected]>
Sent: Thursday, February 24, 2022 3:41 PM
To: Slava Ovsiienko <[email protected]>; Matan Azrad <[email protected]>
Cc: Raslan Darawsheh <[email protected]>; Ori Kam <[email protected]>; [email protected]
Subject: [PATCH v4 00/14] net/mlx5: add hardware steering

The Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:

- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly

However, there are still some disadvantages:

- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects

- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion

- security, exposing the low-level steering entries directly to the userspace may cause security risks

A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.

In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.

---

v4:
v3:
- rebase to the latest version.

v2:
- New HW steering low-level abstract code added.
- commit message improvement.
- add protection for rte_flow and rte_flow_async callbacks.
- rebase to rte_flow_async v9.
- fix some rte_flow error not filled bugs.

Suanming Mou (14):
net/mlx5: introduce hardware steering operation net/mlx5: add HW steering low-level abstract code net/mlx5: introduce hardware steering enable routine net/mlx5: add port flow configuration net/mlx5: add pattern template management net/mlx5: add action template management net/mlx5: add table management
net/mlx5: add basic flow queue operation net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action

 doc/guides/nics/mlx5.rst                |   19 +-
 doc/guides/rel_notes/release_22_03.rst  |    1 +
 drivers/net/mlx5/linux/mlx5_flow_os.h   |    1 +
 drivers/net/mlx5/linux/mlx5_os.c        |   22 +-
 drivers/net/mlx5/meson.build            |    2 +
 drivers/net/mlx5/mlx5.c                 |   55 +-
 drivers/net/mlx5/mlx5.h                 |   66 +-
 drivers/net/mlx5/mlx5_devx.c            |   10 +
 drivers/net/mlx5/mlx5_dr.c              |  383 ++++
 drivers/net/mlx5/mlx5_dr.h              |  456 +++++
 drivers/net/mlx5/mlx5_flow.c            |  721 ++++++-
 drivers/net/mlx5/mlx5_flow.h            |  287 +++
 drivers/net/mlx5/mlx5_flow_dv.c         |  186 +-
 drivers/net/mlx5/mlx5_flow_hw.c         | 2337 +++++++++++++++++++++++
 drivers/net/mlx5/mlx5_flow_verbs.c      |    7 +-
 drivers/net/mlx5/mlx5_rx.h              |    9 +-
 drivers/net/mlx5/mlx5_rxq.c             |   85 +-
 drivers/net/mlx5/windows/mlx5_flow_os.h |    1 +
 18 files changed, 4468 insertions(+), 180 deletions(-)  create mode 100644
Quoted 5 lines

drivers/net/mlx5/mlx5_dr.c create mode 100644 drivers/net/mlx5/mlx5_dr.h create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c

--
2.25.1

Series applied to next-net-mlx,

Kindest regards,
Raslan Darawsheh