net/mlx5: add hardware steering
Subsystems (2)
cover net/mlx5: add hardware steering
As the new queue-based flow rules management mechanism is introduced to the rte_flow. A new steering mode PMD code is also developed to take advantage of the new API.
The HW steering prepares the required flow resources in advanced from the hints when create template table. That means the matcher and most of the actions have been prepared during flow table creation. Limited left actions will be constructed during flow creation if needed.
A flow postpone attribute bit describes if flow creation/destruction should be applied to the HW directly or not. An extra push function has also been prepared to force push all the enqueued flows to HW.
Once the flow has been applied to the HW, the pull function will be called to get the enqueued creation/destruction flows.
The asynchronous flow creation and destruction are handled by queue jobs. The queue job descriptor is currently introduced to convey the flow information and operation type between the flow management in in pull function.
Flow rule memory is allocated in PMD layer instead of allocating from HW layer. While destroying the flow, the flow rule memory can only be freed after the event received.
*** THIS PATCH SET DEPENDS ON THE NEW RTE_FLOW API ***
Suanming Mou (13):
net/mlx5: introduce hardware steering operation net/mlx5: introduce hardware steering enable routine net/mlx5: add port flow configuration net/mlx5: add pattern template management net/mlx5: add action template management net/mlx5: add table management
net/mlx5: add basic flow queue operation net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action
drivers/net/mlx5/linux/mlx5_flow_os.h | 1 + drivers/net/mlx5/linux/mlx5_os.c | 22 +- drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5.c | 50 +- drivers/net/mlx5/mlx5.h | 63 +- drivers/net/mlx5/mlx5_devx.c | 10 + drivers/net/mlx5/mlx5_flow.c | 569 +++++- drivers/net/mlx5/mlx5_flow.h | 276 +++ drivers/net/mlx5/mlx5_flow_dv.c | 175 +- drivers/net/mlx5/mlx5_flow_hw.c | 2284 +++++++++++++++++++++++++ drivers/net/mlx5/mlx5_flow_verbs.c | 7 +- drivers/net/mlx5/mlx5_rx.h | 9 +- drivers/net/mlx5/mlx5_rxq.c | 78 +- 13 files changed, 3374 insertions(+), 171 deletions(-) create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
Signature · 1 line
2.25.1
PATCH 1/13 net/mlx5: introduce hardware steering operation
A new hardware based steering operation is going to be introduced for high insertion rate.
This commit adds the basic driver operation.
--- drivers/net/mlx5/linux/mlx5_flow_os.h | 1 + drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5_flow.c | 1 + drivers/net/mlx5/mlx5_flow.h | 1 + drivers/net/mlx5/mlx5_flow_hw.c | 13 +++++++++++++ 5 files changed, 17 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
drivers/net/mlx5/linux/mlx5_flow_os.h +1 -0
@@ -9,6 +9,7 @@#ifdef HAVE_IBV_FLOW_DV_SUPPORT extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops; +extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;#endif /**
drivers/net/mlx5/meson.build +1 -0
@@ -16,6 +16,7 @@ sources = files( 'mlx5_flow.c', 'mlx5_flow_meter.c', 'mlx5_flow_dv.c', + 'mlx5_flow_hw.c', 'mlx5_flow_aso.c', 'mlx5_flow_flex.c', 'mlx5_mac.c',
drivers/net/mlx5/mlx5_flow.c +1 -0
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = { [MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) [MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops, + [MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops, #endif [MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops, [MLX5_FLOW_TYPE_MAX ] = &mlx5_flow_null_drv_ops
drivers/net/mlx5/mlx5_flow.h +1 -0
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {MLX5_FLOW_TYPE_MIN ,MLX5_FLOW_TYPE_DV ,MLX5_FLOW_TYPE_VERBS , +MLX5_FLOW_TYPE_HW , MLX5_FLOW_TYPE_MAX, };
drivers/net/mlx5/mlx5_flow_hw.c new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright 2020 Mellanox Technologies, Ltd +*/ + +#include <rte_flow.h> + +#include "mlx5_flow.h" + +#ifdef HAVE_IBV_FLOW_DV_SUPPORT + +const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops; + +#endif
Signature · 1 line
2.25.1
PATCH 2/13 net/mlx5: introduce hardware steering enable routine
In reply to Suanming MouAs the new hardware steering operation will be implemented under the new rte_flow_q APIs. This is not compatible with the existing rte_flow PMD's Direct Rules flow operation routine.
This commit introduces an extra dv_flow_en = 2 to specify the new flow operation initialize routine.
--- drivers/net/mlx5/linux/mlx5_os.c | 4 ++++ drivers/net/mlx5/mlx5.c | 2 +- drivers/net/mlx5/mlx5.h | 3 ++- 3 files changed, 7 insertions(+), 2 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +4 -0
@@ -295,6 +295,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv) err =mlx5_alloc_table_hash_list (priv);if (err) goto error; +if (priv->config.dv_flow_en ==2 ) +return 0 ;/* The resources below are only valid with DV support. */ #ifdef HAVE_IBV_FLOW_DV_SUPPORT /* Init port id action list. */
@@ -1712,6 +1714,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->drop_queue.hrxq =mlx5_drop_action_create (eth_dev);if (!priv->drop_queue.hrxq) goto error; +if (priv->config.dv_flow_en ==2 ) +return eth_dev;/* Port representor shares the same max priority with pf port. */ if (!priv->sh->flow_priority_check_flag) {/* Supported Verbs flow priority number detection. */
drivers/net/mlx5/mlx5.c +1 -1
@@ -1933,7 +1933,7 @@ mlx5_args_check(const char *key, const char *val, void *opaque) }else if (strcmp (MLX5_DV_ESW_EN , key) ==0 ) { config->dv_esw_en = !!tmp; }else if (strcmp (MLX5_DV_FLOW_EN , key) ==0 ) { - config->dv_flow_en = !!tmp; + config->dv_flow_en = tmp; }else if (strcmp (MLX5_DV_XMETA_EN , key) ==0 ) { if (tmp != MLX5_XMETA_MODE_LEGACY && tmp != MLX5_XMETA_MODE_META16 &&
drivers/net/mlx5/mlx5.h +2 -1
@@ -259,7 +259,8 @@ struct mlx5_dev_config { unsigned int l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */ unsigned int vf_nl_en:1; /* Enable Netlink requests in VF mode. */ unsigned int dv_esw_en:1; /* Enable E-Switch DV flow. */ - unsigned int dv_flow_en:1; /* Enable DV flow. */ +/* Enable DV flow. 1 means SW steering, 2 means HW steering. */ + unsigned int dv_flow_en:2; unsigned int dv_xmeta_en:2; /* Enable extensive flow metadata. */ unsigned int lacp_by_user:1;/* Enable user to manage LACP traffic. */
Signature · 1 line
2.25.1
PATCH 3/13 net/mlx5: add port flow configuration
In reply to Suanming MouThe hardware steering is backend to support rte_flow_q API in mlx5 PMD. The port configuration function creates the queues and needed flow management resources.
The PMD layer configuration function allocates the queues' context and per-queue job descriptor. The job descriptor size is equal to the queue size, and the job descriptors will be popped from LIFO to convey the flow information during flow insertion/destruction. So when polling the result, the flow information will be extracted from the descriptor and then the job descriptor will be push back to the LIFO.
The commit creates the flow port queue and the job descriptors.
--- drivers/net/mlx5/mlx5.c | 3 + drivers/net/mlx5/mlx5.h | 26 ++++++- drivers/net/mlx5/mlx5_flow.c | 37 +++++++++ drivers/net/mlx5/mlx5_flow.h | 9 +++ drivers/net/mlx5/mlx5_flow_hw.c | 132 ++++++++++++++++++++++++++++++++ 5 files changed, 206 insertions(+), 1 deletion(-)
drivers/net/mlx5/mlx5.c +3 -0
@@ -1567,6 +1567,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)/* Free the eCPRI flex parser resource. */ mlx5_flex_parser_ecpri_release (dev);mlx5_flex_item_port_cleanup (dev); +#ifdef HAVE_IBV_FLOW_DV_SUPPORT +flow_hw_resource_release (dev); +#endif if (priv->rxq_privs !=NULL ) {/* XXX race condition if mlx5_rx_burst() is still running. */ rte_delay_us_sleep (1000 );
drivers/net/mlx5/mlx5.h +25 -1
@@ -33,7 +33,7 @@#include "mlx5_utils.h" #include "mlx5_os.h" #include "mlx5_autoconf.h" - +#include "mlx5dr.h" #define MLX5_SH (dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
@@ -324,6 +324,26 @@ struct mlx5_lb_ctx {uint16_t refcnt;/* Reference count for representors. */ }; +/* HW steering queue job descriptor type. */ +enum { +MLX5_HW_Q_JOB_TYPE_CREATE ,/* Flow create job type. */ +MLX5_HW_Q_JOB_TYPE_DESTROY ,/* Flow destroy job type. */ +}; + +/* HW steering flow management job descriptor. */ +struct mlx5_hw_q_job { +uint32_t type;/* Job type. */ +struct rte_flow *flow;/* Flow attached to the job. */ +void *user_data;/* Job user data. */ +}; + +/* HW steering job descriptor LIFO header . */ +struct mlx5_hw_q { +uint32_t job_idx;/* Free job index. */ +uint32_t size;/* LIFO size. */ +struct mlx5_hw_q_job **job;/* LIFO pointer. */ +} __rte_cache_aligned; +#define MLX5_COUNTERS_PER_POOL 512#define MLX5_MAX_PENDING_QUERIES 4#define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1480,6 +1500,10 @@ struct mlx5_priv {struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM ];/* Flex items have been created on the port. */ uint32_t flex_item_map;/* Map of allocated flex item elements. */ +struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ +uint32_t nb_queue;/* HW steering queue number. */ +/* HW steering queue polling mechanism job descriptor LIFO. */ +struct mlx5_hw_q *hw_q; };#define PORT_ID (priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.c +37 -0
@@ -805,6 +805,12 @@ static int mlx5_flow_flex_item_release(struct rte_eth_dev *dev, const struct rte_flow_item_flex_handle *handle, struct rte_flow_error *error); +static int +mlx5_flow_port_configure (struct rte_eth_dev *dev, +const struct rte_flow_port_attr *port_attr, +uint16_t nb_queue, +const struct rte_flow_queue_attr *queue_attr[], +struct rte_flow_error *err);static const struct rte_flow_ops mlx5_flow_ops = { .validate = mlx5_flow_validate,
@@ -826,6 +832,7 @@ static const struct rte_flow_ops mlx5_flow_ops = { .get_restore_info = mlx5_flow_tunnel_get_restore_info, .flex_item_create = mlx5_flow_flex_item_create, .flex_item_release = mlx5_flow_flex_item_release, + .configure = mlx5_flow_port_configure, };/* Tunnel information. */
@@ -7814,6 +7821,36 @@ mlx5_counter_query(struct rte_eth_dev *dev, uint32_t cnt,return -ENOTSUP ; } +/** +* Configure port HWS resources. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] port_attr +* Port configuration attributes. +* @param[in] nb_queue +* Number of queue. +* @param[in] queue_attr +* Array that holds attributes for each flow queue. +* @param[out] error +* Pointer to error structure. +* +* @return +* 0 on success, a negative errno value otherwise and rte_errno is set. +*/ +static int +mlx5_flow_port_configure (struct rte_eth_dev *dev, +const struct rte_flow_port_attr *port_attr, +uint16_t nb_queue, +const struct rte_flow_queue_attr *queue_attr[], +struct rte_flow_error *err) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->configure (dev, port_attr, nb_queue, queue_attr, err); +} +/** * Allocate a new memory for the counter values wrapped by all the needed * management.
drivers/net/mlx5/mlx5_flow.h +9 -0
@@ -1257,6 +1257,12 @@ typedef int (*mlx5_flow_item_update_t)const struct rte_flow_item_flex_handle *handle, const struct rte_flow_item_flex_conf *conf, struct rte_flow_error *error); +typedef int (*mlx5_flow_port_configure_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_port_attr *port_attr, +uint16_t nb_queue, +const struct rte_flow_queue_attr *queue_attr[], +struct rte_flow_error *err);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1295,6 +1301,7 @@ struct mlx5_flow_driver_ops {mlx5_flow_item_create_t item_create;mlx5_flow_item_release_t item_release;mlx5_flow_item_update_t item_update; +mlx5_flow_port_configure_t configure; };/* mlx5_flow.c */
@@ -1767,4 +1774,6 @@ const struct mlx5_flow_tunnel * mlx5_get_tof(const struct rte_flow_item *items, const struct rte_flow_action *actions, enum mlx5_tof_rule_type *rule_type); +void +flow_hw_resource_release (struct rte_eth_dev *dev);#endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_hw.c +132 -0
142 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 4/13 net/mlx5: add pattern template management
In reply to Suanming MouThe pattern template defines flows that have the same matching fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be (eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while the values for each rule will be different.
Due to the pattern template can be used in different domains, the items will only be cached in pattern template create stage, while the template is binded to a dedicated table, the HW criteria will be created and saved to the table. And different tables may create the same criteria and will not shared between each other in order to have better performance.
This commit adds pattern template management.
--- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_flow.c | 64 +++++++++++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 20 ++++++++ drivers/net/mlx5/mlx5_flow_hw.c | 82 +++++++++++++++++++++++++++++++++ 4 files changed, 168 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
@@ -1500,6 +1500,8 @@ struct mlx5_priv {struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM ];/* Flex items have been created on the port. */ uint32_t flex_item_map;/* Map of allocated flex item elements. */ +/* Item template list. */ + LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +64 -0
82 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +20 -0
@@ -1015,6 +1015,15 @@ struct rte_flow {uint32_t geneve_tlv_option;/**< Holds Geneve TLV option id. > */ } __rte_packed; +/* Flow item template struct. */ +struct rte_flow_pattern_template { +LIST_ENTRY (rte_flow_pattern_template) next; +/* Template attributes. */ +struct rte_flow_pattern_template_attr attr; +struct mlx5dr_match_template *mt;/* mlx5 match template. */ +uint32_t refcnt;/* Reference counter. */ +}; +/* * Define list of valid combinations of RX Hash fields * (see enum ibv_rx_hash_fields).
@@ -1263,6 +1272,15 @@ typedef int (*mlx5_flow_port_configure_t)uint16_t nb_queue, const struct rte_flow_queue_attr *queue_attr[], struct rte_flow_error *err); +typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_pattern_template_attr *attr, +const struct rte_flow_item items[], +struct rte_flow_error *error); +typedef int (*mlx5_flow_pattern_template_destroy_t ) + (struct rte_eth_dev *dev, +struct rte_flow_pattern_template *template, +struct rte_flow_error *error);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1302,6 +1320,8 @@ struct mlx5_flow_driver_ops {mlx5_flow_item_release_t item_release;mlx5_flow_item_update_t item_update;mlx5_flow_port_configure_t configure; +mlx5_flow_pattern_template_create_t pattern_template_create; +mlx5_flow_pattern_template_destroy_t pattern_template_destroy; };/* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +82 -0
103 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 5/13 net/mlx5: add action template management
In reply to Suanming MouThe action template holds a list of action types that will be used together on the same rule. The template's actions instances will be created only when the template bind to the dedicated group. And the created actions will be saved to each individual group in order for best performance. The actions will not be shared with each other.
This commit adds the action template management which caches the flow action template.
--- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_flow.c | 66 ++++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 27 ++++++++ drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++ 4 files changed, 211 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
@@ -1502,6 +1502,8 @@ struct mlx5_priv {uint32_t flex_item_map;/* Map of allocated flex item elements. */ /* Item template list. */ LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt; +/* Action template list. */ + LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +66 -0
84 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +27 -0
@@ -1024,6 +1024,21 @@ struct rte_flow_pattern_template {uint32_t refcnt;/* Reference counter. */ }; +/* Flow action template attribute. */ +struct rte_flow_actions_template_attr { +int32_t reserve; +}; + +/* Flow action template struct. */ +struct rte_flow_actions_template { +LIST_ENTRY (rte_flow_actions_template) next; +/* Template attributes. */ +struct rte_flow_actions_template_attr attr; +struct rte_flow_action *actions;/* Cached flow actions. */ +struct rte_flow_action *masks;/* Cached action masks.*/ +uint32_t refcnt;/* Reference counter. */ +}; +/* * Define list of valid combinations of RX Hash fields * (see enum ibv_rx_hash_fields).
@@ -1281,6 +1296,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t) (struct rte_eth_dev *dev, struct rte_flow_pattern_template *template, struct rte_flow_error *error); +typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_actions_template_attr *attr, +const struct rte_flow_action actions[], +const struct rte_flow_action masks[], +struct rte_flow_error *error); +typedef int (*mlx5_flow_actions_template_destroy_t ) + (struct rte_eth_dev *dev, +struct rte_flow_actions_template *template, +struct rte_flow_error *error);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1322,6 +1347,8 @@ struct mlx5_flow_driver_ops {mlx5_flow_port_configure_t configure;mlx5_flow_pattern_template_create_t pattern_template_create;mlx5_flow_pattern_template_destroy_t pattern_template_destroy; +mlx5_flow_actions_template_create_t actions_template_create; +mlx5_flow_actions_template_destroy_t actions_template_destroy; };/* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +116 -0
140 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 6/13 net/mlx5: add table management
In reply to Suanming MouFlow table is a group of flows with the same matching criteria and the same actions defined for them. The table defines rules that have the same matching fields but with different matching values. For example, matching on 5 tuple, the table will be (IPv4 source + IPv4 dest + s_port + d_port + next_proto) while the values for each rule will be different.
The templates' relevant matching criteria and action instances will be created in the table creation and saved in the table. As table attributes indicate the supported flow number, the flow memory will also be allocated at the same time.
This commit adds the table management functions.
--- drivers/net/mlx5/mlx5.c | 45 ++- drivers/net/mlx5/mlx5.h | 21 +- drivers/net/mlx5/mlx5_flow.c | 81 +++++ drivers/net/mlx5/mlx5_flow.h | 72 +++++ drivers/net/mlx5/mlx5_flow_hw.c | 522 ++++++++++++++++++++++++++++++++ 5 files changed, 735 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.c +41 -4
@@ -1354,12 +1354,46 @@ void mlx5_free_table_hash_list(struct mlx5_priv *priv) {struct mlx5_dev_ctx_shared *sh = priv->sh; - -if (!sh->flow_tbls) +struct mlx5_hlist **tbls = (priv->config.dv_flow_en ==2 ) ? + &sh->groups : &sh->flow_tbls; +if (*tbls ==NULL )return ; -mlx5_hlist_destroy (sh->flow_tbls); - sh->flow_tbls =NULL ; +mlx5_hlist_destroy (*tbls); + *tbls =NULL ; +} + +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +/** +* Allocate HW steering group hash list. +* +* @param[in] priv +* Pointer to the private device data structure. +*/ +static int +mlx5_alloc_hw_group_hash_list (struct mlx5_priv *priv) +{ +int err =0 ; +struct mlx5_dev_ctx_shared *sh = priv->sh; +char s[MLX5_NAME_SIZE ]; + +MLX5_ASSERT (sh); +snprintf (s,sizeof (s),"%s_flow_groups" , priv->sh->ibdev_name); + sh->groups =mlx5_hlist_create + (s,MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE , + false, true, sh, + flow_hw_grp_create_cb, + flow_hw_grp_match_cb, + flow_hw_grp_remove_cb, + flow_hw_grp_clone_cb, + flow_hw_grp_clone_free_cb); +if (!sh->groups) { +DRV_LOG (ERR ,"flow groups with hash creation failed." ); + err =ENOMEM ; + } +return err; } +#endif +/** * Initialize flow table hash list and create the root tables entry
@@ -1375,11 +1409,14 @@ int mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused) {int err =0 ; +/* Tables are only used in DV and DR modes. */ #if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )struct mlx5_dev_ctx_shared *sh = priv->sh;char s[MLX5_NAME_SIZE ]; +if (priv->config.dv_flow_en ==2 ) +return mlx5_alloc_hw_group_hash_list (priv);MLX5_ASSERT (sh);snprintf (s,sizeof (s),"%s_flow_table" , priv->sh->ibdev_name); sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
drivers/net/mlx5/mlx5.h +19 -2
@@ -62,7 +62,9 @@ enum mlx5_ipool_index {MLX5_IPOOL_PUSH_VLAN ,/* Pool for push vlan resource. */ MLX5_IPOOL_TAG ,/* Pool for tag resource. */ MLX5_IPOOL_PORT_ID ,/* Pool for port id resource. */ -MLX5_IPOOL_JUMP ,/* Pool for jump resource. */ +MLX5_IPOOL_JUMP ,/* Pool for SWS jump resource. */ +/* Pool for HWS group. Jump action will be created internally. */ +MLX5_IPOOL_HW_GRP =MLX5_IPOOL_JUMP ,MLX5_IPOOL_SAMPLE ,/* Pool for sample resource. */ MLX5_IPOOL_DEST_ARRAY ,/* Pool for destination array resource. */ MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -106,6 +108,13 @@ enum mlx5_delay_drop_mode { MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */ }; +/* The HWS action type root/non-root. */ +enum mlx5_hw_action_flag_type { +MLX5_HW_ACTION_FLAG_ROOT ,/* Root action. */ +MLX5_HW_ACTION_FLAG_NONE_ROOT ,/* Non-root ation. */ +MLX5_HW_ACTION_FLAG_MAX ,/* Maximum action flag. */ +}; +/* Hlist and list callback context. */ struct mlx5_flow_cb_ctx {struct rte_eth_dev *dev;
@@ -1203,7 +1212,10 @@ struct mlx5_dev_ctx_shared {rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX ];/* UAR same-page access control required in 32bit implementations. */ #endif -struct mlx5_hlist *flow_tbls; +union { +struct mlx5_hlist *flow_tbls;/* SWS flow table. */ +struct mlx5_hlist *groups;/* HWS flow group. */ + };struct mlx5_flow_tunnel_hub *tunnel_hub;/* Direct Rules tables for FDB, NIC TX+RX */ void *dr_drop_action;/* Pointer to DR drop action, any domain. */
@@ -1508,6 +1520,11 @@ struct mlx5_priv {uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */ struct mlx5_hw_q *hw_q; +/* HW steering rte flow table list header. */ + LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl; +/* HW steering global drop action. */ +struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] + [MLX5DR_TABLE_TYPE_MAX ]; };#define PORT_ID (priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.c +81 -0
99 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +72 -0
96 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_hw.c +522 -0
571 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 7/13 net/mlx5: add basic flow queue operation
In reply to Suanming MouThe HW steering uses queue-based flow rules management mechanism. The matcher and part of the actions have been prepared during flow table creation. Some left actions will be constructed during flow creation if needed.
A flow postpone attribute bit decribles if flow creation/destruction should be applied to the HW directly. An extra drain function has also been prepared to force push all the cached flows to the HW.
Once the flow has been applied to the HW, the pull function will be called to get the enqueued creation/destruction flows.
The DR rule flow memory is represented in PMD layer instead of allocating from HW steering layer. While destroying the flow, the flow rule memory can only be freed after the CQE received.
The HW queue job descriptor is currently introduced to convey the flow information and operation type between the flow insertion/destruction in the pull function.
This commit adds the basic flow queue operation for:rte_flow_q_flow_create(); rte_flow_q_flow_destroy(); rte_flow_q_push();rte_flow_q_pull();
--- drivers/net/mlx5/mlx5.h | 2 +- drivers/net/mlx5/mlx5_flow.c | 158 ++++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 39 +++++ drivers/net/mlx5/mlx5_flow_hw.c | 278 +++++++++++++++++++++++++++++++- 4 files changed, 475 insertions(+), 2 deletions(-)
drivers/net/mlx5/mlx5.h +1 -1
@@ -342,7 +342,7 @@ enum {/* HW steering flow management job descriptor. */ struct mlx5_hw_q_job {uint32_t type;/* Job type. */ -struct rte_flow *flow;/* Flow attached to the job. */ +struct rte_flow_hw *flow;/* Flow attached to the job. */ void *user_data;/* Job user data. */ };
drivers/net/mlx5/mlx5_flow.c +158 -0
176 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +39 -0
@@ -1015,6 +1015,13 @@ struct rte_flow {uint32_t geneve_tlv_option;/**< Holds Geneve TLV option id. > */ } __rte_packed; +/* HWS flow struct. */ +struct rte_flow_hw { +uint32_t idx;/* Flow index from indexed pool. */ +struct rte_flow_template_table *table;/* The table flow allcated from. */ +struct mlx5dr_rule rule;/* HWS layer data struct. */ +} __rte_packed; +/* Flow item template struct. */ struct rte_flow_pattern_template {LIST_ENTRY (rte_flow_pattern_template) next;
@@ -1366,6 +1373,32 @@ typedef int (*mlx5_flow_table_destroy_t) (struct rte_eth_dev *dev, struct rte_flow_template_table *table, struct rte_flow_error *error); +typedef struct rte_flow *(*mlx5_flow_q_flow_create_t ) + (struct rte_eth_dev *dev, +uint32_t queue, +const struct rte_flow_q_ops_attr *attr, +struct rte_flow_template_table *table, +const struct rte_flow_item items[], +uint8_t pattern_template_index, +const struct rte_flow_action actions[], +uint8_t action_template_index, +struct rte_flow_error *error); +typedef int (*mlx5_flow_q_flow_destroy_t ) + (struct rte_eth_dev *dev, +uint32_t queue, +const struct rte_flow_q_ops_attr *attr, +struct rte_flow *flow, +struct rte_flow_error *error); +typedef int (*mlx5_flow_q_pull_t ) + (struct rte_eth_dev *dev, +uint32_t queue, +struct rte_flow_q_op_res res[], +uint16_t n_res, +struct rte_flow_error *error); +typedef int (*mlx5_flow_q_push_t ) + (struct rte_eth_dev *dev, +uint32_t queue, +struct rte_flow_error *error);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1411,6 +1444,10 @@ struct mlx5_flow_driver_ops {mlx5_flow_actions_template_destroy_t actions_template_destroy;mlx5_flow_table_create_t template_table_create;mlx5_flow_table_destroy_t template_table_destroy; +mlx5_flow_q_flow_create_t q_flow_create; +mlx5_flow_q_flow_destroy_t q_flow_destroy; +mlx5_flow_q_pull_t q_pull; +mlx5_flow_q_push_t q_push; };/* mlx5_flow.c */
@@ -1581,6 +1618,8 @@ mlx5_translate_tunnel_etypes(uint64_t pattern_flags)return 0 ; } +int flow_hw_q_flow_flush (struct rte_eth_dev *dev, +struct rte_flow_error *error); int mlx5_flow_group_to_table(struct rte_eth_dev *dev, const struct mlx5_flow_tunnel *tunnel, uint32_t group, uint32_t *table,
drivers/net/mlx5/mlx5_flow_hw.c +277 -1
302 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 8/13 net/mlx5: add flow flush function
In reply to Suanming MouIn case port restarting, all created flows should be flushed. This commit adds the flow flush helper function.
--- drivers/net/mlx5/mlx5_flow.c | 8 +++ drivers/net/mlx5/mlx5_flow_hw.c | 117 ++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+)
drivers/net/mlx5/mlx5_flow.c +8 -0
@@ -6991,6 +6991,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,uint32_t num_flushed =0 , fidx =1 ;struct rte_flow *flow; +#ifdef HAVE_IBV_FLOW_DV_SUPPORT +if (priv->config.dv_flow_en ==2 && + type ==MLX5_FLOW_TYPE_GEN ) { +flow_hw_q_flow_flush (dev,NULL ); +return ; + } +#endif + MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {flow_list_destroy (dev, type, fidx); num_flushed++;
drivers/net/mlx5/mlx5_flow_hw.c +117 -0
123 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 9/13 net/mlx5: add flow jump action
In reply to Suanming MouJump action connects different level of flow tables as a complete data flow.
A new action construct data struct is also added in this commit to help handle the dynamic actions.
--- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_flow.h | 25 ++- drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++--- 3 files changed, 275 insertions(+), 21 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
@@ -1525,6 +1525,7 @@ struct mlx5_priv {/* HW steering global drop action. */ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] [MLX5DR_TABLE_TYPE_MAX ]; +struct mlx5_indexed_pool *acts_ipool;/* Action data indexed pool. */ };#define PORT_ID (priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.h +24 -1
@@ -1018,10 +1018,25 @@ struct rte_flow {/* HWS flow struct. */ struct rte_flow_hw {uint32_t idx;/* Flow index from indexed pool. */ +uint32_t fate_type;/* Fate action type. */ +union { +/* Jump action. */ +struct mlx5_hw_jump_action *jump; + };struct rte_flow_template_table *table;/* The table flow allcated from. */ struct mlx5dr_rule rule;/* HWS layer data struct. */ } __rte_packed; +/* rte flow action translate to DR action struct. */ +struct mlx5_action_construct_data { +LIST_ENTRY (mlx5_action_construct_data) next; +/* Ensure the action types are matched. */ +int type; +uint32_t idx;/* Data index. */ +uint16_t action_src;/* rte_flow_action src offset. */ +uint16_t action_dst;/* mlx5dr_rule_action dst offset. */ +}; +/* Flow item template struct. */ struct rte_flow_pattern_template {LIST_ENTRY (rte_flow_pattern_template) next;
@@ -1054,9 +1069,17 @@ struct mlx5_hw_jump_action {struct mlx5dr_action *hws_action; }; +/* The maximum actions support in the flow. */ +#define MLX5_HW_MAX_ACTS 16 +/* DR action set struct. */ struct mlx5_hw_actions { -struct mlx5dr_action *drop;/* Drop action. */ +/* Dynamic action list. */ + LIST_HEAD(act_list, mlx5_action_construct_data) act_list; +struct mlx5_hw_jump_action *jump;/* Jump action. */ +uint32_t acts_num:4 ;/* Total action number. */ +/* Translated DR action array from action template. */ +struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ]; };/* mlx5 action template struct. */
drivers/net/mlx5/mlx5_flow_hw.c +250 -20
399 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 10/13 net/mlx5: add queue and RSS action
In reply to Suanming MouThis commit adds the queue and RSS action. Similar to the jump action, dynamic ones will be added to the action construct list.
Due to the queue and RSS action in template should not be destroyed during port restart, the actions are created with standalone indirect table as indirect action does. When port stops, detaches the indirect table from action, when port starts, attaches the indirect table back to the action.
One more change is made to accelerate the action creation. Currently the mlx5_hrxq_get() function returns the object index instead of object pointer. This introduced an extra converting the index to the object by calling mlx5_ipool_get() in most of the case. And that extra converting hurts multi-thread performance since mlx5_ipool_get() uses the global lock inside. As the hash Rx queue object itself also contains the index, returns the object directly will achieve better performance without the global lock.
--- drivers/net/mlx5/linux/mlx5_os.c | 18 ++-- drivers/net/mlx5/mlx5.h | 4 + drivers/net/mlx5/mlx5_devx.c | 10 ++ drivers/net/mlx5/mlx5_flow.c | 38 +++----- drivers/net/mlx5/mlx5_flow.h | 7 ++ drivers/net/mlx5/mlx5_flow_dv.c | 150 ++++++++++++++--------------- drivers/net/mlx5/mlx5_flow_hw.c | 101 +++++++++++++++++++ drivers/net/mlx5/mlx5_flow_verbs.c | 7 +- drivers/net/mlx5/mlx5_rx.h | 9 +- drivers/net/mlx5/mlx5_rxq.c | 78 +++++++++------ 10 files changed, 271 insertions(+), 151 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +9 -9
@@ -1714,6 +1714,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->drop_queue.hrxq =mlx5_drop_action_create (eth_dev);if (!priv->drop_queue.hrxq) goto error; + priv->hrxqs =mlx5_list_create ("hrxq" , eth_dev, true, + mlx5_hrxq_create_cb, + mlx5_hrxq_match_cb, + mlx5_hrxq_remove_cb, + mlx5_hrxq_clone_cb, + mlx5_hrxq_clone_free_cb); +if (!priv->hrxqs) + goto error; +rte_rwlock_init (&priv->ind_tbls_lock);if (priv->config.dv_flow_en ==2 )return eth_dev;/* Port representor shares the same max priority with pf port. */
@@ -1744,15 +1753,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, err =ENOTSUP ; goto error; } - priv->hrxqs =mlx5_list_create ("hrxq" , eth_dev, true, - mlx5_hrxq_create_cb, - mlx5_hrxq_match_cb, - mlx5_hrxq_remove_cb, - mlx5_hrxq_clone_cb, - mlx5_hrxq_clone_free_cb); -if (!priv->hrxqs) - goto error; -rte_rwlock_init (&priv->ind_tbls_lock);/* Query availability of metadata reg_c's. */ if (!priv->sh->metadata_regc_check_flag) { err =mlx5_flow_discover_mreg_c (eth_dev);
drivers/net/mlx5/mlx5.h +4 -0
@@ -1286,6 +1286,7 @@ struct mlx5_flow_rss_desc {uint64_t hash_fields;/* Verbs Hash fields. */ uint8_t key[MLX5_RSS_HASH_KEY_LEN ];/**< RSS hash key. */ uint32_t key_len;/**< RSS hash key len. */ +uint32_t hws_flags;/**< HW steering action. */ uint32_t tunnel;/**< Queue in tunnel. */ uint32_t shared_rss;/**< Shared RSS index. */ struct mlx5_ind_table_obj *ind_tbl;
@@ -1347,6 +1348,7 @@ struct mlx5_hrxq {#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )void *action;/* DV QP action pointer. */ #endif +uint32_t hws_flags;/* Hw steering flags. */ uint64_t hash_fields;/* Verbs Hash fields. */ uint32_t rss_key_len;/* Hash key length in bytes. */ uint32_t idx;/* Hash Rx queue index. */
@@ -1477,6 +1479,8 @@ struct mlx5_priv { LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. *//* Indirection tables. */ LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls; +/* Standalone indirect tables. */ + LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;/* Pointer to next element. */ rte_rwlock_t ind_tbls_lock;uint32_t refcnt;/**< Reference counter. */
drivers/net/mlx5/mlx5_devx.c +10 -0
@@ -807,6 +807,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq, goto error; }#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +if (hrxq->hws_flags) { + hrxq->action =mlx5dr_action_create_dest_tir + (priv->dr_ctx, + (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags); +if (!hrxq->action) + goto error; +return 0 ; + }if (mlx5_flow_os_create_flow_action_dest_devx_tir (hrxq->tir, &hrxq->action)) { rte_errno = errno;
@@ -1042,6 +1050,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)DRV_LOG (ERR ,"Cannot create drop RX queue" );return ret; } +if (priv->config.dv_flow_en ==2 ) +return 0 ;/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */ ret =mlx5_devx_ind_table_new (dev,0 , hrxq->ind_table);if (ret !=0 ) {
drivers/net/mlx5/mlx5_flow.c +12 -26
@@ -9302,14 +9302,10 @@ int mlx5_action_handle_attach(struct rte_eth_dev *dev) {struct mlx5_priv *priv = dev->data->dev_private; -struct mlx5_indexed_pool *ipool = - priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS ]; -struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;int ret =0 ; -uint32_t idx; +struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {const char *message;uint32_t queue_idx;
@@ -9325,9 +9321,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev) }if (ret !=0 )return ret; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { ret =mlx5_ind_table_obj_attach (dev, ind_tbl);if (ret !=0 ) { DRV_LOG(ERR, "Port %u could not attach "
@@ -9336,13 +9330,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev) goto error; } } +return 0 ; error: - shared_rss_last = shared_rss; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - -if (shared_rss == shared_rss_last) + ind_tbl_last = ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { +if (ind_tbl == ind_tbl_last)break ;if (mlx5_ind_table_obj_detach (dev, ind_tbl) !=0 ) DRV_LOG(CRIT, "Port %u could not detach "
@@ -9365,15 +9358,10 @@ int mlx5_action_handle_detach(struct rte_eth_dev *dev) {struct mlx5_priv *priv = dev->data->dev_private; -struct mlx5_indexed_pool *ipool = - priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS ]; -struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;int ret =0 ; -uint32_t idx; - - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; +struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { ret =mlx5_ind_table_obj_detach (dev, ind_tbl);if (ret !=0 ) { DRV_LOG(ERR, "Port %u could not detach "
@@ -9384,11 +9372,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev) }return 0 ; error: - shared_rss_last = shared_rss; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - -if (shared_rss == shared_rss_last) + ind_tbl_last = ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { +if (ind_tbl == ind_tbl_last)break ;if (mlx5_ind_table_obj_attach (dev, ind_tbl) !=0 ) DRV_LOG(CRIT, "Port %u could not attach "
drivers/net/mlx5/mlx5_flow.h +7 -0
@@ -1022,6 +1022,7 @@ struct rte_flow_hw {union {/* Jump action. */ struct mlx5_hw_jump_action *jump; +struct mlx5_hrxq *hrxq;/* TIR action. */ };struct rte_flow_template_table *table;/* The table flow allcated from. */ struct mlx5dr_rule rule;/* HWS layer data struct. */
@@ -1077,6 +1078,7 @@ struct mlx5_hw_actions {/* Dynamic action list. */ LIST_HEAD(act_list, mlx5_action_construct_data) act_list;struct mlx5_hw_jump_action *jump;/* Jump action. */ +struct mlx5_hrxq *tir;/* TIR action. */ uint32_t acts_num:4; /* Total action number. *//* Translated DR action array from action template. */ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ];
@@ -1907,6 +1909,11 @@ int flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx,int flow_dv_query_count (struct rte_eth_dev *dev,uint32_t cnt_idx,void *data,struct rte_flow_error *error); +void flow_dv_hashfields_set (uint64_t item_flags, +struct mlx5_flow_rss_desc *rss_desc, +uint64_t *hash_fields); +void flow_dv_action_rss_l34_hash_adjust (uint64_t rss_types, +uint64_t *hash_field);struct mlx5_list_entry *flow_hw_grp_create_cb (void *tool_ctx,void *cb_ctx);void flow_hw_grp_remove_cb (void *tool_ctx,struct mlx5_list_entry *entry);
drivers/net/mlx5/mlx5_flow_dv.c +70 -80
336 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_hw.c +101 -0
149 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_verbs.c +2 -5
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,MLX5_ASSERT (priv->drop_queue.hrxq); hrxq = priv->drop_queue.hrxq; }else { -uint32_t hrxq_idx;struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;MLX5_ASSERT (rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow, rss_desc->tunnel = !!(handle->layers &MLX5_FLOW_LAYER_TUNNEL ); rss_desc->shared_rss =0 ; - hrxq_idx =mlx5_hrxq_get (dev, rss_desc); - hrxq =mlx5_ipool_get (priv->sh->ipool[MLX5_IPOOL_HRXQ ], - hrxq_idx); + hrxq =mlx5_hrxq_get (dev, rss_desc);if (!hrxq) { rte_flow_error_set (error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow, "cannot get hash queue"); goto error; } - handle->rix_hrxq = hrxq_idx; + handle->rix_hrxq = hrxq->idx; }MLX5_ASSERT (hrxq); handle->drv_flow = mlx5_glue->create_flow
drivers/net/mlx5/mlx5_rx.h +7 -2
@@ -225,9 +225,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);struct mlx5_ind_table_obj *mlx5_ind_table_obj_get (struct rte_eth_dev *dev,const uint16_t *queues,uint32_t queues_n); +struct mlx5_ind_table_obj *mlx5_ind_table_obj_new (struct rte_eth_dev *dev, +const uint16_t *queues, +uint32_t queues_n, +bool standalone, +bool ref_qs);int mlx5_ind_table_obj_release (struct rte_eth_dev *dev,struct mlx5_ind_table_obj *ind_tbl, -bool standalone,bool deref_rxqs); int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev, struct mlx5_ind_table_obj *ind_tbl,
@@ -250,8 +254,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx, void *cb_ctx __rte_unused); void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,struct mlx5_list_entry *entry); -uint32_t mlx5_hrxq_get (struct rte_eth_dev *dev, +struct mlx5_hrxq *mlx5_hrxq_get (struct rte_eth_dev *dev,struct mlx5_flow_rss_desc *rss_desc); +int mlx5_hrxq_obj_release (struct rte_eth_dev *dev,struct mlx5_hrxq *hrxq);int mlx5_hrxq_release (struct rte_eth_dev *dev,uint32_t hxrq_idx);uint32_t mlx5_hrxq_verify (struct rte_eth_dev *dev);enum mlx5_rxq_type mlx5_rxq_get_type (struct rte_eth_dev *dev,uint16_t idx);
drivers/net/mlx5/mlx5_rxq.c +49 -29
195 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 11/13 net/mlx5: add mark action
In reply to Suanming MouThe mark action is covered by tag action internally. While it is added the HW will add a tag to the packet. The mark value can be set as fixed or dynamic as the action mask indicates.
--- drivers/net/mlx5/mlx5.h | 3 ++ drivers/net/mlx5/mlx5_flow.h | 1 + drivers/net/mlx5/mlx5_flow_hw.c | 87 ++++++++++++++++++++++++++++++--- 3 files changed, 85 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.h +3 -0
@@ -1529,6 +1529,9 @@ struct mlx5_priv {/* HW steering global drop action. */ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] [MLX5DR_TABLE_TYPE_MAX ]; +/* HW steering global drop action. */ +struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX ] + [MLX5DR_TABLE_TYPE_MAX ];struct mlx5_indexed_pool *acts_ipool;/* Action data indexed pool. */ };
drivers/net/mlx5/mlx5_flow.h +1 -0
@@ -1080,6 +1080,7 @@ struct mlx5_hw_actions {struct mlx5_hw_jump_action *jump;/* Jump action. */ struct mlx5_hrxq *tir;/* TIR action. */ uint32_t acts_num:4; /* Total action number. */ + uint32_t mark:1; /* Indicate the mark action. *//* Translated DR action array from action template. */ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ]; };
drivers/net/mlx5/mlx5_flow_hw.c +81 -6
135 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 12/13 net/mlx5: add indirect action
In reply to Suanming MouHW steering can support indirect action as well. With indirect action, the flow can be created with more flexible shared RSS action selection. This will can save the action template with different RSS actions.
This commit adds the flow queue operation callback for:rte_flow_q_action_handle_create(); rte_flow_q_action_handle_destroy(); rte_flow_q_action_handle_update();
--- drivers/net/mlx5/mlx5_flow.c | 116 +++++++++ drivers/net/mlx5/mlx5_flow.h | 56 +++++ drivers/net/mlx5/mlx5_flow_dv.c | 21 +- drivers/net/mlx5/mlx5_flow_hw.c | 402 +++++++++++++++++++++++++++++++- 4 files changed, 582 insertions(+), 13 deletions(-)
drivers/net/mlx5/mlx5_flow.c +116 -0
@@ -873,6 +873,26 @@ mlx5_flow_q_push(struct rte_eth_dev *dev,uint32_t queue, struct rte_flow_error *error); +static struct rte_flow_action_handle * +mlx5_flow_q_action_handle_create (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_q_ops_attr *attr, +const struct rte_flow_indir_action_conf *conf, +const struct rte_flow_action *action, +struct rte_flow_error *error); + +static int +mlx5_flow_q_action_handle_update (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_q_ops_attr *attr, +struct rte_flow_action_handle *handle, +const void *update, +struct rte_flow_error *error); + +static int +mlx5_flow_q_action_handle_destroy (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_q_ops_attr *attr, +struct rte_flow_action_handle *handle, +struct rte_flow_error *error); +static const struct rte_flow_ops mlx5_flow_ops = { .validate = mlx5_flow_validate, .create = mlx5_flow_create,
@@ -904,6 +924,9 @@ static const struct rte_flow_ops mlx5_flow_ops = { .q_flow_destroy = mlx5_flow_q_flow_destroy, .q_pull = mlx5_flow_q_pull, .q_push = mlx5_flow_q_push, + .q_action_handle_create = mlx5_flow_q_action_handle_create, + .q_action_handle_update = mlx5_flow_q_action_handle_update, + .q_action_handle_destroy = mlx5_flow_q_action_handle_destroy, };/* Tunnel information. */
@@ -8228,6 +8251,99 @@ mlx5_flow_q_push(struct rte_eth_dev *dev,return fops->q_push (dev, queue, error); } +/** +* Create shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] conf +* Indirect action configuration. +* @param[in] action +* rte_flow action detail. +* @param[out] error +* Pointer to error structure. +* +* @return +* Action handle on success, NULL otherwise and rte_errno is set. +*/ +static struct rte_flow_action_handle * +mlx5_flow_q_action_handle_create (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_q_ops_attr *attr, +const struct rte_flow_indir_action_conf *conf, +const struct rte_flow_action *action, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->q_action_create (dev, queue, attr, conf, action, error); +} + +/** +* Update shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] handle +* Action handle to be updated. +* @param[in] update +* Update value. +* @param[out] error +* Pointer to error structure. +* +* @return +* 0 on success, negative value otherwise and rte_errno is set. +*/ +static int +mlx5_flow_q_action_handle_update (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_q_ops_attr *attr, +struct rte_flow_action_handle *handle, +const void *update, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->q_action_update (dev, queue, attr, handle, update, error); +} + +/** +* Destroy shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] handle +* Action handle to be destroyed. +* @param[out] error +* Pointer to error structure. +* +* @return +* 0 on success, negative value otherwise and rte_errno is set. +*/ +static int +mlx5_flow_q_action_handle_destroy (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_q_ops_attr *attr, +struct rte_flow_action_handle *handle, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->q_action_destroy (dev, queue, attr, handle, error); +} +/** * Allocate a new memory for the counter values wrapped by all the needed * management.
drivers/net/mlx5/mlx5_flow.h +56 -0
96 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_dv.c +12 -9
@@ -13837,9 +13837,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,* @return * Valid hash RX queue index, otherwise 0. */ -static uint32_t -__flow_dv_action_rss_hrxq_lookup (struct rte_eth_dev *dev,uint32_t idx, -const uint64_t hash_fields) +uint32_t +flow_dv_action_rss_hrxq_lookup (struct rte_eth_dev *dev,uint32_t idx, +const uint64_t hash_fields) {struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_shared_action_rss *shared_rss =
@@ -13967,7 +13967,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,struct mlx5_hrxq *hrxq =NULL ;uint32_t hrxq_idx; - hrxq_idx =__flow_dv_action_rss_hrxq_lookup (dev, + hrxq_idx =flow_dv_action_rss_hrxq_lookup (dev, rss_desc->shared_rss, dev_flow->hash_fields);if (hrxq_idx)
@@ -14691,6 +14691,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,struct mlx5_shared_action_rss *shared_rss, struct rte_flow_error *error) { + struct mlx5_priv *priv = dev->data->dev_private;struct mlx5_flow_rss_desc rss_desc = {0 };size_t i;int err;
@@ -14711,6 +14712,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,/* Set non-zero value to indicate a shared RSS. */ rss_desc.shared_rss = action_idx; rss_desc.ind_tbl = shared_rss->ind_tbl; +if (priv->config.dv_flow_en ==2 ) + rss_desc.hws_flags =MLX5DR_ACTION_FLAG_HWS_RX ;for (i =0 ; i <MLX5_RSS_HASH_FIELDS_LEN ; i++) {struct mlx5_hrxq *hrxq;uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14902,7 +14905,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,* A valid shared action handle in case of success, NULL otherwise and * rte_errno is set. */ -static struct rte_flow_action_handle * +struct rte_flow_action_handle * flow_dv_action_create(struct rte_eth_dev *dev, const struct rte_flow_indir_action_conf *conf, const struct rte_flow_action *action,
@@ -14972,7 +14975,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_destroy (struct rte_eth_dev *dev,struct rte_flow_action_handle *handle,struct rte_flow_error *error)
@@ -15181,7 +15184,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_update(struct rte_eth_dev *dev, struct rte_flow_action_handle *handle, const void *update,
@@ -15895,7 +15898,7 @@ flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx, "counters are not available"); } -static int +int flow_dv_action_query (struct rte_eth_dev *dev,const struct rte_flow_action_handle *handle,void *data,struct rte_flow_error *error)
@@ -17584,7 +17587,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_validate(struct rte_eth_dev *dev, const struct rte_flow_indir_action_conf *conf, const struct rte_flow_action *action,
drivers/net/mlx5/mlx5_flow_hw.c +398 -4
481 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 13/13 net/mlx5: add header reformat action
In reply to Suanming MouHW steering header reformat action can work under bulk mode. In this case, when create the table, bulk size of header reformat actions will be allocated in low level. Afterwards, when create flow, just simply specify the action index in the bulk and the encapsulation data to the action will be enough.
--- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_flow.h | 21 +++ drivers/net/mlx5/mlx5_flow_dv.c | 4 +- drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++- 4 files changed, 251 insertions(+), 3 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
@@ -344,6 +344,7 @@ struct mlx5_hw_q_job {uint32_t type;/* Job type. */ struct rte_flow_hw *flow;/* Flow attached to the job. */ void *user_data;/* Job user data. */ +uint8_t *encap_data;/* Encap data. */ };/* HW steering job descriptor LIFO header . */
drivers/net/mlx5/mlx5_flow.h +21 -0
@@ -1038,6 +1038,14 @@ struct mlx5_action_construct_data {uint16_t action_src;/* rte_flow_action src offset. */ uint16_t action_dst;/* mlx5dr_rule_action dst offset. */ union { +struct { +/* encap src(item) offset. */ +uint16_t src; +/* encap dst data offset. */ +uint16_t dst; +/* encap data len. */ +uint16_t len; + } encap;struct {uint64_t types;/* RSS hash types. */ uint32_t level;/* RSS level. */
@@ -1079,6 +1087,13 @@ struct mlx5_hw_jump_action {struct mlx5dr_action *hws_action; }; +/* Encap decap action struct. */ +struct mlx5_hw_encap_decap_action { +struct mlx5dr_action *action;/* Action object. */ +size_t data_size;/* Action metadata size. */ +uint8_t data[];/* Action data. */ +}; +/* The maximum actions support in the flow. */ #define MLX5_HW_MAX_ACTS 16
@@ -1088,6 +1103,9 @@ struct mlx5_hw_actions { LIST_HEAD(act_list, mlx5_action_construct_data) act_list;struct mlx5_hw_jump_action *jump;/* Jump action. */ struct mlx5_hrxq *tir;/* TIR action. */ +/* Encap/Decap action. */ +struct mlx5_hw_encap_decap_action *encap_decap; +uint16_t encap_decap_pos;/* Encap/Decap action position. */ uint32_t acts_num:4; /* Total action number. */ uint32_t mark:1; /* Indicate the mark action. *//* Translated DR action array from action template. */
@@ -2021,4 +2039,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,const struct rte_flow_action_handle *handle, void *data, struct rte_flow_error *error); +size_t flow_dv_get_item_hdr_len (const enum rte_flow_item_type item_type); +int flow_dv_convert_encap_data (const struct rte_flow_item *items,uint8_t *buf, +size_t *size,struct rte_flow_error *error);#endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_dv.c +2 -2
@@ -4024,7 +4024,7 @@ flow_dv_push_vlan_action_resource_register* @return * sizeof struct item_type, 0 if void or irrelevant. */ -static size_t +size_t flow_dv_get_item_hdr_len (const enum rte_flow_item_type item_type) {size_t retval;
@@ -4090,7 +4090,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)* @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -static int +int flow_dv_convert_encap_data (const struct rte_flow_item *items,uint8_t *buf,size_t *size,struct rte_flow_error *error) {
drivers/net/mlx5/mlx5_flow_hw.c +227 -1
299 lines not on this page · raw message
Signature · 1 line
2.25.1
v2 cover net/mlx5: add hardware steering
In reply to Suanming MouThe Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion
- security, exposing the low-level steering entries directly to the userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.
*** THIS PATCH SET DEPENDS ON THE NEW RTE_FLOW ASYNC API ***
---
v2:
- New HW steering low-level abstract code added.
- commit message improvement.
- add protection for rte_flow and rte_flow_async callbacks.
- rebase to rte_flow_async v9.
- fix some rte_flow error not filled bugs.
Suanming Mou (14):
net/mlx5: introduce hardware steering operation net/mlx5: add HW steering low-level abstract code net/mlx5: introduce hardware steering enable routine net/mlx5: add port flow configuration net/mlx5: add pattern template management net/mlx5: add action template management net/mlx5: add table management
net/mlx5: add basic flow queue operation net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action
doc/guides/nics/mlx5.rst | 19 +- doc/guides/rel_notes/release_22_03.rst | 6 + drivers/net/mlx5/linux/mlx5_flow_os.h | 1 + drivers/net/mlx5/linux/mlx5_os.c | 22 +- drivers/net/mlx5/meson.build | 2 + drivers/net/mlx5/mlx5.c | 55 +- drivers/net/mlx5/mlx5.h | 66 +- drivers/net/mlx5/mlx5_devx.c | 10 + drivers/net/mlx5/mlx5_dr.c | 383 ++++ drivers/net/mlx5/mlx5_dr.h | 456 +++++ drivers/net/mlx5/mlx5_flow.c | 721 ++++++- drivers/net/mlx5/mlx5_flow.h | 287 +++ drivers/net/mlx5/mlx5_flow_dv.c | 186 +- drivers/net/mlx5/mlx5_flow_hw.c | 2335 +++++++++++++++++++++++ drivers/net/mlx5/mlx5_flow_verbs.c | 7 +- drivers/net/mlx5/mlx5_rx.h | 9 +- drivers/net/mlx5/mlx5_rxq.c | 85 +- drivers/net/mlx5/windows/mlx5_flow_os.h | 1 + 18 files changed, 4471 insertions(+), 180 deletions(-) create mode 100644 drivers/net/mlx5/mlx5_dr.c create mode 100644 drivers/net/mlx5/mlx5_dr.h create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
Signature · 1 line
2.25.1
PATCH v2 1/14 net/mlx5: introduce hardware steering operation
The Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion
- security, exposing the low-level steering entries directly to the userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.
This commit adds the basic driver operation.
--- drivers/net/mlx5/linux/mlx5_flow_os.h | 1 + drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5_flow.c | 1 + drivers/net/mlx5/mlx5_flow.h | 1 + drivers/net/mlx5/mlx5_flow_hw.c | 13 +++++++++++++ drivers/net/mlx5/windows/mlx5_flow_os.h | 1 + 6 files changed, 18 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
drivers/net/mlx5/linux/mlx5_flow_os.h +1 -0
@@ -9,6 +9,7 @@#ifdef HAVE_IBV_FLOW_DV_SUPPORT extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops; +extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;#endif /**
drivers/net/mlx5/meson.build +1 -0
@@ -16,6 +16,7 @@ sources = files( 'mlx5_flow.c', 'mlx5_flow_meter.c', 'mlx5_flow_dv.c', + 'mlx5_flow_hw.c', 'mlx5_flow_aso.c', 'mlx5_flow_flex.c', 'mlx5_mac.c',
drivers/net/mlx5/mlx5_flow.c +1 -0
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = { [MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) [MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops, + [MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops, #endif [MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops, [MLX5_FLOW_TYPE_MAX ] = &mlx5_flow_null_drv_ops
drivers/net/mlx5/mlx5_flow.h +1 -0
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {MLX5_FLOW_TYPE_MIN ,MLX5_FLOW_TYPE_DV ,MLX5_FLOW_TYPE_VERBS , +MLX5_FLOW_TYPE_HW , MLX5_FLOW_TYPE_MAX, };
drivers/net/mlx5/mlx5_flow_hw.c new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright (c) 2022 NVIDIA Corporation & Affiliates +*/ + +#include <rte_flow.h> + +#include "mlx5_flow.h" + +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) + +const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops; + +#endif
drivers/net/mlx5/windows/mlx5_flow_os.h +1 -0
@@ -10,6 +10,7 @@#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops; +extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;#endif /**
Signature · 1 line
2.25.1
PATCH v2 2/14 net/mlx5: add HW steering low-level abstract code
In reply to Suanming MouThe HW steering low-level implementation will be added later in another patch series. To avoid the linkage issues the abstract stub replacement is provided currently.
--- drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5_dr.c | 383 +++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_dr.h | 456 +++++++++++++++++++++++++++++++++++ 3 files changed, 840 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_dr.c create mode 100644 drivers/net/mlx5/mlx5_dr.h
drivers/net/mlx5/meson.build +1 -0
@@ -14,6 +14,7 @@ sources = files( 'mlx5.c', 'mlx5_ethdev.c', 'mlx5_flow.c', + 'mlx5_dr.c', 'mlx5_flow_meter.c', 'mlx5_flow_dv.c', 'mlx5_flow_hw.c',
drivers/net/mlx5/mlx5_dr.c new file +383 -0
383 lines not on this page · raw message
drivers/net/mlx5/mlx5_dr.h new file +456 -0
456 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 3/14 net/mlx5: introduce hardware steering enable routine
In reply to Suanming MouThe new hardware steering engine relies on using dedicated steering WQEs instead of direct writing to the low-level steering table entries directly. In the first introduce implementation the hardware steering engine supports the new queue based Flow API, the existing synchronous non-queue based Flow API is not supported.
A new dv_flow_en value 2 is added to manage mlx5 PMD steering engine:
dv_flow_en rte_flow API rte_flow_async API
------------------------------------------------
0 support not support
1 support not support
2 not support support
This commit introduces the extra dv_flow_en = 2 to specify the new flow initialize and manage operation routine.
--- doc/guides/nics/mlx5.rst | 13 ++++++++++--- drivers/net/mlx5/linux/mlx5_os.c | 4 ++++ drivers/net/mlx5/mlx5.c | 7 ++++++- drivers/net/mlx5/mlx5.h | 3 ++- drivers/net/mlx5/mlx5_flow.c | 22 ++++++++++++++++++++++ 5 files changed, 44 insertions(+), 5 deletions(-)
doc/guides/nics/mlx5.rst +10 -3
@@ -1042,10 +1042,17 @@ Driver options - ``dv_flow_en`` parameter [int] - A nonzero value enables the DV flow steering assuming it is supported - by the driver (RDMA Core library version is rdma-core-24.0 or higher). + Value 0 means legacy Verbs flow offloading. - Enabled by default if supported. + Value 1 enables the DV flow steering assuming it is supported by the + driver (RDMA Core library version is rdma-core-24.0 or higher). + + Value 2 enables the WQE based hardware steering. In this mode only + the queue-based rte_flow_q flow management is supported. + + Configured by default to 1 DV flow steering if the driver(RDMA CORE library) + supported. Otherwise, the value will be 0 which indicates legacy Verbs flow + offloading. - ``dv_esw_en`` parameter [int]
drivers/net/mlx5/linux/mlx5_os.c +4 -0
@@ -482,6 +482,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv) err =mlx5_alloc_table_hash_list (priv);if (err) goto error; +if (priv->sh->config.dv_flow_en ==2 ) +return 0 ;/* The resources below are only valid with DV support. */ #ifdef HAVE_IBV_FLOW_DV_SUPPORT /* Init port id action list. */
@@ -1519,6 +1521,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->drop_queue.hrxq =mlx5_drop_action_create (eth_dev);if (!priv->drop_queue.hrxq) goto error; +if (priv->sh->config.dv_flow_en ==2 ) +return eth_dev;/* Port representor shares the same max priority with pf port. */ if (!priv->sh->flow_priority_check_flag) {/* Supported Verbs flow priority number detection. */
drivers/net/mlx5/mlx5.c +6 -1
@@ -1199,7 +1199,12 @@ mlx5_dev_args_check_handler(const char *key, const char *val, void *opaque) }else if (strcmp (MLX5_DV_ESW_EN , key) ==0 ) { config->dv_esw_en = !!tmp; }else if (strcmp (MLX5_DV_FLOW_EN , key) ==0 ) { - config->dv_flow_en = !!tmp; +if (tmp >2 ) { +DRV_LOG (ERR ,"Invalid %s parameter." , key); + rte_errno =EINVAL ; +return -rte_errno; + } + config->dv_flow_en = tmp; }else if (strcmp (MLX5_DV_XMETA_EN , key) ==0 ) { if (tmp != MLX5_XMETA_MODE_LEGACY && tmp != MLX5_XMETA_MODE_META16 &&
drivers/net/mlx5/mlx5.h +2 -1
@@ -287,7 +287,8 @@ struct mlx5_sh_config {int tx_skew;/* Tx scheduling skew between WQE and data on wire. */ uint32_t reclaim_mode:2; /* Memory reclaim mode. */ uint32_t dv_esw_en:1; /* Enable E-Switch DV flow. */ - uint32_t dv_flow_en:1; /* Enable DV flow. */ +/* Enable DV flow. 1 means SW steering, 2 means HW steering. */ + unsigned int dv_flow_en:2; uint32_t dv_xmeta_en:2; /* Enable extensive flow metadata. */ uint32_t dv_miss_info:1; /* Restore packet after partial hw miss. */ uint32_t l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
drivers/net/mlx5/mlx5_flow.c +22 -0
@@ -6838,6 +6838,15 @@ mlx5_flow_create(struct rte_eth_dev *dev,const struct rte_flow_action actions[], struct rte_flow_error *error) { + struct mlx5_priv *priv = dev->data->dev_private; + +if (priv->sh->config.dv_flow_en ==2 ) { +rte_flow_error_set (error,ENOTSUP , +RTE_FLOW_ERROR_TYPE_UNSPECIFIED , +NULL , +"Flow non-Q creation not supported" ); +return NULL ; + }/* * If the device is not started yet, it is not allowed to created a * flow from application. PMD default flows and traffic control flows
@@ -7334,6 +7343,13 @@ mlx5_flow_destroy(struct rte_eth_dev *dev,struct rte_flow *flow, struct rte_flow_error *error __rte_unused) { + struct mlx5_priv *priv = dev->data->dev_private; + +if (priv->sh->config.dv_flow_en ==2 ) +return rte_flow_error_set (error,ENOTSUP , +RTE_FLOW_ERROR_TYPE_UNSPECIFIED , +NULL , +"Flow non-Q destruction not supported" );flow_list_destroy (dev,MLX5_FLOW_TYPE_GEN , (uintptr_t )(void *)flow);return 0 ;
@@ -7431,7 +7447,13 @@ mlx5_flow_query(struct rte_eth_dev *dev, struct rte_flow_error *error) {int ret; +struct mlx5_priv *priv = dev->data->dev_private; +if (priv->sh->config.dv_flow_en ==2 ) +return rte_flow_error_set (error,ENOTSUP , +RTE_FLOW_ERROR_TYPE_UNSPECIFIED , +NULL , +"Flow non-Q query not supported" ); ret =flow_drv_query (dev, (uintptr_t )(void *)flow, actions, data, error);if (ret <0 )
Signature · 1 line
2.25.1
PATCH v2 4/14 net/mlx5: add port flow configuration
In reply to Suanming MouThe hardware steering is backend to support rte_flow_async API in mlx5 PMD. The port configuration function creates the queues and needed flow management resources.
The PMD layer configuration function allocates the queues' context and per-queue job descriptor pool. The job descriptor pool size is equal to the queue size, and the job descriptors will be popped from pool with LIFO strategy to convey the flow information during flow insertion/destruction. Then, while polling the queued operation result, the flow information will be extracted from the job descriptor and the descriptor will be pushed back to the LIFO pool.
The commit creates the flow port queues and the job descriptor pools.
--- drivers/net/mlx5/mlx5.c | 3 + drivers/net/mlx5/mlx5.h | 30 +++++- drivers/net/mlx5/mlx5_flow.c | 86 +++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 15 +++ drivers/net/mlx5/mlx5_flow_hw.c | 159 ++++++++++++++++++++++++++++++++ 5 files changed, 292 insertions(+), 1 deletion(-)
drivers/net/mlx5/mlx5.c +3 -0
@@ -1820,6 +1820,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)/* Free the eCPRI flex parser resource. */ mlx5_flex_parser_ecpri_release (dev);mlx5_flex_item_port_cleanup (dev); +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +flow_hw_resource_release (dev); +#endif if (priv->rxq_privs !=NULL ) {/* XXX race condition if mlx5_rx_burst() is still running. */ rte_delay_us_sleep (1000 );
drivers/net/mlx5/mlx5.h +29 -1
@@ -33,7 +33,9 @@#include "mlx5_utils.h" #include "mlx5_os.h" #include "mlx5_autoconf.h" - +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +#include "mlx5_dr.h" +#endif #define MLX5_SH (dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
@@ -320,6 +322,26 @@ struct mlx5_lb_ctx {uint16_t refcnt;/* Reference count for representors. */ }; +/* HW steering queue job descriptor type. */ +enum { +MLX5_HW_Q_JOB_TYPE_CREATE ,/* Flow create job type. */ +MLX5_HW_Q_JOB_TYPE_DESTROY ,/* Flow destroy job type. */ +}; + +/* HW steering flow management job descriptor. */ +struct mlx5_hw_q_job { +uint32_t type;/* Job type. */ +struct rte_flow *flow;/* Flow attached to the job. */ +void *user_data;/* Job user data. */ +}; + +/* HW steering job descriptor LIFO pool. */ +struct mlx5_hw_q { +uint32_t job_idx;/* Free job index. */ +uint32_t size;/* LIFO size. */ +struct mlx5_hw_q_job **job;/* LIFO header. */ +} __rte_cache_aligned; +#define MLX5_COUNTERS_PER_POOL 512#define MLX5_MAX_PENDING_QUERIES 4#define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1476,6 +1498,12 @@ struct mlx5_priv {struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM ];/* Flex items have been created on the port. */ uint32_t flex_item_map;/* Map of allocated flex item elements. */ +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ +uint32_t nb_queue;/* HW steering queue number. */ +/* HW steering queue polling mechanism job descriptor LIFO. */ +struct mlx5_hw_q *hw_q; +#endif };#define PORT_ID (priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.c +86 -0
110 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +15 -0
@@ -1257,6 +1257,17 @@ typedef int (*mlx5_flow_item_update_t)const struct rte_flow_item_flex_handle *handle, const struct rte_flow_item_flex_conf *conf, struct rte_flow_error *error); +typedef int (*mlx5_flow_info_get_t ) + (struct rte_eth_dev *dev, +struct rte_flow_port_info *port_info, +struct rte_flow_queue_info *queue_info, +struct rte_flow_error *error); +typedef int (*mlx5_flow_port_configure_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_port_attr *port_attr, +uint16_t nb_queue, +const struct rte_flow_queue_attr *queue_attr[], +struct rte_flow_error *err);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1295,6 +1306,8 @@ struct mlx5_flow_driver_ops {mlx5_flow_item_create_t item_create;mlx5_flow_item_release_t item_release;mlx5_flow_item_update_t item_update; +mlx5_flow_info_get_t info_get; +mlx5_flow_port_configure_t configure; };/* mlx5_flow.c */
@@ -1767,4 +1780,6 @@ const struct mlx5_flow_tunnel * mlx5_get_tof(const struct rte_flow_item *items, const struct rte_flow_action *actions, enum mlx5_tof_rule_type *rule_type); +void +flow_hw_resource_release (struct rte_eth_dev *dev);#endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_hw.c +159 -0
169 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 5/14 net/mlx5: add pattern template management
In reply to Suanming MouThe pattern template defines flows that have the same matching fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be (eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while the values for each rule will be different.
Due to the pattern template can be used in different domains, the items will only be cached in pattern template create stage, while the template is binded to a dedicated table, the HW criteria will be created and saved to the table. The pattern templates can be used by multiple tables. But different tables create the same criteria and will not share the matcher between each other in order to have better performance.
This commit adds pattern template management.
--- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_flow.c | 76 +++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 24 +++++++++ drivers/net/mlx5/mlx5_flow_hw.c | 86 +++++++++++++++++++++++++++++++++ 4 files changed, 188 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
@@ -1499,6 +1499,8 @@ struct mlx5_priv {/* Flex items have been created on the port. */ uint32_t flex_item_map;/* Map of allocated flex item elements. */ #if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +/* Item template list. */ + LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +76 -0
94 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +24 -0
@@ -1015,6 +1015,19 @@ struct rte_flow {uint32_t geneve_tlv_option;/**< Holds Geneve TLV option id. > */ } __rte_packed; +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) + +/* Flow item template struct. */ +struct rte_flow_pattern_template { +LIST_ENTRY (rte_flow_pattern_template) next; +/* Template attributes. */ +struct rte_flow_pattern_template_attr attr; +struct mlx5dr_match_template *mt;/* mlx5 match template. */ +uint32_t refcnt;/* Reference counter. */ +}; + +#endif +/* * Define list of valid combinations of RX Hash fields * (see enum ibv_rx_hash_fields).
@@ -1268,6 +1281,15 @@ typedef int (*mlx5_flow_port_configure_t)uint16_t nb_queue, const struct rte_flow_queue_attr *queue_attr[], struct rte_flow_error *err); +typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_pattern_template_attr *attr, +const struct rte_flow_item items[], +struct rte_flow_error *error); +typedef int (*mlx5_flow_pattern_template_destroy_t ) + (struct rte_eth_dev *dev, +struct rte_flow_pattern_template *template, +struct rte_flow_error *error);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1308,6 +1330,8 @@ struct mlx5_flow_driver_ops {mlx5_flow_item_update_t item_update;mlx5_flow_info_get_t info_get;mlx5_flow_port_configure_t configure; +mlx5_flow_pattern_template_create_t pattern_template_create; +mlx5_flow_pattern_template_destroy_t pattern_template_destroy; };/* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +86 -0
107 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 6/14 net/mlx5: add action template management
In reply to Suanming MouThe action template holds a list of action types that will be used together on the same rule. The template's actions instances will be created only when the template bind to the dedicated group. And the created actions will be saved to each individual group in order for best performance. The actions in a group will not be shared with each other unless shared actions are specified.
This commit adds the action template management which stores the flow action template.
--- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_flow.c | 78 +++++++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 22 ++++++ drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++ 4 files changed, 218 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
@@ -1501,6 +1501,8 @@ struct mlx5_priv {#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )/* Item template list. */ LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt; +/* Action template list. */ + LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +78 -0
96 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +22 -0
@@ -1026,6 +1026,16 @@ struct rte_flow_pattern_template { uint32_t refcnt; /* Reference counter. */ }; +/* Flow action template struct. */ +struct rte_flow_actions_template { +LIST_ENTRY (rte_flow_actions_template) next; +/* Template attributes. */ +struct rte_flow_actions_template_attr attr; +struct rte_flow_action *actions;/* Cached flow actions. */ +struct rte_flow_action *masks;/* Cached action masks.*/ +uint32_t refcnt;/* Reference counter. */ +}; +#endif /*
@@ -1290,6 +1300,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t) (struct rte_eth_dev *dev, struct rte_flow_pattern_template *template, struct rte_flow_error *error); +typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_actions_template_attr *attr, +const struct rte_flow_action actions[], +const struct rte_flow_action masks[], +struct rte_flow_error *error); +typedef int (*mlx5_flow_actions_template_destroy_t ) + (struct rte_eth_dev *dev, +struct rte_flow_actions_template *template, +struct rte_flow_error *error);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1332,6 +1352,8 @@ struct mlx5_flow_driver_ops {mlx5_flow_port_configure_t configure;mlx5_flow_pattern_template_create_t pattern_template_create;mlx5_flow_pattern_template_destroy_t pattern_template_destroy; +mlx5_flow_actions_template_create_t actions_template_create; +mlx5_flow_actions_template_destroy_t actions_template_destroy; };/* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +116 -0
140 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 7/14 net/mlx5: add table management
In reply to Suanming MouFlow table is a group of flows with the same matching criteria and the same actions defined for them. The table defines rules that have the same matching fields but with different matching values. For example, matching on 5 tuple, the table will be (IPv4 source + IPv4 dest + s_port + d_port + next_proto) while the values for each rule will be different.
The templates' relevant matching criteria and action instances will be created in the table creation and saved in the table. As table attributes indicate the supported flow number, the flow memory will also be allocated at the same time.
This commit adds the table management functions.
--- drivers/net/mlx5/mlx5.c | 45 ++- drivers/net/mlx5/mlx5.h | 21 +- drivers/net/mlx5/mlx5_flow.c | 93 ++++++ drivers/net/mlx5/mlx5_flow.h | 73 +++++ drivers/net/mlx5/mlx5_flow_hw.c | 525 ++++++++++++++++++++++++++++++++ 5 files changed, 751 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.c +41 -4
@@ -1607,12 +1607,46 @@ void mlx5_free_table_hash_list(struct mlx5_priv *priv) {struct mlx5_dev_ctx_shared *sh = priv->sh; - -if (!sh->flow_tbls) +struct mlx5_hlist **tbls = (priv->sh->config.dv_flow_en ==2 ) ? + &sh->groups : &sh->flow_tbls; +if (*tbls ==NULL )return ; -mlx5_hlist_destroy (sh->flow_tbls); - sh->flow_tbls =NULL ; +mlx5_hlist_destroy (*tbls); + *tbls =NULL ; +} + +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +/** +* Allocate HW steering group hash list. +* +* @param[in] priv +* Pointer to the private device data structure. +*/ +static int +mlx5_alloc_hw_group_hash_list (struct mlx5_priv *priv) +{ +int err =0 ; +struct mlx5_dev_ctx_shared *sh = priv->sh; +char s[MLX5_NAME_SIZE ]; + +MLX5_ASSERT (sh); +snprintf (s,sizeof (s),"%s_flow_groups" , priv->sh->ibdev_name); + sh->groups =mlx5_hlist_create + (s,MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE , + false, true, sh, + flow_hw_grp_create_cb, + flow_hw_grp_match_cb, + flow_hw_grp_remove_cb, + flow_hw_grp_clone_cb, + flow_hw_grp_clone_free_cb); +if (!sh->groups) { +DRV_LOG (ERR ,"flow groups with hash creation failed." ); + err =ENOMEM ; + } +return err; } +#endif +/** * Initialize flow table hash list and create the root tables entry
@@ -1628,11 +1662,14 @@ int mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused) {int err =0 ; +/* Tables are only used in DV and DR modes. */ #if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )struct mlx5_dev_ctx_shared *sh = priv->sh;char s[MLX5_NAME_SIZE ]; +if (priv->sh->config.dv_flow_en ==2 ) +return mlx5_alloc_hw_group_hash_list (priv);MLX5_ASSERT (sh);snprintf (s,sizeof (s),"%s_flow_table" , priv->sh->ibdev_name); sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
drivers/net/mlx5/mlx5.h +19 -2
@@ -64,7 +64,9 @@ enum mlx5_ipool_index {MLX5_IPOOL_PUSH_VLAN ,/* Pool for push vlan resource. */ MLX5_IPOOL_TAG ,/* Pool for tag resource. */ MLX5_IPOOL_PORT_ID ,/* Pool for port id resource. */ -MLX5_IPOOL_JUMP ,/* Pool for jump resource. */ +MLX5_IPOOL_JUMP ,/* Pool for SWS jump resource. */ +/* Pool for HWS group. Jump action will be created internally. */ +MLX5_IPOOL_HW_GRP =MLX5_IPOOL_JUMP ,MLX5_IPOOL_SAMPLE ,/* Pool for sample resource. */ MLX5_IPOOL_DEST_ARRAY ,/* Pool for destination array resource. */ MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -108,6 +110,13 @@ enum mlx5_delay_drop_mode { MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */ }; +/* The HWS action type root/non-root. */ +enum mlx5_hw_action_flag_type { +MLX5_HW_ACTION_FLAG_ROOT ,/* Root action. */ +MLX5_HW_ACTION_FLAG_NONE_ROOT ,/* Non-root ation. */ +MLX5_HW_ACTION_FLAG_MAX ,/* Maximum action flag. */ +}; +/* Hlist and list callback context. */ struct mlx5_flow_cb_ctx {struct rte_eth_dev *dev;
@@ -1201,7 +1210,10 @@ struct mlx5_dev_ctx_shared {rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX ];/* UAR same-page access control required in 32bit implementations. */ #endif -struct mlx5_hlist *flow_tbls; +union { +struct mlx5_hlist *flow_tbls;/* SWS flow table. */ +struct mlx5_hlist *groups;/* HWS flow group. */ + };struct mlx5_flow_tunnel_hub *tunnel_hub;/* Direct Rules tables for FDB, NIC TX+RX */ void *dr_drop_action;/* Pointer to DR drop action, any domain. */
@@ -1507,6 +1519,11 @@ struct mlx5_priv {uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */ struct mlx5_hw_q *hw_q; +/* HW steering rte flow table list header. */ + LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl; +/* HW steering global drop action. */ +struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] + [MLX5DR_TABLE_TYPE_MAX ];#endif };
drivers/net/mlx5/mlx5_flow.c +93 -0
111 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +73 -0
97 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_hw.c +525 -0
574 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 8/14 net/mlx5: add basic flow queue operation
In reply to Suanming MouThe HW steering uses async queue-based flow rules management mechanism. The matcher and part of the actions have been prepared during flow table creation. Some remaining actions will be constructed during flow creation if needed.
A flow postpone attribute bit describes if flow management should be applied to the HW directly. An extra push function is provided to force push all the cached flows to the HW.
Once the flow has been applied to the HW, the pull function will be called to get the queued creation/destruction flows.
The DR rule flow memory is represented in PMD layer instead of allocating from HW steering layer. While destroying the flow, the flow rule memory can only be freed after the CQE received.
The HW queue job descriptor is currently introduced to convey the flow information and operation type between the flow insertion/destruction in the pull function.
This commit adds the basic flow queue operation for:rte_flow_async_create(); rte_flow_async_destroy();rte_flow_push();rte_flow_pull();
--- doc/guides/nics/mlx5.rst | 6 + doc/guides/rel_notes/release_22_03.rst | 6 + drivers/net/mlx5/mlx5.h | 2 +- drivers/net/mlx5/mlx5_flow.c | 188 ++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 41 ++++ drivers/net/mlx5/mlx5_flow_hw.c | 292 ++++++++++++++++++++++++- 6 files changed, 533 insertions(+), 2 deletions(-)
doc/guides/nics/mlx5.rst +6 -0
@@ -511,6 +511,12 @@ Limitations from the reference "Clock Queue" completions, the scheduled send timestamps should not be specified with non-zero MSB. + - HW steering: + + - WQE based high scaling and safer flow insertion/destruction. + - Set ``dv_flow_en`` to 2 in order to enable HW steering. + - Async queue-based ``rte_flow_q`` APIs supported only. + Statistics ----------
doc/guides/rel_notes/release_22_03.rst +6 -0
@@ -156,6 +156,12 @@ New Features * Added LED OEM support. +* **Updated Mellanox mlx5 driver.** + + Updated the Mellanox mlx5 driver with new features and improvements, including: + + * Added WQE based hardware steering support with ``rte_flow_q`` API. + * **Added an API for private user data in asymmetric crypto session.** An API was added to get/set an asymmetric crypto session's user data.
drivers/net/mlx5/mlx5.h +1 -1
@@ -340,7 +340,7 @@ enum {/* HW steering flow management job descriptor. */ struct mlx5_hw_q_job {uint32_t type;/* Job type. */ -struct rte_flow *flow;/* Flow attached to the job. */ +struct rte_flow_hw *flow;/* Flow attached to the job. */ void *user_data;/* Job user data. */ };
drivers/net/mlx5/mlx5_flow.c +188 -0
206 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +41 -0
65 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_hw.c +291 -1
316 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 9/14 net/mlx5: add flow flush function
In reply to Suanming MouIn case port is being stopped, all created flows should be flushed. This commit adds the flow flush helper function.
--- drivers/net/mlx5/mlx5_flow.c | 8 ++ drivers/net/mlx5/mlx5_flow_hw.c | 129 ++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+)
drivers/net/mlx5/mlx5_flow.c +8 -0
@@ -7013,6 +7013,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,uint32_t num_flushed =0 , fidx =1 ;struct rte_flow *flow; +#ifdef HAVE_IBV_FLOW_DV_SUPPORT +if (priv->sh->config.dv_flow_en ==2 && + type ==MLX5_FLOW_TYPE_GEN ) { +flow_hw_q_flow_flush (dev,NULL ); +return ; + } +#endif + MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {flow_list_destroy (dev, type, fidx); num_flushed++;
drivers/net/mlx5/mlx5_flow_hw.c +129 -0
141 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 10/14 net/mlx5: add flow jump action
In reply to Suanming MouJump action connects different level of flow tables and allows packet handling in the chain of flows.
A new action construct data struct is also added in this commit to help to handle not only the dynamic jump action but also for the other generic dynamic actions. The actions with empty mask configuration means dynamic action, and the dedicated action will be created with the flow action configuration during flow creation. In that dynamic action case, the action will be appended to the table template's action list during table creation. When creating the flows, traverse the action list and pick the dynamic action configuration details from flow actions as the action construct data struct describes, then create the dedicated dynamic actions.
This commit adds the jump action and the generic dynamic action construct mechanism.
--- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_flow.h | 25 ++- drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++--- 3 files changed, 275 insertions(+), 21 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
@@ -1524,6 +1524,7 @@ struct mlx5_priv {/* HW steering global drop action. */ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] [MLX5DR_TABLE_TYPE_MAX ]; +struct mlx5_indexed_pool *acts_ipool;/* Action data indexed pool. */ #endif };
drivers/net/mlx5/mlx5_flow.h +24 -1
@@ -1020,10 +1020,25 @@ struct rte_flow {/* HWS flow struct. */ struct rte_flow_hw {uint32_t idx;/* Flow index from indexed pool. */ +uint32_t fate_type;/* Fate action type. */ +union { +/* Jump action. */ +struct mlx5_hw_jump_action *jump; + };struct rte_flow_template_table *table;/* The table flow allcated from. */ struct mlx5dr_rule rule;/* HWS layer data struct. */ } __rte_packed; +/* rte flow action translate to DR action struct. */ +struct mlx5_action_construct_data { +LIST_ENTRY (mlx5_action_construct_data) next; +/* Ensure the action types are matched. */ +int type; +uint32_t idx;/* Data index. */ +uint16_t action_src;/* rte_flow_action src offset. */ +uint16_t action_dst;/* mlx5dr_rule_action dst offset. */ +}; +/* Flow item template struct. */ struct rte_flow_pattern_template {LIST_ENTRY (rte_flow_pattern_template) next;
@@ -1051,9 +1066,17 @@ struct mlx5_hw_jump_action {struct mlx5dr_action *hws_action; }; +/* The maximum actions support in the flow. */ +#define MLX5_HW_MAX_ACTS 16 +/* DR action set struct. */ struct mlx5_hw_actions { -struct mlx5dr_action *drop;/* Drop action. */ +/* Dynamic action list. */ + LIST_HEAD(act_list, mlx5_action_construct_data) act_list; +struct mlx5_hw_jump_action *jump;/* Jump action. */ +uint32_t acts_num:4 ;/* Total action number. */ +/* Translated DR action array from action template. */ +struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ]; };/* mlx5 action template struct. */
drivers/net/mlx5/mlx5_flow_hw.c +250 -20
399 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 11/14 net/mlx5: add queue and RSS action
In reply to Suanming MouThis commit adds the queue and RSS action. Similar to the jump action, dynamic ones will be added to the action construct list.
Due to the queue and RSS action in template should not be destroyed during port restart, the actions are created with standalone indirect table as indirect action does. When port stops, detaches the indirect table from action, when port starts, attaches the indirect table back to the action.
One more change is made to accelerate the action creation. Currently the mlx5_hrxq_get() function returns the object index instead of object pointer. This introduced an extra converting the index to the object by calling mlx5_ipool_get() in most of the case. And that extra converting hurts multi-thread performance since mlx5_ipool_get() uses the global lock inside. As the hash Rx queue object itself also contains the index, returns the object directly will achieve better performance without the global lock.
--- drivers/net/mlx5/linux/mlx5_os.c | 18 ++-- drivers/net/mlx5/mlx5.h | 4 + drivers/net/mlx5/mlx5_devx.c | 10 ++ drivers/net/mlx5/mlx5_flow.c | 38 +++---- drivers/net/mlx5/mlx5_flow.h | 7 ++ drivers/net/mlx5/mlx5_flow_dv.c | 161 ++++++++++++++--------------- drivers/net/mlx5/mlx5_flow_hw.c | 101 ++++++++++++++++++ drivers/net/mlx5/mlx5_flow_verbs.c | 7 +- drivers/net/mlx5/mlx5_rx.h | 9 +- drivers/net/mlx5/mlx5_rxq.c | 85 +++++++++------ 10 files changed, 283 insertions(+), 157 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +9 -9
@@ -1521,6 +1521,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->drop_queue.hrxq =mlx5_drop_action_create (eth_dev);if (!priv->drop_queue.hrxq) goto error; + priv->hrxqs =mlx5_list_create ("hrxq" , eth_dev, true, + mlx5_hrxq_create_cb, + mlx5_hrxq_match_cb, + mlx5_hrxq_remove_cb, + mlx5_hrxq_clone_cb, + mlx5_hrxq_clone_free_cb); +if (!priv->hrxqs) + goto error; +rte_rwlock_init (&priv->ind_tbls_lock);if (priv->sh->config.dv_flow_en ==2 )return eth_dev;/* Port representor shares the same max priority with pf port. */
@@ -1545,15 +1554,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, err =ENOTSUP ; goto error; } - priv->hrxqs =mlx5_list_create ("hrxq" , eth_dev, true, - mlx5_hrxq_create_cb, - mlx5_hrxq_match_cb, - mlx5_hrxq_remove_cb, - mlx5_hrxq_clone_cb, - mlx5_hrxq_clone_free_cb); -if (!priv->hrxqs) - goto error; -rte_rwlock_init (&priv->ind_tbls_lock);/* Query availability of metadata reg_c's. */ if (!priv->sh->metadata_regc_check_flag) { err =mlx5_flow_discover_mreg_c (eth_dev);
drivers/net/mlx5/mlx5.h +4 -0
@@ -1284,6 +1284,7 @@ struct mlx5_flow_rss_desc {uint64_t hash_fields;/* Verbs Hash fields. */ uint8_t key[MLX5_RSS_HASH_KEY_LEN ];/**< RSS hash key. */ uint32_t key_len;/**< RSS hash key len. */ +uint32_t hws_flags;/**< HW steering action. */ uint32_t tunnel;/**< Queue in tunnel. */ uint32_t shared_rss;/**< Shared RSS index. */ struct mlx5_ind_table_obj *ind_tbl;
@@ -1345,6 +1346,7 @@ struct mlx5_hrxq {#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )void *action;/* DV QP action pointer. */ #endif +uint32_t hws_flags;/* Hw steering flags. */ uint64_t hash_fields;/* Verbs Hash fields. */ uint32_t rss_key_len;/* Hash key length in bytes. */ uint32_t idx;/* Hash Rx queue index. */
@@ -1475,6 +1477,8 @@ struct mlx5_priv { LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. *//* Indirection tables. */ LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls; +/* Standalone indirect tables. */ + LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;/* Pointer to next element. */ rte_rwlock_t ind_tbls_lock;uint32_t refcnt;/**< Reference counter. */
drivers/net/mlx5/mlx5_devx.c +10 -0
@@ -807,6 +807,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq, goto error; }#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +if (hrxq->hws_flags) { + hrxq->action =mlx5dr_action_create_dest_tir + (priv->dr_ctx, + (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags); +if (!hrxq->action) + goto error; +return 0 ; + }if (mlx5_flow_os_create_flow_action_dest_devx_tir (hrxq->tir, &hrxq->action)) { rte_errno = errno;
@@ -1042,6 +1050,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)DRV_LOG (ERR ,"Cannot create drop RX queue" );return ret; } +if (priv->sh->config.dv_flow_en ==2 ) +return 0 ;/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */ ret =mlx5_devx_ind_table_new (dev,0 , hrxq->ind_table);if (ret !=0 ) {
drivers/net/mlx5/mlx5_flow.c +12 -26
@@ -9438,14 +9438,10 @@ int mlx5_action_handle_attach(struct rte_eth_dev *dev) {struct mlx5_priv *priv = dev->data->dev_private; -struct mlx5_indexed_pool *ipool = - priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS ]; -struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;int ret =0 ; -uint32_t idx; +struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {const char *message;uint32_t queue_idx;
@@ -9461,9 +9457,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev) }if (ret !=0 )return ret; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { ret =mlx5_ind_table_obj_attach (dev, ind_tbl);if (ret !=0 ) { DRV_LOG(ERR, "Port %u could not attach "
@@ -9472,13 +9466,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev) goto error; } } +return 0 ; error: - shared_rss_last = shared_rss; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - -if (shared_rss == shared_rss_last) + ind_tbl_last = ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { +if (ind_tbl == ind_tbl_last)break ;if (mlx5_ind_table_obj_detach (dev, ind_tbl) !=0 ) DRV_LOG(CRIT, "Port %u could not detach "
@@ -9501,15 +9494,10 @@ int mlx5_action_handle_detach(struct rte_eth_dev *dev) {struct mlx5_priv *priv = dev->data->dev_private; -struct mlx5_indexed_pool *ipool = - priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS ]; -struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;int ret =0 ; -uint32_t idx; - - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; +struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { ret =mlx5_ind_table_obj_detach (dev, ind_tbl);if (ret !=0 ) { DRV_LOG(ERR, "Port %u could not detach "
@@ -9520,11 +9508,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev) }return 0 ; error: - shared_rss_last = shared_rss; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - -if (shared_rss == shared_rss_last) + ind_tbl_last = ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { +if (ind_tbl == ind_tbl_last)break ;if (mlx5_ind_table_obj_attach (dev, ind_tbl) !=0 ) DRV_LOG(CRIT, "Port %u could not attach "
drivers/net/mlx5/mlx5_flow.h +7 -0
@@ -1024,6 +1024,7 @@ struct rte_flow_hw {union {/* Jump action. */ struct mlx5_hw_jump_action *jump; +struct mlx5_hrxq *hrxq;/* TIR action. */ };struct rte_flow_template_table *table;/* The table flow allcated from. */ struct mlx5dr_rule rule;/* HWS layer data struct. */
@@ -1074,6 +1075,7 @@ struct mlx5_hw_actions {/* Dynamic action list. */ LIST_HEAD(act_list, mlx5_action_construct_data) act_list;struct mlx5_hw_jump_action *jump;/* Jump action. */ +struct mlx5_hrxq *tir;/* TIR action. */ uint32_t acts_num:4; /* Total action number. *//* Translated DR action array from action template. */ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ];
@@ -1915,6 +1917,11 @@ int flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx,int flow_dv_query_count (struct rte_eth_dev *dev,uint32_t cnt_idx,void *data,struct rte_flow_error *error); +void flow_dv_hashfields_set (uint64_t item_flags, +struct mlx5_flow_rss_desc *rss_desc, +uint64_t *hash_fields); +void flow_dv_action_rss_l34_hash_adjust (uint64_t rss_types, +uint64_t *hash_field);struct mlx5_list_entry *flow_hw_grp_create_cb (void *tool_ctx,void *cb_ctx);void flow_hw_grp_remove_cb (void *tool_ctx,struct mlx5_list_entry *entry);
drivers/net/mlx5/mlx5_flow_dv.c +76 -85
371 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_hw.c +101 -0
149 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_verbs.c +2 -5
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,MLX5_ASSERT (priv->drop_queue.hrxq); hrxq = priv->drop_queue.hrxq; }else { -uint32_t hrxq_idx;struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;MLX5_ASSERT (rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow, rss_desc->tunnel = !!(handle->layers &MLX5_FLOW_LAYER_TUNNEL ); rss_desc->shared_rss =0 ; - hrxq_idx =mlx5_hrxq_get (dev, rss_desc); - hrxq =mlx5_ipool_get (priv->sh->ipool[MLX5_IPOOL_HRXQ ], - hrxq_idx); + hrxq =mlx5_hrxq_get (dev, rss_desc);if (!hrxq) { rte_flow_error_set (error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow, "cannot get hash queue"); goto error; } - handle->rix_hrxq = hrxq_idx; + handle->rix_hrxq = hrxq->idx; }MLX5_ASSERT (hrxq); handle->drv_flow = mlx5_glue->create_flow
drivers/net/mlx5/mlx5_rx.h +7 -2
@@ -225,9 +225,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);struct mlx5_ind_table_obj *mlx5_ind_table_obj_get (struct rte_eth_dev *dev,const uint16_t *queues,uint32_t queues_n); +struct mlx5_ind_table_obj *mlx5_ind_table_obj_new (struct rte_eth_dev *dev, +const uint16_t *queues, +uint32_t queues_n, +bool standalone, +bool ref_qs);int mlx5_ind_table_obj_release (struct rte_eth_dev *dev,struct mlx5_ind_table_obj *ind_tbl, -bool standalone,bool deref_rxqs); int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev, struct mlx5_ind_table_obj *ind_tbl,
@@ -250,8 +254,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx, void *cb_ctx __rte_unused); void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,struct mlx5_list_entry *entry); -uint32_t mlx5_hrxq_get (struct rte_eth_dev *dev, +struct mlx5_hrxq *mlx5_hrxq_get (struct rte_eth_dev *dev,struct mlx5_flow_rss_desc *rss_desc); +int mlx5_hrxq_obj_release (struct rte_eth_dev *dev,struct mlx5_hrxq *hrxq);int mlx5_hrxq_release (struct rte_eth_dev *dev,uint32_t hxrq_idx);uint32_t mlx5_hrxq_verify (struct rte_eth_dev *dev);enum mlx5_rxq_type mlx5_rxq_get_type (struct rte_eth_dev *dev,uint16_t idx);
drivers/net/mlx5/mlx5_rxq.c +55 -30
209 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 12/14 net/mlx5: add mark action
In reply to Suanming MouThe mark action is covered by tag action internally. While it is added the HW will add a tag to the packet. The mark value can be set as fixed or dynamic as the action mask indicates.
--- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_flow.h | 1 + drivers/net/mlx5/mlx5_flow_hw.c | 66 ++++++++++++++++++++++++++++++--- 3 files changed, 63 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.h +2 -0
@@ -1528,6 +1528,8 @@ struct mlx5_priv {/* HW steering global drop action. */ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] [MLX5DR_TABLE_TYPE_MAX ]; +/* HW steering global drop action. */ +struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX ];struct mlx5_indexed_pool *acts_ipool;/* Action data indexed pool. */ #endif };
drivers/net/mlx5/mlx5_flow.h +1 -0
@@ -1077,6 +1077,7 @@ struct mlx5_hw_actions {struct mlx5_hw_jump_action *jump;/* Jump action. */ struct mlx5_hrxq *tir;/* TIR action. */ uint32_t acts_num:4; /* Total action number. */ + uint32_t mark:1; /* Indicate the mark action. *//* Translated DR action array from action template. */ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ]; };
drivers/net/mlx5/mlx5_flow_hw.c +60 -6
115 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 13/14 net/mlx5: add indirect action
In reply to Suanming MouHW steering can support indirect action as well. With indirect action, the flow can be created with more flexible shared RSS action selection. This will can save the action template with different RSS actions.
This commit adds the flow queue operation callback for:rte_flow_async_action_handle_create(); rte_flow_async_action_handle_destroy(); rte_flow_async_action_handle_update();
--- drivers/net/mlx5/mlx5_flow.c | 131 ++++++++++ drivers/net/mlx5/mlx5_flow.h | 59 +++++ drivers/net/mlx5/mlx5_flow_dv.c | 21 +- drivers/net/mlx5/mlx5_flow_hw.c | 414 +++++++++++++++++++++++++++++++- 4 files changed, 612 insertions(+), 13 deletions(-)
drivers/net/mlx5/mlx5_flow.c +131 -0
@@ -879,6 +879,29 @@ mlx5_flow_push(struct rte_eth_dev *dev,uint32_t queue, struct rte_flow_error *error); +static struct rte_flow_action_handle * +mlx5_flow_async_action_handle_create (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +const struct rte_flow_indir_action_conf *conf, +const struct rte_flow_action *action, +void *user_data, +struct rte_flow_error *error); + +static int +mlx5_flow_async_action_handle_update (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +const void *update, +void *user_data, +struct rte_flow_error *error); + +static int +mlx5_flow_async_action_handle_destroy (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +void *user_data, +struct rte_flow_error *error); +static const struct rte_flow_ops mlx5_flow_ops = { .validate = mlx5_flow_validate, .create = mlx5_flow_create,
@@ -911,6 +934,9 @@ static const struct rte_flow_ops mlx5_flow_ops = { .async_destroy = mlx5_flow_async_flow_destroy, .pull = mlx5_flow_pull, .push = mlx5_flow_push, + .async_action_handle_create = mlx5_flow_async_action_handle_create, + .async_action_handle_update = mlx5_flow_async_action_handle_update, + .async_action_handle_destroy = mlx5_flow_async_action_handle_destroy, };/* Tunnel information. */
@@ -8364,6 +8390,111 @@ mlx5_flow_push(struct rte_eth_dev *dev,return fops->push (dev, queue, error); } +/** +* Create shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] conf +* Indirect action configuration. +* @param[in] action +* rte_flow action detail. +* @param[in] user_data +* Pointer to the user_data. +* @param[out] error +* Pointer to error structure. +* +* @return +* Action handle on success, NULL otherwise and rte_errno is set. +*/ +static struct rte_flow_action_handle * +mlx5_flow_async_action_handle_create (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +const struct rte_flow_indir_action_conf *conf, +const struct rte_flow_action *action, +void *user_data, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->async_action_create (dev, queue, attr, conf, action, + user_data, error); +} + +/** +* Update shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] handle +* Action handle to be updated. +* @param[in] update +* Update value. +* @param[in] user_data +* Pointer to the user_data. +* @param[out] error +* Pointer to error structure. +* +* @return +* 0 on success, negative value otherwise and rte_errno is set. +*/ +static int +mlx5_flow_async_action_handle_update (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +const void *update, +void *user_data, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->async_action_update (dev, queue, attr, handle, + update, user_data, error); +} + +/** +* Destroy shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] handle +* Action handle to be destroyed. +* @param[in] user_data +* Pointer to the user_data. +* @param[out] error +* Pointer to error structure. +* +* @return +* 0 on success, negative value otherwise and rte_errno is set. +*/ +static int +mlx5_flow_async_action_handle_destroy (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +void *user_data, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->async_action_destroy (dev, queue, attr, handle, + user_data, error); +} +/** * Allocate a new memory for the counter values wrapped by all the needed * management.
drivers/net/mlx5/mlx5_flow.h +59 -0
99 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_dv.c +12 -9
@@ -13835,9 +13835,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,* @return * Valid hash RX queue index, otherwise 0. */ -static uint32_t -__flow_dv_action_rss_hrxq_lookup (struct rte_eth_dev *dev,uint32_t idx, -const uint64_t hash_fields) +uint32_t +flow_dv_action_rss_hrxq_lookup (struct rte_eth_dev *dev,uint32_t idx, +const uint64_t hash_fields) {struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_shared_action_rss *shared_rss =
@@ -13965,7 +13965,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,struct mlx5_hrxq *hrxq =NULL ;uint32_t hrxq_idx; - hrxq_idx =__flow_dv_action_rss_hrxq_lookup (dev, + hrxq_idx =flow_dv_action_rss_hrxq_lookup (dev, rss_desc->shared_rss, dev_flow->hash_fields);if (hrxq_idx)
@@ -14689,6 +14689,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,struct mlx5_shared_action_rss *shared_rss, struct rte_flow_error *error) { + struct mlx5_priv *priv = dev->data->dev_private;struct mlx5_flow_rss_desc rss_desc = {0 };size_t i;int err;
@@ -14709,6 +14710,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,/* Set non-zero value to indicate a shared RSS. */ rss_desc.shared_rss = action_idx; rss_desc.ind_tbl = shared_rss->ind_tbl; +if (priv->sh->config.dv_flow_en ==2 ) + rss_desc.hws_flags =MLX5DR_ACTION_FLAG_HWS_RX ;for (i =0 ; i <MLX5_RSS_HASH_FIELDS_LEN ; i++) {struct mlx5_hrxq *hrxq;uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14900,7 +14903,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,* A valid shared action handle in case of success, NULL otherwise and * rte_errno is set. */ -static struct rte_flow_action_handle * +struct rte_flow_action_handle * flow_dv_action_create(struct rte_eth_dev *dev, const struct rte_flow_indir_action_conf *conf, const struct rte_flow_action *action,
@@ -14970,7 +14973,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_destroy (struct rte_eth_dev *dev,struct rte_flow_action_handle *handle,struct rte_flow_error *error)
@@ -15180,7 +15183,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_update(struct rte_eth_dev *dev, struct rte_flow_action_handle *handle, const void *update,
@@ -15894,7 +15897,7 @@ flow_dv_query_count_ptr(struct rte_eth_dev *dev, uint32_t cnt_idx, "counters are not available"); } -static int +int flow_dv_action_query (struct rte_eth_dev *dev,const struct rte_flow_action_handle *handle,void *data,struct rte_flow_error *error)
@@ -17584,7 +17587,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_validate(struct rte_eth_dev *dev, const struct rte_flow_indir_action_conf *conf, const struct rte_flow_action *action,
drivers/net/mlx5/mlx5_flow_hw.c +410 -4
493 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 14/14 net/mlx5: add header reformat action
In reply to Suanming MouHW steering header reformat action can work under bulk mode. In this case, when create the table, bulk size of header reformat actions will be allocated in low level. Afterwards, when create flow, just simply specify the action index in the bulk and the encapsulation data to the action will be enough.
--- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_flow.h | 21 +++ drivers/net/mlx5/mlx5_flow_dv.c | 4 +- drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++- 4 files changed, 251 insertions(+), 3 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
@@ -342,6 +342,7 @@ struct mlx5_hw_q_job {uint32_t type;/* Job type. */ struct rte_flow_hw *flow;/* Flow attached to the job. */ void *user_data;/* Job user data. */ +uint8_t *encap_data;/* Encap data. */ };/* HW steering job descriptor LIFO pool. */
drivers/net/mlx5/mlx5_flow.h +21 -0
@@ -1040,6 +1040,14 @@ struct mlx5_action_construct_data {uint16_t action_src;/* rte_flow_action src offset. */ uint16_t action_dst;/* mlx5dr_rule_action dst offset. */ union { +struct { +/* encap src(item) offset. */ +uint16_t src; +/* encap dst data offset. */ +uint16_t dst; +/* encap data len. */ +uint16_t len; + } encap;struct {uint64_t types;/* RSS hash types. */ uint32_t level;/* RSS level. */
@@ -1076,6 +1084,13 @@ struct mlx5_hw_jump_action {struct mlx5dr_action *hws_action; }; +/* Encap decap action struct. */ +struct mlx5_hw_encap_decap_action { +struct mlx5dr_action *action;/* Action object. */ +size_t data_size;/* Action metadata size. */ +uint8_t data[];/* Action data. */ +}; +/* The maximum actions support in the flow. */ #define MLX5_HW_MAX_ACTS 16
@@ -1085,6 +1100,9 @@ struct mlx5_hw_actions { LIST_HEAD(act_list, mlx5_action_construct_data) act_list;struct mlx5_hw_jump_action *jump;/* Jump action. */ struct mlx5_hrxq *tir;/* TIR action. */ +/* Encap/Decap action. */ +struct mlx5_hw_encap_decap_action *encap_decap; +uint16_t encap_decap_pos;/* Encap/Decap action position. */ uint32_t acts_num:4; /* Total action number. */ uint32_t mark:1; /* Indicate the mark action. *//* Translated DR action array from action template. */
@@ -2032,4 +2050,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,const struct rte_flow_action_handle *handle, void *data, struct rte_flow_error *error); +size_t flow_dv_get_item_hdr_len (const enum rte_flow_item_type item_type); +int flow_dv_convert_encap_data (const struct rte_flow_item *items,uint8_t *buf, +size_t *size,struct rte_flow_error *error);#endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_dv.c +2 -2
@@ -4026,7 +4026,7 @@ flow_dv_push_vlan_action_resource_register* @return * sizeof struct item_type, 0 if void or irrelevant. */ -static size_t +size_t flow_dv_get_item_hdr_len (const enum rte_flow_item_type item_type) {size_t retval;
@@ -4092,7 +4092,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)* @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -static int +int flow_dv_convert_encap_data (const struct rte_flow_item *items,uint8_t *buf,size_t *size,struct rte_flow_error *error) {
drivers/net/mlx5/mlx5_flow_hw.c +227 -1
299 lines not on this page · raw message
Signature · 1 line
2.25.1
v3 cover net/mlx5: add hardware steering
In reply to Suanming MouThe Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion
- security, exposing the low-level steering entries directly to the userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.
*** THIS PATCH SET DEPENDS ON THE NEW RTE_FLOW ASYNC API ***
---
v3:
- rebase to the latest version.
v2:
- New HW steering low-level abstract code added.
- commit message improvement.
- add protection for rte_flow and rte_flow_async callbacks.
- rebase to rte_flow_async v9.
- fix some rte_flow error not filled bugs.
Suanming Mou (14):
net/mlx5: introduce hardware steering operation net/mlx5: add HW steering low-level abstract code net/mlx5: introduce hardware steering enable routine net/mlx5: add port flow configuration net/mlx5: add pattern template management net/mlx5: add action template management net/mlx5: add table management
net/mlx5: add basic flow queue operation net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action
doc/guides/nics/mlx5.rst | 19 +- doc/guides/rel_notes/release_22_03.rst | 6 + drivers/net/mlx5/linux/mlx5_flow_os.h | 1 + drivers/net/mlx5/linux/mlx5_os.c | 22 +- drivers/net/mlx5/meson.build | 2 + drivers/net/mlx5/mlx5.c | 55 +- drivers/net/mlx5/mlx5.h | 66 +- drivers/net/mlx5/mlx5_devx.c | 10 + drivers/net/mlx5/mlx5_dr.c | 383 ++++ drivers/net/mlx5/mlx5_dr.h | 456 +++++ drivers/net/mlx5/mlx5_flow.c | 721 ++++++- drivers/net/mlx5/mlx5_flow.h | 287 +++ drivers/net/mlx5/mlx5_flow_dv.c | 186 +- drivers/net/mlx5/mlx5_flow_hw.c | 2335 +++++++++++++++++++++++ drivers/net/mlx5/mlx5_flow_verbs.c | 7 +- drivers/net/mlx5/mlx5_rx.h | 9 +- drivers/net/mlx5/mlx5_rxq.c | 85 +- drivers/net/mlx5/windows/mlx5_flow_os.h | 1 + 18 files changed, 4471 insertions(+), 180 deletions(-) create mode 100644 drivers/net/mlx5/mlx5_dr.c create mode 100644 drivers/net/mlx5/mlx5_dr.h create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
Signature · 1 line
2.25.1
PATCH v3 1/14 net/mlx5: introduce hardware steering operation
The Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion
- security, exposing the low-level steering entries directly to the userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.
This commit adds the basic driver operation.
--- drivers/net/mlx5/linux/mlx5_flow_os.h | 1 + drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5_flow.c | 1 + drivers/net/mlx5/mlx5_flow.h | 1 + drivers/net/mlx5/mlx5_flow_hw.c | 13 +++++++++++++ drivers/net/mlx5/windows/mlx5_flow_os.h | 1 + 6 files changed, 18 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
drivers/net/mlx5/linux/mlx5_flow_os.h +1 -0
@@ -9,6 +9,7 @@#ifdef HAVE_IBV_FLOW_DV_SUPPORT extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops; +extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;#endif /**
drivers/net/mlx5/meson.build +1 -0
@@ -16,6 +16,7 @@ sources = files( 'mlx5_flow.c', 'mlx5_flow_meter.c', 'mlx5_flow_dv.c', + 'mlx5_flow_hw.c', 'mlx5_flow_aso.c', 'mlx5_flow_flex.c', 'mlx5_mac.c',
drivers/net/mlx5/mlx5_flow.c +1 -0
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = { [MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) [MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops, + [MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops, #endif [MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops, [MLX5_FLOW_TYPE_MAX ] = &mlx5_flow_null_drv_ops
drivers/net/mlx5/mlx5_flow.h +1 -0
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {MLX5_FLOW_TYPE_MIN ,MLX5_FLOW_TYPE_DV ,MLX5_FLOW_TYPE_VERBS , +MLX5_FLOW_TYPE_HW , MLX5_FLOW_TYPE_MAX, };
drivers/net/mlx5/mlx5_flow_hw.c new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright (c) 2022 NVIDIA Corporation & Affiliates +*/ + +#include <rte_flow.h> + +#include "mlx5_flow.h" + +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) + +const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops; + +#endif
drivers/net/mlx5/windows/mlx5_flow_os.h +1 -0
@@ -10,6 +10,7 @@#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops; +extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;#endif /**
Signature · 1 line
2.25.1
PATCH v3 2/14 net/mlx5: add HW steering low-level abstract code
In reply to Suanming MouThe HW steering low-level implementation will be added later in another patch series. To avoid the linkage issues the abstract stub replacement is provided currently.
--- drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5_dr.c | 383 +++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_dr.h | 456 +++++++++++++++++++++++++++++++++++ 3 files changed, 840 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_dr.c create mode 100644 drivers/net/mlx5/mlx5_dr.h
drivers/net/mlx5/meson.build +1 -0
@@ -14,6 +14,7 @@ sources = files( 'mlx5.c', 'mlx5_ethdev.c', 'mlx5_flow.c', + 'mlx5_dr.c', 'mlx5_flow_meter.c', 'mlx5_flow_dv.c', 'mlx5_flow_hw.c',
drivers/net/mlx5/mlx5_dr.c new file +383 -0
383 lines not on this page · raw message
drivers/net/mlx5/mlx5_dr.h new file +456 -0
456 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 3/14 net/mlx5: introduce hardware steering enable routine
In reply to Suanming MouThe new hardware steering engine relies on using dedicated steering WQEs instead of direct writing to the low-level steering table entries directly. In the first introduce implementation the hardware steering engine supports the new queue based Flow API, the existing synchronous non-queue based Flow API is not supported.
A new dv_flow_en value 2 is added to manage mlx5 PMD steering engine:
dv_flow_en rte_flow API rte_flow_async API
------------------------------------------------
0 support not support
1 support not support
2 not support support
This commit introduces the extra dv_flow_en = 2 to specify the new flow initialize and manage operation routine.
--- doc/guides/nics/mlx5.rst | 13 ++++++++++--- drivers/net/mlx5/linux/mlx5_os.c | 4 ++++ drivers/net/mlx5/mlx5.c | 7 ++++++- drivers/net/mlx5/mlx5.h | 3 ++- drivers/net/mlx5/mlx5_flow.c | 22 ++++++++++++++++++++++ 5 files changed, 44 insertions(+), 5 deletions(-)
doc/guides/nics/mlx5.rst +10 -3
@@ -942,10 +942,17 @@ for an additional list of options shared with other mlx5 drivers. - ``dv_flow_en`` parameter [int] - A nonzero value enables the DV flow steering assuming it is supported - by the driver (RDMA Core library version is rdma-core-24.0 or higher). + Value 0 means legacy Verbs flow offloading. - Enabled by default if supported. + Value 1 enables the DV flow steering assuming it is supported by the + driver (RDMA Core library version is rdma-core-24.0 or higher). + + Value 2 enables the WQE based hardware steering. In this mode only + the queue-based rte_flow_q flow management is supported. + + Configured by default to 1 DV flow steering if the driver(RDMA CORE library) + supported. Otherwise, the value will be 0 which indicates legacy Verbs flow + offloading. - ``dv_esw_en`` parameter [int]
drivers/net/mlx5/linux/mlx5_os.c +4 -0
@@ -482,6 +482,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv) err =mlx5_alloc_table_hash_list (priv);if (err) goto error; +if (priv->sh->config.dv_flow_en ==2 ) +return 0 ;/* The resources below are only valid with DV support. */ #ifdef HAVE_IBV_FLOW_DV_SUPPORT /* Init port id action list. */
@@ -1519,6 +1521,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->drop_queue.hrxq =mlx5_drop_action_create (eth_dev);if (!priv->drop_queue.hrxq) goto error; +if (priv->sh->config.dv_flow_en ==2 ) +return eth_dev;/* Port representor shares the same max priority with pf port. */ if (!priv->sh->flow_priority_check_flag) {/* Supported Verbs flow priority number detection. */
drivers/net/mlx5/mlx5.c +6 -1
@@ -1199,7 +1199,12 @@ mlx5_dev_args_check_handler(const char *key, const char *val, void *opaque) }else if (strcmp (MLX5_DV_ESW_EN , key) ==0 ) { config->dv_esw_en = !!tmp; }else if (strcmp (MLX5_DV_FLOW_EN , key) ==0 ) { - config->dv_flow_en = !!tmp; +if (tmp >2 ) { +DRV_LOG (ERR ,"Invalid %s parameter." , key); + rte_errno =EINVAL ; +return -rte_errno; + } + config->dv_flow_en = tmp; }else if (strcmp (MLX5_DV_XMETA_EN , key) ==0 ) { if (tmp != MLX5_XMETA_MODE_LEGACY && tmp != MLX5_XMETA_MODE_META16 &&
drivers/net/mlx5/mlx5.h +2 -1
@@ -287,7 +287,8 @@ struct mlx5_sh_config {int tx_skew;/* Tx scheduling skew between WQE and data on wire. */ uint32_t reclaim_mode:2; /* Memory reclaim mode. */ uint32_t dv_esw_en:1; /* Enable E-Switch DV flow. */ - uint32_t dv_flow_en:1; /* Enable DV flow. */ +/* Enable DV flow. 1 means SW steering, 2 means HW steering. */ + unsigned int dv_flow_en:2; uint32_t dv_xmeta_en:2; /* Enable extensive flow metadata. */ uint32_t dv_miss_info:1; /* Restore packet after partial hw miss. */ uint32_t l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
drivers/net/mlx5/mlx5_flow.c +22 -0
@@ -6840,6 +6840,15 @@ mlx5_flow_create(struct rte_eth_dev *dev,const struct rte_flow_action actions[], struct rte_flow_error *error) { + struct mlx5_priv *priv = dev->data->dev_private; + +if (priv->sh->config.dv_flow_en ==2 ) { +rte_flow_error_set (error,ENOTSUP , +RTE_FLOW_ERROR_TYPE_UNSPECIFIED , +NULL , +"Flow non-Q creation not supported" ); +return NULL ; + }/* * If the device is not started yet, it is not allowed to created a * flow from application. PMD default flows and traffic control flows
@@ -7336,6 +7345,13 @@ mlx5_flow_destroy(struct rte_eth_dev *dev,struct rte_flow *flow, struct rte_flow_error *error __rte_unused) { + struct mlx5_priv *priv = dev->data->dev_private; + +if (priv->sh->config.dv_flow_en ==2 ) +return rte_flow_error_set (error,ENOTSUP , +RTE_FLOW_ERROR_TYPE_UNSPECIFIED , +NULL , +"Flow non-Q destruction not supported" );flow_list_destroy (dev,MLX5_FLOW_TYPE_GEN , (uintptr_t )(void *)flow);return 0 ;
@@ -7433,7 +7449,13 @@ mlx5_flow_query(struct rte_eth_dev *dev, struct rte_flow_error *error) {int ret; +struct mlx5_priv *priv = dev->data->dev_private; +if (priv->sh->config.dv_flow_en ==2 ) +return rte_flow_error_set (error,ENOTSUP , +RTE_FLOW_ERROR_TYPE_UNSPECIFIED , +NULL , +"Flow non-Q query not supported" ); ret =flow_drv_query (dev, (uintptr_t )(void *)flow, actions, data, error);if (ret <0 )
Signature · 1 line
2.25.1
PATCH v3 4/14 net/mlx5: add port flow configuration
In reply to Suanming MouThe hardware steering is backend to support rte_flow_async API in mlx5 PMD. The port configuration function creates the queues and needed flow management resources.
The PMD layer configuration function allocates the queues' context and per-queue job descriptor pool. The job descriptor pool size is equal to the queue size, and the job descriptors will be popped from pool with LIFO strategy to convey the flow information during flow insertion/destruction. Then, while polling the queued operation result, the flow information will be extracted from the job descriptor and the descriptor will be pushed back to the LIFO pool.
The commit creates the flow port queues and the job descriptor pools.
--- drivers/net/mlx5/mlx5.c | 3 + drivers/net/mlx5/mlx5.h | 30 +++++- drivers/net/mlx5/mlx5_flow.c | 86 +++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 15 +++ drivers/net/mlx5/mlx5_flow_hw.c | 159 ++++++++++++++++++++++++++++++++ 5 files changed, 292 insertions(+), 1 deletion(-)
drivers/net/mlx5/mlx5.c +3 -0
@@ -1820,6 +1820,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)/* Free the eCPRI flex parser resource. */ mlx5_flex_parser_ecpri_release (dev);mlx5_flex_item_port_cleanup (dev); +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +flow_hw_resource_release (dev); +#endif if (priv->rxq_privs !=NULL ) {/* XXX race condition if mlx5_rx_burst() is still running. */ rte_delay_us_sleep (1000 );
drivers/net/mlx5/mlx5.h +29 -1
@@ -33,7 +33,9 @@#include "mlx5_utils.h" #include "mlx5_os.h" #include "mlx5_autoconf.h" - +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +#include "mlx5_dr.h" +#endif #define MLX5_SH (dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
@@ -320,6 +322,26 @@ struct mlx5_lb_ctx {uint16_t refcnt;/* Reference count for representors. */ }; +/* HW steering queue job descriptor type. */ +enum { +MLX5_HW_Q_JOB_TYPE_CREATE ,/* Flow create job type. */ +MLX5_HW_Q_JOB_TYPE_DESTROY ,/* Flow destroy job type. */ +}; + +/* HW steering flow management job descriptor. */ +struct mlx5_hw_q_job { +uint32_t type;/* Job type. */ +struct rte_flow *flow;/* Flow attached to the job. */ +void *user_data;/* Job user data. */ +}; + +/* HW steering job descriptor LIFO pool. */ +struct mlx5_hw_q { +uint32_t job_idx;/* Free job index. */ +uint32_t size;/* LIFO size. */ +struct mlx5_hw_q_job **job;/* LIFO header. */ +} __rte_cache_aligned; +#define MLX5_COUNTERS_PER_POOL 512#define MLX5_MAX_PENDING_QUERIES 4#define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1479,6 +1501,12 @@ struct mlx5_priv {struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM ];/* Flex items have been created on the port. */ uint32_t flex_item_map;/* Map of allocated flex item elements. */ +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ +uint32_t nb_queue;/* HW steering queue number. */ +/* HW steering queue polling mechanism job descriptor LIFO. */ +struct mlx5_hw_q *hw_q; +#endif };#define PORT_ID (priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.c +86 -0
110 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +15 -0
@@ -1257,6 +1257,17 @@ typedef int (*mlx5_flow_item_update_t)const struct rte_flow_item_flex_handle *handle, const struct rte_flow_item_flex_conf *conf, struct rte_flow_error *error); +typedef int (*mlx5_flow_info_get_t ) + (struct rte_eth_dev *dev, +struct rte_flow_port_info *port_info, +struct rte_flow_queue_info *queue_info, +struct rte_flow_error *error); +typedef int (*mlx5_flow_port_configure_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_port_attr *port_attr, +uint16_t nb_queue, +const struct rte_flow_queue_attr *queue_attr[], +struct rte_flow_error *err);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1295,6 +1306,8 @@ struct mlx5_flow_driver_ops {mlx5_flow_item_create_t item_create;mlx5_flow_item_release_t item_release;mlx5_flow_item_update_t item_update; +mlx5_flow_info_get_t info_get; +mlx5_flow_port_configure_t configure; };/* mlx5_flow.c */
@@ -1762,4 +1775,6 @@ const struct mlx5_flow_tunnel * mlx5_get_tof(const struct rte_flow_item *items, const struct rte_flow_action *actions, enum mlx5_tof_rule_type *rule_type); +void +flow_hw_resource_release (struct rte_eth_dev *dev);#endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_hw.c +159 -0
169 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 5/14 net/mlx5: add pattern template management
In reply to Suanming MouThe pattern template defines flows that have the same matching fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be (eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while the values for each rule will be different.
Due to the pattern template can be used in different domains, the items will only be cached in pattern template create stage, while the template is binded to a dedicated table, the HW criteria will be created and saved to the table. The pattern templates can be used by multiple tables. But different tables create the same criteria and will not share the matcher between each other in order to have better performance.
This commit adds pattern template management.
--- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_flow.c | 76 +++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 24 +++++++++ drivers/net/mlx5/mlx5_flow_hw.c | 86 +++++++++++++++++++++++++++++++++ 4 files changed, 188 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
@@ -1502,6 +1502,8 @@ struct mlx5_priv {/* Flex items have been created on the port. */ uint32_t flex_item_map;/* Map of allocated flex item elements. */ #if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +/* Item template list. */ + LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +76 -0
94 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +24 -0
@@ -1015,6 +1015,19 @@ struct rte_flow {uint32_t geneve_tlv_option;/**< Holds Geneve TLV option id. > */ } __rte_packed; +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) + +/* Flow item template struct. */ +struct rte_flow_pattern_template { +LIST_ENTRY (rte_flow_pattern_template) next; +/* Template attributes. */ +struct rte_flow_pattern_template_attr attr; +struct mlx5dr_match_template *mt;/* mlx5 match template. */ +uint32_t refcnt;/* Reference counter. */ +}; + +#endif +/* * Define list of valid combinations of RX Hash fields * (see enum ibv_rx_hash_fields).
@@ -1268,6 +1281,15 @@ typedef int (*mlx5_flow_port_configure_t)uint16_t nb_queue, const struct rte_flow_queue_attr *queue_attr[], struct rte_flow_error *err); +typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_pattern_template_attr *attr, +const struct rte_flow_item items[], +struct rte_flow_error *error); +typedef int (*mlx5_flow_pattern_template_destroy_t ) + (struct rte_eth_dev *dev, +struct rte_flow_pattern_template *template, +struct rte_flow_error *error);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1308,6 +1330,8 @@ struct mlx5_flow_driver_ops {mlx5_flow_item_update_t item_update;mlx5_flow_info_get_t info_get;mlx5_flow_port_configure_t configure; +mlx5_flow_pattern_template_create_t pattern_template_create; +mlx5_flow_pattern_template_destroy_t pattern_template_destroy; };/* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +86 -0
107 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 6/14 net/mlx5: add action template management
In reply to Suanming MouThe action template holds a list of action types that will be used together on the same rule. The template's actions instances will be created only when the template bind to the dedicated group. And the created actions will be saved to each individual group in order for best performance. The actions in a group will not be shared with each other unless shared actions are specified.
This commit adds the action template management which stores the flow action template.
--- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_flow.c | 78 +++++++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 22 ++++++ drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++ 4 files changed, 218 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
@@ -1504,6 +1504,8 @@ struct mlx5_priv {#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )/* Item template list. */ LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt; +/* Action template list. */ + LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +78 -0
96 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +22 -0
@@ -1026,6 +1026,16 @@ struct rte_flow_pattern_template { uint32_t refcnt; /* Reference counter. */ }; +/* Flow action template struct. */ +struct rte_flow_actions_template { +LIST_ENTRY (rte_flow_actions_template) next; +/* Template attributes. */ +struct rte_flow_actions_template_attr attr; +struct rte_flow_action *actions;/* Cached flow actions. */ +struct rte_flow_action *masks;/* Cached action masks.*/ +uint32_t refcnt;/* Reference counter. */ +}; +#endif /*
@@ -1290,6 +1300,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t) (struct rte_eth_dev *dev, struct rte_flow_pattern_template *template, struct rte_flow_error *error); +typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_actions_template_attr *attr, +const struct rte_flow_action actions[], +const struct rte_flow_action masks[], +struct rte_flow_error *error); +typedef int (*mlx5_flow_actions_template_destroy_t ) + (struct rte_eth_dev *dev, +struct rte_flow_actions_template *template, +struct rte_flow_error *error);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1332,6 +1352,8 @@ struct mlx5_flow_driver_ops {mlx5_flow_port_configure_t configure;mlx5_flow_pattern_template_create_t pattern_template_create;mlx5_flow_pattern_template_destroy_t pattern_template_destroy; +mlx5_flow_actions_template_create_t actions_template_create; +mlx5_flow_actions_template_destroy_t actions_template_destroy; };/* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +116 -0
140 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 7/14 net/mlx5: add table management
In reply to Suanming MouFlow table is a group of flows with the same matching criteria and the same actions defined for them. The table defines rules that have the same matching fields but with different matching values. For example, matching on 5 tuple, the table will be (IPv4 source + IPv4 dest + s_port + d_port + next_proto) while the values for each rule will be different.
The templates' relevant matching criteria and action instances will be created in the table creation and saved in the table. As table attributes indicate the supported flow number, the flow memory will also be allocated at the same time.
This commit adds the table management functions.
--- drivers/net/mlx5/mlx5.c | 45 ++- drivers/net/mlx5/mlx5.h | 21 +- drivers/net/mlx5/mlx5_flow.c | 93 ++++++ drivers/net/mlx5/mlx5_flow.h | 73 +++++ drivers/net/mlx5/mlx5_flow_hw.c | 525 ++++++++++++++++++++++++++++++++ 5 files changed, 751 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.c +41 -4
@@ -1607,12 +1607,46 @@ void mlx5_free_table_hash_list(struct mlx5_priv *priv) {struct mlx5_dev_ctx_shared *sh = priv->sh; - -if (!sh->flow_tbls) +struct mlx5_hlist **tbls = (priv->sh->config.dv_flow_en ==2 ) ? + &sh->groups : &sh->flow_tbls; +if (*tbls ==NULL )return ; -mlx5_hlist_destroy (sh->flow_tbls); - sh->flow_tbls =NULL ; +mlx5_hlist_destroy (*tbls); + *tbls =NULL ; +} + +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +/** +* Allocate HW steering group hash list. +* +* @param[in] priv +* Pointer to the private device data structure. +*/ +static int +mlx5_alloc_hw_group_hash_list (struct mlx5_priv *priv) +{ +int err =0 ; +struct mlx5_dev_ctx_shared *sh = priv->sh; +char s[MLX5_NAME_SIZE ]; + +MLX5_ASSERT (sh); +snprintf (s,sizeof (s),"%s_flow_groups" , priv->sh->ibdev_name); + sh->groups =mlx5_hlist_create + (s,MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE , + false, true, sh, + flow_hw_grp_create_cb, + flow_hw_grp_match_cb, + flow_hw_grp_remove_cb, + flow_hw_grp_clone_cb, + flow_hw_grp_clone_free_cb); +if (!sh->groups) { +DRV_LOG (ERR ,"flow groups with hash creation failed." ); + err =ENOMEM ; + } +return err; } +#endif +/** * Initialize flow table hash list and create the root tables entry
@@ -1628,11 +1662,14 @@ int mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused) {int err =0 ; +/* Tables are only used in DV and DR modes. */ #if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )struct mlx5_dev_ctx_shared *sh = priv->sh;char s[MLX5_NAME_SIZE ]; +if (priv->sh->config.dv_flow_en ==2 ) +return mlx5_alloc_hw_group_hash_list (priv);MLX5_ASSERT (sh);snprintf (s,sizeof (s),"%s_flow_table" , priv->sh->ibdev_name); sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
drivers/net/mlx5/mlx5.h +19 -2
@@ -64,7 +64,9 @@ enum mlx5_ipool_index {MLX5_IPOOL_PUSH_VLAN ,/* Pool for push vlan resource. */ MLX5_IPOOL_TAG ,/* Pool for tag resource. */ MLX5_IPOOL_PORT_ID ,/* Pool for port id resource. */ -MLX5_IPOOL_JUMP ,/* Pool for jump resource. */ +MLX5_IPOOL_JUMP ,/* Pool for SWS jump resource. */ +/* Pool for HWS group. Jump action will be created internally. */ +MLX5_IPOOL_HW_GRP =MLX5_IPOOL_JUMP ,MLX5_IPOOL_SAMPLE ,/* Pool for sample resource. */ MLX5_IPOOL_DEST_ARRAY ,/* Pool for destination array resource. */ MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -108,6 +110,13 @@ enum mlx5_delay_drop_mode { MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */ }; +/* The HWS action type root/non-root. */ +enum mlx5_hw_action_flag_type { +MLX5_HW_ACTION_FLAG_ROOT ,/* Root action. */ +MLX5_HW_ACTION_FLAG_NONE_ROOT ,/* Non-root ation. */ +MLX5_HW_ACTION_FLAG_MAX ,/* Maximum action flag. */ +}; +/* Hlist and list callback context. */ struct mlx5_flow_cb_ctx {struct rte_eth_dev *dev;
@@ -1204,7 +1213,10 @@ struct mlx5_dev_ctx_shared {rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX ];/* UAR same-page access control required in 32bit implementations. */ #endif -struct mlx5_hlist *flow_tbls; +union { +struct mlx5_hlist *flow_tbls;/* SWS flow table. */ +struct mlx5_hlist *groups;/* HWS flow group. */ + };struct mlx5_flow_tunnel_hub *tunnel_hub;/* Direct Rules tables for FDB, NIC TX+RX */ void *dr_drop_action;/* Pointer to DR drop action, any domain. */
@@ -1510,6 +1522,11 @@ struct mlx5_priv {uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */ struct mlx5_hw_q *hw_q; +/* HW steering rte flow table list header. */ + LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl; +/* HW steering global drop action. */ +struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] + [MLX5DR_TABLE_TYPE_MAX ];#endif };
drivers/net/mlx5/mlx5_flow.c +93 -0
111 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +73 -0
97 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_hw.c +525 -0
574 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 8/14 net/mlx5: add basic flow queue operation
In reply to Suanming MouThe HW steering uses async queue-based flow rules management mechanism. The matcher and part of the actions have been prepared during flow table creation. Some remaining actions will be constructed during flow creation if needed.
A flow postpone attribute bit describes if flow management should be applied to the HW directly. An extra push function is provided to force push all the cached flows to the HW.
Once the flow has been applied to the HW, the pull function will be called to get the queued creation/destruction flows.
The DR rule flow memory is represented in PMD layer instead of allocating from HW steering layer. While destroying the flow, the flow rule memory can only be freed after the CQE received.
The HW queue job descriptor is currently introduced to convey the flow information and operation type between the flow insertion/destruction in the pull function.
This commit adds the basic flow queue operation for:rte_flow_async_create(); rte_flow_async_destroy();rte_flow_push();rte_flow_pull();
--- doc/guides/nics/mlx5.rst | 6 + doc/guides/rel_notes/release_22_03.rst | 6 + drivers/net/mlx5/mlx5.h | 2 +- drivers/net/mlx5/mlx5_flow.c | 188 ++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 41 ++++ drivers/net/mlx5/mlx5_flow_hw.c | 292 ++++++++++++++++++++++++- 6 files changed, 533 insertions(+), 2 deletions(-)
doc/guides/nics/mlx5.rst +6 -0
@@ -487,6 +487,12 @@ Limitations from the reference "Clock Queue" completions, the scheduled send timestamps should not be specified with non-zero MSB. + - HW steering: + + - WQE based high scaling and safer flow insertion/destruction. + - Set ``dv_flow_en`` to 2 in order to enable HW steering. + - Async queue-based ``rte_flow_q`` APIs supported only. + Statistics ----------
doc/guides/rel_notes/release_22_03.rst +6 -0
@@ -162,6 +162,12 @@ New Features * Added LED OEM support. +* **Updated Mellanox mlx5 driver.** + + Updated the Mellanox mlx5 driver with new features and improvements, including: + + * Added WQE based hardware steering support with ``rte_flow_q`` API. + * **Added an API for private user data in asymmetric crypto session.** An API was added to get/set an asymmetric crypto session's user data.
drivers/net/mlx5/mlx5.h +1 -1
@@ -340,7 +340,7 @@ enum {/* HW steering flow management job descriptor. */ struct mlx5_hw_q_job {uint32_t type;/* Job type. */ -struct rte_flow *flow;/* Flow attached to the job. */ +struct rte_flow_hw *flow;/* Flow attached to the job. */ void *user_data;/* Job user data. */ };
drivers/net/mlx5/mlx5_flow.c +188 -0
206 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +41 -0
65 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_hw.c +291 -1
316 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 9/14 net/mlx5: add flow flush function
In reply to Suanming MouIn case port is being stopped, all created flows should be flushed. This commit adds the flow flush helper function.
--- drivers/net/mlx5/mlx5_flow.c | 8 ++ drivers/net/mlx5/mlx5_flow_hw.c | 129 ++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+)
drivers/net/mlx5/mlx5_flow.c +8 -0
@@ -7015,6 +7015,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,uint32_t num_flushed =0 , fidx =1 ;struct rte_flow *flow; +#ifdef HAVE_IBV_FLOW_DV_SUPPORT +if (priv->sh->config.dv_flow_en ==2 && + type ==MLX5_FLOW_TYPE_GEN ) { +flow_hw_q_flow_flush (dev,NULL ); +return ; + } +#endif + MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {flow_list_destroy (dev, type, fidx); num_flushed++;
drivers/net/mlx5/mlx5_flow_hw.c +129 -0
141 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 10/14 net/mlx5: add flow jump action
In reply to Suanming MouJump action connects different level of flow tables and allows packet handling in the chain of flows.
A new action construct data struct is also added in this commit to help to handle not only the dynamic jump action but also for the other generic dynamic actions. The actions with empty mask configuration means dynamic action, and the dedicated action will be created with the flow action configuration during flow creation. In that dynamic action case, the action will be appended to the table template's action list during table creation. When creating the flows, traverse the action list and pick the dynamic action configuration details from flow actions as the action construct data struct describes, then create the dedicated dynamic actions.
This commit adds the jump action and the generic dynamic action construct mechanism.
--- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_flow.h | 25 ++- drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++--- 3 files changed, 275 insertions(+), 21 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
@@ -1527,6 +1527,7 @@ struct mlx5_priv {/* HW steering global drop action. */ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] [MLX5DR_TABLE_TYPE_MAX ]; +struct mlx5_indexed_pool *acts_ipool;/* Action data indexed pool. */ #endif };
drivers/net/mlx5/mlx5_flow.h +24 -1
@@ -1020,10 +1020,25 @@ struct rte_flow {/* HWS flow struct. */ struct rte_flow_hw {uint32_t idx;/* Flow index from indexed pool. */ +uint32_t fate_type;/* Fate action type. */ +union { +/* Jump action. */ +struct mlx5_hw_jump_action *jump; + };struct rte_flow_template_table *table;/* The table flow allcated from. */ struct mlx5dr_rule rule;/* HWS layer data struct. */ } __rte_packed; +/* rte flow action translate to DR action struct. */ +struct mlx5_action_construct_data { +LIST_ENTRY (mlx5_action_construct_data) next; +/* Ensure the action types are matched. */ +int type; +uint32_t idx;/* Data index. */ +uint16_t action_src;/* rte_flow_action src offset. */ +uint16_t action_dst;/* mlx5dr_rule_action dst offset. */ +}; +/* Flow item template struct. */ struct rte_flow_pattern_template {LIST_ENTRY (rte_flow_pattern_template) next;
@@ -1051,9 +1066,17 @@ struct mlx5_hw_jump_action {struct mlx5dr_action *hws_action; }; +/* The maximum actions support in the flow. */ +#define MLX5_HW_MAX_ACTS 16 +/* DR action set struct. */ struct mlx5_hw_actions { -struct mlx5dr_action *drop;/* Drop action. */ +/* Dynamic action list. */ + LIST_HEAD(act_list, mlx5_action_construct_data) act_list; +struct mlx5_hw_jump_action *jump;/* Jump action. */ +uint32_t acts_num:4 ;/* Total action number. */ +/* Translated DR action array from action template. */ +struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ]; };/* mlx5 action template struct. */
drivers/net/mlx5/mlx5_flow_hw.c +250 -20
399 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 11/14 net/mlx5: add queue and RSS action
In reply to Suanming MouThis commit adds the queue and RSS action. Similar to the jump action, dynamic ones will be added to the action construct list.
Due to the queue and RSS action in template should not be destroyed during port restart, the actions are created with standalone indirect table as indirect action does. When port stops, detaches the indirect table from action, when port starts, attaches the indirect table back to the action.
One more change is made to accelerate the action creation. Currently the mlx5_hrxq_get() function returns the object index instead of object pointer. This introduced an extra converting the index to the object by calling mlx5_ipool_get() in most of the case. And that extra converting hurts multi-thread performance since mlx5_ipool_get() uses the global lock inside. As the hash Rx queue object itself also contains the index, returns the object directly will achieve better performance without the global lock.
--- drivers/net/mlx5/linux/mlx5_os.c | 18 ++-- drivers/net/mlx5/mlx5.h | 4 + drivers/net/mlx5/mlx5_devx.c | 10 ++ drivers/net/mlx5/mlx5_flow.c | 38 +++---- drivers/net/mlx5/mlx5_flow.h | 7 ++ drivers/net/mlx5/mlx5_flow_dv.c | 161 ++++++++++++++--------------- drivers/net/mlx5/mlx5_flow_hw.c | 101 ++++++++++++++++++ drivers/net/mlx5/mlx5_flow_verbs.c | 7 +- drivers/net/mlx5/mlx5_rx.h | 9 +- drivers/net/mlx5/mlx5_rxq.c | 85 +++++++++------ 10 files changed, 283 insertions(+), 157 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +9 -9
@@ -1521,6 +1521,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->drop_queue.hrxq =mlx5_drop_action_create (eth_dev);if (!priv->drop_queue.hrxq) goto error; + priv->hrxqs =mlx5_list_create ("hrxq" , eth_dev, true, + mlx5_hrxq_create_cb, + mlx5_hrxq_match_cb, + mlx5_hrxq_remove_cb, + mlx5_hrxq_clone_cb, + mlx5_hrxq_clone_free_cb); +if (!priv->hrxqs) + goto error; +rte_rwlock_init (&priv->ind_tbls_lock);if (priv->sh->config.dv_flow_en ==2 )return eth_dev;/* Port representor shares the same max priority with pf port. */
@@ -1545,15 +1554,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, err =ENOTSUP ; goto error; } - priv->hrxqs =mlx5_list_create ("hrxq" , eth_dev, true, - mlx5_hrxq_create_cb, - mlx5_hrxq_match_cb, - mlx5_hrxq_remove_cb, - mlx5_hrxq_clone_cb, - mlx5_hrxq_clone_free_cb); -if (!priv->hrxqs) - goto error; -rte_rwlock_init (&priv->ind_tbls_lock);/* Query availability of metadata reg_c's. */ if (!priv->sh->metadata_regc_check_flag) { err =mlx5_flow_discover_mreg_c (eth_dev);
drivers/net/mlx5/mlx5.h +4 -0
@@ -1287,6 +1287,7 @@ struct mlx5_flow_rss_desc {uint64_t hash_fields;/* Verbs Hash fields. */ uint8_t key[MLX5_RSS_HASH_KEY_LEN ];/**< RSS hash key. */ uint32_t key_len;/**< RSS hash key len. */ +uint32_t hws_flags;/**< HW steering action. */ uint32_t tunnel;/**< Queue in tunnel. */ uint32_t shared_rss;/**< Shared RSS index. */ struct mlx5_ind_table_obj *ind_tbl;
@@ -1348,6 +1349,7 @@ struct mlx5_hrxq {#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )void *action;/* DV QP action pointer. */ #endif +uint32_t hws_flags;/* Hw steering flags. */ uint64_t hash_fields;/* Verbs Hash fields. */ uint32_t rss_key_len;/* Hash key length in bytes. */ uint32_t idx;/* Hash Rx queue index. */
@@ -1478,6 +1480,8 @@ struct mlx5_priv { LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. *//* Indirection tables. */ LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls; +/* Standalone indirect tables. */ + LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;/* Pointer to next element. */ rte_rwlock_t ind_tbls_lock;uint32_t refcnt;/**< Reference counter. */
drivers/net/mlx5/mlx5_devx.c +10 -0
@@ -807,6 +807,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq, goto error; }#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +if (hrxq->hws_flags) { + hrxq->action =mlx5dr_action_create_dest_tir + (priv->dr_ctx, + (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags); +if (!hrxq->action) + goto error; +return 0 ; + }if (mlx5_flow_os_create_flow_action_dest_devx_tir (hrxq->tir, &hrxq->action)) { rte_errno = errno;
@@ -1042,6 +1050,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)DRV_LOG (ERR ,"Cannot create drop RX queue" );return ret; } +if (priv->sh->config.dv_flow_en ==2 ) +return 0 ;/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */ ret =mlx5_devx_ind_table_new (dev,0 , hrxq->ind_table);if (ret !=0 ) {
drivers/net/mlx5/mlx5_flow.c +12 -26
@@ -9439,14 +9439,10 @@ int mlx5_action_handle_attach(struct rte_eth_dev *dev) {struct mlx5_priv *priv = dev->data->dev_private; -struct mlx5_indexed_pool *ipool = - priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS ]; -struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;int ret =0 ; -uint32_t idx; +struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {const char *message;uint32_t queue_idx;
@@ -9462,9 +9458,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev) }if (ret !=0 )return ret; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { ret =mlx5_ind_table_obj_attach (dev, ind_tbl);if (ret !=0 ) { DRV_LOG(ERR, "Port %u could not attach "
@@ -9473,13 +9467,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev) goto error; } } +return 0 ; error: - shared_rss_last = shared_rss; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - -if (shared_rss == shared_rss_last) + ind_tbl_last = ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { +if (ind_tbl == ind_tbl_last)break ;if (mlx5_ind_table_obj_detach (dev, ind_tbl) !=0 ) DRV_LOG(CRIT, "Port %u could not detach "
@@ -9502,15 +9495,10 @@ int mlx5_action_handle_detach(struct rte_eth_dev *dev) {struct mlx5_priv *priv = dev->data->dev_private; -struct mlx5_indexed_pool *ipool = - priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS ]; -struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;int ret =0 ; -uint32_t idx; - - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; +struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { ret =mlx5_ind_table_obj_detach (dev, ind_tbl);if (ret !=0 ) { DRV_LOG(ERR, "Port %u could not detach "
@@ -9521,11 +9509,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev) }return 0 ; error: - shared_rss_last = shared_rss; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - -if (shared_rss == shared_rss_last) + ind_tbl_last = ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { +if (ind_tbl == ind_tbl_last)break ;if (mlx5_ind_table_obj_attach (dev, ind_tbl) !=0 ) DRV_LOG(CRIT, "Port %u could not attach "
drivers/net/mlx5/mlx5_flow.h +7 -0
@@ -1024,6 +1024,7 @@ struct rte_flow_hw {union {/* Jump action. */ struct mlx5_hw_jump_action *jump; +struct mlx5_hrxq *hrxq;/* TIR action. */ };struct rte_flow_template_table *table;/* The table flow allcated from. */ struct mlx5dr_rule rule;/* HWS layer data struct. */
@@ -1074,6 +1075,7 @@ struct mlx5_hw_actions {/* Dynamic action list. */ LIST_HEAD(act_list, mlx5_action_construct_data) act_list;struct mlx5_hw_jump_action *jump;/* Jump action. */ +struct mlx5_hrxq *tir;/* TIR action. */ uint32_t acts_num:4; /* Total action number. *//* Translated DR action array from action template. */ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ];
@@ -1910,6 +1912,11 @@ struct mlx5_list_entry *flow_dv_dest_array_clone_cb(void *tool_ctx, struct mlx5_list_entry *entry, void *cb_ctx);void flow_dv_dest_array_clone_free_cb (void *tool_ctx,struct mlx5_list_entry *entry); +void flow_dv_hashfields_set (uint64_t item_flags, +struct mlx5_flow_rss_desc *rss_desc, +uint64_t *hash_fields); +void flow_dv_action_rss_l34_hash_adjust (uint64_t rss_types, +uint64_t *hash_field);struct mlx5_list_entry *flow_hw_grp_create_cb (void *tool_ctx,void *cb_ctx);void flow_hw_grp_remove_cb (void *tool_ctx,struct mlx5_list_entry *entry);
drivers/net/mlx5/mlx5_flow_dv.c +76 -85
371 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_hw.c +101 -0
149 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_verbs.c +2 -5
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,MLX5_ASSERT (priv->drop_queue.hrxq); hrxq = priv->drop_queue.hrxq; }else { -uint32_t hrxq_idx;struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;MLX5_ASSERT (rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow, rss_desc->tunnel = !!(handle->layers &MLX5_FLOW_LAYER_TUNNEL ); rss_desc->shared_rss =0 ; - hrxq_idx =mlx5_hrxq_get (dev, rss_desc); - hrxq =mlx5_ipool_get (priv->sh->ipool[MLX5_IPOOL_HRXQ ], - hrxq_idx); + hrxq =mlx5_hrxq_get (dev, rss_desc);if (!hrxq) { rte_flow_error_set (error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow, "cannot get hash queue"); goto error; } - handle->rix_hrxq = hrxq_idx; + handle->rix_hrxq = hrxq->idx; }MLX5_ASSERT (hrxq); handle->drv_flow = mlx5_glue->create_flow
drivers/net/mlx5/mlx5_rx.h +7 -2
@@ -224,9 +224,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);struct mlx5_ind_table_obj *mlx5_ind_table_obj_get (struct rte_eth_dev *dev,const uint16_t *queues,uint32_t queues_n); +struct mlx5_ind_table_obj *mlx5_ind_table_obj_new (struct rte_eth_dev *dev, +const uint16_t *queues, +uint32_t queues_n, +bool standalone, +bool ref_qs);int mlx5_ind_table_obj_release (struct rte_eth_dev *dev,struct mlx5_ind_table_obj *ind_tbl, -bool standalone,bool deref_rxqs); int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev, struct mlx5_ind_table_obj *ind_tbl,
@@ -249,8 +253,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx, void *cb_ctx __rte_unused); void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,struct mlx5_list_entry *entry); -uint32_t mlx5_hrxq_get (struct rte_eth_dev *dev, +struct mlx5_hrxq *mlx5_hrxq_get (struct rte_eth_dev *dev,struct mlx5_flow_rss_desc *rss_desc); +int mlx5_hrxq_obj_release (struct rte_eth_dev *dev,struct mlx5_hrxq *hrxq);int mlx5_hrxq_release (struct rte_eth_dev *dev,uint32_t hxrq_idx);uint32_t mlx5_hrxq_verify (struct rte_eth_dev *dev);enum mlx5_rxq_type mlx5_rxq_get_type (struct rte_eth_dev *dev,uint16_t idx);
drivers/net/mlx5/mlx5_rxq.c +55 -30
209 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 12/14 net/mlx5: add mark action
In reply to Suanming MouThe mark action is covered by tag action internally. While it is added the HW will add a tag to the packet. The mark value can be set as fixed or dynamic as the action mask indicates.
--- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_flow.h | 1 + drivers/net/mlx5/mlx5_flow_hw.c | 66 ++++++++++++++++++++++++++++++--- 3 files changed, 63 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.h +2 -0
@@ -1531,6 +1531,8 @@ struct mlx5_priv {/* HW steering global drop action. */ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] [MLX5DR_TABLE_TYPE_MAX ]; +/* HW steering global drop action. */ +struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX ];struct mlx5_indexed_pool *acts_ipool;/* Action data indexed pool. */ #endif };
drivers/net/mlx5/mlx5_flow.h +1 -0
@@ -1077,6 +1077,7 @@ struct mlx5_hw_actions {struct mlx5_hw_jump_action *jump;/* Jump action. */ struct mlx5_hrxq *tir;/* TIR action. */ uint32_t acts_num:4; /* Total action number. */ + uint32_t mark:1; /* Indicate the mark action. *//* Translated DR action array from action template. */ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ]; };
drivers/net/mlx5/mlx5_flow_hw.c +60 -6
115 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 13/14 net/mlx5: add indirect action
In reply to Suanming MouHW steering can support indirect action as well. With indirect action, the flow can be created with more flexible shared RSS action selection. This will can save the action template with different RSS actions.
This commit adds the flow queue operation callback for:rte_flow_async_action_handle_create(); rte_flow_async_action_handle_destroy(); rte_flow_async_action_handle_update();
--- drivers/net/mlx5/mlx5_flow.c | 131 ++++++++++ drivers/net/mlx5/mlx5_flow.h | 59 +++++ drivers/net/mlx5/mlx5_flow_dv.c | 21 +- drivers/net/mlx5/mlx5_flow_hw.c | 414 +++++++++++++++++++++++++++++++- 4 files changed, 612 insertions(+), 13 deletions(-)
drivers/net/mlx5/mlx5_flow.c +131 -0
@@ -879,6 +879,29 @@ mlx5_flow_push(struct rte_eth_dev *dev,uint32_t queue, struct rte_flow_error *error); +static struct rte_flow_action_handle * +mlx5_flow_async_action_handle_create (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +const struct rte_flow_indir_action_conf *conf, +const struct rte_flow_action *action, +void *user_data, +struct rte_flow_error *error); + +static int +mlx5_flow_async_action_handle_update (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +const void *update, +void *user_data, +struct rte_flow_error *error); + +static int +mlx5_flow_async_action_handle_destroy (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +void *user_data, +struct rte_flow_error *error); +static const struct rte_flow_ops mlx5_flow_ops = { .validate = mlx5_flow_validate, .create = mlx5_flow_create,
@@ -911,6 +934,9 @@ static const struct rte_flow_ops mlx5_flow_ops = { .async_destroy = mlx5_flow_async_flow_destroy, .pull = mlx5_flow_pull, .push = mlx5_flow_push, + .async_action_handle_create = mlx5_flow_async_action_handle_create, + .async_action_handle_update = mlx5_flow_async_action_handle_update, + .async_action_handle_destroy = mlx5_flow_async_action_handle_destroy, };/* Tunnel information. */
@@ -8367,6 +8393,111 @@ mlx5_flow_push(struct rte_eth_dev *dev,return fops->push (dev, queue, error); } +/** +* Create shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] conf +* Indirect action configuration. +* @param[in] action +* rte_flow action detail. +* @param[in] user_data +* Pointer to the user_data. +* @param[out] error +* Pointer to error structure. +* +* @return +* Action handle on success, NULL otherwise and rte_errno is set. +*/ +static struct rte_flow_action_handle * +mlx5_flow_async_action_handle_create (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +const struct rte_flow_indir_action_conf *conf, +const struct rte_flow_action *action, +void *user_data, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->async_action_create (dev, queue, attr, conf, action, + user_data, error); +} + +/** +* Update shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] handle +* Action handle to be updated. +* @param[in] update +* Update value. +* @param[in] user_data +* Pointer to the user_data. +* @param[out] error +* Pointer to error structure. +* +* @return +* 0 on success, negative value otherwise and rte_errno is set. +*/ +static int +mlx5_flow_async_action_handle_update (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +const void *update, +void *user_data, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->async_action_update (dev, queue, attr, handle, + update, user_data, error); +} + +/** +* Destroy shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] handle +* Action handle to be destroyed. +* @param[in] user_data +* Pointer to the user_data. +* @param[out] error +* Pointer to error structure. +* +* @return +* 0 on success, negative value otherwise and rte_errno is set. +*/ +static int +mlx5_flow_async_action_handle_destroy (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +void *user_data, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->async_action_destroy (dev, queue, attr, handle, + user_data, error); +} +/** * Allocate a new memory for the counter values wrapped by all the needed * management.
drivers/net/mlx5/mlx5_flow.h +59 -0
99 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_dv.c +12 -9
@@ -13845,9 +13845,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,* @return * Valid hash RX queue index, otherwise 0. */ -static uint32_t -__flow_dv_action_rss_hrxq_lookup (struct rte_eth_dev *dev,uint32_t idx, -const uint64_t hash_fields) +uint32_t +flow_dv_action_rss_hrxq_lookup (struct rte_eth_dev *dev,uint32_t idx, +const uint64_t hash_fields) {struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_shared_action_rss *shared_rss =
@@ -13975,7 +13975,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,struct mlx5_hrxq *hrxq =NULL ;uint32_t hrxq_idx; - hrxq_idx =__flow_dv_action_rss_hrxq_lookup (dev, + hrxq_idx =flow_dv_action_rss_hrxq_lookup (dev, rss_desc->shared_rss, dev_flow->hash_fields);if (hrxq_idx)
@@ -14699,6 +14699,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,struct mlx5_shared_action_rss *shared_rss, struct rte_flow_error *error) { + struct mlx5_priv *priv = dev->data->dev_private;struct mlx5_flow_rss_desc rss_desc = {0 };size_t i;int err;
@@ -14719,6 +14720,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,/* Set non-zero value to indicate a shared RSS. */ rss_desc.shared_rss = action_idx; rss_desc.ind_tbl = shared_rss->ind_tbl; +if (priv->sh->config.dv_flow_en ==2 ) + rss_desc.hws_flags =MLX5DR_ACTION_FLAG_HWS_RX ;for (i =0 ; i <MLX5_RSS_HASH_FIELDS_LEN ; i++) {struct mlx5_hrxq *hrxq;uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14910,7 +14913,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,* A valid shared action handle in case of success, NULL otherwise and * rte_errno is set. */ -static struct rte_flow_action_handle * +struct rte_flow_action_handle * flow_dv_action_create(struct rte_eth_dev *dev, const struct rte_flow_indir_action_conf *conf, const struct rte_flow_action *action,
@@ -14980,7 +14983,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_destroy (struct rte_eth_dev *dev,struct rte_flow_action_handle *handle,struct rte_flow_error *error)
@@ -15190,7 +15193,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_update(struct rte_eth_dev *dev, struct rte_flow_action_handle *handle, const void *update,
@@ -15862,7 +15865,7 @@ flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data, "counters are not available"); } -static int +int flow_dv_action_query (struct rte_eth_dev *dev,const struct rte_flow_action_handle *handle,void *data,struct rte_flow_error *error)
@@ -17555,7 +17558,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_validate(struct rte_eth_dev *dev, const struct rte_flow_indir_action_conf *conf, const struct rte_flow_action *action,
drivers/net/mlx5/mlx5_flow_hw.c +410 -4
493 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 14/14 net/mlx5: add header reformat action
In reply to Suanming MouHW steering header reformat action can work under bulk mode. In this case, when create the table, bulk size of header reformat actions will be allocated in low level. Afterwards, when create flow, just simply specify the action index in the bulk and the encapsulation data to the action will be enough.
--- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_flow.h | 21 +++ drivers/net/mlx5/mlx5_flow_dv.c | 4 +- drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++- 4 files changed, 251 insertions(+), 3 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
@@ -342,6 +342,7 @@ struct mlx5_hw_q_job {uint32_t type;/* Job type. */ struct rte_flow_hw *flow;/* Flow attached to the job. */ void *user_data;/* Job user data. */ +uint8_t *encap_data;/* Encap data. */ };/* HW steering job descriptor LIFO pool. */
drivers/net/mlx5/mlx5_flow.h +21 -0
@@ -1040,6 +1040,14 @@ struct mlx5_action_construct_data {uint16_t action_src;/* rte_flow_action src offset. */ uint16_t action_dst;/* mlx5dr_rule_action dst offset. */ union { +struct { +/* encap src(item) offset. */ +uint16_t src; +/* encap dst data offset. */ +uint16_t dst; +/* encap data len. */ +uint16_t len; + } encap;struct {uint64_t types;/* RSS hash types. */ uint32_t level;/* RSS level. */
@@ -1076,6 +1084,13 @@ struct mlx5_hw_jump_action {struct mlx5dr_action *hws_action; }; +/* Encap decap action struct. */ +struct mlx5_hw_encap_decap_action { +struct mlx5dr_action *action;/* Action object. */ +size_t data_size;/* Action metadata size. */ +uint8_t data[];/* Action data. */ +}; +/* The maximum actions support in the flow. */ #define MLX5_HW_MAX_ACTS 16
@@ -1085,6 +1100,9 @@ struct mlx5_hw_actions { LIST_HEAD(act_list, mlx5_action_construct_data) act_list;struct mlx5_hw_jump_action *jump;/* Jump action. */ struct mlx5_hrxq *tir;/* TIR action. */ +/* Encap/Decap action. */ +struct mlx5_hw_encap_decap_action *encap_decap; +uint16_t encap_decap_pos;/* Encap/Decap action position. */ uint32_t acts_num:4; /* Total action number. */ uint32_t mark:1; /* Indicate the mark action. *//* Translated DR action array from action template. */
@@ -2027,4 +2045,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,const struct rte_flow_action_handle *handle, void *data, struct rte_flow_error *error); +size_t flow_dv_get_item_hdr_len (const enum rte_flow_item_type item_type); +int flow_dv_convert_encap_data (const struct rte_flow_item *items,uint8_t *buf, +size_t *size,struct rte_flow_error *error);#endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_dv.c +2 -2
@@ -4034,7 +4034,7 @@ flow_dv_push_vlan_action_resource_register* @return * sizeof struct item_type, 0 if void or irrelevant. */ -static size_t +size_t flow_dv_get_item_hdr_len (const enum rte_flow_item_type item_type) {size_t retval;
@@ -4100,7 +4100,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)* @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -static int +int flow_dv_convert_encap_data (const struct rte_flow_item *items,uint8_t *buf,size_t *size,struct rte_flow_error *error) {
drivers/net/mlx5/mlx5_flow_hw.c +227 -1
299 lines not on this page · raw message
Signature · 1 line
2.25.1
v4 cover net/mlx5: add hardware steering
In reply to Suanming MouThe Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion
- security, exposing the low-level steering entries directly to the userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.
---
v4:
v3:
- rebase to the latest version.
v2:
- New HW steering low-level abstract code added.
- commit message improvement.
- add protection for rte_flow and rte_flow_async callbacks.
- rebase to rte_flow_async v9.
- fix some rte_flow error not filled bugs.
Suanming Mou (14):
net/mlx5: introduce hardware steering operation net/mlx5: add HW steering low-level abstract code net/mlx5: introduce hardware steering enable routine net/mlx5: add port flow configuration net/mlx5: add pattern template management net/mlx5: add action template management net/mlx5: add table management
net/mlx5: add basic flow queue operation net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat action
doc/guides/nics/mlx5.rst | 19 +- doc/guides/rel_notes/release_22_03.rst | 1 + drivers/net/mlx5/linux/mlx5_flow_os.h | 1 + drivers/net/mlx5/linux/mlx5_os.c | 22 +- drivers/net/mlx5/meson.build | 2 + drivers/net/mlx5/mlx5.c | 55 +- drivers/net/mlx5/mlx5.h | 66 +- drivers/net/mlx5/mlx5_devx.c | 10 + drivers/net/mlx5/mlx5_dr.c | 383 ++++ drivers/net/mlx5/mlx5_dr.h | 456 +++++ drivers/net/mlx5/mlx5_flow.c | 721 ++++++- drivers/net/mlx5/mlx5_flow.h | 287 +++ drivers/net/mlx5/mlx5_flow_dv.c | 186 +- drivers/net/mlx5/mlx5_flow_hw.c | 2337 +++++++++++++++++++++++ drivers/net/mlx5/mlx5_flow_verbs.c | 7 +- drivers/net/mlx5/mlx5_rx.h | 9 +- drivers/net/mlx5/mlx5_rxq.c | 85 +- drivers/net/mlx5/windows/mlx5_flow_os.h | 1 + 18 files changed, 4468 insertions(+), 180 deletions(-) create mode 100644 drivers/net/mlx5/mlx5_dr.c create mode 100644 drivers/net/mlx5/mlx5_dr.h create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
Signature · 1 line
2.25.1
PATCH v4 1/14 net/mlx5: introduce hardware steering operation
The Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly
However, there are still some disadvantages:
- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion
- security, exposing the low-level steering entries directly to the userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based rte_flow APIs to prepare everything in advance to achieve high insertion rate.
In this new HW steering engine, the original SW steering rte_flow API will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter for dv_flow_en will be introduced and user will be able to engage the new steering engine.
This commit adds the basic driver operation.
--- drivers/net/mlx5/linux/mlx5_flow_os.h | 1 + drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5_flow.c | 1 + drivers/net/mlx5/mlx5_flow.h | 1 + drivers/net/mlx5/mlx5_flow_hw.c | 13 +++++++++++++ drivers/net/mlx5/windows/mlx5_flow_os.h | 1 + 6 files changed, 18 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_flow_hw.c
drivers/net/mlx5/linux/mlx5_flow_os.h +1 -0
@@ -9,6 +9,7 @@#ifdef HAVE_IBV_FLOW_DV_SUPPORT extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops; +extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;#endif /**
drivers/net/mlx5/meson.build +1 -0
@@ -16,6 +16,7 @@ sources = files( 'mlx5_flow.c', 'mlx5_flow_meter.c', 'mlx5_flow_dv.c', + 'mlx5_flow_hw.c', 'mlx5_flow_aso.c', 'mlx5_flow_flex.c', 'mlx5_mac.c',
drivers/net/mlx5/mlx5_flow.c +1 -0
@@ -76,6 +76,7 @@ const struct mlx5_flow_driver_ops *flow_drv_ops[] = { [MLX5_FLOW_TYPE_MIN] = &mlx5_flow_null_drv_ops,#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) [MLX5_FLOW_TYPE_DV] = &mlx5_flow_dv_drv_ops, + [MLX5_FLOW_TYPE_HW] = &mlx5_flow_hw_drv_ops, #endif [MLX5_FLOW_TYPE_VERBS] = &mlx5_flow_verbs_drv_ops, [MLX5_FLOW_TYPE_MAX ] = &mlx5_flow_null_drv_ops
drivers/net/mlx5/mlx5_flow.h +1 -0
@@ -452,6 +452,7 @@ enum mlx5_flow_drv_type {MLX5_FLOW_TYPE_MIN ,MLX5_FLOW_TYPE_DV ,MLX5_FLOW_TYPE_VERBS , +MLX5_FLOW_TYPE_HW , MLX5_FLOW_TYPE_MAX, };
drivers/net/mlx5/mlx5_flow_hw.c new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright (c) 2022 NVIDIA Corporation & Affiliates +*/ + +#include <rte_flow.h> + +#include "mlx5_flow.h" + +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) + +const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops; + +#endif
drivers/net/mlx5/windows/mlx5_flow_os.h +1 -0
@@ -10,6 +10,7 @@#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )extern const struct mlx5_flow_driver_ops mlx5_flow_dv_drv_ops; +extern const struct mlx5_flow_driver_ops mlx5_flow_hw_drv_ops;#endif /**
Signature · 1 line
2.25.1
PATCH v4 2/14 net/mlx5: add HW steering low-level abstract code
In reply to Suanming MouThe HW steering low-level implementation will be added later in another patch series. To avoid the linkage issues the abstract stub replacement is provided currently.
--- drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5_dr.c | 383 +++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_dr.h | 456 +++++++++++++++++++++++++++++++++++ 3 files changed, 840 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_dr.c create mode 100644 drivers/net/mlx5/mlx5_dr.h
drivers/net/mlx5/meson.build +1 -0
@@ -14,6 +14,7 @@ sources = files( 'mlx5.c', 'mlx5_ethdev.c', 'mlx5_flow.c', + 'mlx5_dr.c', 'mlx5_flow_meter.c', 'mlx5_flow_dv.c', 'mlx5_flow_hw.c',
drivers/net/mlx5/mlx5_dr.c new file +383 -0
383 lines not on this page · raw message
drivers/net/mlx5/mlx5_dr.h new file +456 -0
456 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code
Quoted 19 lines · Suanming Mou
The HW steering low-level implementation will be added later in another patch series. To avoid the linkage issues the abstract stub replacement is provided currently.
Signed-off-by Suanming Mou<[email protected]>--- drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5_dr.c | 383 +++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_dr.h | 456 +++++++++++++++++++++++++++++++++++ 3 files changed, 840 insertions(+)create mode 100644
drivers/net/mlx5/mlx5_dr.c
create mode 100644drivers/net/mlx5/mlx5_dr.hdrivers/net/mlx5/meson.build +0
-0
drivers/net/mlx5/meson.build sources = files( · line 20
Quoted 7 lines
drivers/net/mlx5/meson.build +1
-0@@ -14,6 +14,7 @@ sources = files( 'mlx5.c', 'mlx5_ethdev.c', 'mlx5_flow.c', + 'mlx5_dr.c', 'mlx5_flow_meter.c', 'mlx5_flow_dv.c', 'mlx5_flow_hw.c',
gives syntax warning [1], will fix while merging.
[1]./devtools/check-meson.py
Error parsing drivers/net/mlx5/meson.build:16, got some tabulation
Error: Incorrect indent at drivers/net/mlx5/meson.build:17
RE: [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract code
Hi,
Quoted 11 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, February 25, 2022 6:58 AM
To: Suanming Mou <[email protected]>; Slava Ovsiienko <[email protected]>; Matan Azrad <[email protected]>
Cc: Raslan Darawsheh <[email protected]>; Ori Kam <[email protected]>; [email protected]
Subject: Re: [PATCH v4 02/14] net/mlx5: add HW steering low-level abstract codeOn 2/24/2022 1:40 PM, Suanming Mou wrote:
[snip]
Quoted 10 lines
drivers/net/mlx5/meson.build sources = files( · line 20
Quoted 8 lines
drivers/net/mlx5/meson.build +1
-0@@ -14,6 +14,7 @@ sources = files( 'mlx5.c', 'mlx5_ethdev.c', 'mlx5_flow.c', + 'mlx5_dr.c', 'mlx5_flow_meter.c', 'mlx5_flow_dv.c', 'mlx5_flow_hw.c',gives syntax warning [1], will fix while merging.
Just checked, yes, seems vim auto indent placed tab instead of space here. Thank you for the fixing.
Quoted 6 lines
[1]
./devtools/check-meson.py
Error parsingdrivers/net/mlx5/meson.build:16, got some tabulation
Error: Incorrect indent atdrivers/net/mlx5/meson.build:17
PATCH v4 3/14 net/mlx5: introduce hardware steering enable routine
In reply to Suanming MouThe new hardware steering engine relies on using dedicated steering WQEs instead of direct writing to the low-level steering table entries directly. In the first introduce implementation the hardware steering engine supports the new queue based Flow API, the existing synchronous non-queue based Flow API is not supported.
A new dv_flow_en value 2 is added to manage mlx5 PMD steering engine:
dv_flow_en rte_flow API rte_flow_async API
------------------------------------------------
0 support not support
1 support not support
2 not support support
This commit introduces the extra dv_flow_en = 2 to specify the new flow initialize and manage operation routine.
--- doc/guides/nics/mlx5.rst | 13 ++++++++++--- drivers/net/mlx5/linux/mlx5_os.c | 4 ++++ drivers/net/mlx5/mlx5.c | 7 ++++++- drivers/net/mlx5/mlx5.h | 3 ++- drivers/net/mlx5/mlx5_flow.c | 22 ++++++++++++++++++++++ 5 files changed, 44 insertions(+), 5 deletions(-)
doc/guides/nics/mlx5.rst +10 -3
@@ -943,10 +943,17 @@ for an additional list of options shared with other mlx5 drivers. - ``dv_flow_en`` parameter [int] - A nonzero value enables the DV flow steering assuming it is supported - by the driver (RDMA Core library version is rdma-core-24.0 or higher). + Value 0 means legacy Verbs flow offloading. - Enabled by default if supported. + Value 1 enables the DV flow steering assuming it is supported by the + driver (RDMA Core library version is rdma-core-24.0 or higher). + + Value 2 enables the WQE based hardware steering. In this mode only + the queue-based rte_flow_q flow management is supported. + + Configured by default to 1 DV flow steering if the driver(RDMA CORE library) + supported. Otherwise, the value will be 0 which indicates legacy Verbs flow + offloading. - ``dv_esw_en`` parameter [int]
drivers/net/mlx5/linux/mlx5_os.c +4 -0
@@ -482,6 +482,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv) err =mlx5_alloc_table_hash_list (priv);if (err) goto error; +if (priv->sh->config.dv_flow_en ==2 ) +return 0 ;/* The resources below are only valid with DV support. */ #ifdef HAVE_IBV_FLOW_DV_SUPPORT /* Init port id action list. */
@@ -1535,6 +1537,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->drop_queue.hrxq =mlx5_drop_action_create (eth_dev);if (!priv->drop_queue.hrxq) goto error; +if (priv->sh->config.dv_flow_en ==2 ) +return eth_dev;/* Port representor shares the same max priority with pf port. */ if (!priv->sh->flow_priority_check_flag) {/* Supported Verbs flow priority number detection. */
drivers/net/mlx5/mlx5.c +6 -1
@@ -1199,7 +1199,12 @@ mlx5_dev_args_check_handler(const char *key, const char *val, void *opaque) }else if (strcmp (MLX5_DV_ESW_EN , key) ==0 ) { config->dv_esw_en = !!tmp; }else if (strcmp (MLX5_DV_FLOW_EN , key) ==0 ) { - config->dv_flow_en = !!tmp; +if (tmp >2 ) { +DRV_LOG (ERR ,"Invalid %s parameter." , key); + rte_errno =EINVAL ; +return -rte_errno; + } + config->dv_flow_en = tmp; }else if (strcmp (MLX5_DV_XMETA_EN , key) ==0 ) { if (tmp != MLX5_XMETA_MODE_LEGACY && tmp != MLX5_XMETA_MODE_META16 &&
drivers/net/mlx5/mlx5.h +2 -1
@@ -287,7 +287,8 @@ struct mlx5_sh_config {int tx_skew;/* Tx scheduling skew between WQE and data on wire. */ uint32_t reclaim_mode:2; /* Memory reclaim mode. */ uint32_t dv_esw_en:1; /* Enable E-Switch DV flow. */ - uint32_t dv_flow_en:1; /* Enable DV flow. */ +/* Enable DV flow. 1 means SW steering, 2 means HW steering. */ + unsigned int dv_flow_en:2; uint32_t dv_xmeta_en:2; /* Enable extensive flow metadata. */ uint32_t dv_miss_info:1; /* Restore packet after partial hw miss. */ uint32_t l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
drivers/net/mlx5/mlx5_flow.c +22 -0
@@ -6849,6 +6849,15 @@ mlx5_flow_create(struct rte_eth_dev *dev,const struct rte_flow_action actions[], struct rte_flow_error *error) { + struct mlx5_priv *priv = dev->data->dev_private; + +if (priv->sh->config.dv_flow_en ==2 ) { +rte_flow_error_set (error,ENOTSUP , +RTE_FLOW_ERROR_TYPE_UNSPECIFIED , +NULL , +"Flow non-Q creation not supported" ); +return NULL ; + }/* * If the device is not started yet, it is not allowed to created a * flow from application. PMD default flows and traffic control flows
@@ -7345,6 +7354,13 @@ mlx5_flow_destroy(struct rte_eth_dev *dev,struct rte_flow *flow, struct rte_flow_error *error __rte_unused) { + struct mlx5_priv *priv = dev->data->dev_private; + +if (priv->sh->config.dv_flow_en ==2 ) +return rte_flow_error_set (error,ENOTSUP , +RTE_FLOW_ERROR_TYPE_UNSPECIFIED , +NULL , +"Flow non-Q destruction not supported" );flow_list_destroy (dev,MLX5_FLOW_TYPE_GEN , (uintptr_t )(void *)flow);return 0 ;
@@ -7442,7 +7458,13 @@ mlx5_flow_query(struct rte_eth_dev *dev, struct rte_flow_error *error) {int ret; +struct mlx5_priv *priv = dev->data->dev_private; +if (priv->sh->config.dv_flow_en ==2 ) +return rte_flow_error_set (error,ENOTSUP , +RTE_FLOW_ERROR_TYPE_UNSPECIFIED , +NULL , +"Flow non-Q query not supported" ); ret =flow_drv_query (dev, (uintptr_t )(void *)flow, actions, data, error);if (ret <0 )
Signature · 1 line
2.25.1
PATCH v4 4/14 net/mlx5: add port flow configuration
In reply to Suanming MouThe hardware steering is backend to support rte_flow_async API in mlx5 PMD. The port configuration function creates the queues and needed flow management resources.
The PMD layer configuration function allocates the queues' context and per-queue job descriptor pool. The job descriptor pool size is equal to the queue size, and the job descriptors will be popped from pool with LIFO strategy to convey the flow information during flow insertion/destruction. Then, while polling the queued operation result, the flow information will be extracted from the job descriptor and the descriptor will be pushed back to the LIFO pool.
The commit creates the flow port queues and the job descriptor pools.
--- drivers/net/mlx5/mlx5.c | 3 + drivers/net/mlx5/mlx5.h | 30 +++++- drivers/net/mlx5/mlx5_flow.c | 86 +++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 15 +++ drivers/net/mlx5/mlx5_flow_hw.c | 159 ++++++++++++++++++++++++++++++++ 5 files changed, 292 insertions(+), 1 deletion(-)
drivers/net/mlx5/mlx5.c +3 -0
@@ -1820,6 +1820,9 @@ mlx5_dev_close(struct rte_eth_dev *dev)/* Free the eCPRI flex parser resource. */ mlx5_flex_parser_ecpri_release (dev);mlx5_flex_item_port_cleanup (dev); +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +flow_hw_resource_release (dev); +#endif if (priv->rxq_privs !=NULL ) {/* XXX race condition if mlx5_rx_burst() is still running. */ rte_delay_us_sleep (1000 );
drivers/net/mlx5/mlx5.h +29 -1
@@ -33,7 +33,9 @@#include "mlx5_utils.h" #include "mlx5_os.h" #include "mlx5_autoconf.h" - +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +#include "mlx5_dr.h" +#endif #define MLX5_SH (dev) (((struct mlx5_priv *)(dev)->data->dev_private)->sh)
@@ -320,6 +322,26 @@ struct mlx5_lb_ctx {uint16_t refcnt;/* Reference count for representors. */ }; +/* HW steering queue job descriptor type. */ +enum { +MLX5_HW_Q_JOB_TYPE_CREATE ,/* Flow create job type. */ +MLX5_HW_Q_JOB_TYPE_DESTROY ,/* Flow destroy job type. */ +}; + +/* HW steering flow management job descriptor. */ +struct mlx5_hw_q_job { +uint32_t type;/* Job type. */ +struct rte_flow *flow;/* Flow attached to the job. */ +void *user_data;/* Job user data. */ +}; + +/* HW steering job descriptor LIFO pool. */ +struct mlx5_hw_q { +uint32_t job_idx;/* Free job index. */ +uint32_t size;/* LIFO size. */ +struct mlx5_hw_q_job **job;/* LIFO header. */ +} __rte_cache_aligned; +#define MLX5_COUNTERS_PER_POOL 512#define MLX5_MAX_PENDING_QUERIES 4#define MLX5_CNT_CONTAINER_RESIZE 64
@@ -1480,6 +1502,12 @@ struct mlx5_priv {struct mlx5_flex_item flex_item[MLX5_PORT_FLEX_ITEM_NUM ];/* Flex items have been created on the port. */ uint32_t flex_item_map;/* Map of allocated flex item elements. */ +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ +uint32_t nb_queue;/* HW steering queue number. */ +/* HW steering queue polling mechanism job descriptor LIFO. */ +struct mlx5_hw_q *hw_q; +#endif };#define PORT_ID (priv) ((priv)->dev_data->port_id)
drivers/net/mlx5/mlx5_flow.c +86 -0
110 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +15 -0
@@ -1257,6 +1257,17 @@ typedef int (*mlx5_flow_item_update_t)const struct rte_flow_item_flex_handle *handle, const struct rte_flow_item_flex_conf *conf, struct rte_flow_error *error); +typedef int (*mlx5_flow_info_get_t ) + (struct rte_eth_dev *dev, +struct rte_flow_port_info *port_info, +struct rte_flow_queue_info *queue_info, +struct rte_flow_error *error); +typedef int (*mlx5_flow_port_configure_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_port_attr *port_attr, +uint16_t nb_queue, +const struct rte_flow_queue_attr *queue_attr[], +struct rte_flow_error *err);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1295,6 +1306,8 @@ struct mlx5_flow_driver_ops {mlx5_flow_item_create_t item_create;mlx5_flow_item_release_t item_release;mlx5_flow_item_update_t item_update; +mlx5_flow_info_get_t info_get; +mlx5_flow_port_configure_t configure; };/* mlx5_flow.c */
@@ -1762,4 +1775,6 @@ const struct mlx5_flow_tunnel * mlx5_get_tof(const struct rte_flow_item *items, const struct rte_flow_action *actions, enum mlx5_tof_rule_type *rule_type); +void +flow_hw_resource_release (struct rte_eth_dev *dev);#endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_hw.c +159 -0
169 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 5/14 net/mlx5: add pattern template management
In reply to Suanming MouThe pattern template defines flows that have the same matching fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be (eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while the values for each rule will be different.
Due to the pattern template can be used in different domains, the items will only be cached in pattern template create stage, while the template is binded to a dedicated table, the HW criteria will be created and saved to the table. The pattern templates can be used by multiple tables. But different tables create the same criteria and will not share the matcher between each other in order to have better performance.
This commit adds pattern template management.
--- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_flow.c | 76 +++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 24 +++++++++ drivers/net/mlx5/mlx5_flow_hw.c | 86 +++++++++++++++++++++++++++++++++ 4 files changed, 188 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
@@ -1503,6 +1503,8 @@ struct mlx5_priv {/* Flex items have been created on the port. */ uint32_t flex_item_map;/* Map of allocated flex item elements. */ #if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +/* Item template list. */ + LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt;struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +76 -0
94 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +24 -0
@@ -1015,6 +1015,19 @@ struct rte_flow {uint32_t geneve_tlv_option;/**< Holds Geneve TLV option id. > */ } __rte_packed; +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) + +/* Flow item template struct. */ +struct rte_flow_pattern_template { +LIST_ENTRY (rte_flow_pattern_template) next; +/* Template attributes. */ +struct rte_flow_pattern_template_attr attr; +struct mlx5dr_match_template *mt;/* mlx5 match template. */ +uint32_t refcnt;/* Reference counter. */ +}; + +#endif +/* * Define list of valid combinations of RX Hash fields * (see enum ibv_rx_hash_fields).
@@ -1268,6 +1281,15 @@ typedef int (*mlx5_flow_port_configure_t)uint16_t nb_queue, const struct rte_flow_queue_attr *queue_attr[], struct rte_flow_error *err); +typedef struct rte_flow_pattern_template *(*mlx5_flow_pattern_template_create_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_pattern_template_attr *attr, +const struct rte_flow_item items[], +struct rte_flow_error *error); +typedef int (*mlx5_flow_pattern_template_destroy_t ) + (struct rte_eth_dev *dev, +struct rte_flow_pattern_template *template, +struct rte_flow_error *error);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1308,6 +1330,8 @@ struct mlx5_flow_driver_ops {mlx5_flow_item_update_t item_update;mlx5_flow_info_get_t info_get;mlx5_flow_port_configure_t configure; +mlx5_flow_pattern_template_create_t pattern_template_create; +mlx5_flow_pattern_template_destroy_t pattern_template_destroy; };/* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +86 -0
107 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 6/14 net/mlx5: add action template management
In reply to Suanming MouThe action template holds a list of action types that will be used together on the same rule. The template's actions instances will be created only when the template bind to the dedicated group. And the created actions will be saved to each individual group in order for best performance. The actions in a group will not be shared with each other unless shared actions are specified.
This commit adds the action template management which stores the flow action template.
--- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_flow.c | 78 +++++++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 22 ++++++ drivers/net/mlx5/mlx5_flow_hw.c | 116 ++++++++++++++++++++++++++++++++ 4 files changed, 218 insertions(+)
drivers/net/mlx5/mlx5.h +2 -0
@@ -1505,6 +1505,8 @@ struct mlx5_priv {#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )/* Item template list. */ LIST_HEAD(flow_hw_itt, rte_flow_pattern_template) flow_hw_itt; +/* Action template list. */ + LIST_HEAD(flow_hw_at, rte_flow_actions_template) flow_hw_at;struct mlx5dr_context *dr_ctx;/**< HW steering DR context. */ uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */
drivers/net/mlx5/mlx5_flow.c +78 -0
96 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +22 -0
@@ -1026,6 +1026,16 @@ struct rte_flow_pattern_template { uint32_t refcnt; /* Reference counter. */ }; +/* Flow action template struct. */ +struct rte_flow_actions_template { +LIST_ENTRY (rte_flow_actions_template) next; +/* Template attributes. */ +struct rte_flow_actions_template_attr attr; +struct rte_flow_action *actions;/* Cached flow actions. */ +struct rte_flow_action *masks;/* Cached action masks.*/ +uint32_t refcnt;/* Reference counter. */ +}; +#endif /*
@@ -1290,6 +1300,16 @@ typedef int (*mlx5_flow_pattern_template_destroy_t) (struct rte_eth_dev *dev, struct rte_flow_pattern_template *template, struct rte_flow_error *error); +typedef struct rte_flow_actions_template *(*mlx5_flow_actions_template_create_t ) + (struct rte_eth_dev *dev, +const struct rte_flow_actions_template_attr *attr, +const struct rte_flow_action actions[], +const struct rte_flow_action masks[], +struct rte_flow_error *error); +typedef int (*mlx5_flow_actions_template_destroy_t ) + (struct rte_eth_dev *dev, +struct rte_flow_actions_template *template, +struct rte_flow_error *error);struct mlx5_flow_driver_ops {mlx5_flow_validate_t validate;
@@ -1332,6 +1352,8 @@ struct mlx5_flow_driver_ops {mlx5_flow_port_configure_t configure;mlx5_flow_pattern_template_create_t pattern_template_create;mlx5_flow_pattern_template_destroy_t pattern_template_destroy; +mlx5_flow_actions_template_create_t actions_template_create; +mlx5_flow_actions_template_destroy_t actions_template_destroy; };/* mlx5_flow.c */
drivers/net/mlx5/mlx5_flow_hw.c +116 -0
140 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 7/14 net/mlx5: add table management
In reply to Suanming MouFlow table is a group of flows with the same matching criteria and the same actions defined for them. The table defines rules that have the same matching fields but with different matching values. For example, matching on 5 tuple, the table will be (IPv4 source + IPv4 dest + s_port + d_port + next_proto) while the values for each rule will be different.
The templates' relevant matching criteria and action instances will be created in the table creation and saved in the table. As table attributes indicate the supported flow number, the flow memory will also be allocated at the same time.
This commit adds the table management functions.
--- drivers/net/mlx5/mlx5.c | 45 ++- drivers/net/mlx5/mlx5.h | 21 +- drivers/net/mlx5/mlx5_flow.c | 93 ++++++ drivers/net/mlx5/mlx5_flow.h | 73 +++++ drivers/net/mlx5/mlx5_flow_hw.c | 527 ++++++++++++++++++++++++++++++++ 5 files changed, 753 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.c +41 -4
@@ -1607,12 +1607,46 @@ void mlx5_free_table_hash_list(struct mlx5_priv *priv) {struct mlx5_dev_ctx_shared *sh = priv->sh; - -if (!sh->flow_tbls) +struct mlx5_hlist **tbls = (priv->sh->config.dv_flow_en ==2 ) ? + &sh->groups : &sh->flow_tbls; +if (*tbls ==NULL )return ; -mlx5_hlist_destroy (sh->flow_tbls); - sh->flow_tbls =NULL ; +mlx5_hlist_destroy (*tbls); + *tbls =NULL ; +} + +#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +/** +* Allocate HW steering group hash list. +* +* @param[in] priv +* Pointer to the private device data structure. +*/ +static int +mlx5_alloc_hw_group_hash_list (struct mlx5_priv *priv) +{ +int err =0 ; +struct mlx5_dev_ctx_shared *sh = priv->sh; +char s[MLX5_NAME_SIZE ]; + +MLX5_ASSERT (sh); +snprintf (s,sizeof (s),"%s_flow_groups" , priv->sh->ibdev_name); + sh->groups =mlx5_hlist_create + (s,MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE , + false, true, sh, + flow_hw_grp_create_cb, + flow_hw_grp_match_cb, + flow_hw_grp_remove_cb, + flow_hw_grp_clone_cb, + flow_hw_grp_clone_free_cb); +if (!sh->groups) { +DRV_LOG (ERR ,"flow groups with hash creation failed." ); + err =ENOMEM ; + } +return err; } +#endif +/** * Initialize flow table hash list and create the root tables entry
@@ -1628,11 +1662,14 @@ int mlx5_alloc_table_hash_list(struct mlx5_priv *priv __rte_unused) {int err =0 ; +/* Tables are only used in DV and DR modes. */ #if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )struct mlx5_dev_ctx_shared *sh = priv->sh;char s[MLX5_NAME_SIZE ]; +if (priv->sh->config.dv_flow_en ==2 ) +return mlx5_alloc_hw_group_hash_list (priv);MLX5_ASSERT (sh);snprintf (s,sizeof (s),"%s_flow_table" , priv->sh->ibdev_name); sh->flow_tbls = mlx5_hlist_create(s, MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE,
drivers/net/mlx5/mlx5.h +19 -2
@@ -64,7 +64,9 @@ enum mlx5_ipool_index {MLX5_IPOOL_PUSH_VLAN ,/* Pool for push vlan resource. */ MLX5_IPOOL_TAG ,/* Pool for tag resource. */ MLX5_IPOOL_PORT_ID ,/* Pool for port id resource. */ -MLX5_IPOOL_JUMP ,/* Pool for jump resource. */ +MLX5_IPOOL_JUMP ,/* Pool for SWS jump resource. */ +/* Pool for HWS group. Jump action will be created internally. */ +MLX5_IPOOL_HW_GRP =MLX5_IPOOL_JUMP ,MLX5_IPOOL_SAMPLE ,/* Pool for sample resource. */ MLX5_IPOOL_DEST_ARRAY ,/* Pool for destination array resource. */ MLX5_IPOOL_TUNNEL_ID, /* Pool for tunnel offload context */
@@ -108,6 +110,13 @@ enum mlx5_delay_drop_mode { MLX5_DELAY_DROP_HAIRPIN = RTE_BIT32(1), /* Hairpin queues enable. */ }; +/* The HWS action type root/non-root. */ +enum mlx5_hw_action_flag_type { +MLX5_HW_ACTION_FLAG_ROOT ,/* Root action. */ +MLX5_HW_ACTION_FLAG_NONE_ROOT ,/* Non-root ation. */ +MLX5_HW_ACTION_FLAG_MAX ,/* Maximum action flag. */ +}; +/* Hlist and list callback context. */ struct mlx5_flow_cb_ctx {struct rte_eth_dev *dev;
@@ -1204,7 +1213,10 @@ struct mlx5_dev_ctx_shared {rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX ];/* UAR same-page access control required in 32bit implementations. */ #endif -struct mlx5_hlist *flow_tbls; +union { +struct mlx5_hlist *flow_tbls;/* SWS flow table. */ +struct mlx5_hlist *groups;/* HWS flow group. */ + };struct mlx5_flow_tunnel_hub *tunnel_hub;/* Direct Rules tables for FDB, NIC TX+RX */ void *dr_drop_action;/* Pointer to DR drop action, any domain. */
@@ -1511,6 +1523,11 @@ struct mlx5_priv {uint32_t nb_queue;/* HW steering queue number. */ /* HW steering queue polling mechanism job descriptor LIFO. */ struct mlx5_hw_q *hw_q; +/* HW steering rte flow table list header. */ + LIST_HEAD(flow_hw_tbl, rte_flow_template_table) flow_hw_tbl; +/* HW steering global drop action. */ +struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] + [MLX5DR_TABLE_TYPE_MAX ];#endif };
drivers/net/mlx5/mlx5_flow.c +93 -0
111 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +73 -0
97 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_hw.c +527 -0
576 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 8/14 net/mlx5: add basic flow queue operation
In reply to Suanming MouThe HW steering uses async queue-based flow rules management mechanism. The matcher and part of the actions have been prepared during flow table creation. Some remaining actions will be constructed during flow creation if needed.
A flow postpone attribute bit describes if flow management should be applied to the HW directly. An extra push function is provided to force push all the cached flows to the HW.
Once the flow has been applied to the HW, the pull function will be called to get the queued creation/destruction flows.
The DR rule flow memory is represented in PMD layer instead of allocating from HW steering layer. While destroying the flow, the flow rule memory can only be freed after the CQE received.
The HW queue job descriptor is currently introduced to convey the flow information and operation type between the flow insertion/destruction in the pull function.
This commit adds the basic flow queue operation for:rte_flow_async_create(); rte_flow_async_destroy();rte_flow_push();rte_flow_pull();
--- doc/guides/nics/mlx5.rst | 6 + doc/guides/rel_notes/release_22_03.rst | 1 + drivers/net/mlx5/mlx5.h | 2 +- drivers/net/mlx5/mlx5_flow.c | 188 ++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 41 ++++ drivers/net/mlx5/mlx5_flow_hw.c | 292 ++++++++++++++++++++++++- 6 files changed, 528 insertions(+), 2 deletions(-)
doc/guides/nics/mlx5.rst +6 -0
@@ -488,6 +488,12 @@ Limitations from the reference "Clock Queue" completions, the scheduled send timestamps should not be specified with non-zero MSB. + - HW steering: + + - WQE based high scaling and safer flow insertion/destruction. + - Set ``dv_flow_en`` to 2 in order to enable HW steering. + - Async queue-based ``rte_flow_q`` APIs supported only. + Statistics ----------
doc/guides/rel_notes/release_22_03.rst +1 -0
@@ -164,6 +164,7 @@ New Features * Support ConnectX-7 capability to schedule traffic sending on timestamp * Support steering for external Rx queue created outside the PMD. + * Added WQE based hardware steering support with ``rte_flow_async`` API. * **Updated Wangxun ngbe driver.**
drivers/net/mlx5/mlx5.h +1 -1
@@ -340,7 +340,7 @@ enum {/* HW steering flow management job descriptor. */ struct mlx5_hw_q_job {uint32_t type;/* Job type. */ -struct rte_flow *flow;/* Flow attached to the job. */ +struct rte_flow_hw *flow;/* Flow attached to the job. */ void *user_data;/* Job user data. */ };
drivers/net/mlx5/mlx5_flow.c +188 -0
206 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow.h +41 -0
65 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_hw.c +291 -1
316 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 9/14 net/mlx5: add flow flush function
In reply to Suanming MouIn case port is being stopped, all created flows should be flushed. This commit adds the flow flush helper function.
--- drivers/net/mlx5/mlx5_flow.c | 8 ++ drivers/net/mlx5/mlx5_flow_hw.c | 129 ++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+)
drivers/net/mlx5/mlx5_flow.c +8 -0
@@ -7024,6 +7024,14 @@ mlx5_flow_list_flush(struct rte_eth_dev *dev, enum mlx5_flow_type type,uint32_t num_flushed =0 , fidx =1 ;struct rte_flow *flow; +#ifdef HAVE_IBV_FLOW_DV_SUPPORT +if (priv->sh->config.dv_flow_en ==2 && + type ==MLX5_FLOW_TYPE_GEN ) { +flow_hw_q_flow_flush (dev,NULL ); +return ; + } +#endif + MLX5_IPOOL_FOREACH(priv->flows[type], fidx, flow) {flow_list_destroy (dev, type, fidx); num_flushed++;
drivers/net/mlx5/mlx5_flow_hw.c +129 -0
141 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 10/14 net/mlx5: add flow jump action
In reply to Suanming MouJump action connects different level of flow tables and allows packet handling in the chain of flows.
A new action construct data struct is also added in this commit to help to handle not only the dynamic jump action but also for the other generic dynamic actions. The actions with empty mask configuration means dynamic action, and the dedicated action will be created with the flow action configuration during flow creation. In that dynamic action case, the action will be appended to the table template's action list during table creation. When creating the flows, traverse the action list and pick the dynamic action configuration details from flow actions as the action construct data struct describes, then create the dedicated dynamic actions.
This commit adds the jump action and the generic dynamic action construct mechanism.
--- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_flow.h | 25 ++- drivers/net/mlx5/mlx5_flow_hw.c | 270 +++++++++++++++++++++++++++++--- 3 files changed, 275 insertions(+), 21 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
@@ -1528,6 +1528,7 @@ struct mlx5_priv {/* HW steering global drop action. */ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] [MLX5DR_TABLE_TYPE_MAX ]; +struct mlx5_indexed_pool *acts_ipool;/* Action data indexed pool. */ #endif };
drivers/net/mlx5/mlx5_flow.h +24 -1
@@ -1020,10 +1020,25 @@ struct rte_flow {/* HWS flow struct. */ struct rte_flow_hw {uint32_t idx;/* Flow index from indexed pool. */ +uint32_t fate_type;/* Fate action type. */ +union { +/* Jump action. */ +struct mlx5_hw_jump_action *jump; + };struct rte_flow_template_table *table;/* The table flow allcated from. */ struct mlx5dr_rule rule;/* HWS layer data struct. */ } __rte_packed; +/* rte flow action translate to DR action struct. */ +struct mlx5_action_construct_data { +LIST_ENTRY (mlx5_action_construct_data) next; +/* Ensure the action types are matched. */ +int type; +uint32_t idx;/* Data index. */ +uint16_t action_src;/* rte_flow_action src offset. */ +uint16_t action_dst;/* mlx5dr_rule_action dst offset. */ +}; +/* Flow item template struct. */ struct rte_flow_pattern_template {LIST_ENTRY (rte_flow_pattern_template) next;
@@ -1051,9 +1066,17 @@ struct mlx5_hw_jump_action {struct mlx5dr_action *hws_action; }; +/* The maximum actions support in the flow. */ +#define MLX5_HW_MAX_ACTS 16 +/* DR action set struct. */ struct mlx5_hw_actions { -struct mlx5dr_action *drop;/* Drop action. */ +/* Dynamic action list. */ + LIST_HEAD(act_list, mlx5_action_construct_data) act_list; +struct mlx5_hw_jump_action *jump;/* Jump action. */ +uint32_t acts_num:4 ;/* Total action number. */ +/* Translated DR action array from action template. */ +struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ]; };/* mlx5 action template struct. */
drivers/net/mlx5/mlx5_flow_hw.c +250 -20
399 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 11/14 net/mlx5: add queue and RSS action
In reply to Suanming MouThis commit adds the queue and RSS action. Similar to the jump action, dynamic ones will be added to the action construct list.
Due to the queue and RSS action in template should not be destroyed during port restart, the actions are created with standalone indirect table as indirect action does. When port stops, detaches the indirect table from action, when port starts, attaches the indirect table back to the action.
One more change is made to accelerate the action creation. Currently the mlx5_hrxq_get() function returns the object index instead of object pointer. This introduced an extra converting the index to the object by calling mlx5_ipool_get() in most of the case. And that extra converting hurts multi-thread performance since mlx5_ipool_get() uses the global lock inside. As the hash Rx queue object itself also contains the index, returns the object directly will achieve better performance without the global lock.
--- drivers/net/mlx5/linux/mlx5_os.c | 18 ++-- drivers/net/mlx5/mlx5.h | 4 + drivers/net/mlx5/mlx5_devx.c | 10 ++ drivers/net/mlx5/mlx5_flow.c | 38 +++---- drivers/net/mlx5/mlx5_flow.h | 7 ++ drivers/net/mlx5/mlx5_flow_dv.c | 161 ++++++++++++++--------------- drivers/net/mlx5/mlx5_flow_hw.c | 101 ++++++++++++++++++ drivers/net/mlx5/mlx5_flow_verbs.c | 7 +- drivers/net/mlx5/mlx5_rx.h | 9 +- drivers/net/mlx5/mlx5_rxq.c | 85 +++++++++------ 10 files changed, 283 insertions(+), 157 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +9 -9
@@ -1537,6 +1537,15 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->drop_queue.hrxq =mlx5_drop_action_create (eth_dev);if (!priv->drop_queue.hrxq) goto error; + priv->hrxqs =mlx5_list_create ("hrxq" , eth_dev, true, + mlx5_hrxq_create_cb, + mlx5_hrxq_match_cb, + mlx5_hrxq_remove_cb, + mlx5_hrxq_clone_cb, + mlx5_hrxq_clone_free_cb); +if (!priv->hrxqs) + goto error; +rte_rwlock_init (&priv->ind_tbls_lock);if (priv->sh->config.dv_flow_en ==2 )return eth_dev;/* Port representor shares the same max priority with pf port. */
@@ -1561,15 +1570,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, err =ENOTSUP ; goto error; } - priv->hrxqs =mlx5_list_create ("hrxq" , eth_dev, true, - mlx5_hrxq_create_cb, - mlx5_hrxq_match_cb, - mlx5_hrxq_remove_cb, - mlx5_hrxq_clone_cb, - mlx5_hrxq_clone_free_cb); -if (!priv->hrxqs) - goto error; -rte_rwlock_init (&priv->ind_tbls_lock);/* Query availability of metadata reg_c's. */ if (!priv->sh->metadata_regc_check_flag) { err =mlx5_flow_discover_mreg_c (eth_dev);
drivers/net/mlx5/mlx5.h +4 -0
@@ -1287,6 +1287,7 @@ struct mlx5_flow_rss_desc {uint64_t hash_fields;/* Verbs Hash fields. */ uint8_t key[MLX5_RSS_HASH_KEY_LEN ];/**< RSS hash key. */ uint32_t key_len;/**< RSS hash key len. */ +uint32_t hws_flags;/**< HW steering action. */ uint32_t tunnel;/**< Queue in tunnel. */ uint32_t shared_rss;/**< Shared RSS index. */ struct mlx5_ind_table_obj *ind_tbl;
@@ -1348,6 +1349,7 @@ struct mlx5_hrxq {#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H )void *action;/* DV QP action pointer. */ #endif +uint32_t hws_flags;/* Hw steering flags. */ uint64_t hash_fields;/* Verbs Hash fields. */ uint32_t rss_key_len;/* Hash key length in bytes. */ uint32_t idx;/* Hash Rx queue index. */
@@ -1479,6 +1481,8 @@ struct mlx5_priv { LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. *//* Indirection tables. */ LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls; +/* Standalone indirect tables. */ + LIST_HEAD(stdl_ind_tables, mlx5_ind_table_obj) standalone_ind_tbls;/* Pointer to next element. */ rte_rwlock_t ind_tbls_lock;uint32_t refcnt;/**< Reference counter. */
drivers/net/mlx5/mlx5_devx.c +10 -0
@@ -815,6 +815,14 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq, goto error; }#if defined(HAVE_IBV_FLOW_DV_SUPPORT ) || !defined(HAVE_INFINIBAND_VERBS_H ) +if (hrxq->hws_flags) { + hrxq->action =mlx5dr_action_create_dest_tir + (priv->dr_ctx, + (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags); +if (!hrxq->action) + goto error; +return 0 ; + }if (mlx5_flow_os_create_flow_action_dest_devx_tir (hrxq->tir, &hrxq->action)) { rte_errno = errno;
@@ -1050,6 +1058,8 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev)DRV_LOG (ERR ,"Cannot create drop RX queue" );return ret; } +if (priv->sh->config.dv_flow_en ==2 ) +return 0 ;/* hrxq->ind_table queues are NULL, drop RX queue ID will be used */ ret =mlx5_devx_ind_table_new (dev,0 , hrxq->ind_table);if (ret !=0 ) {
drivers/net/mlx5/mlx5_flow.c +12 -26
@@ -9448,14 +9448,10 @@ int mlx5_action_handle_attach(struct rte_eth_dev *dev) {struct mlx5_priv *priv = dev->data->dev_private; -struct mlx5_indexed_pool *ipool = - priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS ]; -struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;int ret =0 ; -uint32_t idx; +struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) {const char *message;uint32_t queue_idx;
@@ -9471,9 +9467,7 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev) }if (ret !=0 )return ret; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { ret =mlx5_ind_table_obj_attach (dev, ind_tbl);if (ret !=0 ) { DRV_LOG(ERR, "Port %u could not attach "
@@ -9482,13 +9476,12 @@ mlx5_action_handle_attach(struct rte_eth_dev *dev) goto error; } } +return 0 ; error: - shared_rss_last = shared_rss; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - -if (shared_rss == shared_rss_last) + ind_tbl_last = ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { +if (ind_tbl == ind_tbl_last)break ;if (mlx5_ind_table_obj_detach (dev, ind_tbl) !=0 ) DRV_LOG(CRIT, "Port %u could not detach "
@@ -9511,15 +9504,10 @@ int mlx5_action_handle_detach(struct rte_eth_dev *dev) {struct mlx5_priv *priv = dev->data->dev_private; -struct mlx5_indexed_pool *ipool = - priv->sh->ipool[MLX5_IPOOL_RSS_SHARED_ACTIONS ]; -struct mlx5_shared_action_rss *shared_rss, *shared_rss_last;int ret =0 ; -uint32_t idx; - - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; +struct mlx5_ind_table_obj *ind_tbl, *ind_tbl_last; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { ret =mlx5_ind_table_obj_detach (dev, ind_tbl);if (ret !=0 ) { DRV_LOG(ERR, "Port %u could not detach "
@@ -9530,11 +9518,9 @@ mlx5_action_handle_detach(struct rte_eth_dev *dev) }return 0 ; error: - shared_rss_last = shared_rss; - ILIST_FOREACH(ipool, priv->rss_shared_actions, idx, shared_rss, next) { -struct mlx5_ind_table_obj *ind_tbl = shared_rss->ind_tbl; - -if (shared_rss == shared_rss_last) + ind_tbl_last = ind_tbl; + LIST_FOREACH(ind_tbl, &priv->standalone_ind_tbls, next) { +if (ind_tbl == ind_tbl_last)break ;if (mlx5_ind_table_obj_attach (dev, ind_tbl) !=0 ) DRV_LOG(CRIT, "Port %u could not attach "
drivers/net/mlx5/mlx5_flow.h +7 -0
@@ -1024,6 +1024,7 @@ struct rte_flow_hw {union {/* Jump action. */ struct mlx5_hw_jump_action *jump; +struct mlx5_hrxq *hrxq;/* TIR action. */ };struct rte_flow_template_table *table;/* The table flow allcated from. */ struct mlx5dr_rule rule;/* HWS layer data struct. */
@@ -1074,6 +1075,7 @@ struct mlx5_hw_actions {/* Dynamic action list. */ LIST_HEAD(act_list, mlx5_action_construct_data) act_list;struct mlx5_hw_jump_action *jump;/* Jump action. */ +struct mlx5_hrxq *tir;/* TIR action. */ uint32_t acts_num:4; /* Total action number. *//* Translated DR action array from action template. */ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ];
@@ -1910,6 +1912,11 @@ struct mlx5_list_entry *flow_dv_dest_array_clone_cb(void *tool_ctx, struct mlx5_list_entry *entry, void *cb_ctx);void flow_dv_dest_array_clone_free_cb (void *tool_ctx,struct mlx5_list_entry *entry); +void flow_dv_hashfields_set (uint64_t item_flags, +struct mlx5_flow_rss_desc *rss_desc, +uint64_t *hash_fields); +void flow_dv_action_rss_l34_hash_adjust (uint64_t rss_types, +uint64_t *hash_field);struct mlx5_list_entry *flow_hw_grp_create_cb (void *tool_ctx,void *cb_ctx);void flow_hw_grp_remove_cb (void *tool_ctx,struct mlx5_list_entry *entry);
drivers/net/mlx5/mlx5_flow_dv.c +76 -85
371 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_hw.c +101 -0
149 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_verbs.c +2 -5
@@ -1943,7 +1943,6 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow,MLX5_ASSERT (priv->drop_queue.hrxq); hrxq = priv->drop_queue.hrxq; }else { -uint32_t hrxq_idx;struct mlx5_flow_rss_desc *rss_desc = &wks->rss_desc;MLX5_ASSERT (rss_desc->queue_num);
@@ -1952,9 +1951,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow, rss_desc->tunnel = !!(handle->layers &MLX5_FLOW_LAYER_TUNNEL ); rss_desc->shared_rss =0 ; - hrxq_idx =mlx5_hrxq_get (dev, rss_desc); - hrxq =mlx5_ipool_get (priv->sh->ipool[MLX5_IPOOL_HRXQ ], - hrxq_idx); + hrxq =mlx5_hrxq_get (dev, rss_desc);if (!hrxq) { rte_flow_error_set (error, rte_errno,
@@ -1962,7 +1959,7 @@ flow_verbs_apply(struct rte_eth_dev *dev, struct rte_flow *flow, "cannot get hash queue"); goto error; } - handle->rix_hrxq = hrxq_idx; + handle->rix_hrxq = hrxq->idx; }MLX5_ASSERT (hrxq); handle->drv_flow = mlx5_glue->create_flow
drivers/net/mlx5/mlx5_rx.h +7 -2
@@ -231,9 +231,13 @@ int mlx5_ind_table_obj_verify(struct rte_eth_dev *dev);struct mlx5_ind_table_obj *mlx5_ind_table_obj_get (struct rte_eth_dev *dev,const uint16_t *queues,uint32_t queues_n); +struct mlx5_ind_table_obj *mlx5_ind_table_obj_new (struct rte_eth_dev *dev, +const uint16_t *queues, +uint32_t queues_n, +bool standalone, +bool ref_qs);int mlx5_ind_table_obj_release (struct rte_eth_dev *dev,struct mlx5_ind_table_obj *ind_tbl, -bool standalone,bool deref_rxqs); int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev, struct mlx5_ind_table_obj *ind_tbl,
@@ -256,8 +260,9 @@ struct mlx5_list_entry *mlx5_hrxq_clone_cb(void *tool_ctx, void *cb_ctx __rte_unused); void mlx5_hrxq_clone_free_cb(void *tool_ctx __rte_unused,struct mlx5_list_entry *entry); -uint32_t mlx5_hrxq_get (struct rte_eth_dev *dev, +struct mlx5_hrxq *mlx5_hrxq_get (struct rte_eth_dev *dev,struct mlx5_flow_rss_desc *rss_desc); +int mlx5_hrxq_obj_release (struct rte_eth_dev *dev,struct mlx5_hrxq *hrxq);int mlx5_hrxq_release (struct rte_eth_dev *dev,uint32_t hxrq_idx);uint32_t mlx5_hrxq_verify (struct rte_eth_dev *dev);bool mlx5_rxq_is_hairpin (struct rte_eth_dev *dev,uint16_t idx);
drivers/net/mlx5/mlx5_rxq.c +55 -30
209 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 12/14 net/mlx5: add mark action
In reply to Suanming MouThe mark action is covered by tag action internally. While it is added the HW will add a tag to the packet. The mark value can be set as fixed or dynamic as the action mask indicates.
--- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_flow.h | 1 + drivers/net/mlx5/mlx5_flow_hw.c | 66 ++++++++++++++++++++++++++++++--- 3 files changed, 63 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5.h +2 -0
@@ -1532,6 +1532,8 @@ struct mlx5_priv {/* HW steering global drop action. */ struct mlx5dr_action *hw_drop[MLX5_HW_ACTION_FLAG_MAX ] [MLX5DR_TABLE_TYPE_MAX ]; +/* HW steering global drop action. */ +struct mlx5dr_action *hw_tag[MLX5_HW_ACTION_FLAG_MAX ];struct mlx5_indexed_pool *acts_ipool;/* Action data indexed pool. */ #endif };
drivers/net/mlx5/mlx5_flow.h +1 -0
@@ -1077,6 +1077,7 @@ struct mlx5_hw_actions {struct mlx5_hw_jump_action *jump;/* Jump action. */ struct mlx5_hrxq *tir;/* TIR action. */ uint32_t acts_num:4; /* Total action number. */ + uint32_t mark:1; /* Indicate the mark action. *//* Translated DR action array from action template. */ struct mlx5dr_rule_action rule_acts[MLX5_HW_MAX_ACTS ]; };
drivers/net/mlx5/mlx5_flow_hw.c +60 -6
115 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 13/14 net/mlx5: add indirect action
In reply to Suanming MouHW steering can support indirect action as well. With indirect action, the flow can be created with more flexible shared RSS action selection. This will can save the action template with different RSS actions.
This commit adds the flow queue operation callback for:rte_flow_async_action_handle_create(); rte_flow_async_action_handle_destroy(); rte_flow_async_action_handle_update();
--- drivers/net/mlx5/mlx5_flow.c | 131 ++++++++++ drivers/net/mlx5/mlx5_flow.h | 59 +++++ drivers/net/mlx5/mlx5_flow_dv.c | 21 +- drivers/net/mlx5/mlx5_flow_hw.c | 414 +++++++++++++++++++++++++++++++- 4 files changed, 612 insertions(+), 13 deletions(-)
drivers/net/mlx5/mlx5_flow.c +131 -0
@@ -879,6 +879,29 @@ mlx5_flow_push(struct rte_eth_dev *dev,uint32_t queue, struct rte_flow_error *error); +static struct rte_flow_action_handle * +mlx5_flow_async_action_handle_create (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +const struct rte_flow_indir_action_conf *conf, +const struct rte_flow_action *action, +void *user_data, +struct rte_flow_error *error); + +static int +mlx5_flow_async_action_handle_update (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +const void *update, +void *user_data, +struct rte_flow_error *error); + +static int +mlx5_flow_async_action_handle_destroy (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +void *user_data, +struct rte_flow_error *error); +static const struct rte_flow_ops mlx5_flow_ops = { .validate = mlx5_flow_validate, .create = mlx5_flow_create,
@@ -911,6 +934,9 @@ static const struct rte_flow_ops mlx5_flow_ops = { .async_destroy = mlx5_flow_async_flow_destroy, .pull = mlx5_flow_pull, .push = mlx5_flow_push, + .async_action_handle_create = mlx5_flow_async_action_handle_create, + .async_action_handle_update = mlx5_flow_async_action_handle_update, + .async_action_handle_destroy = mlx5_flow_async_action_handle_destroy, };/* Tunnel information. */
@@ -8376,6 +8402,111 @@ mlx5_flow_push(struct rte_eth_dev *dev,return fops->push (dev, queue, error); } +/** +* Create shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] conf +* Indirect action configuration. +* @param[in] action +* rte_flow action detail. +* @param[in] user_data +* Pointer to the user_data. +* @param[out] error +* Pointer to error structure. +* +* @return +* Action handle on success, NULL otherwise and rte_errno is set. +*/ +static struct rte_flow_action_handle * +mlx5_flow_async_action_handle_create (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +const struct rte_flow_indir_action_conf *conf, +const struct rte_flow_action *action, +void *user_data, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->async_action_create (dev, queue, attr, conf, action, + user_data, error); +} + +/** +* Update shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] handle +* Action handle to be updated. +* @param[in] update +* Update value. +* @param[in] user_data +* Pointer to the user_data. +* @param[out] error +* Pointer to error structure. +* +* @return +* 0 on success, negative value otherwise and rte_errno is set. +*/ +static int +mlx5_flow_async_action_handle_update (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +const void *update, +void *user_data, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->async_action_update (dev, queue, attr, handle, + update, user_data, error); +} + +/** +* Destroy shared action. +* +* @param[in] dev +* Pointer to the rte_eth_dev structure. +* @param[in] queue +* Which queue to be used.. +* @param[in] attr +* Operation attribute. +* @param[in] handle +* Action handle to be destroyed. +* @param[in] user_data +* Pointer to the user_data. +* @param[out] error +* Pointer to error structure. +* +* @return +* 0 on success, negative value otherwise and rte_errno is set. +*/ +static int +mlx5_flow_async_action_handle_destroy (struct rte_eth_dev *dev,uint32_t queue, +const struct rte_flow_op_attr *attr, +struct rte_flow_action_handle *handle, +void *user_data, +struct rte_flow_error *error) +{ +const struct mlx5_flow_driver_ops *fops = +flow_get_drv_ops (MLX5_FLOW_TYPE_HW ); + +return fops->async_action_destroy (dev, queue, attr, handle, + user_data, error); +} +/** * Allocate a new memory for the counter values wrapped by all the needed * management.
drivers/net/mlx5/mlx5_flow.h +59 -0
99 lines not on this page · raw message
drivers/net/mlx5/mlx5_flow_dv.c +12 -9
@@ -13841,9 +13841,9 @@ __flow_dv_action_rss_hrxq_set(struct mlx5_shared_action_rss *action,* @return * Valid hash RX queue index, otherwise 0. */ -static uint32_t -__flow_dv_action_rss_hrxq_lookup (struct rte_eth_dev *dev,uint32_t idx, -const uint64_t hash_fields) +uint32_t +flow_dv_action_rss_hrxq_lookup (struct rte_eth_dev *dev,uint32_t idx, +const uint64_t hash_fields) {struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_shared_action_rss *shared_rss =
@@ -13971,7 +13971,7 @@ flow_dv_apply(struct rte_eth_dev *dev, struct rte_flow *flow,struct mlx5_hrxq *hrxq =NULL ;uint32_t hrxq_idx; - hrxq_idx =__flow_dv_action_rss_hrxq_lookup (dev, + hrxq_idx =flow_dv_action_rss_hrxq_lookup (dev, rss_desc->shared_rss, dev_flow->hash_fields);if (hrxq_idx)
@@ -14695,6 +14695,7 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,struct mlx5_shared_action_rss *shared_rss, struct rte_flow_error *error) { + struct mlx5_priv *priv = dev->data->dev_private;struct mlx5_flow_rss_desc rss_desc = {0 };size_t i;int err;
@@ -14715,6 +14716,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev,/* Set non-zero value to indicate a shared RSS. */ rss_desc.shared_rss = action_idx; rss_desc.ind_tbl = shared_rss->ind_tbl; +if (priv->sh->config.dv_flow_en ==2 ) + rss_desc.hws_flags =MLX5DR_ACTION_FLAG_HWS_RX ;for (i =0 ; i <MLX5_RSS_HASH_FIELDS_LEN ; i++) {struct mlx5_hrxq *hrxq;uint64_t hash_fields = mlx5_rss_hash_fields[i];
@@ -14906,7 +14909,7 @@ __flow_dv_action_rss_release(struct rte_eth_dev *dev, uint32_t idx,* A valid shared action handle in case of success, NULL otherwise and * rte_errno is set. */ -static struct rte_flow_action_handle * +struct rte_flow_action_handle * flow_dv_action_create(struct rte_eth_dev *dev, const struct rte_flow_indir_action_conf *conf, const struct rte_flow_action *action,
@@ -14976,7 +14979,7 @@ flow_dv_action_create(struct rte_eth_dev *dev,* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_destroy (struct rte_eth_dev *dev,struct rte_flow_action_handle *handle,struct rte_flow_error *error)
@@ -15186,7 +15189,7 @@ __flow_dv_action_ct_update(struct rte_eth_dev *dev, uint32_t idx,* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_update(struct rte_eth_dev *dev, struct rte_flow_action_handle *handle, const void *update,
@@ -15858,7 +15861,7 @@ flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data, "counters are not available"); } -static int +int flow_dv_action_query (struct rte_eth_dev *dev,const struct rte_flow_action_handle *handle,void *data,struct rte_flow_error *error)
@@ -17551,7 +17554,7 @@ flow_dv_counter_allocate(struct rte_eth_dev *dev)* @return * 0 on success, otherwise negative errno value. */ -static int +int flow_dv_action_validate(struct rte_eth_dev *dev, const struct rte_flow_indir_action_conf *conf, const struct rte_flow_action *action,
drivers/net/mlx5/mlx5_flow_hw.c +410 -4
493 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 14/14 net/mlx5: add header reformat action
In reply to Suanming MouHW steering header reformat action can work under bulk mode. In this case, when create the table, bulk size of header reformat actions will be allocated in low level. Afterwards, when create flow, just simply specify the action index in the bulk and the encapsulation data to the action will be enough.
--- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_flow.h | 21 +++ drivers/net/mlx5/mlx5_flow_dv.c | 4 +- drivers/net/mlx5/mlx5_flow_hw.c | 228 +++++++++++++++++++++++++++++++- 4 files changed, 251 insertions(+), 3 deletions(-)
drivers/net/mlx5/mlx5.h +1 -0
@@ -342,6 +342,7 @@ struct mlx5_hw_q_job {uint32_t type;/* Job type. */ struct rte_flow_hw *flow;/* Flow attached to the job. */ void *user_data;/* Job user data. */ +uint8_t *encap_data;/* Encap data. */ };/* HW steering job descriptor LIFO pool. */
drivers/net/mlx5/mlx5_flow.h +21 -0
@@ -1040,6 +1040,14 @@ struct mlx5_action_construct_data {uint16_t action_src;/* rte_flow_action src offset. */ uint16_t action_dst;/* mlx5dr_rule_action dst offset. */ union { +struct { +/* encap src(item) offset. */ +uint16_t src; +/* encap dst data offset. */ +uint16_t dst; +/* encap data len. */ +uint16_t len; + } encap;struct {uint64_t types;/* RSS hash types. */ uint32_t level;/* RSS level. */
@@ -1076,6 +1084,13 @@ struct mlx5_hw_jump_action {struct mlx5dr_action *hws_action; }; +/* Encap decap action struct. */ +struct mlx5_hw_encap_decap_action { +struct mlx5dr_action *action;/* Action object. */ +size_t data_size;/* Action metadata size. */ +uint8_t data[];/* Action data. */ +}; +/* The maximum actions support in the flow. */ #define MLX5_HW_MAX_ACTS 16
@@ -1085,6 +1100,9 @@ struct mlx5_hw_actions { LIST_HEAD(act_list, mlx5_action_construct_data) act_list;struct mlx5_hw_jump_action *jump;/* Jump action. */ struct mlx5_hrxq *tir;/* TIR action. */ +/* Encap/Decap action. */ +struct mlx5_hw_encap_decap_action *encap_decap; +uint16_t encap_decap_pos;/* Encap/Decap action position. */ uint32_t acts_num:4; /* Total action number. */ uint32_t mark:1; /* Indicate the mark action. *//* Translated DR action array from action template. */
@@ -2027,4 +2045,7 @@ int flow_dv_action_query(struct rte_eth_dev *dev,const struct rte_flow_action_handle *handle, void *data, struct rte_flow_error *error); +size_t flow_dv_get_item_hdr_len (const enum rte_flow_item_type item_type); +int flow_dv_convert_encap_data (const struct rte_flow_item *items,uint8_t *buf, +size_t *size,struct rte_flow_error *error);#endif /* RTE_PMD_MLX5_FLOW_H_ */
drivers/net/mlx5/mlx5_flow_dv.c +2 -2
@@ -4034,7 +4034,7 @@ flow_dv_push_vlan_action_resource_register* @return * sizeof struct item_type, 0 if void or irrelevant. */ -static size_t +size_t flow_dv_get_item_hdr_len (const enum rte_flow_item_type item_type) {size_t retval;
@@ -4100,7 +4100,7 @@ flow_dv_get_item_hdr_len(const enum rte_flow_item_type item_type)* @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -static int +int flow_dv_convert_encap_data (const struct rte_flow_item *items,uint8_t *buf,size_t *size,struct rte_flow_error *error) {
drivers/net/mlx5/mlx5_flow_hw.c +227 -1
299 lines not on this page · raw message
Signature · 1 line
2.25.1
Hi,
Quoted 88 lines
-----Original Message-----
From: Suanming Mou <[email protected]>
Sent: Thursday, February 24, 2022 3:41 PM
To: Slava Ovsiienko <[email protected]>; Matan Azrad <[email protected]>
Cc: Raslan Darawsheh <[email protected]>; Ori Kam <[email protected]>; [email protected]
Subject: [PATCH v4 00/14] net/mlx5: add hardware steeringThe Connect-X steering is a lookup hardware mechanism that accesses flow tables, matches packets to the rules, and performs specified actions. Historically, mlx5 PMD implements several software engines to manage steering hardware facility:
- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directlyHowever, there are still some disadvantages:
- performance is limited, we should invoke firmware either to manage the entire flow, or to handle some internal steering objects
- organizing and preparing flow infrastructure (actions, matchers, groups, etc.) on the flow inserting is sure to cause slow flow insertion
- security, exposing the low-level steering entries directly to the userspace may cause security risks
A new hardware WQE based steering operation with codename "HW Steering" is going to be introduced to get rid of the security risks. And it will take advantage of the recently new introduced async queue-based
rte_flowAPIs to prepare everything in advance to achieve high insertion rate.In this new HW steering engine, the original SW steering
rte_flowAPI will not be supported in the first implementation, only the new async queue-based flow operations is going to be supported. A new steering mode parameter fordv_flow_enwill be introduced and user will be able to engage the new steering engine.---
v4:
v3:
- rebase to the latest version.v2:
- New HW steering low-level abstract code added.
- commit message improvement.
- add protection forrte_flowandrte_flow_asynccallbacks.
- rebase torte_flow_asyncv9.
- fix somerte_flowerror not filled bugs.Suanming Mou (14):
net/mlx5: introduce hardware steering operation net/mlx5: add HW steering low-level abstract code net/mlx5: introduce hardware steering enable routine net/mlx5: add port flow configuration net/mlx5: add pattern template management net/mlx5: add action template management net/mlx5: add table management
net/mlx5: add basic flow queue operation net/mlx5: add flow flush function
net/mlx5: add flow jump action
net/mlx5: add queue and RSS action net/mlx5: add mark action
net/mlx5: add indirect action
net/mlx5: add header reformat actiondoc/guides/nics/mlx5.rst | 19 +- doc/guides/rel_notes/release_22_03.rst | 1 + drivers/net/mlx5/linux/mlx5_flow_os.h | 1 + drivers/net/mlx5/linux/mlx5_os.c | 22 +- drivers/net/mlx5/meson.build | 2 + drivers/net/mlx5/mlx5.c | 55 +- drivers/net/mlx5/mlx5.h | 66 +- drivers/net/mlx5/mlx5_devx.c | 10 + drivers/net/mlx5/mlx5_dr.c | 383 ++++ drivers/net/mlx5/mlx5_dr.h | 456 +++++ drivers/net/mlx5/mlx5_flow.c | 721 ++++++- drivers/net/mlx5/mlx5_flow.h | 287 +++ drivers/net/mlx5/mlx5_flow_dv.c | 186 +- drivers/net/mlx5/mlx5_flow_hw.c | 2337 +++++++++++++++++++++++ drivers/net/mlx5/mlx5_flow_verbs.c | 7 +- drivers/net/mlx5/mlx5_rx.h | 9 +- drivers/net/mlx5/mlx5_rxq.c | 85 +- drivers/net/mlx5/windows/mlx5_flow_os.h | 1 + 18 files changed, 4468 insertions(+), 180 deletions(-) create mode 100644
Quoted 5 lines
drivers/net/mlx5/mlx5_dr.ccreate mode 100644drivers/net/mlx5/mlx5_dr.hcreate mode 100644drivers/net/mlx5/mlx5_flow_hw.c--
2.25.1
Series applied to next-net-mlx,
Kindest regards,
Raslan Darawsheh