Skip to contentSkip to conversation

Add support for meter hierarchy

PATCH In review Shun Hao 2 participants 6 messages Jul 6, 2021 – Jul 7, 2021 Network drivers

lore
Share & tools
lore

Shun Hao

author
#0 lore raw

cover Add support for meter hierarchy

This adds support for meter hierarchy, which means multiple meters can be chained together, and using the header meter of hierarchy in a flow will apply all meters in hierarchy to that flow.

Shun Hao (4):
net/mlx5: support meter action in meter policy
net/mlx5: support meter hierarchy drop count
net/mlx5: meter hierarchy destroy and cleanup
net/mlx5: validate meter action in policy

 doc/guides/nics/mlx5.rst               |  15 +
 doc/guides/rel_notes/release_21_08.rst |   6 +
 drivers/net/mlx5/mlx5.h                |  32 +-
 drivers/net/mlx5/mlx5_flow.c           | 123 +++--
 drivers/net/mlx5/mlx5_flow.h           |  12 +
 drivers/net/mlx5/mlx5_flow_dv.c        | 701 +++++++++++++++++++++----
 drivers/net/mlx5/mlx5_flow_meter.c     | 175 +++++-
 7 files changed, 935 insertions(+), 129 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH 1/4 net/mlx5: support meter action in meter policy

This makes the meter policy support meter action. So multiple meters can be chained as a meter hierarchy.

Only termination meter is allowed as the last meter in a hierarchy, and there're two cases:
1. The last meter has non-RSS policy, can directly create sub-policy and color rules during each meter's policy creation.
2. The last meter has RSS policy, don't create sub-policy/rules when creating meter policy. Only when a RTE flow is using the meter hierarchy, will iterate all meters of the hierarchy and create neede sub- policies and color rules for them.

---
 drivers/net/mlx5/mlx5.h            |  12 ++
 drivers/net/mlx5/mlx5_flow.c       |  71 +++++---
 drivers/net/mlx5/mlx5_flow.h       |   5 +
 drivers/net/mlx5/mlx5_flow_dv.c    | 270 ++++++++++++++++++++++++-----
 drivers/net/mlx5/mlx5_flow_meter.c |  43 ++++-
 5 files changed, 332 insertions(+), 69 deletions(-)
drivers/net/mlx5/mlx5.h +12 -0
index 0f4b239142..0c555f0b1f 100644
@@ -677,6 +677,12 @@ struct mlx5_meter_policy_action_container {
 		/* Jump/drop action per color. */
 		uint16_t queue;
 		/* Queue action configuration. */
+		struct {
+			uint32_t next_mtr_id;
+			/* The next meter id. */
+			void *next_sub_policy;
+			/* Next meter's sub-policy. */
+		};
 	};
 };
 
@@ -694,6 +700,8 @@ struct mlx5_flow_meter_policy {
 	/* Rule applies to transfer domain. */
 	uint32_t is_queue:1;
 	/* Is queue action in policy table. */
+	uint32_t is_hierarchy:1;
+	/* Is meter action in policy table. */
 	rte_spinlock_t sl;
 	uint32_t ref_cnt;
 	/* Use count. */
@@ -712,6 +720,7 @@ struct mlx5_flow_meter_policy {
 #define MLX5_MTR_SUB_POLICY_NUM_SHIFT  3
 #define MLX5_MTR_SUB_POLICY_NUM_MASK  0x7
 #define MLX5_MTRS_DEFAULT_RULE_PRIORITY 0xFFFF
+#define MLX5_MTR_CHAIN_MAX_NUM 8
 
 /* Flow meter default policy parameter structure.
  * Policy index 0 is reserved by default policy table.
@@ -1669,6 +1678,9 @@ struct mlx5_flow_meter_policy *mlx5_flow_meter_policy_find
 		(struct rte_eth_dev *dev,
 		uint32_t policy_id,
 		uint32_t *policy_idx);
+struct mlx5_flow_meter_policy *
+mlx5_flow_meter_hierarchy_get_final_policy(struct rte_eth_dev *dev,
+					struct mlx5_flow_meter_policy *policy);
 int mlx5_flow_meter_flush(struct rte_eth_dev *dev,
 			  struct rte_mtr_error *error);
 void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
drivers/net/mlx5/mlx5_flow.c +43 -28

151 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +5 -0
index 2f2aa962f9..09d6d609db 100644
@@ -454,6 +454,7 @@ enum mlx5_flow_fate_type {
 	MLX5_FLOW_FATE_DROP,
 	MLX5_FLOW_FATE_DEFAULT_MISS,
 	MLX5_FLOW_FATE_SHARED_RSS,
+	MLX5_FLOW_FATE_MTR,
 	MLX5_FLOW_FATE_MAX,
 };
 
@@ -1102,6 +1103,10 @@ struct mlx5_flow_workspace {
 	uint32_t rssq_num; /* Allocated queue num in rss_desc. */
 	uint32_t flow_idx; /* Intermediate device flow index. */
 	struct mlx5_flow_meter_info *fm; /* Pointer to the meter in flow. */
+	struct mlx5_flow_meter_policy *policy;
+	/* The meter policy used by meter in flow. */
+	struct mlx5_flow_meter_policy *final_policy;
+	/* The final policy when meter policy is hierarchy. */
 	uint32_t skip_matcher_reg:1;
 	/* Indicates if need to skip matcher register in translate. */
 };
drivers/net/mlx5/mlx5_flow_dv.c +230 -40

361 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_meter.c +42 -1

61 lines not on this page · raw message

Signature · 1 line
2.21.0
author
#2 lore raw

PATCH 2/4 net/mlx5: support meter hierarchy drop count

In reply to Shun Hao

When using meter hierarchy with multiple meters, every meter may have drop counter, so a packet being set red color by one meter should be counted to that specific meter only.

To support this, add tag action in the color rule so packet going to next new meter can have its meter id, so as to be counted to the correct drop counter in drop table.

---
 drivers/net/mlx5/mlx5.h         |  20 +-
 drivers/net/mlx5/mlx5_flow.c    |  52 +++++-
 drivers/net/mlx5/mlx5_flow.h    |   7 +
 drivers/net/mlx5/mlx5_flow_dv.c | 318 ++++++++++++++++++++++++++------
 4 files changed, 339 insertions(+), 58 deletions(-)
drivers/net/mlx5/mlx5.h +16 -4
index 0c555f0b1f..e5c9ec0777 100644
@@ -631,6 +631,20 @@ enum mlx5_meter_domain {
 					MLX5_MTR_DOMAIN_EGRESS_BIT | \
 					MLX5_MTR_DOMAIN_TRANSFER_BIT)
 
+/* The color tag rule structure. */
+struct mlx5_sub_policy_color_rule {
+	void *rule;
+	/* The color rule. */
+	struct mlx5_flow_dv_matcher *matcher;
+	/* The color matcher. */
+	TAILQ_ENTRY(mlx5_sub_policy_color_rule) next_port;
+	/**< Pointer to the next color rule structure. */
+	int32_t src_port;
+	/* On which src port this rule applied. */
+};
+
+TAILQ_HEAD(mlx5_sub_policy_color_rules, mlx5_sub_policy_color_rule);
+
 /*
  * Meter sub-policy structure.
  * Each RSS TIR in meter policy need its own sub-policy resource.
@@ -648,10 +662,8 @@ struct mlx5_flow_meter_sub_policy {
 	/* Index to TIR resource. */
 	struct mlx5_flow_tbl_resource *jump_tbl[MLX5_MTR_RTE_COLORS];
 	/* Meter jump/drop table. */
-	struct mlx5_flow_dv_matcher *color_matcher[RTE_COLORS];
-	/* Matcher for Color. */
-	void *color_rule[RTE_COLORS];
-	/* Meter green/yellow/drop rule. */
+	struct mlx5_sub_policy_color_rules color_rules[RTE_COLORS];
+	/* List for the color rules. */
 };
 
 struct mlx5_meter_policy_acts {
drivers/net/mlx5/mlx5_flow.c +50 -2

83 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow.h +7 -0
index 09d6d609db..7d97c5880f 100644
@@ -1161,6 +1161,12 @@ typedef struct mlx5_flow_meter_sub_policy *
 		(struct rte_eth_dev *dev,
 		struct mlx5_flow_meter_policy *mtr_policy,
 		struct mlx5_flow_rss_desc *rss_desc[MLX5_MTR_RTE_COLORS]);
+typedef int (*mlx5_flow_meter_hierarchy_rule_create_t)
+		(struct rte_eth_dev *dev,
+		struct mlx5_flow_meter_info *fm,
+		int32_t src_port,
+		const struct rte_flow_item *item,
+		struct rte_flow_error *error);
 typedef void (*mlx5_flow_destroy_sub_policy_with_rxq_t)
 	(struct rte_eth_dev *dev,
 	struct mlx5_flow_meter_policy *mtr_policy);
@@ -1257,6 +1263,7 @@ struct mlx5_flow_driver_ops {
 	mlx5_flow_create_def_policy_t create_def_policy;
 	mlx5_flow_destroy_def_policy_t destroy_def_policy;
 	mlx5_flow_meter_sub_policy_rss_prepare_t meter_sub_policy_rss_prepare;
+	mlx5_flow_meter_hierarchy_rule_create_t meter_hierarchy_rule_create;
 	mlx5_flow_destroy_sub_policy_with_rxq_t destroy_sub_policy_with_rxq;
 	mlx5_flow_counter_alloc_t counter_alloc;
 	mlx5_flow_counter_free_t counter_free;
drivers/net/mlx5/mlx5_flow_dv.c +266 -52

466 lines not on this page · raw message

Signature · 1 line
2.21.0
author
#3 lore raw

PATCH 3/4 net/mlx5: meter hierarchy destroy and cleanup

In reply to Shun Hao

When creating hierarchy meter, its color rules will increase next meter's reference count, so when destroy the hierarchy meter, also need to dereference the next meter's count.

During flushing all meters of a port, need to destroy all hierarchy meters and their policies first, to dereference the last meter in hierarchy. Then all meters have no reference and can be destroyed.

---
 drivers/net/mlx5/mlx5_flow_dv.c    |  15 +++-
 drivers/net/mlx5/mlx5_flow_meter.c | 132 +++++++++++++++++++++++++++++
 2 files changed, 145 insertions(+), 2 deletions(-)
drivers/net/mlx5/mlx5_flow_dv.c +13 -2
index 119de09809..681e6fb07c 100644
@@ -14588,12 +14588,20 @@ static void
 __flow_dv_destroy_sub_policy_rules(struct rte_eth_dev *dev,
 			     struct mlx5_flow_meter_sub_policy *sub_policy)
 {
+	struct mlx5_priv *priv = dev->data->dev_private;
 	struct mlx5_flow_tbl_data_entry *tbl;
+	struct mlx5_flow_meter_policy *policy = sub_policy->main_policy;
+	struct mlx5_flow_meter_info *next_fm;
 	struct mlx5_sub_policy_color_rule *color_rule;
 	void *tmp;
-	int i;
+	uint32_t i;
 
 	for (i = 0; i < RTE_COLORS; i++) {
+		next_fm = NULL;
+		if (i == RTE_COLOR_GREEN && policy &&
+		    policy->act_cnt[i].fate_action == MLX5_FLOW_FATE_MTR)
+			next_fm = mlx5_flow_meter_find(priv,
+					policy->act_cnt[i].next_mtr_id, NULL);
 		TAILQ_FOREACH_SAFE(color_rule, &sub_policy->color_rules[i],
 				   next_port, tmp) {
 			claim_zero(mlx5_flow_os_destroy_flow(color_rule->rule));
@@ -14604,11 +14612,14 @@ __flow_dv_destroy_sub_policy_rules(struct rte_eth_dev *dev,
 			TAILQ_REMOVE(&sub_policy->color_rules[i],
 					color_rule, next_port);
 			mlx5_free(color_rule);
+			if (next_fm)
+				mlx5_flow_meter_detach(priv, next_fm);
 		}
 	}
 	for (i = 0; i < MLX5_MTR_RTE_COLORS; i++) {
 		if (sub_policy->rix_hrxq[i]) {
-			mlx5_hrxq_release(dev, sub_policy->rix_hrxq[i]);
+			if (policy && !policy->is_hierarchy)
+				mlx5_hrxq_release(dev, sub_policy->rix_hrxq[i]);
 			sub_policy->rix_hrxq[i] = 0;
 		}
 		if (sub_policy->jump_tbl[i]) {
drivers/net/mlx5/mlx5_flow_meter.c +132 -0

144 lines not on this page · raw message

Signature · 1 line
2.21.0
author
#4 lore raw

PATCH 4/4 net/mlx5: validate meter action in policy

In reply to Shun Hao

This adds the validation when creating a policy with meter action.

Currently meter action is only allowed for green color in policy, and 8 meters are supported at maximum in one meter hierarchy.

---
 doc/guides/nics/mlx5.rst               | 15 ++++
 doc/guides/rel_notes/release_21_08.rst |  6 ++
 drivers/net/mlx5/mlx5_flow_dv.c        | 98 ++++++++++++++++++++++++++
 3 files changed, 119 insertions(+)
doc/guides/nics/mlx5.rst +15 -0
index a16af32e67..de04931f80 100644
@@ -107,6 +107,7 @@ Features
 - 21844 flow priorities for ingress or egress flow groups greater than 0 and for any transfer
   flow group.
 - Flow metering, including meter policy API.
+- Flow meter hierarchy.
 - Flow integrity offload API.
 - Connection tracking.
 - Sub-Function representors.
@@ -1927,3 +1928,17 @@ This section demonstrates how to use the shared meter. A meter M can be created
 on port X and to be shared with a port Y on the same switch domain by the next way:
 
         flow create X ingress transfer pattern eth / port_id id is Y / end actions meter mtr_id M / end
+
+How to use meter hierarchy
+--------------------------
+
+This section demonstrates how to create and use a meter hierarchy.
+A termination meter M can be the policy green action of another termination meter N.
+The two meters are chained together as a chain. Using meter N in a flow will apply
+both the meters in hierarchy on that flow.
+
+        add port meter policy 0 1 g_actions queue index 0 / end y_actions end r_actions drop / end
+        create port meter 0 M 1 1 yes 0xffff 1 0
+        add port meter policy 0 2 g_actions meter mtr_id M / end y_actions end r_actions drop / end
+        create port meter 0 N 2 2 yes 0xffff 1 0
+        flow create 0 ingress group 1 pattern eth / end actions meter mtr_id N / end
doc/guides/rel_notes/release_21_08.rst +6 -0
index 0a05cb02fa..b29d78f4de 100644
@@ -68,6 +68,12 @@ New Features
   usecases. Configuration happens via standard rawdev enq/deq operations. See
   the :doc:`../rawdevs/cnxk_bphy` rawdev guide for more details on this driver.
 
+* **Updated Mellanox mlx5 driver.**
+
+  Updated the Mellanox mlx5 driver with new features and improvements, including:
+
+  * Added support for Meter hierarchy.
+
 
 Removed Items
 -------------
drivers/net/mlx5/mlx5_flow_dv.c +98 -0

122 lines not on this page · raw message

Signature · 1 line
2.21.0
#5 lore raw
In reply to Shun Hao

Hi,

Quoted 28 lines

-----Original Message-----
From: Shun Hao <[email protected]>
Sent: Tuesday, July 6, 2021 4:15 PM
To: Ori Kam <[email protected]>; Slava Ovsiienko <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; NBU-Contact-Thomas Monjalon <[email protected]>; Raslan Darawsheh <[email protected]>
Subject: [PATCH v1 0/4] Add support for meter hierarchy

This adds support for meter hierarchy, which means multiple meters can be chained together, and using the header meter of hierarchy in a flow will apply all meters in hierarchy to that flow.

Shun Hao (4):
net/mlx5: support meter action in meter policy
net/mlx5: support meter hierarchy drop count
net/mlx5: meter hierarchy destroy and cleanup
net/mlx5: validate meter action in policy

 doc/guides/nics/mlx5.rst               |  15 +
 doc/guides/rel_notes/release_21_08.rst |   6 +
 drivers/net/mlx5/mlx5.h                |  32 +-
 drivers/net/mlx5/mlx5_flow.c           | 123 +++--
 drivers/net/mlx5/mlx5_flow.h           |  12 +
 drivers/net/mlx5/mlx5_flow_dv.c        | 701 +++++++++++++++++++++----
 drivers/net/mlx5/mlx5_flow_meter.c     | 175 +++++-
 7 files changed, 935 insertions(+), 129 deletions(-)
Quoted 2 lines

--
2.21.0

Series applied to next-net-mlx,

Kindest regards,
Raslan Darawsheh