Skip to contentSkip to conversation

net/mlx5: fix counter cache starvation

PATCH In review Dariusz Sosnowski 2 participants 2 messages Feb 28, 2024 – Feb 29, 2024 Network drivers

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net/mlx5: fix counter cache starvation

mlx5 PMD maintains a global counter pool and per-queue counter cache, which are used to allocate COUNT flow action objects. Whenever an empty cache is accessed, it is replenished with a pre-defined number of counters.

If number of configured counters was sufficiently small, then it might have happened that caches associated with some queues could get starved because all counters were fetched on other queues.

This patch fixes that by disabling cache at runtime if number of configured counters is not sufficient to avoid such starvation.

Fixes 4d368e1da3a4 ("net/mlx5: support flow counter action for HWS")
---
 drivers/net/mlx5/mlx5_flow_hw.c |  6 +--
 drivers/net/mlx5/mlx5_hws_cnt.c | 72 ++++++++++++++++++++++++---------
 drivers/net/mlx5/mlx5_hws_cnt.h | 25 +++++++++---
 3 files changed, 74 insertions(+), 29 deletions(-)
drivers/net/mlx5/mlx5_flow_hw.c +2 -4
index f778fd0698..8ba3b3321e 100644
@@ -3117,8 +3117,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev,
 				break;
 			/* Fall-through. */
 		case RTE_FLOW_ACTION_TYPE_COUNT:
-			/* If the port is engaged in resource sharing, do not use queue cache. */
-			cnt_queue = mlx5_hws_cnt_is_pool_shared(priv) ? NULL : &queue;
+			cnt_queue = mlx5_hws_cnt_get_queue(priv, &queue);
 			ret = mlx5_hws_cnt_pool_get(priv->hws_cpool, cnt_queue, &cnt_id, age_idx);
 			if (ret != 0)
 				return ret;
@@ -3757,8 +3756,7 @@ flow_hw_age_count_release(struct mlx5_priv *priv, uint32_t queue,
 		}
 		return;
 	}
-	/* If the port is engaged in resource sharing, do not use queue cache. */
-	cnt_queue = mlx5_hws_cnt_is_pool_shared(priv) ? NULL : &queue;
+	cnt_queue = mlx5_hws_cnt_get_queue(priv, &queue);
 	/* Put the counter first to reduce the race risk in BG thread. */
 	mlx5_hws_cnt_pool_put(priv->hws_cpool, cnt_queue, &flow->cnt_id);
 	flow->cnt_id = 0;
drivers/net/mlx5/mlx5_hws_cnt.c +53 -19

96 lines not on this page · raw message

drivers/net/mlx5/mlx5_hws_cnt.h +19 -6
index 585b5a83ad..e00596088f 100644
@@ -557,19 +557,32 @@ mlx5_hws_cnt_pool_get(struct mlx5_hws_cnt_pool *cpool, uint32_t *queue,
 }
 
 /**
- * Check if counter pool allocated for HWS is shared between ports.
+ * Decide if the given queue can be used to perform counter allocation/deallcation
+ * based on counter configuration
  *
  * @param[in] priv
  *   Pointer to the port private data structure.
+ * @param[in] queue
+ *   Pointer to the queue index.
  *
  * @return
- *   True if counter pools is shared between ports. False otherwise.
+ *   @p queue if cache related to the queue can be used. NULL otherwise.
  */
-static __rte_always_inline bool
-mlx5_hws_cnt_is_pool_shared(struct mlx5_priv *priv)
+static __rte_always_inline uint32_t *
+mlx5_hws_cnt_get_queue(struct mlx5_priv *priv, uint32_t *queue)
 {
-	return priv && priv->hws_cpool &&
-	    (priv->shared_refcnt || priv->hws_cpool->cfg.host_cpool != NULL);
+	if (priv && priv->hws_cpool) {
+		/* Do not use queue cache if counter pool is shared. */
+		if (priv->shared_refcnt || priv->hws_cpool->cfg.host_cpool != NULL)
+			return NULL;
+		/* Do not use queue cache if counter cache is disabled. */
+		if (priv->hws_cpool->cache == NULL)
+			return NULL;
+		return queue;
+	}
+	/* This case should not be reached if counter pool was successfully configured. */
+	MLX5_ASSERT(false);
+	return NULL;
 }
 
 static __rte_always_inline unsigned int
Signature · 1 line
2.39.2
#1 lore raw

Hi,

Quoted 21 lines

-----Original Message-----
From: Dariusz Sosnowski <[email protected]>
Sent: Wednesday, February 28, 2024 9:06 PM
To: Slava Ovsiienko <[email protected]>; Ori Kam <[email protected]>; Suanming Mou <[email protected]>; Matan Azrad <[email protected]>; Jack Min <[email protected]>
Cc: [email protected]; [email protected]; Bing Zhao <[email protected]>
Subject: [PATCH] net/mlx5: fix counter cache starvation

mlx5 PMD maintains a global counter pool and per-queue counter cache, which are used to allocate COUNT flow action objects. Whenever an empty cache is accessed, it is replenished with a pre-defined number of counters.

If number of configured counters was sufficiently small, then it might have happened that caches associated with some queues could get starved because all counters were fetched on other queues.

This patch fixes that by disabling cache at runtime if number of configured counters is not sufficient to avoid such starvation.

Quoted 7 lines
Fixes 4d368e1da3a4 ("net/mlx5: support flow counter action for HWS")

Patch applied to next-net-mlx,
Kindest regards
Raslan Darawsheh