Skip to contentSkip to conversation

mlx5: Add netdev-genl queue stats

RFC Superseded Joe Damato 5 participants 22 messages May 29, 2024 – Jun 3, 2024

Subsystems (5)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Joe Damato

author
#0 lore raw

v3 cover mlx5: Add netdev-genl queue stats

Greetings:

Switching to an RFC instead of a PATCH because even though Tariq patiently explained the code to me, I'm sure I probably still missed something ;)

If this turns out to be right and Tariq agrees, I can send a PATCH net-next v4.

This change adds support for the per queue netdev-genl API to mlx5, which seems to output stats:

./cli.py --spec ../../../Documentation/netlink/specs/netdev.yaml \
         --dump qstats-get --json '{"scope": "queue"}'

...snip
{'ifindex': 7,
'queue-id': 28,
'queue-type': 'tx',
'tx-bytes': 399462,
'tx-packets': 3311},
...snip

I've used the suggested tooling to verify the per queue stats match rtnl by doing this:

NETIF=eth0 tools/testing/selftests/drivers/net/stats.py

I've tested the following scenarios:
- The machine at boot (default queue configuration)
- Adjusting the queue configuration to various amounts via ethtool
- Add mqprio TCs
- Removing the mqprio TCs

and in each scenario the stats script above reports that the stats match rtnl.

Worth noting that Tariq suggested I also export HTB/QOS stats in mlx5e_get_base_stats.

I am open to doing this, but I think if I were to do that, HTB/QOS queue stats should also be exported by rtnl so that the script above will continue to show that the output is correct.

I'd like to propose: adding HTB/QOS to both rtnl *and* the netdev-genl code together at the same time, but a later time, separate from this change.

Thanks,
Joe

v2 -> rfcv3:
- Added patch 1/2 which creates some helpers for computing the txq_ix and ch_ix/tc_ix.

- Patch 2/2 modified in several ways:
- Fixed variable declarations in mlx5e_get_queue_stats_rx to be at the start of the function.
- mlx5e_get_queue_stats_tx rewritten to access sq stats directly by using the helpers added in the previous patch.
- mlx5e_get_base_stats modified in several ways:
- Took the state_lock when accessing priv->channels.
- For the base RX stats, code was simplified to call mlx5e_get_queue_stats_rx instead of repeating the same code.
- For the base TX stats, I attempted to implement what I think Tariq suggested in the previous thread:
- for available channels, only unavailable TC stats are summed
- for unavailable channels, all stats for TCs up to max_opened_tc are summed.

v1 - > v2:
- Essentially a full rewrite after comments from Jakub, Tariq, and Zhu.

Joe Damato (2):
net/mlx5e: Add helpers to calculate txq and ch idx net/mlx5e: Add per queue netdev-genl stats

 .../net/ethernet/mellanox/mlx5/core/en_main.c | 150 +++++++++++++++++-
 1 file changed, 149 insertions(+), 1 deletion(-)
Signature · 1 line
2.25.1
author
#1 lore raw

RFC v3 1/2 net/mlx5e: Add helpers to calculate txq and ch idx

Add two helpers to:

1. Compute the txq_ix given a channel and a tc offset (tc_to_txq_ix).
2. Compute the channel index and tc offset given a txq_ix (txq_ix_to_chtc_ix).

The first helper, tc_to_txq_ix, is used in place of the mathematical expressionin mlx5e_open_sqs when txq_ix values are computed.

The second helper, txq_ix_to_chtc_ix, will be used in a following patch.

---
 .../net/ethernet/mellanox/mlx5/core/en_main.c  | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +17 -1
index b758bc72ac36..ce15805ad55a 100644
@@ -2312,6 +2312,22 @@ static int mlx5e_txq_get_qos_node_hw_id(struct mlx5e_params *params, int txq_ix,
 	return 0;
 }
 
+static inline int tc_to_txq_ix(struct mlx5e_channel *c,
+			       struct mlx5e_params *params,
+			       int tc)
+{
+	return c->ix + tc * params->num_channels;
+}
+
+static inline void txq_ix_to_chtc_ix(struct mlx5e_params *params, int txq_ix,
+				     int *ch_ix, int *tc_ix)
+{
+	if (params->num_channels) {
+		*ch_ix = txq_ix % params->num_channels;
+		*tc_ix = txq_ix / params->num_channels;
+	}
+}
+
 static int mlx5e_open_sqs(struct mlx5e_channel *c,
 			  struct mlx5e_params *params,
 			  struct mlx5e_channel_param *cparam)
@@ -2319,7 +2335,7 @@ static int mlx5e_open_sqs(struct mlx5e_channel *c,
 	int err, tc;
 
 	for (tc = 0; tc < mlx5e_get_dcb_num_tc(params); tc++) {
-		int txq_ix = c->ix + tc * params->num_channels;
+		int txq_ix = tc_to_txq_ix(c, params, tc);
 		u32 qos_queue_group_id;
 		u32 tisn;
 
Signature · 1 line
2.25.1
Reviewed-by on v3 1/2
#2 lore raw

Re: [RFC net-next v3 1/2] net/mlx5e: Add helpers to calculate txq and ch idx

Quoted 6 lines · Joe Damato

Add two helpers to:

1. Compute the txq_ix given a channel and a tc offset (tc_to_txq_ix).
2. Compute the channel index and tc offset given a txq_ix (txq_ix_to_chtc_ix).

Quoted 7 lines

The first helper, tc_to_txq_ix, is used in place of the mathematical expressionin mlx5e_open_sqs when txq_ix values are computed.

The second helper, txq_ix_to_chtc_ix, will be used in a following patch.

---

reviewer
#3 lore raw

Re: [RFC net-next v3 1/2] net/mlx5e: Add helpers to calculate txq and ch idx

In reply to Joe Damato
Quoted 10 lines · Joe Damato

Add two helpers to:

1. Compute the txq_ix given a channel and a tc offset (tc_to_txq_ix).
2. Compute the channel index and tc offset given a txq_ix (txq_ix_to_chtc_ix).

The first helper, tc_to_txq_ix, is used in place of the mathematical expressionin mlx5e_open_sqs when txq_ix values are computed.

The second helper, txq_ix_to_chtc_ix, will be used in a following patch.

Hi Joe,

I think it would be best to add txq_ix_to_chtc_ix as part of patch that uses it, because the current arrangement will cause allmodconfigs with clang-18 and W=1 to fail due to txq_ix_to_chtc_ix being unused.

...

reviewer
#4 lore raw

Re: [RFC net-next v3 1/2] net/mlx5e: Add helpers to calculate txq and ch idx

Quoted 12 lines · Simon Horman
Quoted 10 lines · Joe Damato

Add two helpers to:

1. Compute the txq_ix given a channel and a tc offset (tc_to_txq_ix).
2. Compute the channel index and tc offset given a txq_ix (txq_ix_to_chtc_ix).

The first helper, tc_to_txq_ix, is used in place of the mathematical expressionin mlx5e_open_sqs when txq_ix values are computed.

The second helper, txq_ix_to_chtc_ix, will be used in a following patch.

Quoted 7 lines

Hi Joe,

I think it would be best to add txq_ix_to_chtc_ix as part of patch that uses it, because the current arrangement will cause allmodconfigs with clang-18 and W=1 to fail due to txq_ix_to_chtc_ix being unused.

...

Sorry, one more thing.

Please don't use inline in .c files unless there is a demonstrable reason - f.e. performance - to do so. Rather, let the compiler figure out when to inline functions.

author
#5 lore raw

Re: [RFC net-next v3 1/2] net/mlx5e: Add helpers to calculate txq and ch idx

Quoted 19 lines · Simon Horman
Quoted 18 lines · Simon Horman
Quoted 10 lines · Joe Damato

Add two helpers to:

1. Compute the txq_ix given a channel and a tc offset (tc_to_txq_ix).
2. Compute the channel index and tc offset given a txq_ix (txq_ix_to_chtc_ix).

The first helper, tc_to_txq_ix, is used in place of the mathematical expressionin mlx5e_open_sqs when txq_ix values are computed.

The second helper, txq_ix_to_chtc_ix, will be used in a following patch.

Hi Joe,

I think it would be best to add txq_ix_to_chtc_ix as part of patch that uses it, because the current arrangement will cause allmodconfigs with clang-18 and W=1 to fail due to txq_ix_to_chtc_ix being unused.

Quoted 7 lines
Quoted 1 line

...

Sorry, one more thing.

Please don't use inline in .c files unless there is a demonstrable reason - f.e. performance - to do so. Rather, let the compiler figure out when to inline functions.

Sure, I'll make sure in the next revision to include the second helper in the second patch instead and avoid using "inline" in both cases.

author
#6 lore raw

RFC v3 2/2 net/mlx5e: Add per queue netdev-genl stats

In reply to Joe Damato

Add functions to support the netdev-genl per queue stats API.

./cli.py --spec netlink/specs/netdev.yaml \
         --dump qstats-get --json '{"scope": "queue"}'

...snip

{'ifindex': 7,
'queue-id': 62,
'queue-type': 'rx',
'rx-alloc-fail': 0,
'rx-bytes': 105965251,
'rx-packets': 179790},
{'ifindex': 7,
'queue-id': 0,
'queue-type': 'tx',
'tx-bytes': 9402665,
'tx-packets': 17551},

...snip

Also tested with the script tools/testing/selftests/drivers/net/stats.py in several scenarios to ensure stats tallying was correct:

- on boot (default queue counts)
- adjusting queue count up or down (ethtool -L eth0 combined ...)
- adding mqprio TCs

---
 .../net/ethernet/mellanox/mlx5/core/en_main.c | 132 ++++++++++++++++++
 1 file changed, 132 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +132 -0
index ce15805ad55a..515c16a88a6c 100644
@@ -39,6 +39,7 @@
 #include <linux/debugfs.h>
 #include <linux/if_bridge.h>
 #include <linux/filter.h>
+#include <net/netdev_queues.h>
 #include <net/page_pool/types.h>
 #include <net/pkt_sched.h>
 #include <net/xdp_sock_drv.h>
@@ -5293,6 +5294,136 @@ static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev)
 	return (mlx5_vxlan_allowed(mdev->vxlan) || mlx5_geneve_tx_allowed(mdev));
 }
 
+static void mlx5e_get_queue_stats_rx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_rx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_rq_stats *xskrq_stats;
+	struct mlx5e_rq_stats *rq_stats;
+
+	if (mlx5e_is_uplink_rep(priv))
+		return;
+
+	channel_stats = priv->channel_stats[i];
+	xskrq_stats = &channel_stats->xskrq;
+	rq_stats = &channel_stats->rq;
+
+	stats->packets = rq_stats->packets + xskrq_stats->packets;
+	stats->bytes = rq_stats->bytes + xskrq_stats->bytes;
+	stats->alloc_fail = rq_stats->buff_alloc_err +
+			    xskrq_stats->buff_alloc_err;
+}
+
+static void mlx5e_get_queue_stats_tx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_tx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_sq_stats *sq_stats;
+	int ch_ix, tc_ix;
+
+	mutex_lock(&priv->state_lock);
+	txq_ix_to_chtc_ix(&priv->channels.params, i, &ch_ix, &tc_ix);
+	mutex_unlock(&priv->state_lock);
+
+	channel_stats = priv->channel_stats[ch_ix];
+	sq_stats = &channel_stats->sq[tc_ix];
+
+	stats->packets = sq_stats->packets;
+	stats->bytes = sq_stats->bytes;
+}
+
+static void mlx5e_get_base_stats(struct net_device *dev,
+				 struct netdev_queue_stats_rx *rx,
+				 struct netdev_queue_stats_tx *tx)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	int i, j;
+
+	if (!mlx5e_is_uplink_rep(priv)) {
+		rx->packets = 0;
+		rx->bytes = 0;
+		rx->alloc_fail = 0;
+
+		/* compute stats for deactivated RX queues
+		 *
+		 * if priv->channels.num == 0 the device is down, so compute
+		 * stats for every queue.
+		 *
+		 * otherwise, compute only the queues which have been deactivated.
+		 */
+		mutex_lock(&priv->state_lock);
+		if (priv->channels.num == 0)
+			i = 0;
+		else
+			i = priv->channels.params.num_channels;
+		mutex_unlock(&priv->state_lock);
+
+		for (; i < priv->stats_nch; i++) {
+			struct netdev_queue_stats_rx rx_i = {0};
+
+			mlx5e_get_queue_stats_rx(dev, i, &rx_i);
+
+			rx->packets += rx_i.packets;
+			rx->bytes += rx_i.bytes;
+			rx->alloc_fail += rx_i.alloc_fail;
+		}
+
+		if (priv->rx_ptp_opened) {
+			struct mlx5e_rq_stats *rq_stats = &priv->ptp_stats.rq;
+
+			rx->packets += rq_stats->packets;
+			rx->bytes += rq_stats->bytes;
+		}
+	}
+
+	tx->packets = 0;
+	tx->bytes = 0;
+
+	mutex_lock(&priv->state_lock);
+	for (i = 0; i < priv->stats_nch; i++) {
+		struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i];
+
+		/* while iterating through all channels [0, stats_nch], there
+		 * are two cases to handle:
+		 *
+		 *  1. the channel is available, so sum only the unavailable TCs
+		 *     [mlx5e_get_dcb_num_tc, max_opened_tc).
+		 *
+		 *  2. the channel is unavailable, so sum all TCs [0, max_opened_tc).
+		 */
+		if (i < priv->channels.params.num_channels) {
+			j = mlx5e_get_dcb_num_tc(&priv->channels.params);
+		} else {
+			j = 0;
+		}
+
+		for (; j < priv->max_opened_tc; j++) {
+			struct mlx5e_sq_stats *sq_stats = &channel_stats->sq[j];
+
+			tx->packets += sq_stats->packets;
+			tx->bytes += sq_stats->bytes;
+		}
+	}
+	mutex_unlock(&priv->state_lock);
+
+	if (priv->tx_ptp_opened) {
+		for (j = 0; j < priv->max_opened_tc; j++) {
+			struct mlx5e_sq_stats *sq_stats = &priv->ptp_stats.sq[j];
+
+			tx->packets    += sq_stats->packets;
+			tx->bytes      += sq_stats->bytes;
+		}
+	}
+}
+
+static const struct netdev_stat_ops mlx5e_stat_ops = {
+	.get_queue_stats_rx     = mlx5e_get_queue_stats_rx,
+	.get_queue_stats_tx     = mlx5e_get_queue_stats_tx,
+	.get_base_stats         = mlx5e_get_base_stats,
+};
+
 static void mlx5e_build_nic_netdev(struct net_device *netdev)
 {
 	struct mlx5e_priv *priv = netdev_priv(netdev);
@@ -5310,6 +5441,7 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev)
 
 	netdev->watchdog_timeo    = 15 * HZ;
 
+	netdev->stat_ops          = &mlx5e_stat_ops;
 	netdev->ethtool_ops	  = &mlx5e_ethtool_ops;
 
 	netdev->vlan_features    |= NETIF_F_SG;
Signature · 1 line
2.25.1
Reviewed-by on v3 2/2
#7 lore raw

Re: [RFC net-next v3 2/2] net/mlx5e: Add per queue netdev-genl stats

Quoted 24 lines · Joe Damato

Add functions to support the netdev-genl per queue stats API.

./cli.py --spec netlink/specs/netdev.yaml \
         --dump qstats-get --json '{"scope": "queue"}'

...snip

{'ifindex': 7,
'queue-id': 62,
'queue-type': 'rx',
'rx-alloc-fail': 0,
'rx-bytes': 105965251,
'rx-packets': 179790},
{'ifindex': 7,
'queue-id': 0,
'queue-type': 'tx',
'tx-bytes': 9402665,
'tx-packets': 17551},

...snip

Also tested with the script tools/testing/selftests/drivers/net/stats.py in several scenarios to ensure stats tallying was correct:

Quoted 6 lines

- on boot (default queue counts)
- adjusting queue count up or down (ethtool -L eth0 combined ...)
- adding mqprio TCs

---

#8 lore raw

Re: [RFC net-next v3 2/2] net/mlx5e: Add per queue netdev-genl stats

In reply to Joe Damato
Quoted 19 lines · Joe Damato

Add functions to support the netdev-genl per queue stats API.

./cli.py --spec netlink/specs/netdev.yaml \
          --dump qstats-get --json '{"scope": "queue"}'

...snip

{'ifindex': 7,
'queue-id': 62,
'queue-type': 'rx',
'rx-alloc-fail': 0,
'rx-bytes': 105965251,
'rx-packets': 179790},
{'ifindex': 7,
'queue-id': 0,
'queue-type': 'tx',
'tx-bytes': 9402665,
'tx-packets': 17551},

Quoted 8 lines

...snip

Also tested with the script tools/testing/selftests/drivers/net/stats.py in several scenarios to ensure stats tallying was correct:

- on boot (default queue counts)
- adjusting queue count up or down (ethtool -L eth0 combined ...)
- adding mqprio TCs

Please test also with interface down.

Quoted 74 lines
---
  .../net/ethernet/mellanox/mlx5/core/en_main.c | 132 ++++++++++++++++++
  1 file changed, 132 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +53 -0
index ce15805ad55a..515c16a88a6c 100644
@@ -39,6 +39,7 @@
  #include <linux/debugfs.h>
  #include <linux/if_bridge.h>
  #include <linux/filter.h>
+#include <net/netdev_queues.h>
  #include <net/page_pool/types.h>
  #include <net/pkt_sched.h>
  #include <net/xdp_sock_drv.h>
@@ -5293,6 +5294,136 @@ static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev)
  	return (mlx5_vxlan_allowed(mdev->vxlan) || mlx5_geneve_tx_allowed(mdev));
  }
 
+static void mlx5e_get_queue_stats_rx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_rx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_rq_stats *xskrq_stats;
+	struct mlx5e_rq_stats *rq_stats;
+
+	if (mlx5e_is_uplink_rep(priv))
+		return;
+
+	channel_stats = priv->channel_stats[i];
+	xskrq_stats = &channel_stats->xskrq;
+	rq_stats = &channel_stats->rq;
+
+	stats->packets = rq_stats->packets + xskrq_stats->packets;
+	stats->bytes = rq_stats->bytes + xskrq_stats->bytes;
+	stats->alloc_fail = rq_stats->buff_alloc_err +
+			    xskrq_stats->buff_alloc_err;
+}
+
+static void mlx5e_get_queue_stats_tx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_tx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_sq_stats *sq_stats;
+	int ch_ix, tc_ix;
+
+	mutex_lock(&priv->state_lock);
+	txq_ix_to_chtc_ix(&priv->channels.params, i, &ch_ix, &tc_ix);
+	mutex_unlock(&priv->state_lock);
+
+	channel_stats = priv->channel_stats[ch_ix];
+	sq_stats = &channel_stats->sq[tc_ix];
+
+	stats->packets = sq_stats->packets;
+	stats->bytes = sq_stats->bytes;
+}
+
+static void mlx5e_get_base_stats(struct net_device *dev,
+				 struct netdev_queue_stats_rx *rx,
+				 struct netdev_queue_stats_tx *tx)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	int i, j;
+
+	if (!mlx5e_is_uplink_rep(priv)) {
+		rx->packets = 0;
+		rx->bytes = 0;
+		rx->alloc_fail = 0;
+

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5358

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +10 -0
+		/* compute stats for deactivated RX queues
+		 *
+		 * if priv->channels.num == 0 the device is down, so compute
+		 * stats for every queue.
+		 *
+		 * otherwise, compute only the queues which have been deactivated.
+		 */
+		mutex_lock(&priv->state_lock);
+		if (priv->channels.num == 0)
+			i = 0;

This is not consistent with the above implementation of mlx5e_get_queue_stats_rx(), which always returns the stats even if the channel is down. This way, you'll double count the down channels.

I think you should always start from priv->channels.params.num_channels.

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5361

Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +3 -0
+		else
+			i = priv->channels.params.num_channels;
+		mutex_unlock(&priv->state_lock);

I understand that you're following the guidelines by taking the lock here, I just don't think this improves anything... If channels can be modified in between calls to mlx5e_get_base_stats / mlx5e_get_queue_stats_rx, then wrapping the priv->channels access with a lock can help protect each single deref, but not necessarily in giving a consistent "screenshot" of the stats.

The rtnl_lock should take care of that, as the driver holds it when changing the number of channels and updating the real_numrx/tx_queues.

This said, I would carefully say you can drop the mutex once following the requested changes above.

Quoted 24 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +24 -0
+
+		for (; i < priv->stats_nch; i++) {
+			struct netdev_queue_stats_rx rx_i = {0};
+
+			mlx5e_get_queue_stats_rx(dev, i, &rx_i);
+
+			rx->packets += rx_i.packets;
+			rx->bytes += rx_i.bytes;
+			rx->alloc_fail += rx_i.alloc_fail;
+		}
+
+		if (priv->rx_ptp_opened) {
+			struct mlx5e_rq_stats *rq_stats = &priv->ptp_stats.rq;
+
+			rx->packets += rq_stats->packets;
+			rx->bytes += rq_stats->bytes;
+		}
+	}
+
+	tx->packets = 0;
+	tx->bytes = 0;
+
+	mutex_lock(&priv->state_lock);
+	for (i = 0; i < priv->stats_nch; i++) {

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5395

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +10 -0
+		struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i];
+
+		/* while iterating through all channels [0, stats_nch], there
+		 * are two cases to handle:
+		 *
+		 *  1. the channel is available, so sum only the unavailable TCs
+		 *     [mlx5e_get_dcb_num_tc, max_opened_tc).
+		 *
+		 *  2. the channel is unavailable, so sum all TCs [0, max_opened_tc).
+		 */

I wonder why not call the local var 'tc'?

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5400

Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +5 -0
+		if (i < priv->channels.params.num_channels) {
+			j = mlx5e_get_dcb_num_tc(&priv->channels.params);
+		} else {
+			j = 0;
+		}

Remove parenthesis, or use ternary op.

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5410

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +10 -0
+
+		for (; j < priv->max_opened_tc; j++) {
+			struct mlx5e_sq_stats *sq_stats = &channel_stats->sq[j];
+
+			tx->packets += sq_stats->packets;
+			tx->bytes += sq_stats->bytes;
+		}
+	}
+	mutex_unlock(&priv->state_lock);
+

Same comment regarding dropping the mutex.

Quoted 27 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +17 -0
+	if (priv->tx_ptp_opened) {
+		for (j = 0; j < priv->max_opened_tc; j++) {
+			struct mlx5e_sq_stats *sq_stats = &priv->ptp_stats.sq[j];
+
+			tx->packets    += sq_stats->packets;
+			tx->bytes      += sq_stats->bytes;
+		}
+	}
+}
+
+static const struct netdev_stat_ops mlx5e_stat_ops = {
+	.get_queue_stats_rx     = mlx5e_get_queue_stats_rx,
+	.get_queue_stats_tx     = mlx5e_get_queue_stats_tx,
+	.get_base_stats         = mlx5e_get_base_stats,
+};
+
  static void mlx5e_build_nic_netdev(struct net_device *netdev)
  {
  	struct mlx5e_priv *priv = netdev_priv(netdev);
@@ -5310,6 +5441,7 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev)
 
  	netdev->watchdog_timeo    = 15 * HZ;
 
+	netdev->stat_ops          = &mlx5e_stat_ops;
  	netdev->ethtool_ops	  = &mlx5e_ethtool_ops;
 
  	netdev->vlan_features    |= NETIF_F_SG;
author
#9 lore raw

Re: [RFC net-next v3 2/2] net/mlx5e: Add per queue netdev-genl stats

Quoted 24 lines · Tariq Toukan
Quoted 21 lines · Joe Damato

Add functions to support the netdev-genl per queue stats API.

./cli.py --spec netlink/specs/netdev.yaml \
          --dump qstats-get --json '{"scope": "queue"}'

...snip

{'ifindex': 7,
'queue-id': 62,
'queue-type': 'rx',
'rx-alloc-fail': 0,
'rx-bytes': 105965251,
'rx-packets': 179790},
{'ifindex': 7,
'queue-id': 0,
'queue-type': 'tx',
'tx-bytes': 9402665,
'tx-packets': 17551},

...snip

Quoted 8 lines
Quoted 6 lines

Also tested with the script tools/testing/selftests/drivers/net/stats.py in several scenarios to ensure stats tallying was correct:

- on boot (default queue counts)
- adjusting queue count up or down (ethtool -L eth0 combined ...)
- adding mqprio TCs

Please test also with interface down.

OK. I'll test with the interface down.

Is there some publicly available Mellanox script I can run to test all the different cases? That would make this much easier. Maybe this is something to include in mlnx-tools on github?

The mlnx-tools scripts that includes some python scripts for setting up QoS doesn't seem to work on my system, and outputs vague error messages. I have no idea if I'm missing some kernel option, if the device doesn't support it, or if I need some other dependency installed.

I have been testing these patches on a:

Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
firmware-version: 16.29.2002 (MT_0000000013)

Quoted 89 lines
Quoted 73 lines
---
  .../net/ethernet/mellanox/mlx5/core/en_main.c | 132 ++++++++++++++++++
  1 file changed, 132 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +53 -0
index ce15805ad55a..515c16a88a6c 100644
@@ -39,6 +39,7 @@
  #include <linux/debugfs.h>
  #include <linux/if_bridge.h>
  #include <linux/filter.h>
+#include <net/netdev_queues.h>
  #include <net/page_pool/types.h>
  #include <net/pkt_sched.h>
  #include <net/xdp_sock_drv.h>
@@ -5293,6 +5294,136 @@ static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev)
  	return (mlx5_vxlan_allowed(mdev->vxlan) || mlx5_geneve_tx_allowed(mdev));
  }
+static void mlx5e_get_queue_stats_rx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_rx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_rq_stats *xskrq_stats;
+	struct mlx5e_rq_stats *rq_stats;
+
+	if (mlx5e_is_uplink_rep(priv))
+		return;
+
+	channel_stats = priv->channel_stats[i];
+	xskrq_stats = &channel_stats->xskrq;
+	rq_stats = &channel_stats->rq;
+
+	stats->packets = rq_stats->packets + xskrq_stats->packets;
+	stats->bytes = rq_stats->bytes + xskrq_stats->bytes;
+	stats->alloc_fail = rq_stats->buff_alloc_err +
+			    xskrq_stats->buff_alloc_err;
+}
+
+static void mlx5e_get_queue_stats_tx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_tx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_sq_stats *sq_stats;
+	int ch_ix, tc_ix;
+
+	mutex_lock(&priv->state_lock);
+	txq_ix_to_chtc_ix(&priv->channels.params, i, &ch_ix, &tc_ix);
+	mutex_unlock(&priv->state_lock);
+
+	channel_stats = priv->channel_stats[ch_ix];
+	sq_stats = &channel_stats->sq[tc_ix];
+
+	stats->packets = sq_stats->packets;
+	stats->bytes = sq_stats->bytes;
+}
+
+static void mlx5e_get_base_stats(struct net_device *dev,
+				 struct netdev_queue_stats_rx *rx,
+				 struct netdev_queue_stats_tx *tx)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	int i, j;
+
+	if (!mlx5e_is_uplink_rep(priv)) {
+		rx->packets = 0;
+		rx->bytes = 0;
+		rx->alloc_fail = 0;
+

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5357

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +10 -0
+		/* compute stats for deactivated RX queues
+		 *
+		 * if priv->channels.num == 0 the device is down, so compute
+		 * stats for every queue.
+		 *
+		 * otherwise, compute only the queues which have been deactivated.
+		 */
+		mutex_lock(&priv->state_lock);
+		if (priv->channels.num == 0)
+			i = 0;

This is not consistent with the above implementation of mlx5e_get_queue_stats_rx(), which always returns the stats even if the channel is down.
This way, you'll double count the down channels.

Quoted 1 line

I think you should always start from priv->channels.params.num_channels.

OK, I'll do that.

Quoted 10 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5361

Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +3 -0
+		else
+			i = priv->channels.params.num_channels;
+		mutex_unlock(&priv->state_lock);

I understand that you're following the guidelines by taking the lock here, I just don't think this improves anything... If channels can be modified in between calls to mlx5e_get_base_stats / mlx5e_get_queue_stats_rx, then wrapping the priv->channels access with a lock can help protect each single deref, but not necessarily in giving a consistent "screenshot" of the stats.

Quoted 5 lines

The rtnl_lock should take care of that, as the driver holds it when changing the number of channels and updating the real_numrx/tx_queues.

This said, I would carefully say you can drop the mutex once following the requested changes above.

OK, that makes sense to me.

So then I assume I can drop the mutex in mlx5e_get_queue_stats_tx above, as well, for the same reasons?

Does this mean then that you are in favor of the implementation for tx stats provided in this RFC and that I've implemented option 1 as you described in the previous thread correctly?

Quoted 28 lines
Quoted 28 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +28 -0
+
+		for (; i < priv->stats_nch; i++) {
+			struct netdev_queue_stats_rx rx_i = {0};
+
+			mlx5e_get_queue_stats_rx(dev, i, &rx_i);
+
+			rx->packets += rx_i.packets;
+			rx->bytes += rx_i.bytes;
+			rx->alloc_fail += rx_i.alloc_fail;
+		}
+
+		if (priv->rx_ptp_opened) {
+			struct mlx5e_rq_stats *rq_stats = &priv->ptp_stats.rq;
+
+			rx->packets += rq_stats->packets;
+			rx->bytes += rq_stats->bytes;
+		}
+	}
+
+	tx->packets = 0;
+	tx->bytes = 0;
+
+	mutex_lock(&priv->state_lock);
+	for (i = 0; i < priv->stats_nch; i++) {
+		struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i];
+
+		/* while iterating through all channels [0, stats_nch], there
+		 * are two cases to handle:
Quoted 8 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5395

Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +6 -0
+		 *
+		 *  1. the channel is available, so sum only the unavailable TCs
+		 *     [mlx5e_get_dcb_num_tc, max_opened_tc).
+		 *
+		 *  2. the channel is unavailable, so sum all TCs [0, max_opened_tc).
+		 */

I wonder why not call the local var 'tc'?

OK.

Quoted 7 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5400

Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +5 -0
+		if (i < priv->channels.params.num_channels) {
+			j = mlx5e_get_dcb_num_tc(&priv->channels.params);
+		} else {
+			j = 0;
+		}

Remove parenthesis, or use ternary op.

I'll remove the parenthesis; I didn't run checkpatch.pl on this RFC (which catches this), but I should have.

Quoted 4 lines
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +4 -0
+
+		for (; j < priv->max_opened_tc; j++) {
+			struct mlx5e_sq_stats *sq_stats = &channel_stats->sq[j];
+
Quoted 8 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5410

Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +6 -0
+			tx->packets += sq_stats->packets;
+			tx->bytes += sq_stats->bytes;
+		}
+	}
+	mutex_unlock(&priv->state_lock);
+

Same comment regarding dropping the mutex.

OK.

#10 lore raw

Re: [RFC net-next v3 2/2] net/mlx5e: Add per queue netdev-genl stats

Quoted 32 lines · Joe Damato
Quoted 31 lines · Tariq Toukan
Quoted 27 lines · Joe Damato

Add functions to support the netdev-genl per queue stats API.

./cli.py --spec netlink/specs/netdev.yaml \
           --dump qstats-get --json '{"scope": "queue"}'

...snip

   {'ifindex': 7,
    'queue-id': 62,
    'queue-type': 'rx',
    'rx-alloc-fail': 0,
    'rx-bytes': 105965251,
    'rx-packets': 179790},
   {'ifindex': 7,
    'queue-id': 0,
    'queue-type': 'tx',
    'tx-bytes': 9402665,
    'tx-packets': 17551},

...snip

Also tested with the script tools/testing/selftests/drivers/net/stats.py in several scenarios to ensure stats tallying was correct:

- on boot (default queue counts)
- adjusting queue count up or down (ethtool -L eth0 combined ...)
- adding mqprio TCs

Quoted 8 lines
Quoted 1 line

Please test also with interface down.

OK. I'll test with the interface down.

Is there some publicly available Mellanox script I can run to test all the different cases? That would make this much easier. Maybe this is something to include in mlnx-tools on github?

You're testing some new functionality. We don't have something for it.

Quoted 6 lines

The mlnx-tools scripts that includes some python scripts for setting up QoS doesn't seem to work on my system, and outputs vague error messages. I have no idea if I'm missing some kernel option, if the device doesn't support it, or if I need some other dependency installed.

Can you share the command you use, and the output?

Quoted 94 lines

I have been testing these patches on a:

Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
firmware-version: 16.29.2002 (MT_0000000013)

Quoted 89 lines
Quoted 73 lines
---
   .../net/ethernet/mellanox/mlx5/core/en_main.c | 132 ++++++++++++++++++
   1 file changed, 132 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +53 -0
index ce15805ad55a..515c16a88a6c 100644
@@ -39,6 +39,7 @@
   #include <linux/debugfs.h>
   #include <linux/if_bridge.h>
   #include <linux/filter.h>
+#include <net/netdev_queues.h>
   #include <net/page_pool/types.h>
   #include <net/pkt_sched.h>
   #include <net/xdp_sock_drv.h>
@@ -5293,6 +5294,136 @@ static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev)
   	return (mlx5_vxlan_allowed(mdev->vxlan) || mlx5_geneve_tx_allowed(mdev));
   }
+static void mlx5e_get_queue_stats_rx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_rx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_rq_stats *xskrq_stats;
+	struct mlx5e_rq_stats *rq_stats;
+
+	if (mlx5e_is_uplink_rep(priv))
+		return;
+
+	channel_stats = priv->channel_stats[i];
+	xskrq_stats = &channel_stats->xskrq;
+	rq_stats = &channel_stats->rq;
+
+	stats->packets = rq_stats->packets + xskrq_stats->packets;
+	stats->bytes = rq_stats->bytes + xskrq_stats->bytes;
+	stats->alloc_fail = rq_stats->buff_alloc_err +
+			    xskrq_stats->buff_alloc_err;
+}
+
+static void mlx5e_get_queue_stats_tx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_tx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_sq_stats *sq_stats;
+	int ch_ix, tc_ix;
+
+	mutex_lock(&priv->state_lock);
+	txq_ix_to_chtc_ix(&priv->channels.params, i, &ch_ix, &tc_ix);
+	mutex_unlock(&priv->state_lock);
+
+	channel_stats = priv->channel_stats[ch_ix];
+	sq_stats = &channel_stats->sq[tc_ix];
+
+	stats->packets = sq_stats->packets;
+	stats->bytes = sq_stats->bytes;
+}
+
+static void mlx5e_get_base_stats(struct net_device *dev,
+				 struct netdev_queue_stats_rx *rx,
+				 struct netdev_queue_stats_tx *tx)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	int i, j;
+
+	if (!mlx5e_is_uplink_rep(priv)) {
+		rx->packets = 0;
+		rx->bytes = 0;
+		rx->alloc_fail = 0;
+

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5357

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +10 -0
+		/* compute stats for deactivated RX queues
+		 *
+		 * if priv->channels.num == 0 the device is down, so compute
+		 * stats for every queue.
+		 *
+		 * otherwise, compute only the queues which have been deactivated.
+		 */
+		mutex_lock(&priv->state_lock);
+		if (priv->channels.num == 0)
+			i = 0;

This is not consistent with the above implementation of mlx5e_get_queue_stats_rx(), which always returns the stats even if the channel is down.
This way, you'll double count the down channels.

Quoted 19 lines
Quoted 1 line

I think you should always start from priv->channels.params.num_channels.

OK, I'll do that.

Quoted 15 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5361

Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +3 -0
+		else
+			i = priv->channels.params.num_channels;
+		mutex_unlock(&priv->state_lock);

I understand that you're following the guidelines by taking the lock here, I just don't think this improves anything... If channels can be modified in between calls to mlx5e_get_base_stats / mlx5e_get_queue_stats_rx, then wrapping the priv->channels access with a lock can help protect each single deref, but not necessarily in giving a consistent "screenshot" of the stats.

The rtnl_lock should take care of that, as the driver holds it when changing the number of channels and updating the real_numrx/tx_queues.

This said, I would carefully say you can drop the mutex once following the requested changes above.

I still don't really like this design, so I gave some more thought on this...

I think we should come up with a new mapping array under priv, that maps i (from real_num_tx_queues) to the matching sq_stats struct. This array would be maintained in the channels open/close functions, similarly to priv->txq2sq.

Then, we would not calculate the mapping per call, but just get the proper pointer from the array. This eases the handling of htb and ptp queues, which were missed in your txq_ix_to_chtc_ix().

This handles mapped SQs.

Now, regarding unmapped ones, they must be handled in the "base" function call. We'd still need to access channels->params, to:
1. read params.num_channels to iterate until priv->stats_nch, and
2. read mlx5e_get_dcb_num_tc(params) to iterate until priv->max_opened_tc.

I think we can live with this without holding the mutex, given that this runs under the rtnl lock. We can add ASSERT_RTNL() to verify the assumption.

Quoted 10 lines

OK, that makes sense to me.

So then I assume I can drop the mutex in mlx5e_get_queue_stats_tx above, as well, for the same reasons?

Does this mean then that you are in favor of the implementation for tx stats provided in this RFC and that I've implemented option 1 as you described in the previous thread correctly?

Yes, but I wasn't happy enough with the design.
Thanks for your contribution.

Quoted 64 lines
Quoted 28 lines
Quoted 28 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +28 -0
+
+		for (; i < priv->stats_nch; i++) {
+			struct netdev_queue_stats_rx rx_i = {0};
+
+			mlx5e_get_queue_stats_rx(dev, i, &rx_i);
+
+			rx->packets += rx_i.packets;
+			rx->bytes += rx_i.bytes;
+			rx->alloc_fail += rx_i.alloc_fail;
+		}
+
+		if (priv->rx_ptp_opened) {
+			struct mlx5e_rq_stats *rq_stats = &priv->ptp_stats.rq;
+
+			rx->packets += rq_stats->packets;
+			rx->bytes += rq_stats->bytes;
+		}
+	}
+
+	tx->packets = 0;
+	tx->bytes = 0;
+
+	mutex_lock(&priv->state_lock);
+	for (i = 0; i < priv->stats_nch; i++) {
+		struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i];
+
+		/* while iterating through all channels [0, stats_nch], there
+		 * are two cases to handle:
Quoted 8 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5395

Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +6 -0
+		 *
+		 *  1. the channel is available, so sum only the unavailable TCs
+		 *     [mlx5e_get_dcb_num_tc, max_opened_tc).
+		 *
+		 *  2. the channel is unavailable, so sum all TCs [0, max_opened_tc).
+		 */

I wonder why not call the local var 'tc'?

OK.

Quoted 7 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5400

Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +5 -0
+		if (i < priv->channels.params.num_channels) {
+			j = mlx5e_get_dcb_num_tc(&priv->channels.params);
+		} else {
+			j = 0;
+		}

Remove parenthesis, or use ternary op.

I'll remove the parenthesis; I didn't run checkpatch.pl on this RFC (which catches this), but I should have.

Quoted 4 lines
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +4 -0
+
+		for (; j < priv->max_opened_tc; j++) {
+			struct mlx5e_sq_stats *sq_stats = &channel_stats->sq[j];
+
Quoted 8 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5410

Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +6 -0
+			tx->packets += sq_stats->packets;
+			tx->bytes += sq_stats->bytes;
+		}
+	}
+	mutex_unlock(&priv->state_lock);
+

Same comment regarding dropping the mutex.

OK.

author
#11 lore raw

Re: [RFC net-next v3 2/2] net/mlx5e: Add per queue netdev-genl stats

Quoted 47 lines · Tariq Toukan
Quoted 32 lines · Joe Damato
Quoted 31 lines · Tariq Toukan
Quoted 27 lines · Joe Damato

Add functions to support the netdev-genl per queue stats API.

./cli.py --spec netlink/specs/netdev.yaml \
           --dump qstats-get --json '{"scope": "queue"}'

...snip

   {'ifindex': 7,
    'queue-id': 62,
    'queue-type': 'rx',
    'rx-alloc-fail': 0,
    'rx-bytes': 105965251,
    'rx-packets': 179790},
   {'ifindex': 7,
    'queue-id': 0,
    'queue-type': 'tx',
    'tx-bytes': 9402665,
    'tx-packets': 17551},

...snip

Also tested with the script tools/testing/selftests/drivers/net/stats.py in several scenarios to ensure stats tallying was correct:

- on boot (default queue counts)
- adjusting queue count up or down (ethtool -L eth0 combined ...)
- adding mqprio TCs

Quoted 8 lines
Quoted 1 line

Please test also with interface down.

OK. I'll test with the interface down.

Is there some publicly available Mellanox script I can run to test all the different cases? That would make this much easier. Maybe this is something to include in mlnx-tools on github?

You're testing some new functionality. We don't have something for it.

Quoted 8 lines
Quoted 6 lines

The mlnx-tools scripts that includes some python scripts for setting up QoS doesn't seem to work on my system, and outputs vague error messages. I have no idea if I'm missing some kernel option, if the device doesn't support it, or if I need some other dependency installed.

Can you share the command you use, and the output?

Sure:

jdamato@test:~/mlnx-tools/python$ python --version
Python 3.12.3
jdamato@test:~/mlnx-tools/python$ ./mlnx_qos -i vlan401
Priority trust state is not supported on your system
Buffers commands are not supported on your system
Rate limit is not supported on your system!
ETS features are not supported on your system
jdamato@test:~/mlnx-tools/python$ echo $?
1

This is mlnx-tools at SHA 641718b13f71 ("Merge pull request #87 from ahlabenadam/no_remove_folder")

You can feel free to follow up with me about this off-list if you like.

Quoted 108 lines
Quoted 108 lines

I have been testing these patches on a:

Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
firmware-version: 16.29.2002 (MT_0000000013)

Quoted 89 lines
Quoted 73 lines
---
   .../net/ethernet/mellanox/mlx5/core/en_main.c | 132 ++++++++++++++++++
   1 file changed, 132 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +53 -0
index ce15805ad55a..515c16a88a6c 100644
@@ -39,6 +39,7 @@
   #include <linux/debugfs.h>
   #include <linux/if_bridge.h>
   #include <linux/filter.h>
+#include <net/netdev_queues.h>
   #include <net/page_pool/types.h>
   #include <net/pkt_sched.h>
   #include <net/xdp_sock_drv.h>
@@ -5293,6 +5294,136 @@ static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev)
   	return (mlx5_vxlan_allowed(mdev->vxlan) || mlx5_geneve_tx_allowed(mdev));
   }
+static void mlx5e_get_queue_stats_rx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_rx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_rq_stats *xskrq_stats;
+	struct mlx5e_rq_stats *rq_stats;
+
+	if (mlx5e_is_uplink_rep(priv))
+		return;
+
+	channel_stats = priv->channel_stats[i];
+	xskrq_stats = &channel_stats->xskrq;
+	rq_stats = &channel_stats->rq;
+
+	stats->packets = rq_stats->packets + xskrq_stats->packets;
+	stats->bytes = rq_stats->bytes + xskrq_stats->bytes;
+	stats->alloc_fail = rq_stats->buff_alloc_err +
+			    xskrq_stats->buff_alloc_err;
+}
+
+static void mlx5e_get_queue_stats_tx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_tx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_sq_stats *sq_stats;
+	int ch_ix, tc_ix;
+
+	mutex_lock(&priv->state_lock);
+	txq_ix_to_chtc_ix(&priv->channels.params, i, &ch_ix, &tc_ix);
+	mutex_unlock(&priv->state_lock);
+
+	channel_stats = priv->channel_stats[ch_ix];
+	sq_stats = &channel_stats->sq[tc_ix];
+
+	stats->packets = sq_stats->packets;
+	stats->bytes = sq_stats->bytes;
+}
+
+static void mlx5e_get_base_stats(struct net_device *dev,
+				 struct netdev_queue_stats_rx *rx,
+				 struct netdev_queue_stats_tx *tx)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	int i, j;
+
+	if (!mlx5e_is_uplink_rep(priv)) {
+		rx->packets = 0;
+		rx->bytes = 0;
+		rx->alloc_fail = 0;
+

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5357

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +10 -0
+		/* compute stats for deactivated RX queues
+		 *
+		 * if priv->channels.num == 0 the device is down, so compute
+		 * stats for every queue.
+		 *
+		 * otherwise, compute only the queues which have been deactivated.
+		 */
+		mutex_lock(&priv->state_lock);
+		if (priv->channels.num == 0)
+			i = 0;

This is not consistent with the above implementation of mlx5e_get_queue_stats_rx(), which always returns the stats even if the channel is down.
This way, you'll double count the down channels.

Quoted 1 line

I think you should always start from priv->channels.params.num_channels.

OK, I'll do that.

Quoted 10 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5361

Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +3 -0
+		else
+			i = priv->channels.params.num_channels;
+		mutex_unlock(&priv->state_lock);

I understand that you're following the guidelines by taking the lock here, I just don't think this improves anything... If channels can be modified in between calls to mlx5e_get_base_stats / mlx5e_get_queue_stats_rx, then wrapping the priv->channels access with a lock can help protect each single deref, but not necessarily in giving a consistent "screenshot" of the stats.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

The rtnl_lock should take care of that, as the driver holds it when changing the number of channels and updating the real_numrx/tx_queues.

This said, I would carefully say you can drop the mutex once following the requested changes above.

I still don't really like this design, so I gave some more thought on this...

Thanks, again, for your careful thoughts and review on this. I do appreciate it and this functionality will be extremely useful for me (and I suspect many others).

Quoted 10 lines

I think we should come up with a new mapping array under priv, that maps i (from real_num_tx_queues) to the matching sq_stats struct. This array would be maintained in the channels open/close functions, similarly to priv->txq2sq.

Then, we would not calculate the mapping per call, but just get the proper pointer from the array. This eases the handling of htb and ptp queues, which were missed in your txq_ix_to_chtc_ix().

This handles mapped SQs.

OK, the above makes sense. I'll give that a try.

Quoted 9 lines

Now, regarding unmapped ones, they must be handled in the "base" function call.
We'd still need to access channels->params, to:
1. read params.num_channels to iterate until priv->stats_nch, and
2. read mlx5e_get_dcb_num_tc(params) to iterate until priv->max_opened_tc.

I think we can live with this without holding the mutex, given that this runs under the rtnl lock.
We can add ASSERT_RTNL() to verify the assumption.

OK. I'll try the above and propose an rfc v4.

Quoted 7 lines
Quoted 7 lines

OK, that makes sense to me.

So then I assume I can drop the mutex in mlx5e_get_queue_stats_tx above, as well, for the same reasons?

Quoted 7 lines
Quoted 4 lines

Does this mean then that you are in favor of the implementation for tx stats provided in this RFC and that I've implemented option 1 as you described in the previous thread correctly?

Yes, but I wasn't happy enough with the design.
Thanks for your contribution.

Thanks for your review and patience.

author
#12 lore raw

Re: [RFC net-next v3 2/2] net/mlx5e: Add per queue netdev-genl stats

In reply to Tariq Toukan
Quoted 173 lines · Tariq Toukan
Quoted 32 lines · Joe Damato
Quoted 31 lines · Tariq Toukan
Quoted 27 lines · Joe Damato

Add functions to support the netdev-genl per queue stats API.

./cli.py --spec netlink/specs/netdev.yaml \
           --dump qstats-get --json '{"scope": "queue"}'

...snip

   {'ifindex': 7,
    'queue-id': 62,
    'queue-type': 'rx',
    'rx-alloc-fail': 0,
    'rx-bytes': 105965251,
    'rx-packets': 179790},
   {'ifindex': 7,
    'queue-id': 0,
    'queue-type': 'tx',
    'tx-bytes': 9402665,
    'tx-packets': 17551},

...snip

Also tested with the script tools/testing/selftests/drivers/net/stats.py in several scenarios to ensure stats tallying was correct:

- on boot (default queue counts)
- adjusting queue count up or down (ethtool -L eth0 combined ...)
- adding mqprio TCs

Quoted 8 lines
Quoted 1 line

Please test also with interface down.

OK. I'll test with the interface down.

Is there some publicly available Mellanox script I can run to test all the different cases? That would make this much easier. Maybe this is something to include in mlnx-tools on github?

You're testing some new functionality. We don't have something for it.

Quoted 6 lines

The mlnx-tools scripts that includes some python scripts for setting up QoS doesn't seem to work on my system, and outputs vague error messages. I have no idea if I'm missing some kernel option, if the device doesn't support it, or if I need some other dependency installed.

Can you share the command you use, and the output?

Quoted 94 lines

I have been testing these patches on a:

Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
firmware-version: 16.29.2002 (MT_0000000013)

Quoted 89 lines
Quoted 73 lines
---
   .../net/ethernet/mellanox/mlx5/core/en_main.c | 132 ++++++++++++++++++
   1 file changed, 132 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +53 -0
index ce15805ad55a..515c16a88a6c 100644
@@ -39,6 +39,7 @@
   #include <linux/debugfs.h>
   #include <linux/if_bridge.h>
   #include <linux/filter.h>
+#include <net/netdev_queues.h>
   #include <net/page_pool/types.h>
   #include <net/pkt_sched.h>
   #include <net/xdp_sock_drv.h>
@@ -5293,6 +5294,136 @@ static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev)
   	return (mlx5_vxlan_allowed(mdev->vxlan) || mlx5_geneve_tx_allowed(mdev));
   }
+static void mlx5e_get_queue_stats_rx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_rx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_rq_stats *xskrq_stats;
+	struct mlx5e_rq_stats *rq_stats;
+
+	if (mlx5e_is_uplink_rep(priv))
+		return;
+
+	channel_stats = priv->channel_stats[i];
+	xskrq_stats = &channel_stats->xskrq;
+	rq_stats = &channel_stats->rq;
+
+	stats->packets = rq_stats->packets + xskrq_stats->packets;
+	stats->bytes = rq_stats->bytes + xskrq_stats->bytes;
+	stats->alloc_fail = rq_stats->buff_alloc_err +
+			    xskrq_stats->buff_alloc_err;
+}
+
+static void mlx5e_get_queue_stats_tx(struct net_device *dev, int i,
+				     struct netdev_queue_stats_tx *stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel_stats *channel_stats;
+	struct mlx5e_sq_stats *sq_stats;
+	int ch_ix, tc_ix;
+
+	mutex_lock(&priv->state_lock);
+	txq_ix_to_chtc_ix(&priv->channels.params, i, &ch_ix, &tc_ix);
+	mutex_unlock(&priv->state_lock);
+
+	channel_stats = priv->channel_stats[ch_ix];
+	sq_stats = &channel_stats->sq[tc_ix];
+
+	stats->packets = sq_stats->packets;
+	stats->bytes = sq_stats->bytes;
+}
+
+static void mlx5e_get_base_stats(struct net_device *dev,
+				 struct netdev_queue_stats_rx *rx,
+				 struct netdev_queue_stats_tx *tx)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	int i, j;
+
+	if (!mlx5e_is_uplink_rep(priv)) {
+		rx->packets = 0;
+		rx->bytes = 0;
+		rx->alloc_fail = 0;
+

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5357

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +10 -0
+		/* compute stats for deactivated RX queues
+		 *
+		 * if priv->channels.num == 0 the device is down, so compute
+		 * stats for every queue.
+		 *
+		 * otherwise, compute only the queues which have been deactivated.
+		 */
+		mutex_lock(&priv->state_lock);
+		if (priv->channels.num == 0)
+			i = 0;

This is not consistent with the above implementation of mlx5e_get_queue_stats_rx(), which always returns the stats even if the channel is down.
This way, you'll double count the down channels.

Quoted 19 lines
Quoted 1 line

I think you should always start from priv->channels.params.num_channels.

OK, I'll do that.

Quoted 15 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5361

Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +3 -0
+		else
+			i = priv->channels.params.num_channels;
+		mutex_unlock(&priv->state_lock);

I understand that you're following the guidelines by taking the lock here, I just don't think this improves anything... If channels can be modified in between calls to mlx5e_get_base_stats / mlx5e_get_queue_stats_rx, then wrapping the priv->channels access with a lock can help protect each single deref, but not necessarily in giving a consistent "screenshot" of the stats.

The rtnl_lock should take care of that, as the driver holds it when changing the number of channels and updating the real_numrx/tx_queues.

This said, I would carefully say you can drop the mutex once following the requested changes above.

I still don't really like this design, so I gave some more thought on this...

Quoted 8 lines

I think we should come up with a new mapping array under priv, that maps i (from real_num_tx_queues) to the matching sq_stats struct. This array would be maintained in the channels open/close functions, similarly to priv->txq2sq.

Then, we would not calculate the mapping per call, but just get the proper pointer from the array. This eases the handling of htb and ptp queues, which were missed in your txq_ix_to_chtc_ix().

Maybe I am just getting way off track here, but I noticed this in drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c in set_pflag_tx_port_ts:

  if (!err)
    priv->tx_ptp_opened = true;

but what if enable is false, meaning the user is disabling ptp via ethtool?

In that case, shouldn't this be:

  if (!err)
    priv->tx_ptp_opened = enabled;

Because it seems like the user could pass 0 to disable ptp, and with the current code then trigger mlx5e_close_channels which would call mlx5e_ptp_close, but priv->tx_ptp_opened might still be true even though ptp has been closed. Just a guess as I really don't know much about this code at all, but it seems that the MLX5E_PFLAG_TX_PORT_TS will be set in set_pflag_tx_port_ts based on enable, which then affects how mlx5e_open_channels behaves.

Likewise in mlx5e_ptp_close_queues, maybe rx_ptp_opened and tx_ptp_opened should be set to false there?

It seems like the user could get the driver into an inconsistent state by enabling then disabling ptp, but maybe I'm just reading this all wrong.

Is that a bug? If so, I can submit:

1. ethtool tx_ptp_opened = false
2. rx_ptp_opened = tx_ptp_opened = false in mlx5e_ptp_close_queues

to net as a Fixes ?

I am asking about this from the perspective of stats, because in the qos stuff, the txq2sq_stats mapping is created or removed (set to NULL) in mlx5e_activate_qos_sq and mlx5e_deactivate_qos_sq / mlx5e_qos_deactivate_queues, respectively.

This means that priv->txq2sq_stats could be a valid sq_stats or invalid for deactivated TCs and qos. It seems like ptp should work the same way, but I have no idea.

If ptp did work the same way, then in base the code would check if ptp was disabled and obtain the stats from it, if there are any from it being previously enabled.

That seems like more consistent behavior, but sorry if I'm totally off on all of this.

Quoted 92 lines

This handles mapped SQs.

Now, regarding unmapped ones, they must be handled in the "base" function call.
We'd still need to access channels->params, to:
1. read params.num_channels to iterate until priv->stats_nch, and
2. read mlx5e_get_dcb_num_tc(params) to iterate until priv->max_opened_tc.

I think we can live with this without holding the mutex, given that this runs under the rtnl lock.
We can add ASSERT_RTNL() to verify the assumption.

Quoted 10 lines

OK, that makes sense to me.

So then I assume I can drop the mutex in mlx5e_get_queue_stats_tx above, as well, for the same reasons?

Does this mean then that you are in favor of the implementation for tx stats provided in this RFC and that I've implemented option 1 as you described in the previous thread correctly?

Yes, but I wasn't happy enough with the design.
Thanks for your contribution.

Quoted 64 lines
Quoted 28 lines
Quoted 28 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +28 -0
+
+		for (; i < priv->stats_nch; i++) {
+			struct netdev_queue_stats_rx rx_i = {0};
+
+			mlx5e_get_queue_stats_rx(dev, i, &rx_i);
+
+			rx->packets += rx_i.packets;
+			rx->bytes += rx_i.bytes;
+			rx->alloc_fail += rx_i.alloc_fail;
+		}
+
+		if (priv->rx_ptp_opened) {
+			struct mlx5e_rq_stats *rq_stats = &priv->ptp_stats.rq;
+
+			rx->packets += rq_stats->packets;
+			rx->bytes += rq_stats->bytes;
+		}
+	}
+
+	tx->packets = 0;
+	tx->bytes = 0;
+
+	mutex_lock(&priv->state_lock);
+	for (i = 0; i < priv->stats_nch; i++) {
+		struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i];
+
+		/* while iterating through all channels [0, stats_nch], there
+		 * are two cases to handle:
Quoted 8 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5395

Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +6 -0
+		 *
+		 *  1. the channel is available, so sum only the unavailable TCs
+		 *     [mlx5e_get_dcb_num_tc, max_opened_tc).
+		 *
+		 *  2. the channel is unavailable, so sum all TCs [0, max_opened_tc).
+		 */

I wonder why not call the local var 'tc'?

OK.

Quoted 7 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5400

Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +5 -0
+		if (i < priv->channels.params.num_channels) {
+			j = mlx5e_get_dcb_num_tc(&priv->channels.params);
+		} else {
+			j = 0;
+		}

Remove parenthesis, or use ternary op.

I'll remove the parenthesis; I didn't run checkpatch.pl on this RFC (which catches this), but I should have.

Quoted 4 lines
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +4 -0
+
+		for (; j < priv->max_opened_tc; j++) {
+			struct mlx5e_sq_stats *sq_stats = &channel_stats->sq[j];
+
Quoted 8 lines

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static bool mlx5e_tunnel_any_tx_proto_supported(struct mlx5_core_dev *mdev) · line 5410

Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +6 -0
+			tx->packets += sq_stats->packets;
+			tx->bytes += sq_stats->bytes;
+		}
+	}
+	mutex_unlock(&priv->state_lock);
+

Same comment regarding dropping the mutex.

OK.

#13 lore raw

Re: [RFC net-next v3 2/2] net/mlx5e: Add per queue netdev-genl stats

Quoted 2 lines · Joe Damato
Quoted 1 line · Tariq Toukan

...

Quoted 17 lines
Quoted 4 lines

I still don't really like this design, so I gave some more thought on this...

Quoted 8 lines

I think we should come up with a new mapping array under priv, that maps i (from real_num_tx_queues) to the matching sq_stats struct. This array would be maintained in the channels open/close functions, similarly to priv->txq2sq.

Then, we would not calculate the mapping per call, but just get the proper pointer from the array. This eases the handling of htb and ptp queues, which were missed in your txq_ix_to_chtc_ix().

Maybe I am just getting way off track here, but I noticed this in drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c in set_pflag_tx_port_ts:

Quoted 6 lines
   if (!err)
     priv->tx_ptp_opened = true;

but what if enable is false, meaning the user is disabling ptp via ethtool?

This bool field under priv acts as a flag, means: ptp was ever opened. It's the same idea as max_opened_tc, but with boolean instead of numeric.

Quoted 43 lines

In that case, shouldn't this be:

   if (!err)
     priv->tx_ptp_opened = enabled;

Because it seems like the user could pass 0 to disable ptp, and with the current code then trigger mlx5e_close_channels which would call mlx5e_ptp_close, but priv->tx_ptp_opened might still be true even though ptp has been closed. Just a guess as I really don't know much about this code at all, but it seems that the MLX5E_PFLAG_TX_PORT_TS will be set in set_pflag_tx_port_ts based on enable, which then affects how mlx5e_open_channels behaves.

Likewise in mlx5e_ptp_close_queues, maybe rx_ptp_opened and tx_ptp_opened should be set to false there?

It seems like the user could get the driver into an inconsistent state by enabling then disabling ptp, but maybe I'm just reading this all wrong.

Is that a bug? If so, I can submit:

1. ethtool tx_ptp_opened = false
2. rx_ptp_opened = tx_ptp_opened = false in mlx5e_ptp_close_queues

to net as a Fixes ?

I am asking about this from the perspective of stats, because in the qos stuff, the txq2sq_stats mapping is created or removed (set to NULL) in mlx5e_activate_qos_sq and mlx5e_deactivate_qos_sq / mlx5e_qos_deactivate_queues, respectively.

This means that priv->txq2sq_stats could be a valid sq_stats or invalid for deactivated TCs and qos. It seems like ptp should work the same way, but I have no idea.

If ptp did work the same way, then in base the code would check if ptp was disabled and obtain the stats from it, if there are any from it being previously enabled.

That seems like more consistent behavior, but sorry if I'm totally off on all of this.

author
#14 lore raw

Re: [RFC net-next v3 2/2] net/mlx5e: Add per queue netdev-genl stats

Quoted 28 lines · Tariq Toukan
Quoted 2 lines · Joe Damato
Quoted 1 line · Tariq Toukan

...

Quoted 20 lines
Quoted 4 lines

I still don't really like this design, so I gave some more thought on this...

Quoted 8 lines

I think we should come up with a new mapping array under priv, that maps i (from real_num_tx_queues) to the matching sq_stats struct. This array would be maintained in the channels open/close functions, similarly to priv->txq2sq.

Then, we would not calculate the mapping per call, but just get the proper pointer from the array. This eases the handling of htb and ptp queues, which were missed in your txq_ix_to_chtc_ix().

Maybe I am just getting way off track here, but I noticed this in drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c in set_pflag_tx_port_ts:

   if (!err)
     priv->tx_ptp_opened = true;
Quoted 6 lines
Quoted 3 lines

but what if enable is false, meaning the user is disabling ptp via ethtool?

This bool field under priv acts as a flag, means: ptp was ever opened. It's the same idea as max_opened_tc, but with boolean instead of numeric.

OK, but then to avoid double counting ptp, I suppose we don't output ptp stats in base and only when the correct txq_ix is passed in to the tx stats function which refers to the PTP queue?

It's either that or we dont include ptp's sqstats in the txq2sq_stats mapping you've proposed, if I understand correctly.

maintainer
#15 lore raw
In reply to Joe Damato
Quoted 2 lines · On Wed, 29 May 2024 03:16:25 +0000 Joe Damato wrote:

Worth noting that Tariq suggested I also export HTB/QOS stats in mlx5e_get_base_stats.

Why to base, and not report them as queue stats?

Judging by mlx5e_update_tx_netdev_queues() calls sprinkled in ../mlx5/core/en/htb.c it seems that the driver will update the real_num_tx_queues accordingly. And from mlx5e_qid_from_qos() it seems like the inverse calculation is:

i - (chs->params.num_channels + is_ptp)*mlx5e_get_dcb_num_tc(&chs->params)

But really, isn't it enough to use priv->txq2sq[i] for the active queues, and not active ones you've already covered?

Quoted 7 lines

I am open to doing this, but I think if I were to do that, HTB/QOS queue stats should also be exported by rtnl so that the script above will continue to show that the output is correct.

I'd like to propose: adding HTB/QOS to both rtnl *and* the netdev-genl code together at the same time, but a later time, separate from this change.

Hm, are HTB queues really not counted in rtnl? That'd be pretty wrong.

author
#16 lore raw
Quoted 11 lines · Jakub Kicinski
Quoted 2 lines · On Wed, 29 May 2024 03:16:25 +0000 Joe Damato wrote:

Worth noting that Tariq suggested I also export HTB/QOS stats in mlx5e_get_base_stats.

Why to base, and not report them as queue stats?

Judging by mlx5e_update_tx_netdev_queues() calls sprinkled in ../mlx5/core/en/htb.c it seems that the driver will update the real_num_tx_queues accordingly. And from mlx5e_qid_from_qos() it seems like the inverse calculation is:

Quoted 4 lines

i - (chs->params.num_channels + is_ptp)*mlx5e_get_dcb_num_tc(&chs->params)

But really, isn't it enough to use priv->txq2sq[i] for the active queues, and not active ones you've already covered?

This is what I proposed in the thread for the v2, but Tariq suggested a different approach he liked more, please see this message for more details:

https://lore.kernel.org/netdev/[email protected]/

I attempted to implement option 1 as he described in his message.

Quoted 9 lines
Quoted 7 lines

I am open to doing this, but I think if I were to do that, HTB/QOS queue stats should also be exported by rtnl so that the script above will continue to show that the output is correct.

I'd like to propose: adding HTB/QOS to both rtnl *and* the netdev-genl code together at the same time, but a later time, separate from this change.

Hm, are HTB queues really not counted in rtnl? That'd be pretty wrong.

As far as I can tell (and I could be wrong) I didn't see them included in the rtnl stats, but I'll take another look now.

maintainer
#17 lore raw
Quoted 12 lines · On Thu, 30 May 2024 17:15:25 -0700 Joe Damato wrote:
Quoted 11 lines

Why to base, and not report them as queue stats?

Judging by mlx5e_update_tx_netdev_queues() calls sprinkled in ../mlx5/core/en/htb.c it seems that the driver will update the real_num_tx_queues accordingly. And from mlx5e_qid_from_qos() it seems like the inverse calculation is:

i - (chs->params.num_channels + is_ptp)*mlx5e_get_dcb_num_tc(&chs->params)

But really, isn't it enough to use priv->txq2sq[i] for the active queues, and not active ones you've already covered?

Quoted 7 lines

This is what I proposed in the thread for the v2, but Tariq suggested a different approach he liked more, please see this message for more details:

https://lore.kernel.org/netdev/[email protected]/

I attempted to implement option 1 as he described in his message.

I see, although it sounds like option 2 would also work.

Saeed can you shine any light here? I'm worried Tariq is already AFK for the weekend and we'll make no progress until Monday...

author
#18 lore raw
Quoted 17 lines · Jakub Kicinski
Quoted 16 lines · On Thu, 30 May 2024 17:15:25 -0700 Joe Damato wrote:
Quoted 11 lines

Why to base, and not report them as queue stats?

Judging by mlx5e_update_tx_netdev_queues() calls sprinkled in ../mlx5/core/en/htb.c it seems that the driver will update the real_num_tx_queues accordingly. And from mlx5e_qid_from_qos() it seems like the inverse calculation is:

i - (chs->params.num_channels + is_ptp)*mlx5e_get_dcb_num_tc(&chs->params)

But really, isn't it enough to use priv->txq2sq[i] for the active queues, and not active ones you've already covered?

This is what I proposed in the thread for the v2, but Tariq suggested a different approach he liked more, please see this message for more details:

Quoted 5 lines
Quoted 3 lines

https://lore.kernel.org/netdev/[email protected]/

I attempted to implement option 1 as he described in his message.

I see, although it sounds like option 2 would also work.

I don't really mind either way; from Tariq's message it sounded like he preferred option 1, so I tried to implement that thinking that it would be my best bet at getting this done.

If option 2 is easier/preferred for some reason... it seems like (other than the locking I forgot to include) the base implementation in v2 was correct and I could use what I proposed in the thread for the tx stats, which was:

  mutex_lock(&priv->state_lock);
  if (priv->channels.num > 0) {
          sq = priv->txq2sq[i];
          stats->packets = sq->stats->packets;
          stats->bytes = sq->stats->bytes;
  }
  mutex_unlock(&priv->state_lock);

And I would have implemented option 2... IIUC.

author
#19 lore raw
In reply to Joe Damato
Quoted 29 lines · Joe Damato
Quoted 11 lines · Jakub Kicinski
Quoted 2 lines · On Wed, 29 May 2024 03:16:25 +0000 Joe Damato wrote:

Worth noting that Tariq suggested I also export HTB/QOS stats in mlx5e_get_base_stats.

Why to base, and not report them as queue stats?

Judging by mlx5e_update_tx_netdev_queues() calls sprinkled in ../mlx5/core/en/htb.c it seems that the driver will update the real_num_tx_queues accordingly. And from mlx5e_qid_from_qos() it seems like the inverse calculation is:

Quoted 4 lines

i - (chs->params.num_channels + is_ptp)*mlx5e_get_dcb_num_tc(&chs->params)

But really, isn't it enough to use priv->txq2sq[i] for the active queues, and not active ones you've already covered?

This is what I proposed in the thread for the v2, but Tariq suggested a different approach he liked more, please see this message for more details:

https://lore.kernel.org/netdev/[email protected]/

I attempted to implement option 1 as he described in his message.

Quoted 4 lines
Quoted 4 lines

I am open to doing this, but I think if I were to do that, HTB/QOS queue stats should also be exported by rtnl so that the script above will continue to show that the output is correct.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I'd like to propose: adding HTB/QOS to both rtnl *and* the netdev-genl code together at the same time, but a later time, separate from this change.

Hm, are HTB queues really not counted in rtnl? That'd be pretty wrong.

As far as I can tell (and I could be wrong) I didn't see them included in the rtnl stats, but I'll take another look now.

I looked and it seems like the htb stats are included in ethtool's output if you look at mlx5/core/en_stats.c, it appears that mlx5e_stats_grp_sw_update_stats_qos rolls up the htb stats.

However, the rtnl stats, seem to be computed in mlx5/core/en_main.c via mlx5e_get_stats calling mlx5e_fold_sw_stats64, which appears to gather stats for rx, tx, and ptp, but it doesn't seem like qos/htb is handled.

Unless I am missing something, I think mlx5e_fold_sw_stats64 would need code similar to mlx5e_stats_grp_sw_update_stats_qos and then rtnl would account for htb stats.

That said: since it seems the htb numbers are not included right now, I was proposing adding that in later both to rtnl and netdev-genl together, hoping that would keep the proposed simpler/easier to get accepted.

LMK what you think.

maintainer
#20 lore raw
Quoted 3 lines · On Thu, 30 May 2024 18:07:31 -0700 Joe Damato wrote:

Unless I am missing something, I think mlx5e_fold_sw_stats64 would need code similar to mlx5e_stats_grp_sw_update_stats_qos and then rtnl would account for htb stats.

Hm, I think you're right. I'm just surprised this could have gone unnoticed for so long.

Quoted 4 lines

That said: since it seems the htb numbers are not included right now, I was proposing adding that in later both to rtnl and netdev-genl together, hoping that would keep the proposed simpler/easier to get accepted.

SGTM.

author
#21 lore raw
Quoted 8 lines · Jakub Kicinski
Quoted 3 lines · On Thu, 30 May 2024 18:07:31 -0700 Joe Damato wrote:

Unless I am missing something, I think mlx5e_fold_sw_stats64 would need code similar to mlx5e_stats_grp_sw_update_stats_qos and then rtnl would account for htb stats.

Hm, I think you're right. I'm just surprised this could have gone unnoticed for so long.

Quoted 6 lines
Quoted 4 lines

That said: since it seems the htb numbers are not included right now, I was proposing adding that in later both to rtnl and netdev-genl together, hoping that would keep the proposed simpler/easier to get accepted.

SGTM.

Cool, so based on that it seems like I just need to figure out the correct implementation for base and tx stats that is correct and that will be accepted.

Hoping to hear back from them soon as I'd personally love to have this API available on our systems.