Skip to contentSkip to conversation

devlink: Add port stats

RFC PATCH In review Michael Guralnik 3 participants 10 messages Apr 7, 2022 – Apr 12, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover devlink: Add port stats

This patch set adds port statistics to the devlink port object. It allows device drivers to dynamically attach and detach counters from a devlink port object.

The approach of adding object-attached statistics is already supported for trap with traffic statistics and for the dev object with reload statistics. For the port object, this will allow the device driver to expose and dynamicly control a set of metrics related to the port.
Currently we add support only for counters, but later API extensions can be made to support histograms or configurable counters.

The statistics are exposed to the user with the port get command.

Example:

# devlink -s port show
pci/0000:00:0b.0/65535: type eth netdev eth1 flavour physical port 0 splittable false
  stats:
    counter1 235
    counter2 18

Michael Guralnik (2):
devlink: Introduce stats to the port object
devlink: Add statistics to port get

 include/net/devlink.h        |  28 +++++++++
 include/uapi/linux/devlink.h |   4 ++
 net/core/devlink.c           | 119 +++++++++++++++++++++++++++++++++++
 3 files changed, 151 insertions(+)
Signature · 1 line
2.17.2
#1 lore raw

RFC PATCH 1/2 devlink: Introduce stats to the port object

Introducing a mechanism for drivers to register statistics counters to devlink ports.
Drivers to supply only name and a get method for the counter.

---
 include/net/devlink.h | 28 +++++++++++++++
 net/core/devlink.c    | 80 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 108 insertions(+)
include/net/devlink.h +28 -0
index 8d5349d2fb68..e9c3996ed359 100644
@@ -115,6 +115,11 @@ struct devlink_rate {
 	};
 };
 
+struct devlink_port_stats {
+	struct list_head stats_list;
+	int (*get)(struct devlink_port *port);
+};
+
 struct devlink_port {
 	struct list_head list;
 	struct list_head param_list;
@@ -135,6 +140,7 @@ struct devlink_port {
 	struct mutex reporters_lock; /* Protects reporter_list */
 
 	struct devlink_rate *devlink_rate;
+	struct devlink_port_stats stats;
 };
 
 struct devlink_port_new_attrs {
@@ -387,6 +393,16 @@ struct devlink_param_gset_ctx {
 	enum devlink_param_cmode cmode;
 };
 
+struct devlink_port_stat {
+	char *name;
+	u64 val;
+};
+
+struct devlink_port_stat_item {
+	struct list_head list;
+	const struct devlink_port_stat *stat;
+};
+
 /**
  * struct devlink_flash_notify - devlink dev flash notify data
  * @status_msg: current status string
@@ -1712,6 +1728,18 @@ devlink_trap_policers_unregister(struct devlink *devlink,
 				 const struct devlink_trap_policer *policers,
 				 size_t policers_count);
 
+void devlink_port_stat_unregister(struct devlink_port *port,
+				  const struct devlink_port_stat *stat);
+
+int devlink_port_stats_register(struct devlink_port *port,
+				const struct devlink_port_stat *stats,
+				size_t stats_count,
+				int (*get)(struct devlink_port *port));
+
+void devlink_port_stats_unregister(struct devlink_port *port,
+				   const struct devlink_port_stat *stats,
+				   size_t stats_count);
+
 #if IS_ENABLED(CONFIG_NET_DEVLINK)
 
 struct devlink *__must_check devlink_try_get(struct devlink *devlink);
net/core/devlink.c +80 -0

92 lines not on this page · raw message

Signature · 1 line
2.17.2
#2 lore raw

RFC PATCH 2/2 devlink: Add statistics to port get

In reply to Michael Guralnik

Add a list of all registered stats and their value to the port get command.

---
 include/uapi/linux/devlink.h |  4 ++++
 net/core/devlink.c           | 39 ++++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+)
include/uapi/linux/devlink.h +4 -0
index b897b80770f6..8da01e551695 100644
@@ -553,6 +553,10 @@ enum devlink_attr {
 
 	DEVLINK_ATTR_REGION_MAX_SNAPSHOTS,	/* u32 */
 
+	DEVLINK_ATTR_STATS_ENTRY,		/* nested */
+	DEVLINK_ATTR_STATS_NAME,		/* string */
+	DEVLINK_ATTR_STATS_VALUE,		/* u64 */
+
 	/* add new attributes above here, update the policy in devlink.c */
 
 	__DEVLINK_ATTR_MAX,
net/core/devlink.c +39 -0
index 9636d7998630..b735cca727a7 100644
@@ -1080,6 +1080,41 @@ devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *por
 	return err;
 }
 
+static int devlink_port_stats_get(struct devlink_port *port)
+{
+	if (!port->stats.get)
+		return -EOPNOTSUPP;
+	return port->stats.get(port);
+}
+
+static int devlink_nl_port_stats_put(struct sk_buff *msg, struct devlink_port *port)
+{
+	struct nlattr *stats_attr, *stats_entry_attr;
+	struct devlink_port_stat_item *stats_item;
+	int err;
+
+	stats_attr = nla_nest_start(msg, DEVLINK_ATTR_STATS);
+	if (!stats_attr)
+		return -EMSGSIZE;
+
+	err = devlink_port_stats_get(port);
+	if (err)
+		return err;
+
+	list_for_each_entry(stats_item, &port->stats.stats_list, list) {
+		stats_entry_attr = nla_nest_start(msg, DEVLINK_ATTR_STATS_ENTRY);
+		if (!stats_entry_attr)
+			return -EMSGSIZE;
+
+		nla_put_string(msg, DEVLINK_ATTR_STATS_NAME, stats_item->stat->name);
+		nla_put_u32(msg, DEVLINK_ATTR_STATS_VALUE, stats_item->stat->val);
+
+		nla_nest_end(msg, stats_entry_attr);
+	}
+	nla_nest_end(msg, stats_attr);
+	return 0;
+}
+
 static int devlink_nl_port_fill(struct sk_buff *msg,
 				struct devlink_port *devlink_port,
 				enum devlink_command cmd, u32 portid, u32 seq,
@@ -1132,6 +1167,8 @@ static int devlink_nl_port_fill(struct sk_buff *msg,
 	if (devlink_nl_port_function_attrs_put(msg, devlink_port, extack))
 		goto nla_put_failure;
 
+	if (devlink_nl_port_stats_put(msg, devlink_port))
+		goto nla_put_failure;
 	genlmsg_end(msg, hdr);
 	return 0;
 
@@ -8670,6 +8707,8 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = {
 	[DEVLINK_ATTR_RATE_TX_MAX] = { .type = NLA_U64 },
 	[DEVLINK_ATTR_RATE_NODE_NAME] = { .type = NLA_NUL_STRING },
 	[DEVLINK_ATTR_RATE_PARENT_NODE_NAME] = { .type = NLA_NUL_STRING },
+	[DEVLINK_ATTR_STATS_NAME] = {.type = NLA_NUL_STRING },
+	[DEVLINK_ATTR_STATS_VALUE] = {.type = NLA_U64 },
 };
 
 static const struct genl_small_ops devlink_nl_ops[] = {
Signature · 1 line
2.17.2
maintainer
#3 lore raw
In reply to Michael Guralnik
Quoted 3 lines · On Thu, 7 Apr 2022 11:40:48 +0300 Michael Guralnik wrote:

This patch set adds port statistics to the devlink port object. It allows device drivers to dynamically attach and detach counters from a devlink port object.

The challenge in defining APIs for stats is not in how to wrap a free form string in a netlink message but how do define values that have clear semantics and are of value to the user.

Start from that, discuss what you have with the customer who requested the feature. Then think about the API.

I have said this multiple times to multiple people on your team.

Quoted 2 lines

The approach of adding object-attached statistics is already supported for trap with traffic statistics and for the dev object with reload statistics.

That's an entirely false comparison.

Quoted 13 lines

For the port object, this will allow the device driver to expose and dynamicly control a set of metrics related to the port.
Currently we add support only for counters, but later API extensions can be made to support histograms or configurable counters.

The statistics are exposed to the user with the port get command.

Example:

# devlink -s port show
pci/0000:00:0b.0/65535: type eth netdev eth1 flavour physical port 0 splittable false
  stats:
    counter1 235
    counter2 18
maintainer
#4 lore raw
Quoted 12 lines · Fri, Apr 08, 2022 at 05:16:38AM CEST, [email protected] wrote:
Quoted 3 lines · On Thu, 7 Apr 2022 11:40:48 +0300 Michael Guralnik wrote:

This patch set adds port statistics to the devlink port object. It allows device drivers to dynamically attach and detach counters from a devlink port object.

The challenge in defining APIs for stats is not in how to wrap a free form string in a netlink message but how do define values that have clear semantics and are of value to the user.

Start from that, discuss what you have with the customer who requested the feature. Then think about the API.

Quoted 6 lines

I have said this multiple times to multiple people on your team.

Quoted 2 lines

The approach of adding object-attached statistics is already supported for trap with traffic statistics and for the dev object with reload statistics.

That's an entirely false comparison.

The trap stats are there already, why do you thing it is a "false comparison" to that?

They use DEVLINK_ATTR_STATS attribute to carry the stats nest and then:

        DEVLINK_ATTR_STATS_RX_PACKETS,          /* u64 */
        DEVLINK_ATTR_STATS_RX_BYTES,            /* u64 */
        DEVLINK_ATTR_STATS_RX_DROPPED,          /* u64 */

I think that the semantics of these are quite clear.

Quoted 14 lines
Quoted 14 lines

For the port object, this will allow the device driver to expose and dynamicly control a set of metrics related to the port.
Currently we add support only for counters, but later API extensions can be made to support histograms or configurable counters.

The statistics are exposed to the user with the port get command.

Example:

# devlink -s port show
pci/0000:00:0b.0/65535: type eth netdev eth1 flavour physical port 0 splittable false
  stats:
    counter1 235
    counter2 18
maintainer
#5 lore raw
Quoted 5 lines · On Mon, 11 Apr 2022 12:28:40 +0200 Jiri Pirko wrote:
Quoted 4 lines
Quoted 2 lines

The approach of adding object-attached statistics is already supported for trap with traffic statistics and for the dev object with reload statistics.

That's an entirely false comparison.

Quoted 8 lines

The trap stats are there already, why do you thing it is a "false comparison" to that?

They use DEVLINK_ATTR_STATS attribute to carry the stats nest and then:

        DEVLINK_ATTR_STATS_RX_PACKETS,          /* u64 */
        DEVLINK_ATTR_STATS_RX_BYTES,            /* u64 */
        DEVLINK_ATTR_STATS_RX_DROPPED,          /* u64 */

I think that the semantics of these are quite clear.

Exactly, (1) the semantics of statistics on traps and health are clearly dictated by the object, there is no ambiguity what rx packets for a trap means; (2) the statistics are enumerated by the core...

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

For the port object, this will allow the device driver to expose and dynamicly control a set of metrics related to the port.

..this is like saying "it was 14C in Prague and San Francisco today, so the weather was the same. It was sunny in Prague and raining in SF".

maintainer
#6 lore raw
In reply to Jakub Kicinski
Quoted 8 lines · Fri, Apr 08, 2022 at 05:16:38AM CEST, [email protected] wrote:
Quoted 3 lines · On Thu, 7 Apr 2022 11:40:48 +0300 Michael Guralnik wrote:

This patch set adds port statistics to the devlink port object. It allows device drivers to dynamically attach and detach counters from a devlink port object.

The challenge in defining APIs for stats is not in how to wrap a free form string in a netlink message but how do define values that have clear semantics and are of value to the user.

Wait, does all stats have to be well-defined? I mean, look at the ethtool stats. They are free-form strings too. Do you mean that in devlink, we can only have well-defines enum-based stats?

Quoted 24 lines

Start from that, discuss what you have with the customer who requested the feature. Then think about the API.

I have said this multiple times to multiple people on your team.

Quoted 2 lines

The approach of adding object-attached statistics is already supported for trap with traffic statistics and for the dev object with reload statistics.

That's an entirely false comparison.

Quoted 13 lines

For the port object, this will allow the device driver to expose and dynamicly control a set of metrics related to the port.
Currently we add support only for counters, but later API extensions can be made to support histograms or configurable counters.

The statistics are exposed to the user with the port get command.

Example:

# devlink -s port show
pci/0000:00:0b.0/65535: type eth netdev eth1 flavour physical port 0 splittable false
  stats:
    counter1 235
    counter2 18
maintainer
#7 lore raw
Quoted 6 lines · On Mon, 11 Apr 2022 12:31:53 +0200 Jiri Pirko wrote:
Quoted 5 lines · Fri, Apr 08, 2022 at 05:16:38AM CEST, [email protected] wrote:
Quoted 3 lines · On Thu, 7 Apr 2022 11:40:48 +0300 Michael Guralnik wrote:

This patch set adds port statistics to the devlink port object. It allows device drivers to dynamically attach and detach counters from a devlink port object.

Quoted 7 lines
Quoted 3 lines

The challenge in defining APIs for stats is not in how to wrap a free form string in a netlink message but how do define values that have clear semantics and are of value to the user.

Wait, does all stats have to be well-defined? I mean, look at the ethtool stats. They are free-form strings too. Do you mean that in devlink, we can only have well-defines enum-based stats?

That's my strong preference, yes.

First, and obvious argument is that it make lazy coding less likely (see devlink params).

More importantly, tho, if your stats are not well defined - users don't need to seem them. Really! If I can't draw a line between a statistic and device behavior then keep that stat in the register dump, debugfs or /dev/null.

That's why it's important that we talk about _what_ you're trying to expose.

maintainer
#8 lore raw
Quoted 15 lines · Mon, Apr 11, 2022 at 08:01:57PM CEST, [email protected] wrote:
Quoted 6 lines · On Mon, 11 Apr 2022 12:31:53 +0200 Jiri Pirko wrote:
Quoted 5 lines · Fri, Apr 08, 2022 at 05:16:38AM CEST, [email protected] wrote:
Quoted 3 lines · On Thu, 7 Apr 2022 11:40:48 +0300 Michael Guralnik wrote:

This patch set adds port statistics to the devlink port object. It allows device drivers to dynamically attach and detach counters from a devlink port object.

Quoted 7 lines
Quoted 3 lines

The challenge in defining APIs for stats is not in how to wrap a free form string in a netlink message but how do define values that have clear semantics and are of value to the user.

Wait, does all stats have to be well-defined? I mean, look at the ethtool stats. They are free-form strings too. Do you mean that in devlink, we can only have well-defines enum-based stats?

Quoted 8 lines

That's my strong preference, yes.

First, and obvious argument is that it make lazy coding less likely (see devlink params).

More importantly, tho, if your stats are not well defined - users don't need to seem them. Really! If I can't draw a line between a statistic and device behavior then keep that stat in the register dump, debugfs

During the DaveM's-only era, there was quite strict policy against any debugfs usage. As far as I remember the claim was, find of define the proper api or do your debug things out-of-tree.

Does that changed? I just want to make sure that we are now free to use debugfs for exposuse of debugging info as "odd vendor stats". Personally, I think it is good idea. I think that the rest of the kernel actually uses debugfs like that.

Thanks!

Quoted 4 lines

or /dev/null.

That's why it's important that we talk about _what_ you're trying to expose.

Basically a mixture of quite generic things and very obscure device specific items.

maintainer
#9 lore raw
Quoted 14 lines · On Tue, 12 Apr 2022 10:16:26 +0200 Jiri Pirko wrote:
Quoted 4 lines · Mon, Apr 11, 2022 at 08:01:57PM CEST, [email protected] wrote:
Quoted 3 lines

Wait, does all stats have to be well-defined? I mean, look at the ethtool stats. They are free-form strings too. Do you mean that in devlink, we can only have well-defines enum-based stats?

Quoted 8 lines

That's my strong preference, yes.

First, and obvious argument is that it make lazy coding less likely (see devlink params).

More importantly, tho, if your stats are not well defined - users don't need to seem them. Really! If I can't draw a line between a statistic and device behavior then keep that stat in the register dump, debugfs

Quoted 8 lines

During the DaveM's-only era, there was quite strict policy against any debugfs usage. As far as I remember the claim was, find of define the proper api or do your debug things out-of-tree.

Does that changed? I just want to make sure that we are now free to use debugfs for exposuse of debugging info as "odd vendor stats". Personally, I think it is good idea. I think that the rest of the kernel actually uses debugfs like that.

I think the policy is "it's fine as long as it's read-only". Which could be a problem if you want to parametrize the counters. But then again based on this RFC IDK if you do.

Quoted 7 lines
Quoted 4 lines

or /dev/null.

That's why it's important that we talk about _what_ you're trying to expose.

Basically a mixture of quite generic things and very obscure device specific items.