Skip to contentSkip to conversation

Configuring congestion watermarks on ocelot switch using devlink-sb

PATCH Superseded Vladimir Oltean 4 participants 36 messages Jan 8, 2021 – Jan 15, 2021

Subsystems (4)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Vladimir Oltean

author maintainer
#0 lore raw

v3 cover Configuring congestion watermarks on ocelot switch using devlink-sb

From: Vladimir Oltean <[email protected]>

In some applications, it is important to create resource reservations in the Ethernet switches, to prevent background traffic, or deliberate attacks, from inducing denial of service into the high-priority traffic.

These patches give the user some knobs to turn. The ocelot switches support per-port and per-port-tc reservations, on ingress and on egress. The resources that are monitored are packet buffers (in cells of 60 bytes each) and frame references.

The frames that exceed the reservations can optionally consume from sharing watermarks which are not per-port but global across the switch. There are 10 sharing watermarks, 8 of them are per traffic class and 2 are per drop priority.

I am configuring the hardware using the best of my knowledge, and mostly through trial and error. Same goes for devlink-sb integration. Feedback is welcome.

Vladimir Oltean (10):
net: mscc: ocelot: auto-detect packet buffer size and number of frame
references
net: mscc: ocelot: add ops for decoding watermark threshold and
occupancy
net: dsa: add ops for devlink-sb
net: dsa: felix: reindent struct dsa_switch_ops
net: dsa: felix: perform teardown in reverse order of setup
net: mscc: ocelot: export NUM_TC constant from felix to common switch
lib
net: mscc: ocelot: delete unused ocelot_set_cpu_port prototype
net: mscc: ocelot: register devlink ports
net: mscc: ocelot: initialize watermarks to sane defaults
net: mscc: ocelot: configure watermarks using devlink-sb

 drivers/net/dsa/ocelot/felix.c             | 210 +++--
 drivers/net/dsa/ocelot/felix.h             |   2 -
 drivers/net/dsa/ocelot/felix_vsc9959.c     |  23 +-
 drivers/net/dsa/ocelot/seville_vsc9953.c   |  20 +-
 drivers/net/ethernet/mscc/Makefile         |   3 +-
 drivers/net/ethernet/mscc/ocelot.c         |  18 +-
 drivers/net/ethernet/mscc/ocelot.h         |   8 +-
 drivers/net/ethernet/mscc/ocelot_devlink.c | 885 +++++++++++++++++++++
 drivers/net/ethernet/mscc/ocelot_net.c     | 282 ++++++-
 drivers/net/ethernet/mscc/ocelot_vsc7514.c |  47 +-
 include/net/dsa.h                          |  34 +
 include/soc/mscc/ocelot.h                  |  54 +-
 include/soc/mscc/ocelot_qsys.h             |   7 +-
 net/dsa/dsa2.c                             | 159 +++-
 14 files changed, 1657 insertions(+), 95 deletions(-)
 create mode 100644 drivers/net/ethernet/mscc/ocelot_devlink.c
Signature · 1 line
2.25.1
author maintainer
#1 lore raw

PATCH v3 1/10 net: mscc: ocelot: auto-detect packet buffer size and number of frame references

From: Vladimir Oltean <[email protected]>

Instead of reading these values from the reference manual and writing them down into the driver, it appears that the hardware gives us the option of detecting them dynamically.

The number of frame references corresponds to what the reference manual notes, however it seems that the frame buffers are reported as slightly less than the books would indicate. On VSC9959 (Felix), the books say it should have 128KB of packet buffer, but the registers indicate only 129840 bytes (126.79 KB). Also, the unit of measurement for FREECNT from the documentation of all these devices is incorrect (taken from an older generation). This was confirmed by Younes Leroul from Microchip support.

Not having anything better to do with these values at the moment* (this will change soon), let's just print them.

*The frame buffer size is, in fact, used to calculate the tail dropping watermarks.

---
Changes in v3:
None.

Changes in v2:
- Fixed FREECNT multiplier after consulting with Microchip support.

 drivers/net/dsa/ocelot/felix.c             |  1 -
 drivers/net/dsa/ocelot/felix.h             |  1 -
 drivers/net/dsa/ocelot/felix_vsc9959.c     |  1 -
 drivers/net/dsa/ocelot/seville_vsc9953.c   |  1 -
 drivers/net/ethernet/mscc/ocelot.c         | 22 +++++++++++++++++++++-
 drivers/net/ethernet/mscc/ocelot_vsc7514.c |  1 -
 include/soc/mscc/ocelot.h                  |  3 ++-
 include/soc/mscc/ocelot_qsys.h             |  3 +++
 8 files changed, 26 insertions(+), 7 deletions(-)
drivers/net/dsa/ocelot/felix.c +0 -1
index 90c3c76f21b2..a2e06a0d1509 100644
@@ -451,7 +451,6 @@ static int felix_init_structs(struct felix *felix, int num_phys_ports)
 	ocelot->map		= felix->info->map;
 	ocelot->stats_layout	= felix->info->stats_layout;
 	ocelot->num_stats	= felix->info->num_stats;
-	ocelot->shared_queue_sz	= felix->info->shared_queue_sz;
 	ocelot->num_mact_rows	= felix->info->num_mact_rows;
 	ocelot->vcap		= felix->info->vcap;
 	ocelot->ops		= felix->info->ops;
drivers/net/dsa/ocelot/felix.h +0 -1
index 4c717324ac2f..5434fe278d2c 100644
@@ -15,7 +15,6 @@ struct felix_info {
 	const struct reg_field		*regfields;
 	const u32 *const		*map;
 	const struct ocelot_ops		*ops;
-	int				shared_queue_sz;
 	int				num_mact_rows;
 	const struct ocelot_stat_layout	*stats_layout;
 	unsigned int			num_stats;
drivers/net/dsa/ocelot/felix_vsc9959.c +0 -1
index 2e5bbdca5ea4..9fffbad6ef9b 100644
@@ -1356,7 +1356,6 @@ static const struct felix_info felix_info_vsc9959 = {
 	.stats_layout		= vsc9959_stats_layout,
 	.num_stats		= ARRAY_SIZE(vsc9959_stats_layout),
 	.vcap			= vsc9959_vcap_props,
-	.shared_queue_sz	= 128 * 1024,
 	.num_mact_rows		= 2048,
 	.num_ports		= 6,
 	.num_tx_queues		= FELIX_NUM_TC,
drivers/net/dsa/ocelot/seville_vsc9953.c +0 -1
index ebbaf6817ec8..b72813da6d9f 100644
@@ -1181,7 +1181,6 @@ static const struct felix_info seville_info_vsc9953 = {
 	.stats_layout		= vsc9953_stats_layout,
 	.num_stats		= ARRAY_SIZE(vsc9953_stats_layout),
 	.vcap			= vsc9953_vcap_props,
-	.shared_queue_sz	= 256 * 1024,
 	.num_mact_rows		= 2048,
 	.num_ports		= 10,
 	.mdio_bus_alloc		= vsc9953_mdio_bus_alloc,
drivers/net/ethernet/mscc/ocelot.c +21 -1
index 0b9992bd6626..876c03e51bdc 100644
@@ -1366,7 +1366,7 @@ void ocelot_port_set_maxlen(struct ocelot *ocelot, int port, size_t sdu)
 			    pause_stop);
 
 	/* Tail dropping watermarks */
-	atop_tot = (ocelot->shared_queue_sz - 9 * maxlen) /
+	atop_tot = (ocelot->packet_buffer_size - 9 * maxlen) /
 		   OCELOT_BUFFER_CELL_SZ;
 	atop = (9 * maxlen) / OCELOT_BUFFER_CELL_SZ;
 	ocelot_write_rix(ocelot, ocelot->ops->wm_enc(atop), SYS_ATOP, port);
@@ -1479,6 +1479,25 @@ static void ocelot_cpu_port_init(struct ocelot *ocelot)
 			 ANA_PORT_VLAN_CFG, cpu);
 }
 
+static void ocelot_detect_features(struct ocelot *ocelot)
+{
+	int mmgt, eq_ctrl;
+
+	/* For Ocelot, Felix, Seville, Serval etc, SYS:MMGT:MMGT:FREECNT holds
+	 * the number of 240-byte free memory words (aka 4-cell chunks) and not
+	 * 192 bytes as the documentation incorrectly says.
+	 */
+	mmgt = ocelot_read(ocelot, SYS_MMGT);
+	ocelot->packet_buffer_size = 240 * SYS_MMGT_FREECNT(mmgt);
+
+	eq_ctrl = ocelot_read(ocelot, QSYS_EQ_CTRL);
+	ocelot->num_frame_refs = QSYS_MMGT_EQ_CTRL_FP_FREE_CNT(eq_ctrl);
+
+	dev_info(ocelot->dev,
+		 "Detected %d bytes of packet buffer and %d frame references\n",
+		 ocelot->packet_buffer_size, ocelot->num_frame_refs);
+}
+
 int ocelot_init(struct ocelot *ocelot)
 {
 	char queue_name[32];
@@ -1521,6 +1540,7 @@ int ocelot_init(struct ocelot *ocelot)
 
 	INIT_LIST_HEAD(&ocelot->multicast);
 	INIT_LIST_HEAD(&ocelot->pgids);
+	ocelot_detect_features(ocelot);
 	ocelot_mact_init(ocelot);
 	ocelot_vlan_init(ocelot);
 	ocelot_vcap_init(ocelot);
drivers/net/ethernet/mscc/ocelot_vsc7514.c +0 -1
index 9cf2bc5f4289..7135ad18affe 100644
@@ -517,7 +517,6 @@ static int ocelot_chip_init(struct ocelot *ocelot, const struct ocelot_ops *ops)
 	ocelot->map = ocelot_regmap;
 	ocelot->stats_layout = ocelot_stats_layout;
 	ocelot->num_stats = ARRAY_SIZE(ocelot_stats_layout);
-	ocelot->shared_queue_sz = 224 * 1024;
 	ocelot->num_mact_rows = 1024;
 	ocelot->ops = ops;
 
include/soc/mscc/ocelot.h +2 -1
index 2f4cd3288bcc..c6c131142195 100644
@@ -607,7 +607,8 @@ struct ocelot {
 	const struct ocelot_stat_layout	*stats_layout;
 	unsigned int			num_stats;
 
-	int				shared_queue_sz;
+	int				packet_buffer_size;
+	int				num_frame_refs;
 	int				num_mact_rows;
 
 	struct net_device		*hw_bridge_dev;
include/soc/mscc/ocelot_qsys.h +3 -0
index a814bc2017d8..b7b263a19068 100644
@@ -77,6 +77,9 @@
 #define QSYS_RES_STAT_MAXUSE(x)                           ((x) & GENMASK(11, 0))
 #define QSYS_RES_STAT_MAXUSE_M                            GENMASK(11, 0)
 
+#define QSYS_MMGT_EQ_CTRL_FP_FREE_CNT(x)                  ((x) & GENMASK(15, 0))
+#define QSYS_MMGT_EQ_CTRL_FP_FREE_CNT_M                   GENMASK(15, 0)
+
 #define QSYS_EVENTS_CORE_EV_FDC(x)                        (((x) << 2) & GENMASK(4, 2))
 #define QSYS_EVENTS_CORE_EV_FDC_M                         GENMASK(4, 2)
 #define QSYS_EVENTS_CORE_EV_FDC_X(x)                      (((x) & GENMASK(4, 2)) >> 2)
Signature · 1 line
2.25.1
Reviewed-by on v3 1/10
#2 lore raw

Re: [PATCH v3 net-next 01/10] net: mscc: ocelot: auto-detect packet buffer size and number of frame references

Quoted 14 lines · Vladimir Oltean

From: Vladimir Oltean <[email protected]>

Instead of reading these values from the reference manual and writing them down into the driver, it appears that the hardware gives us the option of detecting them dynamically.

The number of frame references corresponds to what the reference manual notes, however it seems that the frame buffers are reported as slightly less than the books would indicate. On VSC9959 (Felix), the books say it should have 128KB of packet buffer, but the registers indicate only 129840 bytes (126.79 KB). Also, the unit of measurement for FREECNT from the documentation of all these devices is incorrect (taken from an older generation). This was confirmed by Younes Leroul from Microchip support.

Quoted 7 lines

Not having anything better to do with these values at the moment* (this will change soon), let's just print them.

*The frame buffer size is, in fact, used to calculate the tail dropping watermarks.

Signature · 1 line
Florian
author maintainer
#3 lore raw

PATCH v3 2/10 net: mscc: ocelot: add ops for decoding watermark threshold and occupancy

In reply to Vladimir Oltean

From: Vladimir Oltean <[email protected]>

We'll need to read back the watermark thresholds and occupancy from hardware (for devlink-sb integration), not only to write them as we did so far in ocelot_port_set_maxlen. So introduce 2 new functions in struct ocelot_ops, similar to wm_enc, and implement them for the 3 supported mscc_ocelot switches.

Remove the INUSE and MAXUSE unpacking helpers for the QSYS_RES_STAT register, because that doesn't scale with the number of switches that mscc_ocelot supports now. They have different bit widths for the watermarks, and we need function pointers to abstract that difference away.

---
Changes in v3:
None.

Changes in v2:
None.

 drivers/net/dsa/ocelot/felix_vsc9959.c     | 18 ++++++++++++++++++
 drivers/net/dsa/ocelot/seville_vsc9953.c   | 18 ++++++++++++++++++
 drivers/net/ethernet/mscc/ocelot_vsc7514.c | 16 ++++++++++++++++
 include/soc/mscc/ocelot.h                  |  2 ++
 include/soc/mscc/ocelot_qsys.h             |  6 ------
 5 files changed, 54 insertions(+), 6 deletions(-)
drivers/net/dsa/ocelot/felix_vsc9959.c +18 -0
index 9fffbad6ef9b..540b86edbbb0 100644
@@ -1006,9 +1006,27 @@ static u16 vsc9959_wm_enc(u16 value)
 	return value;
 }
 
+static u16 vsc9959_wm_dec(u16 wm)
+{
+	WARN_ON(wm & ~GENMASK(8, 0));
+
+	if (wm & BIT(8))
+		return (wm & GENMASK(7, 0)) * 16;
+
+	return wm;
+}
+
+static void vsc9959_wm_stat(u32 val, u32 *inuse, u32 *maxuse)
+{
+	*inuse = (val & GENMASK(23, 12)) >> 12;
+	*maxuse = val & GENMASK(11, 0);
+}
+
 static const struct ocelot_ops vsc9959_ops = {
 	.reset			= vsc9959_reset,
 	.wm_enc			= vsc9959_wm_enc,
+	.wm_dec			= vsc9959_wm_dec,
+	.wm_stat		= vsc9959_wm_stat,
 	.port_to_netdev		= felix_port_to_netdev,
 	.netdev_to_port		= felix_netdev_to_port,
 };
drivers/net/dsa/ocelot/seville_vsc9953.c +18 -0
index b72813da6d9f..8dad0c894eca 100644
@@ -1057,9 +1057,27 @@ static u16 vsc9953_wm_enc(u16 value)
 	return value;
 }
 
+static u16 vsc9953_wm_dec(u16 wm)
+{
+	WARN_ON(wm & ~GENMASK(9, 0));
+
+	if (wm & BIT(9))
+		return (wm & GENMASK(8, 0)) * 16;
+
+	return wm;
+}
+
+static void vsc9953_wm_stat(u32 val, u32 *inuse, u32 *maxuse)
+{
+	*inuse = (val & GENMASK(25, 13)) >> 13;
+	*maxuse = val & GENMASK(12, 0);
+}
+
 static const struct ocelot_ops vsc9953_ops = {
 	.reset			= vsc9953_reset,
 	.wm_enc			= vsc9953_wm_enc,
+	.wm_dec			= vsc9953_wm_dec,
+	.wm_stat		= vsc9953_wm_stat,
 	.port_to_netdev		= felix_port_to_netdev,
 	.netdev_to_port		= felix_netdev_to_port,
 };
drivers/net/ethernet/mscc/ocelot_vsc7514.c +16 -0
index 7135ad18affe..ecd474476cc6 100644
@@ -763,9 +763,25 @@ static u16 ocelot_wm_enc(u16 value)
 	return value;
 }
 
+static u16 ocelot_wm_dec(u16 wm)
+{
+	if (wm & BIT(8))
+		return (wm & GENMASK(7, 0)) * 16;
+
+	return wm;
+}
+
+static void ocelot_wm_stat(u32 val, u32 *inuse, u32 *maxuse)
+{
+	*inuse = (val & GENMASK(23, 12)) >> 12;
+	*maxuse = val & GENMASK(11, 0);
+}
+
 static const struct ocelot_ops ocelot_ops = {
 	.reset			= ocelot_reset,
 	.wm_enc			= ocelot_wm_enc,
+	.wm_dec			= ocelot_wm_dec,
+	.wm_stat		= ocelot_wm_stat,
 	.port_to_netdev		= ocelot_port_to_netdev,
 	.netdev_to_port		= ocelot_netdev_to_port,
 };
include/soc/mscc/ocelot.h +2 -0
index c6c131142195..8eb134cd8d9d 100644
@@ -563,6 +563,8 @@ struct ocelot_ops {
 	int (*netdev_to_port)(struct net_device *dev);
 	int (*reset)(struct ocelot *ocelot);
 	u16 (*wm_enc)(u16 value);
+	u16 (*wm_dec)(u16 value);
+	void (*wm_stat)(u32 val, u32 *inuse, u32 *maxuse);
 };
 
 struct ocelot_vcap_block {
include/soc/mscc/ocelot_qsys.h +0 -6
index b7b263a19068..9731895be643 100644
@@ -71,12 +71,6 @@
 
 #define QSYS_RES_STAT_GSZ                                 0x8
 
-#define QSYS_RES_STAT_INUSE(x)                            (((x) << 12) & GENMASK(23, 12))
-#define QSYS_RES_STAT_INUSE_M                             GENMASK(23, 12)
-#define QSYS_RES_STAT_INUSE_X(x)                          (((x) & GENMASK(23, 12)) >> 12)
-#define QSYS_RES_STAT_MAXUSE(x)                           ((x) & GENMASK(11, 0))
-#define QSYS_RES_STAT_MAXUSE_M                            GENMASK(11, 0)
-
 #define QSYS_MMGT_EQ_CTRL_FP_FREE_CNT(x)                  ((x) & GENMASK(15, 0))
 #define QSYS_MMGT_EQ_CTRL_FP_FREE_CNT_M                   GENMASK(15, 0)
 
Signature · 1 line
2.25.1
Reviewed-by on v3 2/10
#4 lore raw

Re: [PATCH v3 net-next 02/10] net: mscc: ocelot: add ops for decoding watermark threshold and occupancy

Quoted 8 lines · Vladimir Oltean

From: Vladimir Oltean <[email protected]>

We'll need to read back the watermark thresholds and occupancy from hardware (for devlink-sb integration), not only to write them as we did so far in ocelot_port_set_maxlen. So introduce 2 new functions in struct ocelot_ops, similar to wm_enc, and implement them for the 3 supported mscc_ocelot switches.

Quoted 7 lines

Remove the INUSE and MAXUSE unpacking helpers for the QSYS_RES_STAT register, because that doesn't scale with the number of switches that mscc_ocelot supports now. They have different bit widths for the watermarks, and we need function pointers to abstract that difference away.

Signature · 1 line
Florian
maintainer
#5 lore raw

Re: [PATCH v3 net-next 02/10] net: mscc: ocelot: add ops for decoding watermark threshold and occupancy

In reply to Vladimir Oltean

Review

Quoted 1 line · On Fri, 8 Jan 2021 19:59:42 +0200 Vladimir Oltean wrote:
(quoted diff) +1 -0
+	*inuse = (val & GENMASK(23, 12)) >> 12;

FWIW FIELD_GET()

author maintainer
#6 lore raw

Re: [PATCH v3 net-next 02/10] net: mscc: ocelot: add ops for decoding watermark threshold and occupancy

Quoted 4 lines · Jakub Kicinski

Review

Quoted 1 line · On Fri, 8 Jan 2021 19:59:42 +0200 Vladimir Oltean wrote:
(quoted diff) +1 -0
+	*inuse = (val & GENMASK(23, 12)) >> 12;

FWIW FIELD_GET()

Do you mind if I don't use it? I don't feel that:

	*inuse = FIELD_GET(GENMASK(23, 12), val);

looks any better.

maintainer
#7 lore raw

Re: [PATCH v3 net-next 02/10] net: mscc: ocelot: add ops for decoding watermark threshold and occupancy

Quoted 9 lines · On Mon, 11 Jan 2021 18:53:21 +0200 Vladimir Oltean wrote:
Quoted 4 lines · Jakub Kicinski

Review

Quoted 1 line · On Fri, 8 Jan 2021 19:59:42 +0200 Vladimir Oltean wrote:
(quoted diff) +1 -0
+	*inuse = (val & GENMASK(23, 12)) >> 12;  

FWIW FIELD_GET()

Do you mind if I don't use it? I don't feel that:

	*inuse = FIELD_GET(GENMASK(23, 12), val);

looks any better.

Not at all, your call.

author maintainer
#8 lore raw

PATCH v3 3/10 net: dsa: add ops for devlink-sb

In reply to Vladimir Oltean

From: Vladimir Oltean <[email protected]>

Switches that care about QoS might have hardware support for reserving buffer pools for individual ports or traffic classes, and configuring their sizes and thresholds. Through devlink-sb (shared buffers), this is all configurable, as well as their occupancy being viewable.

Add the plumbing in DSA for these operations.

Individual drivers still need to call devlink_sb_register() with the shared buffers they want to expose. A helper was not created in DSA for this purpose (unlike, say, dsa_devlink_params_register), since in my opinion it does not bring any benefit over plainly calling devlink_sb_register() directly.

---
Changes in v3:
Use the helpers added by Andrew to convert from devlink and devlink_port to DSA structures rather than open-coding them.

Changes in v2:
None.

 include/net/dsa.h |  34 ++++++++++
 net/dsa/dsa2.c    | 159 +++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 192 insertions(+), 1 deletion(-)
include/net/dsa.h +34 -0
index 9a0cab5fb7c4..b64fd53d7fce 100644
@@ -650,6 +650,40 @@ struct dsa_switch_ops {
 	int	(*devlink_info_get)(struct dsa_switch *ds,
 				    struct devlink_info_req *req,
 				    struct netlink_ext_ack *extack);
+	int	(*devlink_sb_pool_get)(struct dsa_switch *ds,
+				       unsigned int sb_index, u16 pool_index,
+				       struct devlink_sb_pool_info *pool_info);
+	int	(*devlink_sb_pool_set)(struct dsa_switch *ds, unsigned int sb_index,
+				       u16 pool_index, u32 size,
+				       enum devlink_sb_threshold_type threshold_type,
+				       struct netlink_ext_ack *extack);
+	int	(*devlink_sb_port_pool_get)(struct dsa_switch *ds, int port,
+					    unsigned int sb_index, u16 pool_index,
+					    u32 *p_threshold);
+	int	(*devlink_sb_port_pool_set)(struct dsa_switch *ds, int port,
+					    unsigned int sb_index, u16 pool_index,
+					    u32 threshold,
+					    struct netlink_ext_ack *extack);
+	int	(*devlink_sb_tc_pool_bind_get)(struct dsa_switch *ds, int port,
+					       unsigned int sb_index, u16 tc_index,
+					       enum devlink_sb_pool_type pool_type,
+					       u16 *p_pool_index, u32 *p_threshold);
+	int	(*devlink_sb_tc_pool_bind_set)(struct dsa_switch *ds, int port,
+					       unsigned int sb_index, u16 tc_index,
+					       enum devlink_sb_pool_type pool_type,
+					       u16 pool_index, u32 threshold,
+					       struct netlink_ext_ack *extack);
+	int	(*devlink_sb_occ_snapshot)(struct dsa_switch *ds,
+					   unsigned int sb_index);
+	int	(*devlink_sb_occ_max_clear)(struct dsa_switch *ds,
+					    unsigned int sb_index);
+	int	(*devlink_sb_occ_port_pool_get)(struct dsa_switch *ds, int port,
+						unsigned int sb_index, u16 pool_index,
+						u32 *p_cur, u32 *p_max);
+	int	(*devlink_sb_occ_tc_port_bind_get)(struct dsa_switch *ds, int port,
+						   unsigned int sb_index, u16 tc_index,
+						   enum devlink_sb_pool_type pool_type,
+						   u32 *p_cur, u32 *p_max);
 
 	/*
 	 * MTU change functionality. Switches can also adjust their MRU through
net/dsa/dsa2.c +158 -1

166 lines not on this page · raw message

Signature · 1 line
2.25.1
maintainer Reviewed-by on v3 3/10
#9 lore raw

Re: [PATCH v3 net-next 03/10] net: dsa: add ops for devlink-sb

Quoted 9 lines · Vladimir Oltean

From: Vladimir Oltean <[email protected]>

Switches that care about QoS might have hardware support for reserving buffer pools for individual ports or traffic classes, and configuring their sizes and thresholds. Through devlink-sb (shared buffers), this is all configurable, as well as their occupancy being viewable.

Add the plumbing in DSA for these operations.

Quoted 7 lines

Individual drivers still need to call devlink_sb_register() with the shared buffers they want to expose. A helper was not created in DSA for this purpose (unlike, say, dsa_devlink_params_register), since in my opinion it does not bring any benefit over plainly calling devlink_sb_register() directly.

Andrew

Reviewed-by on v3 3/10
#10 lore raw

Re: [PATCH v3 net-next 03/10] net: dsa: add ops for devlink-sb

In reply to Vladimir Oltean
Quoted 9 lines · Vladimir Oltean

From: Vladimir Oltean <[email protected]>

Switches that care about QoS might have hardware support for reserving buffer pools for individual ports or traffic classes, and configuring their sizes and thresholds. Through devlink-sb (shared buffers), this is all configurable, as well as their occupancy being viewable.

Add the plumbing in DSA for these operations.

Quoted 7 lines

Individual drivers still need to call devlink_sb_register() with the shared buffers they want to expose. A helper was not created in DSA for this purpose (unlike, say, dsa_devlink_params_register), since in my opinion it does not bring any benefit over plainly calling devlink_sb_register() directly.

Signature · 1 line
Florian
author maintainer
#11 lore raw

PATCH v3 4/10 net: dsa: felix: reindent struct dsa_switch_ops

In reply to Vladimir Oltean

From: Vladimir Oltean <[email protected]>

The devlink function pointer names are super long, and they would break the alignment. So reindent the existing ops now by adding one tab.

---
Changes in v3:
None.

Changes in v2:
None.

 drivers/net/dsa/ocelot/felix.c | 78 +++++++++++++++++-----------------
 1 file changed, 39 insertions(+), 39 deletions(-)
drivers/net/dsa/ocelot/felix.c +39 -39
index a2e06a0d1509..79d0508e5910 100644
@@ -781,45 +781,45 @@ static int felix_port_setup_tc(struct dsa_switch *ds, int port,
 }
 
 const struct dsa_switch_ops felix_switch_ops = {
-	.get_tag_protocol	= felix_get_tag_protocol,
-	.setup			= felix_setup,
-	.teardown		= felix_teardown,
-	.set_ageing_time	= felix_set_ageing_time,
-	.get_strings		= felix_get_strings,
-	.get_ethtool_stats	= felix_get_ethtool_stats,
-	.get_sset_count		= felix_get_sset_count,
-	.get_ts_info		= felix_get_ts_info,
-	.phylink_validate	= felix_phylink_validate,
-	.phylink_mac_config	= felix_phylink_mac_config,
-	.phylink_mac_link_down	= felix_phylink_mac_link_down,
-	.phylink_mac_link_up	= felix_phylink_mac_link_up,
-	.port_enable		= felix_port_enable,
-	.port_disable		= felix_port_disable,
-	.port_fdb_dump		= felix_fdb_dump,
-	.port_fdb_add		= felix_fdb_add,
-	.port_fdb_del		= felix_fdb_del,
-	.port_mdb_prepare	= felix_mdb_prepare,
-	.port_mdb_add		= felix_mdb_add,
-	.port_mdb_del		= felix_mdb_del,
-	.port_bridge_join	= felix_bridge_join,
-	.port_bridge_leave	= felix_bridge_leave,
-	.port_stp_state_set	= felix_bridge_stp_state_set,
-	.port_vlan_prepare	= felix_vlan_prepare,
-	.port_vlan_filtering	= felix_vlan_filtering,
-	.port_vlan_add		= felix_vlan_add,
-	.port_vlan_del		= felix_vlan_del,
-	.port_hwtstamp_get	= felix_hwtstamp_get,
-	.port_hwtstamp_set	= felix_hwtstamp_set,
-	.port_rxtstamp		= felix_rxtstamp,
-	.port_txtstamp		= felix_txtstamp,
-	.port_change_mtu	= felix_change_mtu,
-	.port_max_mtu		= felix_get_max_mtu,
-	.port_policer_add	= felix_port_policer_add,
-	.port_policer_del	= felix_port_policer_del,
-	.cls_flower_add		= felix_cls_flower_add,
-	.cls_flower_del		= felix_cls_flower_del,
-	.cls_flower_stats	= felix_cls_flower_stats,
-	.port_setup_tc		= felix_port_setup_tc,
+	.get_tag_protocol		= felix_get_tag_protocol,
+	.setup				= felix_setup,
+	.teardown			= felix_teardown,
+	.set_ageing_time		= felix_set_ageing_time,
+	.get_strings			= felix_get_strings,
+	.get_ethtool_stats		= felix_get_ethtool_stats,
+	.get_sset_count			= felix_get_sset_count,
+	.get_ts_info			= felix_get_ts_info,
+	.phylink_validate		= felix_phylink_validate,
+	.phylink_mac_config		= felix_phylink_mac_config,
+	.phylink_mac_link_down		= felix_phylink_mac_link_down,
+	.phylink_mac_link_up		= felix_phylink_mac_link_up,
+	.port_enable			= felix_port_enable,
+	.port_disable			= felix_port_disable,
+	.port_fdb_dump			= felix_fdb_dump,
+	.port_fdb_add			= felix_fdb_add,
+	.port_fdb_del			= felix_fdb_del,
+	.port_mdb_prepare		= felix_mdb_prepare,
+	.port_mdb_add			= felix_mdb_add,
+	.port_mdb_del			= felix_mdb_del,
+	.port_bridge_join		= felix_bridge_join,
+	.port_bridge_leave		= felix_bridge_leave,
+	.port_stp_state_set		= felix_bridge_stp_state_set,
+	.port_vlan_prepare		= felix_vlan_prepare,
+	.port_vlan_filtering		= felix_vlan_filtering,
+	.port_vlan_add			= felix_vlan_add,
+	.port_vlan_del			= felix_vlan_del,
+	.port_hwtstamp_get		= felix_hwtstamp_get,
+	.port_hwtstamp_set		= felix_hwtstamp_set,
+	.port_rxtstamp			= felix_rxtstamp,
+	.port_txtstamp			= felix_txtstamp,
+	.port_change_mtu		= felix_change_mtu,
+	.port_max_mtu			= felix_get_max_mtu,
+	.port_policer_add		= felix_port_policer_add,
+	.port_policer_del		= felix_port_policer_del,
+	.cls_flower_add			= felix_cls_flower_add,
+	.cls_flower_del			= felix_cls_flower_del,
+	.cls_flower_stats		= felix_cls_flower_stats,
+	.port_setup_tc			= felix_port_setup_tc,
 };
 
 struct net_device *felix_port_to_netdev(struct ocelot *ocelot, int port)
Signature · 1 line
2.25.1
maintainer
#13 lore raw

Re: [PATCH v3 net-next 04/10] net: dsa: felix: reindent struct dsa_switch_ops

In reply to Vladimir Oltean
Quoted 4 lines · On Fri, 8 Jan 2021 19:59:44 +0200 Vladimir Oltean wrote:

From: Vladimir Oltean <[email protected]>

The devlink function pointer names are super long, and they would break the alignment. So reindent the existing ops now by adding one tab.

Therefore it'd be tempting to prefix them with dl_ rather than full devlink_

author maintainer
#14 lore raw

Re: [PATCH v3 net-next 04/10] net: dsa: felix: reindent struct dsa_switch_ops

Quoted 8 lines · Jakub Kicinski
Quoted 4 lines · On Fri, 8 Jan 2021 19:59:44 +0200 Vladimir Oltean wrote:

From: Vladimir Oltean <[email protected]>

The devlink function pointer names are super long, and they would break the alignment. So reindent the existing ops now by adding one tab.

Therefore it'd be tempting to prefix them with dl_ rather than full devlink_

Indentation is broken even with devlink_sb_occ_tc_port_bind_get reduced to dl_sb_occ_tc_port_bind_get.

maintainer
#15 lore raw

Re: [PATCH v3 net-next 04/10] net: dsa: felix: reindent struct dsa_switch_ops

Quoted 4 lines · On Mon, 11 Jan 2021 19:01:58 +0200 Vladimir Oltean wrote:
Quoted 3 lines · Jakub Kicinski
Quoted 2 lines · On Fri, 8 Jan 2021 19:59:44 +0200 Vladimir Oltean wrote:

From: Vladimir Oltean <[email protected]>

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

The devlink function pointer names are super long, and they would break the alignment. So reindent the existing ops now by adding one tab.

Therefore it'd be tempting to prefix them with dl_ rather than full devlink_

Indentation is broken even with devlink_sb_occ_tc_port_bind_get reduced to dl_sb_occ_tc_port_bind_get.

Ack, still, shorter is better IMHO.

author maintainer
#16 lore raw

PATCH v3 5/10 net: dsa: felix: perform teardown in reverse order of setup

In reply to Vladimir Oltean

From: Vladimir Oltean <[email protected]>

In general it is desirable that cleanup is the reverse process of setup. In this case I am not seeing any particular issue, but with the introduction of devlink-sb for felix, a non-obvious decision had to be made as to where to put its cleanup method. When there's a convention in place, that decision becomes obvious.

---
Changes in v3:
None.

Changes in v2:
None.

 drivers/net/dsa/ocelot/felix.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
drivers/net/dsa/ocelot/felix.c +5 -6
index 79d0508e5910..42770a86e871 100644
@@ -639,14 +639,13 @@ static void felix_teardown(struct dsa_switch *ds)
 	struct felix *felix = ocelot_to_felix(ocelot);
 	int port;
 
-	if (felix->info->mdio_bus_free)
-		felix->info->mdio_bus_free(ocelot);
-
+	if (ocelot->ptp)
+		ocelot_deinit_timestamp(ocelot);
+	ocelot_deinit(ocelot);
 	for (port = 0; port < ocelot->num_phys_ports; port++)
 		ocelot_deinit_port(ocelot, port);
-	ocelot_deinit_timestamp(ocelot);
-	/* stop workqueue thread */
-	ocelot_deinit(ocelot);
+	if (felix->info->mdio_bus_free)
+		felix->info->mdio_bus_free(ocelot);
 }
 
 static int felix_hwtstamp_get(struct dsa_switch *ds, int port,
Signature · 1 line
2.25.1
Reviewed-by on v3 5/10
#17 lore raw

Re: [PATCH v3 net-next 05/10] net: dsa: felix: perform teardown in reverse order of setup

Quoted 9 lines · Vladimir Oltean

From: Vladimir Oltean <[email protected]>

In general it is desirable that cleanup is the reverse process of setup. In this case I am not seeing any particular issue, but with the introduction of devlink-sb for felix, a non-obvious decision had to be made as to where to put its cleanup method. When there's a convention in place, that decision becomes obvious.

Signature · 1 line
Florian
author maintainer
#18 lore raw

PATCH v3 6/10 net: mscc: ocelot: export NUM_TC constant from felix to common switch lib

In reply to Vladimir Oltean

From: Vladimir Oltean <[email protected]>

We should be moving anything that isn't DSA-specific or SoC-specific out of the felix DSA driver, and into the common mscc_ocelot switch library.

The number of traffic classes is one of the aspects that is common between all ocelot switches, so it belongs in the library.

This patch also makes seville use 8 TX queues, and therefore enables prioritization via the QOS_CLASS field in the NPI injection header.

---
Changes in v3:
None.

Changes in v2:
Rebased on top of commit edd2410b165e ("net: mscc: ocelot: fix dropping of unknown IPv4 multicast on Seville").

 drivers/net/dsa/ocelot/felix.c           | 2 +-
 drivers/net/dsa/ocelot/felix.h           | 1 -
 drivers/net/dsa/ocelot/felix_vsc9959.c   | 4 ++--
 drivers/net/dsa/ocelot/seville_vsc9953.c | 1 +
 include/soc/mscc/ocelot.h                | 1 +
 5 files changed, 5 insertions(+), 4 deletions(-)
drivers/net/dsa/ocelot/felix.c +1 -1
index 42770a86e871..f41f3476ae06 100644
@@ -328,7 +328,7 @@ static void felix_port_qos_map_init(struct ocelot *ocelot, int port)
 		       ANA_PORT_QOS_CFG,
 		       port);
 
-	for (i = 0; i < FELIX_NUM_TC * 2; i++) {
+	for (i = 0; i < OCELOT_NUM_TC * 2; i++) {
 		ocelot_rmw_ix(ocelot,
 			      (ANA_PORT_PCP_DEI_MAP_DP_PCP_DEI_VAL & i) |
 			      ANA_PORT_PCP_DEI_MAP_QOS_PCP_DEI_VAL(i),
drivers/net/dsa/ocelot/felix.h +0 -1
index 5434fe278d2c..994835cb9307 100644
@@ -5,7 +5,6 @@
 #define _MSCC_FELIX_H
 
 #define ocelot_to_felix(o)		container_of((o), struct felix, ocelot)
-#define FELIX_NUM_TC			8
 
 /* Platform-specific information */
 struct felix_info {
drivers/net/dsa/ocelot/felix_vsc9959.c +2 -2
index 540b86edbbb0..57d1d6767867 100644
@@ -1376,7 +1376,7 @@ static const struct felix_info felix_info_vsc9959 = {
 	.vcap			= vsc9959_vcap_props,
 	.num_mact_rows		= 2048,
 	.num_ports		= 6,
-	.num_tx_queues		= FELIX_NUM_TC,
+	.num_tx_queues		= OCELOT_NUM_TC,
 	.switch_pci_bar		= 4,
 	.imdio_pci_bar		= 0,
 	.ptp_caps		= &vsc9959_ptp_caps,
@@ -1446,7 +1446,7 @@ static int felix_pci_probe(struct pci_dev *pdev,
 	pci_set_drvdata(pdev, felix);
 	ocelot = &felix->ocelot;
 	ocelot->dev = &pdev->dev;
-	ocelot->num_flooding_pgids = FELIX_NUM_TC;
+	ocelot->num_flooding_pgids = OCELOT_NUM_TC;
 	felix->info = &felix_info_vsc9959;
 	felix->switch_base = pci_resource_start(pdev,
 						felix->info->switch_pci_bar);
drivers/net/dsa/ocelot/seville_vsc9953.c +1 -0
index 8dad0c894eca..5e9bfdea50be 100644
@@ -1201,6 +1201,7 @@ static const struct felix_info seville_info_vsc9953 = {
 	.vcap			= vsc9953_vcap_props,
 	.num_mact_rows		= 2048,
 	.num_ports		= 10,
+	.num_tx_queues		= OCELOT_NUM_TC,
 	.mdio_bus_alloc		= vsc9953_mdio_bus_alloc,
 	.mdio_bus_free		= vsc9953_mdio_bus_free,
 	.phylink_validate	= vsc9953_phylink_validate,
include/soc/mscc/ocelot.h +1 -0
index 8eb134cd8d9d..9a46787c679b 100644
@@ -98,6 +98,7 @@
 #define IFH_REW_OP_TWO_STEP_PTP		0x3
 #define IFH_REW_OP_ORIGIN_PTP		0x5
 
+#define OCELOT_NUM_TC			8
 #define OCELOT_TAG_LEN			16
 #define OCELOT_SHORT_PREFIX_LEN		4
 #define OCELOT_LONG_PREFIX_LEN		16
Signature · 1 line
2.25.1
Reviewed-by on v3 6/10
#19 lore raw

Re: [PATCH v3 net-next 06/10] net: mscc: ocelot: export NUM_TC constant from felix to common switch lib

Quoted 5 lines · Vladimir Oltean

From: Vladimir Oltean <[email protected]>

We should be moving anything that isn't DSA-specific or SoC-specific out of the felix DSA driver, and into the common mscc_ocelot switch library.

Quoted 7 lines

The number of traffic classes is one of the aspects that is common between all ocelot switches, so it belongs in the library.

This patch also makes seville use 8 TX queues, and therefore enables prioritization via the QOS_CLASS field in the NPI injection header.

Signature · 1 line
Florian
author maintainer
#20 lore raw

PATCH v3 7/10 net: mscc: ocelot: delete unused ocelot_set_cpu_port prototype

In reply to Vladimir Oltean

From: Vladimir Oltean <[email protected]>

This is a leftover of commit 69df578c5f4b ("net: mscc: ocelot: eliminate confusion between CPU and NPI port") which renamed that function.

---
Changes in v3:
None.

Changes in v2:
None.

 drivers/net/ethernet/mscc/ocelot.h | 4 ----
 1 file changed, 4 deletions(-)
drivers/net/ethernet/mscc/ocelot.h +0 -4
index 291d39d49c4e..519335676c24 100644
@@ -122,10 +122,6 @@ void ocelot_port_writel(struct ocelot_port *port, u32 val, u32 reg);
 int ocelot_probe_port(struct ocelot *ocelot, int port, struct regmap *target,
 		      struct phy_device *phy);
 
-void ocelot_set_cpu_port(struct ocelot *ocelot, int cpu,
-			 enum ocelot_tag_prefix injection,
-			 enum ocelot_tag_prefix extraction);
-
 extern struct notifier_block ocelot_netdevice_nb;
 extern struct notifier_block ocelot_switchdev_nb;
 extern struct notifier_block ocelot_switchdev_blocking_nb;
Signature · 1 line
2.25.1
Reviewed-by on v3 7/10
#21 lore raw

Re: [PATCH v3 net-next 07/10] net: mscc: ocelot: delete unused ocelot_set_cpu_port prototype

Quoted 6 lines · Vladimir Oltean

From: Vladimir Oltean <[email protected]>

This is a leftover of commit 69df578c5f4b ("net: mscc: ocelot: eliminate confusion between CPU and NPI port") which renamed that function.

Signature · 1 line
Florian
author maintainer
#22 lore raw

PATCH v3 8/10 net: mscc: ocelot: register devlink ports

In reply to Vladimir Oltean

From: Vladimir Oltean <[email protected]>

Add devlink integration into the mscc_ocelot switchdev driver. Only the probed interfaces are registered with devlink, because for convenience, struct devlink_port was included into struct ocelot_port_private, which is only initialized for the ports that are used.

Since we use devlink_port_type_eth_set to link the devlink port to the net_device, we can as well remove the .ndo_get_phys_port_name and .ndo_get_port_parent_id implementations, since devlink takes care of retrieving the port name and number automatically, once .ndo_get_devlink_port is implemented.

Note that the felix DSA driver is already integrated with devlink by default, since that is a thing that the DSA core takes care of. This is the reason why these devlink stubs were put in ocelot_net.c and not in the common library.

---
Changes in v3:
None.

Changes in v2:
Using devlink_port_type_eth_set as per Jiri's suggestion.

 drivers/net/ethernet/mscc/ocelot.h         |   4 +
 drivers/net/ethernet/mscc/ocelot_net.c     | 139 ++++++++++++++++-----
 drivers/net/ethernet/mscc/ocelot_vsc7514.c |   7 ++
 include/soc/mscc/ocelot.h                  |   1 +
 4 files changed, 123 insertions(+), 28 deletions(-)
drivers/net/ethernet/mscc/ocelot.h +4 -0
index 519335676c24..2e9a3c4697c8 100644
@@ -57,6 +57,8 @@ struct ocelot_port_private {
 	struct phy *serdes;
 
 	struct ocelot_port_tc tc;
+
+	struct devlink_port devlink_port;
 };
 
 struct ocelot_dump_ctx {
@@ -121,6 +123,8 @@ void ocelot_port_writel(struct ocelot_port *port, u32 val, u32 reg);
 
 int ocelot_probe_port(struct ocelot *ocelot, int port, struct regmap *target,
 		      struct phy_device *phy);
+int ocelot_devlink_init(struct ocelot *ocelot);
+void ocelot_devlink_teardown(struct ocelot *ocelot);
 
 extern struct notifier_block ocelot_netdevice_nb;
 extern struct notifier_block ocelot_switchdev_nb;
drivers/net/ethernet/mscc/ocelot_net.c +111 -28

171 lines not on this page · raw message

drivers/net/ethernet/mscc/ocelot_vsc7514.c +7 -0
index ecd474476cc6..80fdf971d573 100644
@@ -1293,6 +1293,12 @@ static int mscc_ocelot_probe(struct platform_device *pdev)
 		}
 	}
 
+	err = ocelot_devlink_init(ocelot);
+	if (err) {
+		mscc_ocelot_release_ports(ocelot);
+		goto out_ocelot_deinit;
+	}
+
 	register_netdevice_notifier(&ocelot_netdevice_nb);
 	register_switchdev_notifier(&ocelot_switchdev_nb);
 	register_switchdev_blocking_notifier(&ocelot_switchdev_blocking_nb);
@@ -1314,6 +1320,7 @@ static int mscc_ocelot_remove(struct platform_device *pdev)
 {
 	struct ocelot *ocelot = platform_get_drvdata(pdev);
 
+	ocelot_devlink_teardown(ocelot);
 	ocelot_deinit_timestamp(ocelot);
 	mscc_ocelot_release_ports(ocelot);
 	ocelot_deinit(ocelot);
include/soc/mscc/ocelot.h +1 -0
index 9a46787c679b..75cd457b99b9 100644
@@ -602,6 +602,7 @@ struct ocelot_port {
 
 struct ocelot {
 	struct device			*dev;
+	struct devlink			*devlink;
 
 	const struct ocelot_ops		*ops;
 	struct regmap			*targets[TARGET_MAX];
Signature · 1 line
2.25.1
Reviewed-by on v3 8/10
#23 lore raw

Re: [PATCH v3 net-next 08/10] net: mscc: ocelot: register devlink ports

Quoted 13 lines · Vladimir Oltean

From: Vladimir Oltean <[email protected]>

Add devlink integration into the mscc_ocelot switchdev driver. Only the probed interfaces are registered with devlink, because for convenience, struct devlink_port was included into struct ocelot_port_private, which is only initialized for the ports that are used.

Since we use devlink_port_type_eth_set to link the devlink port to the net_device, we can as well remove the .ndo_get_phys_port_name and .ndo_get_port_parent_id implementations, since devlink takes care of retrieving the port name and number automatically, once .ndo_get_devlink_port is implemented.

Quoted 6 lines

Note that the felix DSA driver is already integrated with devlink by default, since that is a thing that the DSA core takes care of. This is the reason why these devlink stubs were put in ocelot_net.c and not in the common library.

Signature · 1 line
Florian
maintainer
#24 lore raw

Re: [PATCH v3 net-next 08/10] net: mscc: ocelot: register devlink ports

In reply to Vladimir Oltean
Quoted 25 lines · On Fri, 8 Jan 2021 19:59:48 +0200 Vladimir Oltean wrote:

From: Vladimir Oltean <[email protected]>

Add devlink integration into the mscc_ocelot switchdev driver. Only the probed interfaces are registered with devlink, because for convenience, struct devlink_port was included into struct ocelot_port_private, which is only initialized for the ports that are used.

Since we use devlink_port_type_eth_set to link the devlink port to the net_device, we can as well remove the .ndo_get_phys_port_name and .ndo_get_port_parent_id implementations, since devlink takes care of retrieving the port name and number automatically, once .ndo_get_devlink_port is implemented.

Note that the felix DSA driver is already integrated with devlink by default, since that is a thing that the DSA core takes care of. This is the reason why these devlink stubs were put in ocelot_net.c and not in the common library.

drivers/net/ethernet/mscc/ocelot_net.c +0 -0
index 2bd2840d88bd..d0d98c6adea8 100644

drivers/net/ethernet/mscc/ocelot_net.c line 13

Quoted 6 lines
drivers/net/ethernet/mscc/ocelot_net.c +3 -0
@@ -8,6 +8,116 @@
 #include "ocelot.h"
 #include "ocelot_vcap.h"
 
+struct ocelot_devlink_private {
+	struct ocelot *ocelot;
+};

Why not make struct ocelot part of devlink_priv?

Quoted 26 lines
drivers/net/ethernet/mscc/ocelot_net.c +26 -0
+static const struct devlink_ops ocelot_devlink_ops = {
+};
+
+static int ocelot_port_devlink_init(struct ocelot *ocelot, int port)
+{
+	struct ocelot_port *ocelot_port = ocelot->ports[port];
+	int id_len = sizeof(ocelot->base_mac);
+	struct devlink *dl = ocelot->devlink;
+	struct devlink_port_attrs attrs = {};
+	struct ocelot_port_private *priv;
+	struct devlink_port *dlp;
+	int err;
+
+	if (!ocelot_port)
+		return 0;
+
+	priv = container_of(ocelot_port, struct ocelot_port_private, port);
+	dlp = &priv->devlink_port;
+
+	memcpy(attrs.switch_id.id, &ocelot->base_mac, id_len);
+	attrs.switch_id.id_len = id_len;
+	attrs.phys.port_number = port;
+
+	if (priv->dev)
+		attrs.flavour = DEVLINK_PORT_FLAVOUR_PHYSICAL;
+	else

drivers/net/ethernet/mscc/ocelot_net.c line 50

Quoted 10 lines
drivers/net/ethernet/mscc/ocelot_net.c +10 -0
+		attrs.flavour = DEVLINK_PORT_FLAVOUR_UNUSED;
+
+	devlink_port_attrs_set(dlp, &attrs);
+
+	err = devlink_port_register(dl, dlp, port);
+	if (err)
+		return err;
+
+	if (priv->dev)
+		devlink_port_type_eth_set(dlp, priv->dev);

devlink_port_attrs_set() should be called before netdev is registered, and devlink_port_type_eth_set() after. So this sequence makes me a tad suspicious.

In particular IIRC devlink's .ndo_get_phys_port_name depends on it, because udev event needs to carry the right info for interface renaming to work reliably. No?

Quoted 29 lines
drivers/net/ethernet/mscc/ocelot_net.c +29 -0
+	return 0;
+}
+
+static void ocelot_port_devlink_teardown(struct ocelot *ocelot, int port)
+{
+	struct ocelot_port *ocelot_port = ocelot->ports[port];
+	struct ocelot_port_private *priv;
+	struct devlink_port *dlp;
+
+	if (!ocelot_port)
+		return;
+
+	priv = container_of(ocelot_port, struct ocelot_port_private, port);
+	dlp = &priv->devlink_port;
+
+	devlink_port_unregister(dlp);
+}
+
+int ocelot_devlink_init(struct ocelot *ocelot)
+{
+	struct ocelot_devlink_private *dl_priv;
+	int port, err;
+
+	ocelot->devlink = devlink_alloc(&ocelot_devlink_ops, sizeof(*dl_priv));
+	if (!ocelot->devlink)
+		return -ENOMEM;
+	dl_priv = devlink_priv(ocelot->devlink);
+	dl_priv->ocelot = ocelot;
+

drivers/net/ethernet/mscc/ocelot_net.c line 90

Quoted 10 lines
drivers/net/ethernet/mscc/ocelot_net.c +10 -0
+	err = devlink_register(ocelot->devlink, ocelot->dev);
+	if (err)
+		goto free_devlink;
+
+	for (port = 0; port < ocelot->num_phys_ports; port++) {
+		err = ocelot_port_devlink_init(ocelot, port);
+		if (err) {
+			while (port-- > 0)
+				ocelot_port_devlink_teardown(ocelot, port);
+			goto unregister_devlink;

nit: should this also be on the error path below?

Quoted 15 lines
drivers/net/ethernet/mscc/ocelot_net.c +11 -0
+		}
+	}
+
+	return 0;
+
+unregister_devlink:
+	devlink_unregister(ocelot->devlink);
+free_devlink:
+	devlink_free(ocelot->devlink);
+	return err;
+}
 

drivers/net/ethernet/mscc/ocelot_vsc7514.c static int mscc_ocelot_probe(struct platform_device *pdev) · line 1299

Quoted 7 lines
drivers/net/ethernet/mscc/ocelot_vsc7514.c +4 -0
@@ -1293,6 +1293,12 @@ static int mscc_ocelot_probe(struct platform_device *pdev)
 		}
 	}
 
+	err = ocelot_devlink_init(ocelot);
+	if (err) {
+		mscc_ocelot_release_ports(ocelot);
+		goto out_ocelot_deinit;

No need to add ocelot_deinit_timestamp(ocelot); to the error path?

Quoted 5 lines
drivers/net/ethernet/mscc/ocelot_vsc7514.c +2 -0
+	}
+
 	register_netdevice_notifier(&ocelot_netdevice_nb);
 	register_switchdev_notifier(&ocelot_switchdev_nb);
 	register_switchdev_blocking_notifier(&ocelot_switchdev_blocking_nb);
author maintainer
#25 lore raw

Re: [PATCH v3 net-next 08/10] net: mscc: ocelot: register devlink ports

Quoted 29 lines · Jakub Kicinski
Quoted 28 lines · On Fri, 8 Jan 2021 19:59:48 +0200 Vladimir Oltean wrote:

From: Vladimir Oltean <[email protected]>

Add devlink integration into the mscc_ocelot switchdev driver. Only the probed interfaces are registered with devlink, because for convenience, struct devlink_port was included into struct ocelot_port_private, which is only initialized for the ports that are used.

Since we use devlink_port_type_eth_set to link the devlink port to the net_device, we can as well remove the .ndo_get_phys_port_name and .ndo_get_port_parent_id implementations, since devlink takes care of retrieving the port name and number automatically, once .ndo_get_devlink_port is implemented.

Note that the felix DSA driver is already integrated with devlink by default, since that is a thing that the DSA core takes care of. This is the reason why these devlink stubs were put in ocelot_net.c and not in the common library.

drivers/net/ethernet/mscc/ocelot_net.c +0 -0
index 2bd2840d88bd..d0d98c6adea8 100644
@@ -8,6 +8,116 @@
 #include "ocelot.h"
 #include "ocelot_vcap.h"
 
Quoted 5 lines

drivers/net/ethernet/mscc/ocelot_net.c line 13

Quoted 3 lines
drivers/net/ethernet/mscc/ocelot_net.c +3 -0
+struct ocelot_devlink_private {
+	struct ocelot *ocelot;
+};

Why not make struct ocelot part of devlink_priv?

I am not sure what you mean.

Quoted 41 lines
Quoted 26 lines
drivers/net/ethernet/mscc/ocelot_net.c +26 -0
+static const struct devlink_ops ocelot_devlink_ops = {
+};
+
+static int ocelot_port_devlink_init(struct ocelot *ocelot, int port)
+{
+	struct ocelot_port *ocelot_port = ocelot->ports[port];
+	int id_len = sizeof(ocelot->base_mac);
+	struct devlink *dl = ocelot->devlink;
+	struct devlink_port_attrs attrs = {};
+	struct ocelot_port_private *priv;
+	struct devlink_port *dlp;
+	int err;
+
+	if (!ocelot_port)
+		return 0;
+
+	priv = container_of(ocelot_port, struct ocelot_port_private, port);
+	dlp = &priv->devlink_port;
+
+	memcpy(attrs.switch_id.id, &ocelot->base_mac, id_len);
+	attrs.switch_id.id_len = id_len;
+	attrs.phys.port_number = port;
+
+	if (priv->dev)
+		attrs.flavour = DEVLINK_PORT_FLAVOUR_PHYSICAL;
+	else

drivers/net/ethernet/mscc/ocelot_net.c line 50

Quoted 10 lines
drivers/net/ethernet/mscc/ocelot_net.c +10 -0
+		attrs.flavour = DEVLINK_PORT_FLAVOUR_UNUSED;
+
+	devlink_port_attrs_set(dlp, &attrs);
+
+	err = devlink_port_register(dl, dlp, port);
+	if (err)
+		return err;
+
+	if (priv->dev)
+		devlink_port_type_eth_set(dlp, priv->dev);

devlink_port_attrs_set() should be called before netdev is registered, and devlink_port_type_eth_set() after. So this sequence makes me a tad suspicious.

Quoted 4 lines

In particular IIRC devlink's .ndo_get_phys_port_name depends on it, because udev event needs to carry the right info for interface renaming to work reliably. No?

If I change the driver's Kconfig from tristate to bool, all is fine, isn't it?

Quoted 60 lines
Quoted 29 lines
drivers/net/ethernet/mscc/ocelot_net.c +29 -0
+	return 0;
+}
+
+static void ocelot_port_devlink_teardown(struct ocelot *ocelot, int port)
+{
+	struct ocelot_port *ocelot_port = ocelot->ports[port];
+	struct ocelot_port_private *priv;
+	struct devlink_port *dlp;
+
+	if (!ocelot_port)
+		return;
+
+	priv = container_of(ocelot_port, struct ocelot_port_private, port);
+	dlp = &priv->devlink_port;
+
+	devlink_port_unregister(dlp);
+}
+
+int ocelot_devlink_init(struct ocelot *ocelot)
+{
+	struct ocelot_devlink_private *dl_priv;
+	int port, err;
+
+	ocelot->devlink = devlink_alloc(&ocelot_devlink_ops, sizeof(*dl_priv));
+	if (!ocelot->devlink)
+		return -ENOMEM;
+	dl_priv = devlink_priv(ocelot->devlink);
+	dl_priv->ocelot = ocelot;
+

drivers/net/ethernet/mscc/ocelot_net.c line 90

Quoted 10 lines
drivers/net/ethernet/mscc/ocelot_net.c +10 -0
+	err = devlink_register(ocelot->devlink, ocelot->dev);
+	if (err)
+		goto free_devlink;
+
+	for (port = 0; port < ocelot->num_phys_ports; port++) {
+		err = ocelot_port_devlink_init(ocelot, port);
+		if (err) {
+			while (port-- > 0)
+				ocelot_port_devlink_teardown(ocelot, port);
+			goto unregister_devlink;

nit: should this also be on the error path below?

Quoted 18 lines
drivers/net/ethernet/mscc/ocelot_net.c +11 -0
+		}
+	}
+
+	return 0;
+
+unregister_devlink:
+	devlink_unregister(ocelot->devlink);
+free_devlink:
+	devlink_free(ocelot->devlink);
+	return err;
+}
 
drivers/net/ethernet/mscc/ocelot_vsc7514.c +0 -0
@@ -1293,6 +1293,12 @@ static int mscc_ocelot_probe(struct platform_device *pdev)
 		}
 	}
 
Quoted 6 lines

drivers/net/ethernet/mscc/ocelot_vsc7514.c static int mscc_ocelot_probe(struct platform_device *pdev) · line 1299

Quoted 4 lines
drivers/net/ethernet/mscc/ocelot_vsc7514.c +4 -0
+	err = ocelot_devlink_init(ocelot);
+	if (err) {
+		mscc_ocelot_release_ports(ocelot);
+		goto out_ocelot_deinit;

No need to add ocelot_deinit_timestamp(ocelot); to the error path?

Yes, the error handling could be better.

maintainer
#26 lore raw

Re: [PATCH v3 net-next 08/10] net: mscc: ocelot: register devlink ports

Quoted 30 lines · On Mon, 11 Jan 2021 19:13:44 +0200 Vladimir Oltean wrote:
Quoted 29 lines · Jakub Kicinski
Quoted 28 lines · On Fri, 8 Jan 2021 19:59:48 +0200 Vladimir Oltean wrote:

From: Vladimir Oltean <[email protected]>

Add devlink integration into the mscc_ocelot switchdev driver. Only the probed interfaces are registered with devlink, because for convenience, struct devlink_port was included into struct ocelot_port_private, which is only initialized for the ports that are used.

Since we use devlink_port_type_eth_set to link the devlink port to the net_device, we can as well remove the .ndo_get_phys_port_name and .ndo_get_port_parent_id implementations, since devlink takes care of retrieving the port name and number automatically, once .ndo_get_devlink_port is implemented.

Note that the felix DSA driver is already integrated with devlink by default, since that is a thing that the DSA core takes care of. This is the reason why these devlink stubs were put in ocelot_net.c and not in the common library.

drivers/net/ethernet/mscc/ocelot_net.c +0 -0
index 2bd2840d88bd..d0d98c6adea8 100644
@@ -8,6 +8,116 @@
 #include "ocelot.h"
 #include "ocelot_vcap.h"
 
Quoted 7 lines
Quoted 5 lines

drivers/net/ethernet/mscc/ocelot_net.c line 13

Quoted 3 lines
drivers/net/ethernet/mscc/ocelot_net.c +3 -0
+struct ocelot_devlink_private {
+	struct ocelot *ocelot;
+};  

Why not make struct ocelot part of devlink_priv?

I am not sure what you mean.

You put a pointer to struct ocelot inside devlink->priv, why not put the actual struct ocelot there?

Quoted 41 lines
Quoted 41 lines
Quoted 26 lines
drivers/net/ethernet/mscc/ocelot_net.c +26 -0
+static const struct devlink_ops ocelot_devlink_ops = {
+};
+
+static int ocelot_port_devlink_init(struct ocelot *ocelot, int port)
+{
+	struct ocelot_port *ocelot_port = ocelot->ports[port];
+	int id_len = sizeof(ocelot->base_mac);
+	struct devlink *dl = ocelot->devlink;
+	struct devlink_port_attrs attrs = {};
+	struct ocelot_port_private *priv;
+	struct devlink_port *dlp;
+	int err;
+
+	if (!ocelot_port)
+		return 0;
+
+	priv = container_of(ocelot_port, struct ocelot_port_private, port);
+	dlp = &priv->devlink_port;
+
+	memcpy(attrs.switch_id.id, &ocelot->base_mac, id_len);
+	attrs.switch_id.id_len = id_len;
+	attrs.phys.port_number = port;
+
+	if (priv->dev)
+		attrs.flavour = DEVLINK_PORT_FLAVOUR_PHYSICAL;
+	else

drivers/net/ethernet/mscc/ocelot_net.c line 50

Quoted 10 lines
drivers/net/ethernet/mscc/ocelot_net.c +10 -0
+		attrs.flavour = DEVLINK_PORT_FLAVOUR_UNUSED;
+
+	devlink_port_attrs_set(dlp, &attrs);
+
+	err = devlink_port_register(dl, dlp, port);
+	if (err)
+		return err;
+
+	if (priv->dev)
+		devlink_port_type_eth_set(dlp, priv->dev);  

devlink_port_attrs_set() should be called before netdev is registered, and devlink_port_type_eth_set() after. So this sequence makes me a tad suspicious.

Quoted 6 lines
Quoted 3 lines

In particular IIRC devlink's .ndo_get_phys_port_name depends on it, because udev event needs to carry the right info for interface renaming to work reliably. No?

If I change the driver's Kconfig from tristate to bool, all is fine, isn't it?

How does Kconfig change the order of registration of objects to subsystems _within_ the driver?

Can you unbind and bind the driver back and see if phys_port_name always gets the correct value? (replay/udevadm test is not sufficient)

author maintainer
#27 lore raw

Re: [PATCH v3 net-next 08/10] net: mscc: ocelot: register devlink ports

Quoted 8 lines · Jakub Kicinski
Quoted 8 lines
Quoted 7 lines

devlink_port_attrs_set() should be called before netdev is registered, and devlink_port_type_eth_set() after. So this sequence makes me a tad suspicious.

In particular IIRC devlink's .ndo_get_phys_port_name depends on it, because udev event needs to carry the right info for interface renaming to work reliably. No?

Quoted 5 lines
Quoted 2 lines

If I change the driver's Kconfig from tristate to bool, all is fine, isn't it?

How does Kconfig change the order of registration of objects to subsystems _within_ the driver?

The registration order within the driver is not what matters. What matters is that the devlink_port and net_device are both registered _before_ udev is up.

Quoted 2 lines

Can you unbind and bind the driver back and see if phys_port_name always gets the correct value? (replay/udevadm test is not sufficient)

Yes, and that udev renaming test failed miserably still.

I have dhcpcd in my system, and it races with my udev script by auto-upping the interfaces when they probe. Then, dev_change_name() returns -EBUSY because the interfaces are up but its priv_flags do not declare IFF_LIVE_RENAME_OK.

How is that one solved?

maintainer
#28 lore raw

Re: [PATCH v3 net-next 08/10] net: mscc: ocelot: register devlink ports

Quoted 12 lines · On Thu, 14 Jan 2021 12:34:05 +0200 Vladimir Oltean wrote:
Quoted 11 lines · Jakub Kicinski
Quoted 10 lines
Quoted 7 lines

devlink_port_attrs_set() should be called before netdev is registered, and devlink_port_type_eth_set() after. So this sequence makes me a tad suspicious.

In particular IIRC devlink's .ndo_get_phys_port_name depends on it, because udev event needs to carry the right info for interface renaming to work reliably. No?

If I change the driver's Kconfig from tristate to bool, all is fine, isn't it?

Quoted 6 lines
Quoted 2 lines

How does Kconfig change the order of registration of objects to subsystems _within_ the driver?

The registration order within the driver is not what matters. What matters is that the devlink_port and net_device are both registered _before_ udev is up.

I see.

Quoted 11 lines
Quoted 2 lines

Can you unbind and bind the driver back and see if phys_port_name always gets the correct value? (replay/udevadm test is not sufficient)

Yes, and that udev renaming test failed miserably still.

I have dhcpcd in my system, and it races with my udev script by auto-upping the interfaces when they probe. Then, dev_change_name() returns -EBUSY because the interfaces are up but its priv_flags do not declare IFF_LIVE_RENAME_OK.

How is that one solved?

Yeah, that's one of those perennial problems we never found a strong answer to. IIRC IFF_LIVE_RENAME_OK was more of a dirty hack than a serious answer. I think we should allow renaming interfaces while they're up, and see if anything breaks. We'd just need to add the right netlink notifications to dev_change_name(), maybe?

author maintainer
#29 lore raw

Renaming interfaces that are up (Was "Re: [PATCH v3 net-next 08/10] net: mscc: ocelot: register devlink") ports

Quoted 10 lines · Jakub Kicinski
Quoted 10 lines
Quoted 2 lines

Can you unbind and bind the driver back and see if phys_port_name always gets the correct value? (replay/udevadm test is not sufficient)

Yes, and that udev renaming test failed miserably still.

I have dhcpcd in my system, and it races with my udev script by auto-upping the interfaces when they probe. Then, dev_change_name() returns -EBUSY because the interfaces are up but its priv_flags do not declare IFF_LIVE_RENAME_OK.

Quoted 7 lines
Quoted 1 line

How is that one solved?

Yeah, that's one of those perennial problems we never found a strong answer to. IIRC IFF_LIVE_RENAME_OK was more of a dirty hack than a serious answer. I think we should allow renaming interfaces while they're up, and see if anything breaks. We'd just need to add the right netlink notifications to dev_change_name(), maybe?

I'm probably crazy for even indulging in this, since it is likely going to just be a huge time sink. But I need to ask anyway: what netlink notification are you thinking of adding? Do you want me to call dev_close and then dev_open, like what was discussed in the "[virtio-dev] Re: net_failover slave udev renaming (was Re: [RFC PATCH net-next v6 4/4] netvsc: refactor notifier/event handling code to use the bypass framework)" thread and then in the "failover: allow name change on IFF_UP slave interfaces" email threads?

By the way I removed the if condition and added nothing in its place, just to see what would happen. I see a lot of these messages, I did not investigate where they are coming from and why they are emitted. They go away when I rename the interface back to swp0.

# ip monitor &
[1] 26931
# ip link set swp0 name random
[ 2857.570246] mscc_felix 0000:00:00.5 random: renamed from swp0
32: random: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default
    link/ether 7a:1e:87:48:79:c0 brd ff:ff:ff:ff:ff:ff
Deleted inet swp0
inet swp0 forwarding off rp_filter off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
Deleted inet6 swp0
inet6 swp0 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
32: random: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default
    link/ether 7a:1e:87:48:79:c0 brd ff:ff:ff:ff:ff:ff
fe80::/64 dev swp0 proto kernel metric 256 pref medium
32: swp0    inet6 fe80::e4fb:6ac5:7211:7981/64 scope link tentative
       valid_lft forever preferred_lft forever
Deleted 32: swp0    inet6 fe80::e4fb:6ac5:7211:7981/64 scope link tentative
       valid_lft forever preferred_lft forever
Deleted fe80::/64 dev swp0 proto kernel metric 256 pref medium
32: random: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default
    link/ether 7a:1e:87:48:79:c0 brd ff:ff:ff:ff:ff:ff
fe80::/64 dev swp0 proto kernel metric 256 pref medium
32: swp0    inet6 fe80::e4fb:6ac5:7211:7981/64 scope link tentative
       valid_lft forever preferred_lft forever
Deleted 32: swp0    inet6 fe80::e4fb:6ac5:7211:7981/64 scope link tentative
       valid_lft forever preferred_lft forever
Deleted fe80::/64 dev swp0 proto kernel metric 256 pref medium
32: random: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default
    link/ether 7a:1e:87:48:79:c0 brd ff:ff:ff:ff:ff:ff
fe80::/64 dev swp0 proto kernel metric 256 pref medium
32: swp0    inet6 fe80::e4fb:6ac5:7211:7981/64 scope link tentative
       valid_lft forever preferred_lft forever
Deleted 32: swp0    inet6 fe80::e4fb:6ac5:7211:7981/64 scope link tentative
       valid_lft forever preferred_lft forever
Deleted fe80::/64 dev swp0 proto kernel metric 256 pref medium
^C32: random: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default
    link/ether 7a:1e:87:48:79:c0 brd ff:ff:ff:ff:ff:ff
fe80::/64 dev swp0 proto kernel metric 256 pref medium
32: swp0    inet6 fe80::e4fb:6ac5:7211:7981/64 scope link tentative
       valid_lft forever preferred_lft forever
Deleted 32: swp0    inet6 fe80::e4fb:6ac5:7211:7981/64 scope link tentative
       valid_lft forever preferred_lft forever
Deleted fe80::/64 dev swp0 proto kernel metric 256 pref medium
32: random: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default
    link/ether 7a:1e:87:48:79:c0 brd ff:ff:ff:ff:ff:ff
fe80::/64 dev swp0 proto kernel metric 256 pref medium
32: swp0    inet6 fe80::e4fb:6ac5:7211:7981/64 scope link tentative
       valid_lft forever preferred_lft forever
Deleted 32: swp0    inet6 fe80::e4fb:6ac5:7211:7981/64 scope link tentative
       valid_lft forever preferred_lft forever
Deleted fe80::/64 dev swp0 proto kernel metric 256 pref medium
author maintainer
#30 lore raw

Re: Renaming interfaces that are up (Was "Re: [PATCH v3 net-next 08/10] net: mscc: ocelot: register devlink") ports

Quoted 4 lines · Vladimir Oltean

By the way I removed the if condition and added nothing in its place, just to see what would happen. I see a lot of these messages, I did not investigate where they are coming from and why they are emitted. They go away when I rename the interface back to swp0.

Looks like it's again related to dhcpcd. If I rename the interface while it's down, the link-local IPv6 addresses don't get endlessly deleted as they do if I rename it while it's up. Also, if I live-rename the interface while dhcpcd isn't up, that behavior disappears too.

Still not sure why it happens.

maintainer
#31 lore raw

Re: [PATCH v3 net-next 08/10] net: mscc: ocelot: register devlink ports

In reply to Vladimir Oltean
Quoted 6 lines · On Fri, 8 Jan 2021 19:59:48 +0200 Vladimir Oltean wrote:

From: Vladimir Oltean <[email protected]>

Add devlink integration into the mscc_ocelot switchdev driver. Only the probed interfaces are registered with devlink, because for convenience, struct devlink_port was included into struct ocelot_port_private, which is only initialized for the ports that are used.

This sounds like something that DSA should have a general policy on. I actually feel like it was discussed in the past.. Do you know what other drivers do?

author maintainer
#32 lore raw

PATCH v3 9/10 net: mscc: ocelot: initialize watermarks to sane defaults

In reply to Vladimir Oltean

From: Vladimir Oltean <[email protected]>

This is meant to be a gentle introduction into the world of watermarks on ocelot. The code is placed in ocelot_devlink.c because it will be integrated with devlink, even if it isn't right now.

My first step was intended to be to replicate the default configuration of the congestion watermarks programatically, since they are now going to be tuned by the user.

But after studying and understanding through trial and error how they work, I now believe that the configuration used out of reset does not do justice to the word "reservation", since the sum of all reservations exceeds the total amount of resources (otherwise said, all reservations cannot be fulfilled at the same time, which means that, contrary to the reference manual, they don't guarantee anything).

As an example, here's a dump of the reservation watermarks for frame buffers, for port 0 (for brevity, the ports 1-6 were omitted, but they have the same configuration):

BUF_Q_RSRV_I(port 0, prio 0) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 1) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 2) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 3) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 4) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 5) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 6) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 7) = max 3000 bytes

Otherwise said, every port-tc has an ingress reservation of 3000 bytes, and there are 7 ports in VSC9959 Felix (6 user ports and 1 CPU port). Concentrating only on the ingress reservations, there are, in total, 8 [traffic classes] x 7 [ports] x 3000 [bytes] = 168,000 bytes of memory reserved on ingress.
But, surprise, Felix only has 128 KB of packet buffer in total... A similar thing happens with Seville, which has a larger packet buffer, but also more ports, and the default configuration is also overcommitted.

This patch disables the (apparently) bogus reservations and moves all resources to the shared area. This way, real reservations can be set up by the user, using devlink-sb.

---
Changes in v3:
None.

Changes in v2:
None.

 drivers/net/ethernet/mscc/Makefile         |   3 +-
 drivers/net/ethernet/mscc/ocelot.c         |   1 +
 drivers/net/ethernet/mscc/ocelot.h         |   1 +
 drivers/net/ethernet/mscc/ocelot_devlink.c | 442 +++++++++++++++++++++
 4 files changed, 446 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/mscc/ocelot_devlink.c
drivers/net/ethernet/mscc/Makefile +2 -1
index 58f94c3d80f9..346bba2730ad 100644
@@ -6,7 +6,8 @@ mscc_ocelot_switch_lib-y := \
 	ocelot_police.o \
 	ocelot_vcap.o \
 	ocelot_flower.o \
-	ocelot_ptp.o
+	ocelot_ptp.o \
+	ocelot_devlink.o
 obj-$(CONFIG_MSCC_OCELOT_SWITCH) += mscc_ocelot.o
 mscc_ocelot-y := \
 	ocelot_vsc7514.o \
drivers/net/ethernet/mscc/ocelot.c +1 -0
index 876c03e51bdc..f547728c6718 100644
@@ -1636,6 +1636,7 @@ int ocelot_init(struct ocelot *ocelot)
 	INIT_DELAYED_WORK(&ocelot->stats_work, ocelot_check_stats_work);
 	queue_delayed_work(ocelot->stats_queue, &ocelot->stats_work,
 			   OCELOT_STATS_CHECK_DELAY);
+	ocelot_watermark_init(ocelot);
 
 	return 0;
 }
drivers/net/ethernet/mscc/ocelot.h +1 -0
index 2e9a3c4697c8..a9d60ef30344 100644
@@ -125,6 +125,7 @@ int ocelot_probe_port(struct ocelot *ocelot, int port, struct regmap *target,
 		      struct phy_device *phy);
 int ocelot_devlink_init(struct ocelot *ocelot);
 void ocelot_devlink_teardown(struct ocelot *ocelot);
+void ocelot_watermark_init(struct ocelot *ocelot);
 
 extern struct notifier_block ocelot_netdevice_nb;
 extern struct notifier_block ocelot_switchdev_nb;
drivers/net/ethernet/mscc/ocelot_devlink.c new file +442 -0

442 lines not on this page · raw message

Signature · 1 line
2.25.1
Reviewed-by on v3 9/10
#33 lore raw

Re: [PATCH v3 net-next 09/10] net: mscc: ocelot: initialize watermarks to sane defaults

Quoted 39 lines · Vladimir Oltean

From: Vladimir Oltean <[email protected]>

This is meant to be a gentle introduction into the world of watermarks on ocelot. The code is placed in ocelot_devlink.c because it will be integrated with devlink, even if it isn't right now.

My first step was intended to be to replicate the default configuration of the congestion watermarks programatically, since they are now going to be tuned by the user.

But after studying and understanding through trial and error how they work, I now believe that the configuration used out of reset does not do justice to the word "reservation", since the sum of all reservations exceeds the total amount of resources (otherwise said, all reservations cannot be fulfilled at the same time, which means that, contrary to the reference manual, they don't guarantee anything).

As an example, here's a dump of the reservation watermarks for frame buffers, for port 0 (for brevity, the ports 1-6 were omitted, but they have the same configuration):

BUF_Q_RSRV_I(port 0, prio 0) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 1) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 2) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 3) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 4) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 5) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 6) = max 3000 bytes
BUF_Q_RSRV_I(port 0, prio 7) = max 3000 bytes

Otherwise said, every port-tc has an ingress reservation of 3000 bytes, and there are 7 ports in VSC9959 Felix (6 user ports and 1 CPU port). Concentrating only on the ingress reservations, there are, in total, 8 [traffic classes] x 7 [ports] x 3000 [bytes] = 168,000 bytes of memory reserved on ingress.
But, surprise, Felix only has 128 KB of packet buffer in total... A similar thing happens with Seville, which has a larger packet buffer, but also more ports, and the default configuration is also overcommitted.

Quoted 5 lines

This patch disables the (apparently) bogus reservations and moves all resources to the shared area. This way, real reservations can be set up by the user, using devlink-sb.

Signature · 1 line
Florian
author maintainer
#34 lore raw

PATCH v3 10/10 net: mscc: ocelot: configure watermarks using devlink-sb

In reply to Vladimir Oltean

From: Vladimir Oltean <[email protected]>

Using devlink-sb, we can configure 12/16 (the important 75%) of the switch's controlling watermarks for congestion drops, and we can monitor 50% of the watermark occupancies (we can monitor the reservation watermarks, but not the sharing watermarks, which are exposed as pool sizes).

The following definitions can be made:

SB_BUF=0 # The devlink-sb for frame buffers
SB_REF=1 # The devlink-sb for frame references
POOL_ING=0 # The pool for ingress traffic. Both devlink-sb instances

           # have one of these.
POOL_EGR=1 # The pool for egress traffic. Both devlink-sb instances
           # have one of these.

Editing the hardware watermarks is done in the following way:
BUF_xxxx_I is accessed when sb=$SB_BUF and pool=$POOL_ING REF_xxxx_I is accessed when sb=$SB_REF and pool=$POOL_ING BUF_xxxx_E is accessed when sb=$SB_BUF and pool=$POOL_EGR REF_xxxx_E is accessed when sb=$SB_REF and pool=$POOL_EGR

Configuring the sharing watermarks for COL_SHR(dp=0) is done implicitly by modifying the corresponding pool size. By default, the pool size has maximum size, so this can be skipped.

devlink sb pool set pci/0000:00:00.5 sb $SB_BUF pool $POOL_ING \
size 103872 thtype static

Since by default there is no buffer reservation, the above command has maxed out BUF_COL_SHR_I(dp=0).

Configuring the per-port reservation watermark (P_RSRV) is done in the following way:

devlink sb port pool set pci/0000:00:00.5/0 sb $SB_BUF \

	pool $POOL_ING th 1000

The above command sets BUF_P_RSRV_I(port 0) to 1000 bytes. After this command, the sharing watermarks are internally reconfigured with 1000 bytes less, i.e. from 103872 bytes to 102872 bytes.

Configuring the per-port-tc reservation watermarks (Q_RSRV) is done in the following way:

for tc in {0..7}; do

	devlink sb tc bind set pci/0000:00:00.5/0 sb 0 tc $tc \
		type ingress pool $POOL_ING \
		th 3000

done

The above command sets BUF_Q_RSRV_I(port 0, tc 0..7) to 3000 bytes. The sharing watermarks are again reconfigured with 24000 bytes less.

---
Changes in v3:
None.

Changes in v2:
- Calling ocelot_watermark_init from ocelot_devlink_sb_register, since there was a bug in v1 where ocelot_watermark_init would use an uninitialized value from ocelot->pool_size, so we need to ensure that ocelot_watermark_init is called after ocelot_devlink_sb_register actually initializes the pool sizes.
- Deleted the "detected N bytes and M frame references" message since that information is now available through devlink-sb.

 drivers/net/dsa/ocelot/felix.c             | 118 ++++++
 drivers/net/ethernet/mscc/ocelot.c         |   5 -
 drivers/net/ethernet/mscc/ocelot.h         |   1 -
 drivers/net/ethernet/mscc/ocelot_devlink.c | 453 ++++++++++++++++++++-
 drivers/net/ethernet/mscc/ocelot_net.c     | 143 +++++++
 drivers/net/ethernet/mscc/ocelot_vsc7514.c |  27 +-
 include/soc/mscc/ocelot.h                  |  47 +++
 7 files changed, 781 insertions(+), 13 deletions(-)
drivers/net/dsa/ocelot/felix.c +118 -0
index f41f3476ae06..8c421165213a 100644
@@ -456,6 +456,7 @@ static int felix_init_structs(struct felix *felix, int num_phys_ports)
 	ocelot->ops		= felix->info->ops;
 	ocelot->inj_prefix	= OCELOT_TAG_PREFIX_SHORT;
 	ocelot->xtr_prefix	= OCELOT_TAG_PREFIX_SHORT;
+	ocelot->devlink		= felix->ds->devlink;
 
 	port_phy_modes = kcalloc(num_phys_ports, sizeof(phy_interface_t),
 				 GFP_KERNEL);
@@ -617,6 +618,10 @@ static int felix_setup(struct dsa_switch *ds)
 		felix_port_qos_map_init(ocelot, port);
 	}
 
+	err = ocelot_devlink_sb_register(ocelot);
+	if (err)
+		return err;
+
 	/* Include the CPU port module in the forwarding mask for unknown
 	 * unicast - the hardware default value for ANA_FLOODING_FLD_UNICAST
 	 * excludes BIT(ocelot->num_phys_ports), and so does ocelot_init, since
@@ -639,6 +644,7 @@ static void felix_teardown(struct dsa_switch *ds)
 	struct felix *felix = ocelot_to_felix(ocelot);
 	int port;
 
+	ocelot_devlink_sb_unregister(ocelot);
 	if (ocelot->ptp)
 		ocelot_deinit_timestamp(ocelot);
 	ocelot_deinit(ocelot);
@@ -779,6 +785,108 @@ static int felix_port_setup_tc(struct dsa_switch *ds, int port,
 		return -EOPNOTSUPP;
 }
 
+static int felix_sb_pool_get(struct dsa_switch *ds, unsigned int sb_index,
+			     u16 pool_index,
+			     struct devlink_sb_pool_info *pool_info)
+{
+	struct ocelot *ocelot = ds->priv;
+
+	return ocelot_sb_pool_get(ocelot, sb_index, pool_index, pool_info);
+}
+
+static int felix_sb_pool_set(struct dsa_switch *ds, unsigned int sb_index,
+			     u16 pool_index, u32 size,
+			     enum devlink_sb_threshold_type threshold_type,
+			     struct netlink_ext_ack *extack)
+{
+	struct ocelot *ocelot = ds->priv;
+
+	return ocelot_sb_pool_set(ocelot, sb_index, pool_index, size,
+				  threshold_type, extack);
+}
+
+static int felix_sb_port_pool_get(struct dsa_switch *ds, int port,
+				  unsigned int sb_index, u16 pool_index,
+				  u32 *p_threshold)
+{
+	struct ocelot *ocelot = ds->priv;
+
+	return ocelot_sb_port_pool_get(ocelot, port, sb_index, pool_index,
+				       p_threshold);
+}
+
+static int felix_sb_port_pool_set(struct dsa_switch *ds, int port,
+				  unsigned int sb_index, u16 pool_index,
+				  u32 threshold, struct netlink_ext_ack *extack)
+{
+	struct ocelot *ocelot = ds->priv;
+
+	return ocelot_sb_port_pool_set(ocelot, port, sb_index, pool_index,
+				       threshold, extack);
+}
+
+static int felix_sb_tc_pool_bind_get(struct dsa_switch *ds, int port,
+				     unsigned int sb_index, u16 tc_index,
+				     enum devlink_sb_pool_type pool_type,
+				     u16 *p_pool_index, u32 *p_threshold)
+{
+	struct ocelot *ocelot = ds->priv;
+
+	return ocelot_sb_tc_pool_bind_get(ocelot, port, sb_index, tc_index,
+					  pool_type, p_pool_index,
+					  p_threshold);
+}
+
+static int felix_sb_tc_pool_bind_set(struct dsa_switch *ds, int port,
+				     unsigned int sb_index, u16 tc_index,
+				     enum devlink_sb_pool_type pool_type,
+				     u16 pool_index, u32 threshold,
+				     struct netlink_ext_ack *extack)
+{
+	struct ocelot *ocelot = ds->priv;
+
+	return ocelot_sb_tc_pool_bind_set(ocelot, port, sb_index, tc_index,
+					  pool_type, pool_index, threshold,
+					  extack);
+}
+
+static int felix_sb_occ_snapshot(struct dsa_switch *ds,
+				 unsigned int sb_index)
+{
+	struct ocelot *ocelot = ds->priv;
+
+	return ocelot_sb_occ_snapshot(ocelot, sb_index);
+}
+
+static int felix_sb_occ_max_clear(struct dsa_switch *ds,
+				  unsigned int sb_index)
+{
+	struct ocelot *ocelot = ds->priv;
+
+	return ocelot_sb_occ_max_clear(ocelot, sb_index);
+}
+
+static int felix_sb_occ_port_pool_get(struct dsa_switch *ds, int port,
+				      unsigned int sb_index, u16 pool_index,
+				      u32 *p_cur, u32 *p_max)
+{
+	struct ocelot *ocelot = ds->priv;
+
+	return ocelot_sb_occ_port_pool_get(ocelot, port, sb_index, pool_index,
+					   p_cur, p_max);
+}
+
+static int felix_sb_occ_tc_port_bind_get(struct dsa_switch *ds, int port,
+					 unsigned int sb_index, u16 tc_index,
+					 enum devlink_sb_pool_type pool_type,
+					 u32 *p_cur, u32 *p_max)
+{
+	struct ocelot *ocelot = ds->priv;
+
+	return ocelot_sb_occ_tc_port_bind_get(ocelot, port, sb_index, tc_index,
+					      pool_type, p_cur, p_max);
+}
+
 const struct dsa_switch_ops felix_switch_ops = {
 	.get_tag_protocol		= felix_get_tag_protocol,
 	.setup				= felix_setup,
@@ -819,6 +927,16 @@ const struct dsa_switch_ops felix_switch_ops = {
 	.cls_flower_del			= felix_cls_flower_del,
 	.cls_flower_stats		= felix_cls_flower_stats,
 	.port_setup_tc			= felix_port_setup_tc,
+	.devlink_sb_pool_get		= felix_sb_pool_get,
+	.devlink_sb_pool_set		= felix_sb_pool_set,
+	.devlink_sb_port_pool_get	= felix_sb_port_pool_get,
+	.devlink_sb_port_pool_set	= felix_sb_port_pool_set,
+	.devlink_sb_tc_pool_bind_get	= felix_sb_tc_pool_bind_get,
+	.devlink_sb_tc_pool_bind_set	= felix_sb_tc_pool_bind_set,
+	.devlink_sb_occ_snapshot	= felix_sb_occ_snapshot,
+	.devlink_sb_occ_max_clear	= felix_sb_occ_max_clear,
+	.devlink_sb_occ_port_pool_get	= felix_sb_occ_port_pool_get,
+	.devlink_sb_occ_tc_port_bind_get= felix_sb_occ_tc_port_bind_get,
 };
 
 struct net_device *felix_port_to_netdev(struct ocelot *ocelot, int port)
drivers/net/ethernet/mscc/ocelot.c +0 -5
index f547728c6718..02fa86c72eb5 100644
@@ -1492,10 +1492,6 @@ static void ocelot_detect_features(struct ocelot *ocelot)
 
 	eq_ctrl = ocelot_read(ocelot, QSYS_EQ_CTRL);
 	ocelot->num_frame_refs = QSYS_MMGT_EQ_CTRL_FP_FREE_CNT(eq_ctrl);
-
-	dev_info(ocelot->dev,
-		 "Detected %d bytes of packet buffer and %d frame references\n",
-		 ocelot->packet_buffer_size, ocelot->num_frame_refs);
 }
 
 int ocelot_init(struct ocelot *ocelot)
@@ -1636,7 +1632,6 @@ int ocelot_init(struct ocelot *ocelot)
 	INIT_DELAYED_WORK(&ocelot->stats_work, ocelot_check_stats_work);
 	queue_delayed_work(ocelot->stats_queue, &ocelot->stats_work,
 			   OCELOT_STATS_CHECK_DELAY);
-	ocelot_watermark_init(ocelot);
 
 	return 0;
 }
drivers/net/ethernet/mscc/ocelot.h +0 -1
index a9d60ef30344..2e9a3c4697c8 100644
@@ -125,7 +125,6 @@ int ocelot_probe_port(struct ocelot *ocelot, int port, struct regmap *target,
 		      struct phy_device *phy);
 int ocelot_devlink_init(struct ocelot *ocelot);
 void ocelot_devlink_teardown(struct ocelot *ocelot);
-void ocelot_watermark_init(struct ocelot *ocelot);
 
 extern struct notifier_block ocelot_netdevice_nb;
 extern struct notifier_block ocelot_switchdev_nb;
drivers/net/ethernet/mscc/ocelot_devlink.c +448 -5

480 lines not on this page · raw message

drivers/net/ethernet/mscc/ocelot_net.c +143 -0

157 lines not on this page · raw message

drivers/net/ethernet/mscc/ocelot_vsc7514.c +25 -2
index 80fdf971d573..7bb80144dd1d 100644
@@ -1173,6 +1173,29 @@ static int mscc_ocelot_init_ports(struct platform_device *pdev,
 	return 0;
 }
 
+static int mscc_ocelot_devlink_setup(struct ocelot *ocelot)
+{
+	int err;
+
+	err = ocelot_devlink_init(ocelot);
+	if (err)
+		return err;
+
+	err = ocelot_devlink_sb_register(ocelot);
+	if (err) {
+		ocelot_devlink_teardown(ocelot);
+		return err;
+	}
+
+	return 0;
+}
+
+static void mscc_ocelot_devlink_cleanup(struct ocelot *ocelot)
+{
+	ocelot_devlink_sb_unregister(ocelot);
+	ocelot_devlink_teardown(ocelot);
+}
+
 static int mscc_ocelot_probe(struct platform_device *pdev)
 {
 	struct device_node *np = pdev->dev.of_node;
@@ -1293,7 +1316,7 @@ static int mscc_ocelot_probe(struct platform_device *pdev)
 		}
 	}
 
-	err = ocelot_devlink_init(ocelot);
+	err = mscc_ocelot_devlink_setup(ocelot);
 	if (err) {
 		mscc_ocelot_release_ports(ocelot);
 		goto out_ocelot_deinit;
@@ -1320,7 +1343,7 @@ static int mscc_ocelot_remove(struct platform_device *pdev)
 {
 	struct ocelot *ocelot = platform_get_drvdata(pdev);
 
-	ocelot_devlink_teardown(ocelot);
+	mscc_ocelot_devlink_cleanup(ocelot);
 	ocelot_deinit_timestamp(ocelot);
 	mscc_ocelot_release_ports(ocelot);
 	ocelot_deinit(ocelot);
include/soc/mscc/ocelot.h +47 -0

63 lines not on this page · raw message

Signature · 1 line
2.25.1
Reviewed-by on v3 10/10
#35 lore raw

Re: [PATCH v3 net-next 10/10] net: mscc: ocelot: configure watermarks using devlink-sb

Quoted 50 lines · Vladimir Oltean

From: Vladimir Oltean <[email protected]>

Using devlink-sb, we can configure 12/16 (the important 75%) of the switch's controlling watermarks for congestion drops, and we can monitor 50% of the watermark occupancies (we can monitor the reservation watermarks, but not the sharing watermarks, which are exposed as pool sizes).

The following definitions can be made:

SB_BUF=0 # The devlink-sb for frame buffers
SB_REF=1 # The devlink-sb for frame references
POOL_ING=0 # The pool for ingress traffic. Both devlink-sb instances

           # have one of these.
POOL_EGR=1 # The pool for egress traffic. Both devlink-sb instances
           # have one of these.

Editing the hardware watermarks is done in the following way:
BUF_xxxx_I is accessed when sb=$SB_BUF and pool=$POOL_ING REF_xxxx_I is accessed when sb=$SB_REF and pool=$POOL_ING BUF_xxxx_E is accessed when sb=$SB_BUF and pool=$POOL_EGR REF_xxxx_E is accessed when sb=$SB_REF and pool=$POOL_EGR

Configuring the sharing watermarks for COL_SHR(dp=0) is done implicitly by modifying the corresponding pool size. By default, the pool size has maximum size, so this can be skipped.

devlink sb pool set pci/0000:00:00.5 sb $SB_BUF pool $POOL_ING \
size 103872 thtype static

Since by default there is no buffer reservation, the above command has maxed out BUF_COL_SHR_I(dp=0).

Configuring the per-port reservation watermark (P_RSRV) is done in the following way:

devlink sb port pool set pci/0000:00:00.5/0 sb $SB_BUF \

	pool $POOL_ING th 1000

The above command sets BUF_P_RSRV_I(port 0) to 1000 bytes. After this command, the sharing watermarks are internally reconfigured with 1000 bytes less, i.e. from 103872 bytes to 102872 bytes.

Configuring the per-port-tc reservation watermarks (Q_RSRV) is done in the following way:

for tc in {0..7}; do

	devlink sb tc bind set pci/0000:00:00.5/0 sb 0 tc $tc \
		type ingress pool $POOL_ING \
		th 3000
Quoted 6 lines

done

The above command sets BUF_Q_RSRV_I(port 0, tc 0..7) to 3000 bytes. The sharing watermarks are again reconfigured with 24000 bytes less.

Signature · 1 line
Florian