mlxsw: Support dcbnl_setbuffer, dcbnl_getbuffer
Subsystems (3)
cover mlxsw: Support dcbnl_setbuffer, dcbnl_getbuffer
From: Ido Schimmel <[email protected]>
Petr says:
On Spectrum, port buffers, also called port headroom, is where packets are stored while they are parsed and the forwarding decision is being made. For lossless traffic flows, in case shared buffer admission is not allowed, headroom is also where to put the extra traffic received before the sent PAUSE takes effect.
Linux supports two DCB interfaces related to the headroom: dcbnl_setbuffer for configuration, and dcbnl_getbuffer for inspection. This patch set implements them.
With dcbnl_setbuffer in place, there will be two sources of authority over the ingress configuration: the DCB ETS hook, because ETS configuration is mirrored to ingress, and the DCB setbuffer hook. mlxsw is in a similar situation on the egress side, where there are two sources of the ETS
configuration: the DCB ETS hook, and the TC qdisc hooks. This is a non-intuitive situation, because the way the ASIC ends up being configured depends not only on the actual configured bits, but also on the order in which they were configured.
To prevent these issues on the ingress side, two configuration modes will
exist: DCB mode and TC mode. DCB ETS will keep getting projected to ingress in the (default) DCB mode. When a qdisc is installed on a port, it will be switched to the TC mode, the ingress configuration will be done through the dcbnl_setbuffer callback. The reason is that the dcbnl_setbuffer hook is not standardized and supported by lldpad. Projecting DCB ETS configuration to ingress is a reasonable heuristic to configure ingress especially when PFC is in effect.
In patch #1, the toggle between the DCB and TC modes of headroom configuration, described above, is introduced.
Patch #2 implements dcbnl_getbuffer and dcbnl_setbuffer. dcbnl_getbuffer can be always used to determine the current port headroom configuration. dcbnl_setbuffer is only permitted in the TC mode.
In patch #3, make the qdisc module toggle the headroom mode from DCB to TC and back, depending on whether there is an offloaded qdisc on the port.
Petr Machata (3):
mlxsw: spectrum_buffers: Support two headroom modes
mlxsw: spectrum_dcb: Implement dcbnl_setbuffer / getbuffer
mlxsw: spectrum_qdisc: Disable port buffer autoresize with qdiscs
.../net/ethernet/mellanox/mlxsw/spectrum.h | 11 ++++ .../mellanox/mlxsw/spectrum_buffers.c | 22 ++++++- .../ethernet/mellanox/mlxsw/spectrum_dcb.c | 59 +++++++++++++++++++ .../ethernet/mellanox/mlxsw/spectrum_qdisc.c | 34 ++++++++++- 4 files changed, 122 insertions(+), 4 deletions(-)
Signature · 1 line
2.26.2
PATCH 1/3 mlxsw: spectrum_buffers: Support two headroom modes
From: Petr Machata <[email protected]>
There are two interfaces to configure ETS: qdiscs and DCB. Historically, DCB ETS configuration was projected to ingress as well, and configured port buffers. Qdisc was not.
So as not to break clients that today use DCB ETS and PFC and rely on getting a reasonable ingress buffer priomap, keep the ETS mirroring in effect.
Since qdiscs have not done this mirroring historically, it is reasonable not to introduce it, but rather permit manual ingress configuration through dcbnl_setbuffer only in the qdisc mode.
This will require a toggle to indicate whether buffer sizes should be autocomputed or taken from dcbnl_setbuffer, and likewise for priomaps. Introduce such and initialize it, and guard port buffer size configuration as appropriate. The toggle is currently left in the DCB position. In a following patch, qdisc code will switch it.
--- .../net/ethernet/mellanox/mlxsw/spectrum.h | 11 ++++++++++ .../mellanox/mlxsw/spectrum_buffers.c | 22 ++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlxsw/spectrum.h +11 -0
@@ -434,18 +434,29 @@ struct mlxsw_sp_hdroom_prio {u8 buf_idx;/* Value of buf_idx deduced from the DCB ETS configuration. */ u8 ets_buf_idx; +/* Value of buf_idx taken from the dcbnl_setbuffer configuration. */ +u8 set_buf_idx;bool lossy; };struct mlxsw_sp_hdroom_buf {u32 thres_cells;u32 size_cells; +/* Size requirement form dcbnl_setbuffer. */ +u32 set_size_cells;bool lossy; }; +enum mlxsw_sp_hdroom_mode { +MLXSW_SP_HDROOM_MODE_DCB , +MLXSW_SP_HDROOM_MODE_TC , +}; +#define MLXSW_SP_PB_COUNT 10struct mlxsw_sp_hdroom { +enum mlxsw_sp_hdroom_mode mode; +struct {struct mlxsw_sp_hdroom_prio prio[IEEE_8021Q_MAX_PRIORITIES ]; } prios;
drivers/net/ethernet/mellanox/mlxsw/spectrum_buffers.c +19 -3
@@ -304,8 +304,16 @@ void mlxsw_sp_hdroom_prios_reset_buf_idx(struct mlxsw_sp_hdroom *hdroom) {int prio; -for (prio =0 ; prio <IEEE_8021QAZ_MAX_TCS ; prio++) - hdroom->prios.prio[prio].buf_idx = hdroom->prios.prio[prio].ets_buf_idx; +for (prio =0 ; prio <IEEE_8021QAZ_MAX_TCS ; prio++) { +switch (hdroom->mode) { +case MLXSW_SP_HDROOM_MODE_DCB : + hdroom->prios.prio[prio].buf_idx = hdroom->prios.prio[prio].ets_buf_idx; +break ; +case MLXSW_SP_HDROOM_MODE_TC : + hdroom->prios.prio[prio].buf_idx = hdroom->prios.prio[prio].set_buf_idx; +break ; + } + } }void mlxsw_sp_hdroom_bufs_reset_lossiness (struct mlxsw_sp_hdroom *hdroom)
@@ -411,7 +419,14 @@ void mlxsw_sp_hdroom_bufs_reset_sizes(struct mlxsw_sp_port *mlxsw_sp_port, delay_cells =mlxsw_sp_port_headroom_8x_adjust (mlxsw_sp_port, delay_cells); buf->thres_cells = thres_cells; - buf->size_cells = thres_cells + delay_cells; +if (hdroom->mode ==MLXSW_SP_HDROOM_MODE_DCB ) { + buf->size_cells = thres_cells + delay_cells; + }else { +/* Do not allow going below the minimum size, even if +* the user requested it. +*/ + buf->size_cells =max (buf->set_size_cells, buf->thres_cells); + } } }
@@ -575,6 +590,7 @@ static int mlxsw_sp_port_headroom_init(struct mlxsw_sp_port *mlxsw_sp_port)int prio; hdroom.mtu = mlxsw_sp_port->dev->mtu; + hdroom.mode =MLXSW_SP_HDROOM_MODE_DCB ;for (prio =0 ; prio <IEEE_8021QAZ_MAX_TCS ; prio++) hdroom.prios.prio[prio].lossy = true;
Signature · 1 line
2.26.2
PATCH 2/3 mlxsw: spectrum_dcb: Implement dcbnl_setbuffer / getbuffer
In reply to Ido SchimmelFrom: Petr Machata <[email protected]>
Add dcbnl_setbuffer, which bounces requests if a headroom is in DCB mode. Implement dcbnl_getbuffer such that it can always be used to determine port-buffer configuration, regardless of headroom mode.
--- .../ethernet/mellanox/mlxsw/spectrum_dcb.c | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+)
drivers/net/ethernet/mellanox/mlxsw/spectrum_dcb.c +59 -0
@@ -592,6 +592,62 @@ static int mlxsw_sp_dcbnl_ieee_setpfc(struct net_device *dev,return err; } +static int mlxsw_sp_dcbnl_getbuffer (struct net_device *dev,struct dcbnl_buffer *buf) +{ +struct mlxsw_sp_port *mlxsw_sp_port =netdev_priv (dev); +struct mlxsw_sp_hdroom *hdroom = mlxsw_sp_port->hdroom; +struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp; +int prio; +int i; + + buf->total_size =0 ; + +BUILD_BUG_ON (DCBX_MAX_BUFFERS >MLXSW_SP_PB_COUNT ); +for (i =0 ; i <MLXSW_SP_PB_COUNT ; i++) { +u32 bytes =mlxsw_sp_cells_bytes (mlxsw_sp, hdroom->bufs.buf[i].size_cells); + +if (i <DCBX_MAX_BUFFERS ) + buf->buffer_size[i] = bytes; + buf->total_size += bytes; + } + + buf->total_size +=mlxsw_sp_cells_bytes (mlxsw_sp, hdroom->int_buf.size_cells); + +for (prio =0 ; prio <IEEE_8021Q_MAX_PRIORITIES ; prio++) + buf->prio2buffer[prio] = hdroom->prios.prio[prio].buf_idx; + +return 0 ; +} + +static int mlxsw_sp_dcbnl_setbuffer (struct net_device *dev,struct dcbnl_buffer *buf) +{ +struct mlxsw_sp_port *mlxsw_sp_port =netdev_priv (dev); +struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp; +struct mlxsw_sp_hdroom hdroom; +int prio; +int i; + + hdroom = *mlxsw_sp_port->hdroom; + +if (hdroom.mode !=MLXSW_SP_HDROOM_MODE_TC ) { +netdev_err (dev,"The use of dcbnl_setbuffer is only allowed if egress is configured using TC\n" ); +return -EINVAL ; + } + +for (prio =0 ; prio <IEEE_8021Q_MAX_PRIORITIES ; prio++) + hdroom.prios.prio[prio].set_buf_idx = buf->prio2buffer[prio]; + +BUILD_BUG_ON (DCBX_MAX_BUFFERS >MLXSW_SP_PB_COUNT ); +for (i =0 ; i <DCBX_MAX_BUFFERS ; i++) + hdroom.bufs.buf[i].set_size_cells =mlxsw_sp_bytes_cells (mlxsw_sp, + buf->buffer_size[i]); + +mlxsw_sp_hdroom_prios_reset_buf_idx (&hdroom); +mlxsw_sp_hdroom_bufs_reset_lossiness (&hdroom); +mlxsw_sp_hdroom_bufs_reset_sizes (mlxsw_sp_port, &hdroom); +return mlxsw_sp_hdroom_configure (mlxsw_sp_port, &hdroom); +} +static const struct dcbnl_rtnl_ops mlxsw_sp_dcbnl_ops = { .ieee_getets = mlxsw_sp_dcbnl_ieee_getets, .ieee_setets = mlxsw_sp_dcbnl_ieee_setets,
@@ -604,6 +660,9 @@ static const struct dcbnl_rtnl_ops mlxsw_sp_dcbnl_ops = { .getdcbx = mlxsw_sp_dcbnl_getdcbx, .setdcbx = mlxsw_sp_dcbnl_setdcbx, + + .dcbnl_getbuffer = mlxsw_sp_dcbnl_getbuffer, + .dcbnl_setbuffer = mlxsw_sp_dcbnl_setbuffer, };static int mlxsw_sp_port_ets_init (struct mlxsw_sp_port *mlxsw_sp_port)
Signature · 1 line
2.26.2
PATCH 3/3 mlxsw: spectrum_qdisc: Disable port buffer autoresize with qdiscs
In reply to Ido SchimmelFrom: Petr Machata <[email protected]>
There are two interfaces to configure ETS: qdiscs and DCB. Historically, DCB ETS configuration was projected to ingress as well, and configured port buffers. Qdisc was not.
Keep qdiscs behaving this way, and if an offloaded qdisc is configured on a port, move this port's headroom to a manual mode, thus allowing configuration of port buffers through dcbnl_setbuffer.
--- .../ethernet/mellanox/mlxsw/spectrum_qdisc.c | 34 ++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlxsw/spectrum_qdisc.c +33 -1
@@ -140,18 +140,31 @@ static int mlxsw_sp_qdisc_destroy(struct mlxsw_sp_port *mlxsw_sp_port, struct mlxsw_sp_qdisc *mlxsw_sp_qdisc) { + struct mlxsw_sp_qdisc *root_qdisc = &mlxsw_sp_port->qdisc->root_qdisc; +int err_hdroom =0 ;int err =0 ;if (!mlxsw_sp_qdisc)return 0 ; +if (root_qdisc == mlxsw_sp_qdisc) { +struct mlxsw_sp_hdroom hdroom = *mlxsw_sp_port->hdroom; + + hdroom.mode =MLXSW_SP_HDROOM_MODE_DCB ; +mlxsw_sp_hdroom_prios_reset_buf_idx (&hdroom); +mlxsw_sp_hdroom_bufs_reset_lossiness (&hdroom); +mlxsw_sp_hdroom_bufs_reset_sizes (mlxsw_sp_port, &hdroom); + err_hdroom =mlxsw_sp_hdroom_configure (mlxsw_sp_port, &hdroom); + } +if (mlxsw_sp_qdisc->ops && mlxsw_sp_qdisc->ops->destroy) err = mlxsw_sp_qdisc->ops->destroy (mlxsw_sp_port, mlxsw_sp_qdisc); mlxsw_sp_qdisc->handle =TC_H_UNSPEC ; mlxsw_sp_qdisc->ops =NULL ; -return err; + +return err_hdroom ?: err; } static int
@@ -159,6 +172,8 @@ mlxsw_sp_qdisc_replace(struct mlxsw_sp_port *mlxsw_sp_port, u32 handle,struct mlxsw_sp_qdisc *mlxsw_sp_qdisc, struct mlxsw_sp_qdisc_ops *ops, void *params) { + struct mlxsw_sp_qdisc *root_qdisc = &mlxsw_sp_port->qdisc->root_qdisc; +struct mlxsw_sp_hdroom orig_hdroom;int err;if (mlxsw_sp_qdisc->ops && mlxsw_sp_qdisc->ops->type != ops->type)
@@ -168,6 +183,21 @@ mlxsw_sp_qdisc_replace(struct mlxsw_sp_port *mlxsw_sp_port, u32 handle,* new one. */ mlxsw_sp_qdisc_destroy (mlxsw_sp_port, mlxsw_sp_qdisc); + + orig_hdroom = *mlxsw_sp_port->hdroom; +if (root_qdisc == mlxsw_sp_qdisc) { +struct mlxsw_sp_hdroom hdroom = orig_hdroom; + + hdroom.mode =MLXSW_SP_HDROOM_MODE_TC ; +mlxsw_sp_hdroom_prios_reset_buf_idx (&hdroom); +mlxsw_sp_hdroom_bufs_reset_lossiness (&hdroom); +mlxsw_sp_hdroom_bufs_reset_sizes (mlxsw_sp_port, &hdroom); + + err =mlxsw_sp_hdroom_configure (mlxsw_sp_port, &hdroom); +if (err) + goto err_hdroom_configure; + } + err = ops->check_params (mlxsw_sp_port, mlxsw_sp_qdisc, params);if (err) goto err_bad_param;
@@ -191,6 +221,8 @@ mlxsw_sp_qdisc_replace(struct mlxsw_sp_port *mlxsw_sp_port, u32 handle, err_bad_param: err_config: +mlxsw_sp_hdroom_configure (mlxsw_sp_port, &orig_hdroom); +err_hdroom_configure:if (mlxsw_sp_qdisc->handle == handle && ops->unoffload) ops->unoffload (mlxsw_sp_port, mlxsw_sp_qdisc, params);
Signature · 1 line
2.26.2
David Miller
maintainerFrom: Ido Schimmel <[email protected]>
Date: Thu, 17 Sep 2020 09:49:00 +0300
Quoted 35 lines
From: Ido Schimmel <[email protected]>
Petr says:
On Spectrum, port buffers, also called port headroom, is where packets are stored while they are parsed and the forwarding decision is being made. For lossless traffic flows, in case shared buffer admission is not allowed, headroom is also where to put the extra traffic received before the sent PAUSE takes effect.
Linux supports two DCB interfaces related to the headroom:
dcbnl_setbufferfor configuration, anddcbnl_getbufferfor inspection. This patch set implements them.With
dcbnl_setbufferin place, there will be two sources of authority over the ingress configuration: the DCB ETS hook, because ETS configuration is mirrored to ingress, and the DCB setbuffer hook. mlxsw is in a similar situation on the egress side, where there are two sources of the ETS
configuration: the DCB ETS hook, and the TC qdisc hooks. This is a non-intuitive situation, because the way the ASIC ends up being configured depends not only on the actual configured bits, but also on the order in which they were configured.To prevent these issues on the ingress side, two configuration modes will
exist: DCB mode and TC mode. DCB ETS will keep getting projected to ingress in the (default) DCB mode. When a qdisc is installed on a port, it will be switched to the TC mode, the ingress configuration will be done through thedcbnl_setbuffercallback. The reason is that thedcbnl_setbufferhook is not standardized and supported by lldpad. Projecting DCB ETS configuration to ingress is a reasonable heuristic to configure ingress especially when PFC is in effect.In patch #1, the toggle between the DCB and TC modes of headroom configuration, described above, is introduced.
Quoted 6 lines
Patch #2 implements
dcbnl_getbufferanddcbnl_setbuffer.dcbnl_getbuffercan be always used to determine the current port headroom configuration.dcbnl_setbufferis only permitted in the TC mode.In patch #3, make the qdisc module toggle the headroom mode from DCB to TC and back, depending on whether there is an offloaded qdisc on the port.
Series applied, thank you.