devlink: Extend the devlink rate API to support DCB ETS
Subsystems (5)
RFC PATCH 1/1 devlink: Extend the devlink rate API to support DCB ETS
Hi,
We have support for DCB-ETS for quite a while: mapping priorities to TC (traffic class) and setting min-share per TC. The configuration is set on the PF/uplink and affects all the traffic/queues including any VFs/SFs instantiated from that PF.
We have a customer requirement to apply ETS for a group of VFs. For example, TC0 and TC5 for TCP/UDP and RoCE, 20 / 80 respectively.
Two options we considered that didn’t meet the bar:
1. MQPRIO: the DSCP or VLAN-PCP values are set by the VM, one can use MQPRIO qdisc from inside the VM to implement ETS, however, it is not possible to share a qdisc across multiple net-devices.
2. TC police action: use TC filters (on the VF-representors) to classify packet based on DSCP/PCP and apply a policer (a policer action can be shared across multiple filters). However, this is policing and not traffic shaping (no backpressure to the VM).
To this end, devlink-rate seems to be the most suitable interface – has support for group of VFs. Following are two options to extend it:
1. In addition to leaf and node, expose a new type: TC. Each VF will expose TC0 to TC7:
pci/0000:04:00.0/1/tc0
…
pci/0000:04:00.0/1/tc7
Example:
DEV=pci/0000:04:00.0
# Creating a group:
devlink port function rate add $DEV/vfs_group tx_share 10Gbit tx_max 50Gbit
# Creating two groups TC0 and TC5:
devlink port function rate add $DEV/group_tc0 tx_weight 20 parent vfs_group devlink port function rate add $DEV/group_tc5 tx_weight 80 parent vfs_group
# Adding TCs
devlink port function rate set $DEV/1/tc0 parent group_tc0 devlink port function rate set $DEV/2/tc0 parent group_tc0
devlink port function rate set $DEV/1/tc5 parent group_tc5 devlink port function rate set $DEV/2/tc5 parent group_tc5
2. New option to specify the bandwidth proportions between the TCs:
devlink port function rate add $DEV/vfs_group \
tx_share 10Gbit tx_max 50Gbit tc-bw 0:20 1:0 2:0 3:0 4:0 5:80 6:0 7:0
All traffic (group of VFs) will be subjected to share (10), max (50) and adhere to the specified TC weights.
Moreover, we need a mechanism to configure TC-priority mapping and selecting the DSCP or PCP (trust state); this could be either using the existing tools (e.g., dcb-ets/lldp) or by other means.
This patch is based on a previous attempt at extending devlink rate to support per queue rate limiting ([1]). In that case, a different approach was chosen. Here, we propose extending devlink rate with a new node type to represent traffic classes, necessary because of the interactions with VF groups.
We’ll appreciate any feedback.
Cosmin.
PATCH 1/1 devlink: Extend the devlink rate API to support rate management on traffic classes
From: Carolina Jubran <[email protected]>
Introduce support for traffic classes (TC) in the devlink-rate API, expanding beyond the current two object types: nodes and leafs. The support of traffic classes provides more granular control, especially in scenarios where customers need to implement Enhanced Transmission Selection (ETS) for specific groups of Virtual Functions (VFs).
For instance, users can now allocate specific traffic classes, such as TC0 and TC5, to handle TCP/UDP and RoCE traffic respectively, with defined bandwidth shares (e.g., 20% for TC0 and 80% for TC5).
Example:
DEV=pci/0000:08:00.0
devlink port function rate add $DEV/vfs_group tx_share 10Gbit tx_max 50Gbit devlink port function rate add $DEV/group_tc0 tx_weight 20 parent vfs_group devlink port function rate add $DEV/group_tc5 tx_weight 80 parent vfs_group devlink port function rate set $DEV/1/tc0 parent group_tc0 devlink port function rate set $DEV/2/tc0 parent group_tc0
devlink port function rate set $DEV/1/tc5 parent group_tc5 devlink port function rate set $DEV/2/tc5 parent group_tc5
--- Documentation/netlink/specs/devlink.yaml | 8 ++ .../networking/devlink/devlink-port.rst | 18 ++- include/net/devlink.h | 16 +++ include/uapi/linux/devlink.h | 2 + net/devlink/netlink_gen.c | 3 + net/devlink/rate.c | 128 ++++++++++++++++++ 6 files changed, 170 insertions(+), 5 deletions(-)
Documentation/netlink/specs/devlink.yaml +8 -0
@@ -83,6 +83,8 @@ definitions: name: leaf - name: node + - + name: traffic-class - type: enum name: sb-threshold-type
@@ -781,6 +783,9 @@ attribute-sets: - name: rate-tx-max type: u64 + - + name: rate-traffic-class-index + type: u16 - name: rate-node-name type: string
@@ -2121,6 +2126,7 @@ operations: - bus-name - dev-name - port-index + - rate-traffic-class-index - rate-node-name reply: &rate-get-reply value: 76
@@ -2143,6 +2149,7 @@ operations: attributes: - bus-name - dev-name + - rate-traffic-class-index - rate-node-name - rate-tx-share - rate-tx-max
@@ -2163,6 +2170,7 @@ operations: attributes: - bus-name - dev-name + - rate-traffic-class-index - rate-node-name - rate-tx-share - rate-tx-max
Documentation/networking/devlink/devlink-port.rst +13 -5
@@ -374,14 +374,21 @@ At this point a matching subfunction driver binds to the subfunction's auxiliary Rate object management ====================== -Devlink provides API to manage tx rates of single devlink port or a group. -This is done through rate objects, which can be one of the two types: +Devlink provides API to manage tx rates of single devlink port, specific traffic classes or a group. +This is done through rate objects, which can be one of the three types: ``leaf`` Represents a single devlink port; created/destroyed by the driver. Since leaf have 1to1 mapping to its devlink port, in user space it is referred as ``pci/<bus_addr>/<port_index>``; +``traffic class (tc)`` + Represents a traffic class on a devlink port; created/destroyed by the + driver. The traffic class object is referred to in userspace as + ``pci/<bus_addr>/<port_index>/tc<traffic_class_index>``. This object allows + for the management of TX rates at the traffic class level on a specific + devlink port. + ``node`` Represents a group of rate objects (leafs and/or nodes); created/deleted by request from the userspace; initially empty (no rate objects added). In
@@ -437,9 +444,10 @@ Arbitration flow from the high level: #. If all the nodes from the highest priority sub-group are satisfied, or overused their assigned BW, move to the lower priority nodes. -Driver implementations are allowed to support both or either rate object types -and setting methods of their parameters. Additionally driver implementation -may export nodes/leafs and their child-parent relationships. +Driver implementations are allowed to support any combination of the rate +object types and setting methods of their parameters. Additionally driver +implementation may export nodes, leafs, traffic classes, and their +child-parent relationships. Terms and Definitions =====================
include/net/devlink.h +16 -0
@@ -117,6 +117,7 @@ struct devlink_rate {u32 tx_priority;u32 tx_weight; +u16 tc_id; };struct devlink_port {
@@ -1477,6 +1478,14 @@ struct devlink_ops { u32 tx_priority, struct netlink_ext_ack *extack);int (*rate_node_tx_weight_set)(struct devlink_rate *devlink_rate,void *priv,u32 tx_weight,struct netlink_ext_ack *extack); +int (*rate_traffic_class_tx_share_set)(struct devlink_rate *devlink_rate,void *priv, +u64 tx_share,struct netlink_ext_ack *extack); +int (*rate_traffic_class_tx_max_set)(struct devlink_rate *devlink_rate,void *priv, +u64 tx_max,struct netlink_ext_ack *extack); +int (*rate_traffic_class_tx_priority_set)(struct devlink_rate *devlink_rate,void *priv, +u32 tx_priority,struct netlink_ext_ack *extack); +int (*rate_traffic_class_tx_weight_set)(struct devlink_rate *devlink_rate,void *priv, +u32 tx_weight,struct netlink_ext_ack *extack);int (*rate_node_new)(struct devlink_rate *rate_node,void **priv,struct netlink_ext_ack *extack); int (*rate_node_del)(struct devlink_rate *rate_node, void *priv,
@@ -1489,6 +1498,10 @@ struct devlink_ops {struct devlink_rate *parent, void *priv_child, void *priv_parent, struct netlink_ext_ack *extack); +int (*rate_traffic_class_parent_set)(struct devlink_rate *child, +struct devlink_rate *parent, +void *priv_child,void *priv_parent, +struct netlink_ext_ack *extack);/** * selftests_check() - queries if selftest is supported * @devlink: devlink instance
@@ -1723,6 +1736,9 @@ devl_rate_node_create(struct devlink *devlink, void *priv, char *node_name,int devl_rate_leaf_create (struct devlink_port *devlink_port,void *priv,struct devlink_rate *parent); +int +devl_rate_traffic_class_create (struct devlink_port *devlink_port,void *priv,u16 tc_id, +struct devlink_rate *parent);void devl_rate_leaf_destroy (struct devlink_port *devlink_port);void devl_rate_nodes_destroy (struct devlink *devlink); void devlink_port_linecard_set(struct devlink_port *devlink_port,
include/uapi/linux/devlink.h +2 -0
@@ -224,6 +224,7 @@ enum devlink_port_flavour {enum devlink_rate_type {DEVLINK_RATE_TYPE_LEAF ,DEVLINK_RATE_TYPE_NODE , +DEVLINK_RATE_TYPE_TRAFFIC_CLASS , };enum devlink_param_cmode {
@@ -595,6 +596,7 @@ enum devlink_attr {DEVLINK_ATTR_RATE_TYPE ,/* u16 */ DEVLINK_ATTR_RATE_TX_SHARE ,/* u64 */ DEVLINK_ATTR_RATE_TX_MAX ,/* u64 */ +DEVLINK_ATTR_RATE_TRAFFIC_CLASS_INDEX ,/* u16 */ DEVLINK_ATTR_RATE_NODE_NAME ,/* string */ DEVLINK_ATTR_RATE_PARENT_NODE_NAME, /* string */
net/devlink/netlink_gen.c +3 -0
@@ -486,6 +486,7 @@ static const struct nla_policy devlink_rate_get_do_nl_policy[DEVLINK_ATTR_RATE_N [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING, }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING, }, [DEVLINK_ATTR_PORT_INDEX] = { .type = NLA_U32, }, + [DEVLINK_ATTR_RATE_TRAFFIC_CLASS_INDEX] = { .type = NLA_U16, }, [DEVLINK_ATTR_RATE_NODE_NAME] = { .type = NLA_NUL_STRING, }, };
@@ -499,6 +500,7 @@ static const struct nla_policy devlink_rate_get_dump_nl_policy[DEVLINK_ATTR_DEV_static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TX_WEIGHT +1 ] = { [DEVLINK_ATTR_BUS_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_DEV_NAME ] = { .type =NLA_NUL_STRING , }, + [DEVLINK_ATTR_RATE_TRAFFIC_CLASS_INDEX ] = { .type =NLA_U16 , }, [DEVLINK_ATTR_RATE_NODE_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_RATE_TX_SHARE ] = { .type =NLA_U64 , }, [DEVLINK_ATTR_RATE_TX_MAX ] = { .type =NLA_U64 , },
@@ -511,6 +513,7 @@ static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TX_Wstatic const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TX_WEIGHT +1 ] = { [DEVLINK_ATTR_BUS_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_DEV_NAME ] = { .type =NLA_NUL_STRING , }, + [DEVLINK_ATTR_RATE_TRAFFIC_CLASS_INDEX ] = { .type =NLA_U16 , }, [DEVLINK_ATTR_RATE_NODE_NAME ] = { .type =NLA_NUL_STRING , }, [DEVLINK_ATTR_RATE_TX_SHARE ] = { .type =NLA_U64 , }, [DEVLINK_ATTR_RATE_TX_MAX ] = { .type =NLA_U64 , },
net/devlink/rate.c +128 -0
182 lines not on this page · raw message
Signature · 1 line
2.43.2
kernel test robot
build botRe: [PATCH] devlink: Extend the devlink rate API to support rate management on traffic classes
kernel test robot noticed the following build warnings:
Hi Cosmin,
kernel test robot noticed the following build warnings:
[auto build test WARNING on linus/master]
[also build test WARNING on v6.11-rc3 next-20240814]
[cannot apply to horms-ipvs/master]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Cosmin-Ratiu/devlink-Extend-the-devlink-rate-API-to-support-rate-management-on-traffic-classes/20240814-232253 base: linus/master patch link: https://lore.kernel.org/r/20240814085320.134075-2-cratiu%40nvidia.com patch subject: [PATCH] devlink: Extend the devlink rate API to support rate management on traffic classes config: arc-randconfig-001-20240815 (https://download.01.org/0day-ci/archive/20240815/[email protected]/config) compiler: arc-elf-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240815/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> net/devlink/rate.c:755: warning: Function parameter or struct member 'devlink_port' not described in 'devl_rate_traffic_class_create' >> net/devlink/rate.c:755: warning: Function parameter or struct member 'parent' not described in 'devl_rate_traffic_class_create' >> net/devlink/rate.c:755: warning: Excess function parameter 'devlink' description in 'devl_rate_traffic_class_create'
vim +755 net/devlink/rate.c
744
745 /**
746 * devl_rate_traffic_class_create - create devlink rate queue
747 * @devlink: devlink instance
748 * @priv: driver private data
749 * @tc_id: identifier of the new traffic class
750 *
751 * Create devlink rate object of type node
752 */
753 int devl_rate_traffic_class_create(struct devlink_port *devlink_port, void *priv, u16 tc_id,
754 struct devlink_rate *parent)
> 755 { 756 struct devlink *devlink = devlink_port->devlink;
757 struct devlink_rate *devlink_rate;
758
759 devl_assert_locked(devlink);
760
761 devlink_rate = devlink_rate_traffic_class_get_by_id(devlink, tc_id);
762 if (!IS_ERR(devlink_rate))
763 return -EEXIST;
764
765 devlink_rate = kzalloc(sizeof(*devlink_rate), GFP_KERNEL);
766 if (!devlink_rate)
767 return -ENOMEM;
768
769 if (parent) {
770 devlink_rate->parent = parent;
771 refcount_inc(&devlink_rate->parent->refcnt);
772 }
773
774 devlink_rate->type = DEVLINK_RATE_TYPE_TRAFFIC_CLASS;
775 devlink_rate->devlink = devlink;
776 devlink_rate->devlink_port = devlink_port;
777 devlink_rate->tc_id = tc_id;
778 devlink_rate->priv = priv;
779 list_add_tail(&devlink_rate->list, &devlink->rate_list);
780 devlink_port->devlink_rate = devlink_rate;
781 devlink_rate_notify(devlink_rate, DEVLINK_CMD_RATE_NEW);
782
783 return 0;
784 }
785 EXPORT_SYMBOL_GPL(devl_rate_traffic_class_create);
786
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki