ethtool: Add support for frame preemption
Subsystems (4)
cover ethtool: Add support for frame preemption
Hi,
This series adds support for configuring frame preemption, as defined by IEEE 802.1Q-2018 (previously IEEE 802.1Qbu) and IEEE 802.3br.
Frame preemption allows a packet from a higher priority queue marked as "express" to preempt a packet from lower priority queue marked as "preemptible". The idea is that this can help reduce the latency for higher priority traffic.
Previously, the proposed interface for configuring these features was using the qdisc layer. But as this is very hardware dependent and all that qdisc did was pass the information to the driver, it makes sense to have this in ethtool.
One example, for retrieving and setting the configuration:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: active supported queues: 0xf supported queues: 0xe minimum fragment size: 68
$ ethtool --set-frame-preemption enp3s0 fp on min-frag-size 68 preemptible-queues-mask 0xe
This is a RFC because I wanted to have feedback on some points:
- The parameters added are enough for the hardware I have, is it enough in general?
- even with the ethtool via netlink effort, I chose to keep the ioctl() way, in case someone wants to backport this to an older kernel, is there a problem with this?
- Some space for bikeshedding the names and location (for example, does it make sense for these settings to be per-queue?), as I am not quite happy with them, one example, is the use of preemptible vs. preemptable;
About the patches, should be quite straightforward:
Patch 1, adds the ETHTOOL_GFP and ETHOOL_SFP commands and the associated data structures;
Patch 2, adds the ETHTOOL_MSG_PREEMPT_GET and ETHTOOL_MSG_PREEMPT_SET netlink messages and the associated attributes;
Patch 3, is the example implementation for the igc driver, the catch here is that frame preemption in igc is dependent on the TSN "mode" being enabled;
Patch 4, adds some registers that helped during implementation.
Another thing is that because igc is still under development, to avoid conflicts, I think it might be easier for the PATCH version of this series to go through Jeff Kirsher's tree.
Vinicius Costa Gomes (4):
ethtool: Add support for configuring frame preemption
ethtool: Add support for configuring frame preemption via netlink
igc: Add support for configuring frame preemption
igc: Add support for exposing frame preemption stats registers
drivers/net/ethernet/intel/igc/igc.h | 3 + drivers/net/ethernet/intel/igc/igc_defines.h | 6 + drivers/net/ethernet/intel/igc/igc_ethtool.c | 77 ++++++++ drivers/net/ethernet/intel/igc/igc_regs.h | 10 + drivers/net/ethernet/intel/igc/igc_tsn.c | 46 ++++- include/linux/ethtool.h | 6 + include/uapi/linux/ethtool.h | 25 +++ include/uapi/linux/ethtool_netlink.h | 19 ++ net/ethtool/Makefile | 3 +- net/ethtool/ioctl.c | 36 ++++ net/ethtool/netlink.c | 15 ++ net/ethtool/netlink.h | 2 + net/ethtool/preempt.c | 181 +++++++++++++++++++ 13 files changed, 423 insertions(+), 6 deletions(-) create mode 100644 net/ethtool/preempt.c
Signature · 1 line
2.26.2
RFC 1/4 ethtool: Add support for configuring frame preemption
Frame preemption (described in IEEE 802.3br-2016) defines the concept of preemptible and express queues. It allows traffic from express queues to "interrupt" traffic from preemptible queues, which are "resumed" after the express traffic has finished transmitting.
Frame preemption can only be used when both the local device and the link partner support it.
A new ethtool command was added to support the configuration parameters.
--- include/linux/ethtool.h | 6 ++++++ include/uapi/linux/ethtool.h | 25 +++++++++++++++++++++++++ net/ethtool/ioctl.c | 36 ++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+)
include/linux/ethtool.h +6 -0
@@ -360,6 +360,8 @@ bool ethtool_convert_link_mode_to_legacy_u32(u32 *legacy_u32,* @get_ethtool_phy_stats: Return extended statistics about the PHY device. * This is only useful if the device maintains PHY statistics and * cannot use the standard PHY library helpers. +* @get_preempt: Get the network device Frame Preemption parameters. +* @set_preempt: Set the network device Frame Preemption parameters. * * All operations are optional (i.e. the function pointer may be set * to %NULL) and callers must take this into account. Callers must
@@ -454,6 +456,10 @@ struct ethtool_ops { struct ethtool_fecparam *);int (*set_fecparam)(struct net_device *,struct ethtool_fecparam *); +int (*get_preempt)(struct net_device *, +struct ethtool_fp *); +int (*set_preempt)(struct net_device *, +struct ethtool_fp *);void (*get_ethtool_phy_stats)(struct net_device *,struct ethtool_stats *,u64 *); };
include/uapi/linux/ethtool.h +25 -0
@@ -369,6 +369,28 @@ struct ethtool_eee {__u32 reserved[2 ]; }; +/** +* struct ethtool_fp - Frame Preemption information +* @cmd: ETHTOOL_{G,S}FP +* @fp_supported: If frame preemption is supported. +* @fp_enabled: If frame preemption should be advertised to the link partner +* as enabled. +* @supported_queues_mask: Bitmask indicating which queues support being +* configured as preemptible (bit 0 -> queue 0, bit N -> queue N). +* @preemptible_queues_mask: Bitmask indicating which queues are +* configured as preemptible (bit 0 -> queue 0, bit N -> queue N). +* @min_frag_size: Minimum size for all non-final fragment size. +*/ +struct ethtool_fp { +__u32 cmd; +__u8 fp_supported; +__u8 fp_enabled; +__u32 supported_queues_mask; +__u32 preemptible_queues_mask; +__u32 min_frag_size; +__u32 reserved[2 ]; +}; +/** * struct ethtool_modinfo - plugin module eeprom information * @cmd: %ETHTOOL_GMODULEINFO
@@ -1441,6 +1463,9 @@ enum ethtool_fec_config_bits {#define ETHTOOL_GFECPARAM 0x00000050/* Get FEC settings */ #define ETHTOOL_SFECPARAM 0x00000051/* Set FEC settings */ +#define ETHTOOL_GFP 0x00000052/* Get Frame Preemption settings */ +#define ETHTOOL_SFP 0x00000053/* Set Frame Preemption settings */ +/* compatibility with older code */ #define SPARC_ETH_GSET ETHTOOL_GSET#define SPARC_ETH_SSET ETHTOOL_SSET
net/ethtool/ioctl.c +36 -0
@@ -2531,6 +2531,36 @@ static int ethtool_set_fecparam(struct net_device *dev, void __user *useraddr)return dev->ethtool_ops->set_fecparam (dev, &fecparam); } +static int ethtool_get_preempt (struct net_device *dev,void __user *useraddr) +{ +struct ethtool_fp fpparam = { .cmd =ETHTOOL_GFP }; +int rc; + +if (!dev->ethtool_ops->get_preempt) +return -EOPNOTSUPP ; + + rc = dev->ethtool_ops->get_preempt (dev, &fpparam); +if (rc) +return rc; + +if (copy_to_user (useraddr, &fpparam,sizeof (fpparam))) +return -EFAULT ; +return 0 ; +} + +static int ethtool_set_preempt (struct net_device *dev,void __user *useraddr) +{ +struct ethtool_fp fpparam; + +if (!dev->ethtool_ops->set_preempt) +return -EOPNOTSUPP ; + +if (copy_from_user (&fpparam, useraddr,sizeof (fpparam))) +return -EFAULT ; + +return dev->ethtool_ops->set_preempt (dev, &fpparam); +} +/* The main entry point in this file. Called from net/core/dev_ioctl.c */ int dev_ethtool (struct net *net,struct ifreq *ifr)
@@ -2810,6 +2840,12 @@ int dev_ethtool(struct net *net, struct ifreq *ifr)case ETHTOOL_SFECPARAM : rc =ethtool_set_fecparam (dev, useraddr);break ; +case ETHTOOL_GFP : + rc =ethtool_get_preempt (dev, useraddr); +break ; +case ETHTOOL_SFP : + rc =ethtool_set_preempt (dev, useraddr); +break ;default : rc = -EOPNOTSUPP ; }
Signature · 1 line
2.26.2
Re: [next-queue RFC 1/4] ethtool: Add support for configuring frame preemption
Hi Vinicius,
Quoted 32 lines · Vinicius Costa Gomes
Frame preemption (described in IEEE 802.3br-2016) defines the concept of preemptible and express queues. It allows traffic from express queues to "interrupt" traffic from preemptible queues, which are "resumed" after the express traffic has finished transmitting.
Frame preemption can only be used when both the local device and the link partner support it.
A new ethtool command was added to support the configuration parameters.
Signed-off-by Vinicius Costa Gomes <[email protected]>--- include/linux/ethtool.h | 6 ++++++ include/uapi/linux/ethtool.h | 25 +++++++++++++++++++++++++ net/ethtool/ioctl.c | 36 ++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+)include/linux/ethtool.h +2
-0@@ -360,6 +360,8 @@ bool ethtool_convert_link_mode_to_legacy_u32(u32 *legacy_u32,* @get_ethtool_phy_stats: Return extended statistics about the PHY device. * This is only useful if the device maintains PHY statistics and * cannot use the standard PHY library helpers. +* @get_preempt: Get the network device Frame Preemption parameters. +* @set_preempt: Set the network device Frame Preemption parameters. * * All operations are optional (i.e. the function pointer may be set * to %NULL) and callers must take this into account. Callers must
include/linux/ethtool.h struct ethtool_ops { · line 464
Quoted 9 lines
include/linux/ethtool.h +4
-0@@ -454,6 +456,10 @@ struct ethtool_ops { struct ethtool_fecparam *);int (*set_fecparam)(struct net_device *,struct ethtool_fecparam *); +int (*get_preempt)(struct net_device *, +struct ethtool_fp *); +int (*set_preempt)(struct net_device *, +struct ethtool_fp *);void (*get_ethtool_phy_stats)(struct net_device *,struct ethtool_stats *,u64 *);
I understand this series for IET Preemption. But want to see if we can also add EST parameter, queueMaxSDU, that is also configurable.
This is defined as up to 8 entries of queueMaxSDU (unsigned int) defined in 12.29.1.1,12.29.1.1.1 of 802.1Q 2018 edition. May be set_queue_max_sdu() with per traffic class queue value as an array of __u16 values?
Quoted 15 lines
include/linux/ethtool.h +0
-0};include/uapi/linux/ethtool.h +6
-0@@ -369,6 +369,28 @@ struct ethtool_eee {__u32 reserved[2 ]; }; +/** +* struct ethtool_fp - Frame Preemption information +* @cmd: ETHTOOL_{G,S}FP +* @fp_supported: If frame preemption is supported. +* @fp_enabled: If frame preemption should be advertised to the link partner +* as enabled.
include/uapi/linux/ethtool.h struct ethtool_eee { · line 387
Quoted 10 lines
include/uapi/linux/ethtool.h +10
-0+* @supported_queues_mask: Bitmask indicating which queues support being +* configured as preemptible (bit 0 -> queue 0, bit N -> queue N). +* @preemptible_queues_mask: Bitmask indicating which queues are +* configured as preemptible (bit 0 -> queue 0, bit N -> queue N). +* @min_frag_size: Minimum size for all non-final fragment size. +*/ +struct ethtool_fp { +__u32 cmd; +__u8 fp_supported; +__u8 fp_enabled;
Could we add verify_supported and verify_enabled?
Quoted 74 lines
include/uapi/linux/ethtool.h +9
-0+__u32 supported_queues_mask; +__u32 preemptible_queues_mask; +__u32 min_frag_size; +__u32 reserved[2 ]; +}; +/** * struct ethtool_modinfo - plugin module eeprom information * @cmd: %ETHTOOL_GMODULEINFO @@ -1441,6 +1463,9 @@ enum ethtool_fec_config_bits {#define ETHTOOL_GFECPARAM 0x00000050/* Get FEC settings */ #define ETHTOOL_SFECPARAM 0x00000051/* Set FEC settings */ +#define ETHTOOL_GFP 0x00000052/* Get Frame Preemption settings */ +#define ETHTOOL_SFP 0x00000053/* Set Frame Preemption settings */ +/* compatibility with older code */ #define SPARC_ETH_GSET ETHTOOL_GSET#define SPARC_ETH_SSET ETHTOOL_SSETnet/ethtool/ioctl.c +36
-0@@ -2531,6 +2531,36 @@ static int ethtool_set_fecparam(struct net_device *dev, void __user *useraddr)return dev->ethtool_ops->set_fecparam (dev, &fecparam); } +static int ethtool_get_preempt (struct net_device *dev,void __user *useraddr) +{ +struct ethtool_fp fpparam = { .cmd =ETHTOOL_GFP }; +int rc; + +if (!dev->ethtool_ops->get_preempt) +return -EOPNOTSUPP ; + + rc = dev->ethtool_ops->get_preempt (dev, &fpparam); +if (rc) +return rc; + +if (copy_to_user (useraddr, &fpparam,sizeof (fpparam))) +return -EFAULT ; +return 0 ; +} + +static int ethtool_set_preempt (struct net_device *dev,void __user *useraddr) +{ +struct ethtool_fp fpparam; + +if (!dev->ethtool_ops->set_preempt) +return -EOPNOTSUPP ; + +if (copy_from_user (&fpparam, useraddr,sizeof (fpparam))) +return -EFAULT ; + +return dev->ethtool_ops->set_preempt (dev, &fpparam); +} +/* The main entry point in this file. Called from net/core/dev_ioctl.c */ int dev_ethtool (struct net *net,struct ifreq *ifr)@@ -2810,6 +2840,12 @@ int dev_ethtool(struct net *net, struct ifreq *ifr)case ETHTOOL_SFECPARAM : rc =ethtool_set_fecparam (dev, useraddr);break ; +case ETHTOOL_GFP : + rc =ethtool_get_preempt (dev, useraddr); +break ; +case ETHTOOL_SFP : + rc =ethtool_set_preempt (dev, useraddr); +break ;default : rc = -EOPNOTSUPP ; }
Signature · 2 lines
Murali Karicheri Texas Instruments
RFC 2/4 ethtool: Add support for configuring frame preemption via netlink
In reply to Vinicius Costa Gomesethtool is gaining support for using netlink as transport for its messages, being an alternative to ioctl() calls.
Frame preemption, being new, makes a good target for being added to the list of features that are also supported via the netlink transport.
--- include/uapi/linux/ethtool_netlink.h | 19 +++ net/ethtool/Makefile | 3 +- net/ethtool/netlink.c | 15 +++ net/ethtool/netlink.h | 2 + net/ethtool/preempt.c | 181 +++++++++++++++++++++++++++ 5 files changed, 219 insertions(+), 1 deletion(-) create mode 100644 net/ethtool/preempt.c
include/uapi/linux/ethtool_netlink.h +19 -0
@@ -40,6 +40,8 @@ enum {ETHTOOL_MSG_EEE_SET ,ETHTOOL_MSG_TSINFO_GET ,ETHTOOL_MSG_CABLE_TEST_ACT , +ETHTOOL_MSG_PREEMPT_GET , +ETHTOOL_MSG_PREEMPT_SET ,/* add new constants above here */ __ETHTOOL_MSG_USER_CNT,
@@ -76,6 +78,8 @@ enum {ETHTOOL_MSG_EEE_NTF ,ETHTOOL_MSG_TSINFO_GET_REPLY ,ETHTOOL_MSG_CABLE_TEST_NTF , +ETHTOOL_MSG_PREEMPT_GET_REPLY , +ETHTOOL_MSG_PREEMPT_NTF ,/* add new constants above here */ __ETHTOOL_MSG_KERNEL_CNT,
@@ -476,6 +480,21 @@ enum { ETHTOOL_A_CABLE_TEST_NTF_MAX = (__ETHTOOL_A_CABLE_TEST_NTF_CNT - 1) }; +/* FRAME PREEMPTION */ +enum { +ETHTOOL_A_PREEMPT_UNSPEC , +ETHTOOL_A_PREEMPT_HEADER ,/* nest - _A_HEADER_* */ +ETHTOOL_A_PREEMPT_SUPPORTED ,/* u8 */ +ETHTOOL_A_PREEMPT_ACTIVE ,/* u8 */ +ETHTOOL_A_PREEMPT_MIN_FRAG_SIZE ,/* u32 */ +ETHTOOL_A_PREEMPT_QUEUES_SUPPORTED ,/* u32 */ +ETHTOOL_A_PREEMPT_QUEUES_PREEMPTIBLE ,/* u32 */ + +/* add new constants above here */ + __ETHTOOL_A_PREEMPT_CNT, +ETHTOOL_A_PREEMPT_MAX = (__ETHTOOL_A_PREEMPT_CNT -1 ) +}; +/* generic netlink info */ #define ETHTOOL_GENL_NAME "ethtool"#define ETHTOOL_GENL_VERSION 1
net/ethtool/Makefile +2 -1
@@ -6,4 +6,5 @@ obj-$(CONFIG_ETHTOOL_NETLINK) += ethtool_nl.o ethtool_nl-y := netlink.o bitset.o strset.o linkinfo.o linkmodes.o \ linkstate.o debug.o wol.o features.o privflags.o rings.o \ - channels.o coalesce.o pause.o eee.o tsinfo.o cabletest.o + channels.o coalesce.o pause.o eee.o tsinfo.o cabletest.o \ + preempt.o
net/ethtool/netlink.c +15 -0
@@ -231,6 +231,7 @@ ethnl_default_requests[__ETHTOOL_MSG_USER_CNT] = { [ETHTOOL_MSG_PAUSE_GET] = ðnl_pause_request_ops, [ETHTOOL_MSG_EEE_GET] = ðnl_eee_request_ops, [ETHTOOL_MSG_TSINFO_GET] = ðnl_tsinfo_request_ops, + [ETHTOOL_MSG_PREEMPT_GET] = ðnl_preempt_request_ops, };static struct ethnl_dump_ctx *ethnl_dump_context (struct netlink_callback *cb)
@@ -550,6 +551,7 @@ ethnl_default_notify_ops[ETHTOOL_MSG_KERNEL_MAX + 1] = { [ETHTOOL_MSG_COALESCE_NTF] = ðnl_coalesce_request_ops, [ETHTOOL_MSG_PAUSE_NTF] = ðnl_pause_request_ops, [ETHTOOL_MSG_EEE_NTF] = ðnl_eee_request_ops, + [ETHTOOL_MSG_PREEMPT_NTF] = ðnl_preempt_request_ops, };/* default notification handler */
@@ -642,6 +644,7 @@ static const ethnl_notify_handler_t ethnl_notify_handlers[] = { [ETHTOOL_MSG_COALESCE_NTF] = ethnl_default_notify, [ETHTOOL_MSG_PAUSE_NTF] = ethnl_default_notify, [ETHTOOL_MSG_EEE_NTF] = ethnl_default_notify, + [ETHTOOL_MSG_PREEMPT_NTF] = ethnl_default_notify, };void ethtool_notify (struct net_device *dev,unsigned int cmd,const void *data)
@@ -844,6 +847,18 @@ static const struct genl_ops ethtool_genl_ops[] = { .flags = GENL_UNS_ADMIN_PERM, .doit = ethnl_act_cable_test, }, + { + .cmd = ETHTOOL_MSG_PREEMPT_GET, + .doit = ethnl_default_doit, + .start = ethnl_default_start, + .dumpit = ethnl_default_dumpit, + .done = ethnl_default_done, + }, + { + .cmd = ETHTOOL_MSG_PREEMPT_SET, + .flags = GENL_UNS_ADMIN_PERM, + .doit = ethnl_set_preempt, + }, }; static const struct genl_multicast_group ethtool_nl_mcgrps[] = {
net/ethtool/netlink.h +2 -0
@@ -347,6 +347,7 @@ extern const struct ethnl_request_ops ethnl_coalesce_request_ops;extern const struct ethnl_request_ops ethnl_pause_request_ops;extern const struct ethnl_request_ops ethnl_eee_request_ops;extern const struct ethnl_request_ops ethnl_tsinfo_request_ops; +extern const struct ethnl_request_ops ethnl_preempt_request_ops;int ethnl_set_linkinfo (struct sk_buff *skb,struct genl_info *info);int ethnl_set_linkmodes (struct sk_buff *skb,struct genl_info *info);
@@ -360,5 +361,6 @@ int ethnl_set_coalesce(struct sk_buff *skb, struct genl_info *info);int ethnl_set_pause (struct sk_buff *skb,struct genl_info *info);int ethnl_set_eee (struct sk_buff *skb,struct genl_info *info);int ethnl_act_cable_test (struct sk_buff *skb,struct genl_info *info); +int ethnl_set_preempt (struct sk_buff *skb,struct genl_info *info);#endif /* _NET_ETHTOOL_NETLINK_H */
net/ethtool/preempt.c new file +181 -0
181 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [next-queue RFC 2/4] ethtool: Add support for configuring frame preemption via netlink
Hi Vinicius,
url: https://github.com/0day-ci/linux/commits/Vinicius-Costa-Gomes/ethtool-Add-support-for-frame-preemption/20200516-093235 base: https://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/next-queue.git dev-queue
If you fix the issue, kindly add following tag as appropriate
smatch warnings:net/ethtool/preempt.c:152 ethnl_set_preempt() warn: variable dereferenced before check 'info' (see line 127)
# https://github.com/0day-ci/linux/commit/5d130ebd7a21741d4fda1c8829a32353e10d49d5
git remote add linux-review https://github.com/0day-ci/linux
git remote update linux-review
git checkout 5d130ebd7a21741d4fda1c8829a32353e10d49d5
vim +/info +152 net/ethtool/preempt.c
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 118 int ethnl_set_preempt(struct sk_buff *skb, struct genl_info *info)
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 119 {
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 120 struct nlattr *tb[ETHTOOL_A_LINKINFO_MAX + 1];
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 121 struct ethtool_fp preempt = {};
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 122 struct ethnl_req_info req_info = {};
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 123 struct net_device *dev;
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 124 bool mod = false;
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 125 int ret;
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 126
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 @127 ret = nlmsg_parse(info->nlhdr, GENL_HDRLEN, tb,
^^^^^^^^^^^Unchecked dereference.
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 128 ETHTOOL_A_PREEMPT_MAX, preempt_set_policy,
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 129 info->extack);
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 130 if (ret < 0)
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 131 return ret;
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 132
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 133 ret = ethnl_parse_header_dev_get(&req_info,
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 134 tb[ETHTOOL_A_PREEMPT_HEADER],
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 135 genl_info_net(info), info->extack,
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 136 true);
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 137 if (ret < 0)
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 138 return ret;
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 139 dev = req_info.dev;
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 140 ret = -EOPNOTSUPP;
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 141 if (!dev->ethtool_ops->get_preempt ||
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 142 !dev->ethtool_ops->set_preempt)
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 143 goto out_dev;
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 144
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 145 rtnl_lock();
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 146 ret = ethnl_ops_begin(dev);
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 147 if (ret < 0)
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 148 goto out_rtnl;
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 149
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 150 ret = dev->ethtool_ops->get_preempt(dev, &preempt);
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 151 if (ret < 0) {
5d130ebd7a2174 Vinicius Costa Gomes 2020-05-15 @152 if (info)
^^^^^^^^^Check is too late to help.
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]
RFC 3/4 igc: Add support for configuring frame preemption
In reply to Vinicius Costa GomesWIP
--- drivers/net/ethernet/intel/igc/igc.h | 3 + drivers/net/ethernet/intel/igc/igc_defines.h | 6 ++ drivers/net/ethernet/intel/igc/igc_ethtool.c | 68 ++++++++++++++++++++ drivers/net/ethernet/intel/igc/igc_tsn.c | 46 +++++++++++-- 4 files changed, 118 insertions(+), 5 deletions(-)
drivers/net/ethernet/intel/igc/igc.h +3 -0
@@ -87,6 +87,7 @@ struct igc_ring {u8 queue_index;/* logical index of the ring*/ u8 reg_idx;/* physical index of the ring */ bool launchtime_enable;/* true if LaunchTime is enabled */ +bool preemptible;/* true if not express */ u32 start_time;u32 end_time;
@@ -162,6 +163,8 @@ struct igc_adapter {ktime_t base_time;ktime_t cycle_time; +bool frame_preemption_active; +u32 min_frag_size;/* Frame preemption minimum fragment size */ /* OS defined structs */ struct pci_dev *pdev;
drivers/net/ethernet/intel/igc/igc_defines.h +6 -0
@@ -364,6 +364,8 @@#define IGC_RXPBS_CFG_TS_EN 0x80000000/* Timestamp in Rx buffer */ #define IGC_TXPBSIZE_TSN 0x04145145/* 5k bytes buffer for each queue */ +#define IGC_RXPBSIZE_TSN 0x00010090/* 16KB for EXP + 16KB for BE + 2KB for BMC */ +#define IGC_RXPBSIZE_SIZE_MASK 0x0001FFFF#define IGC_DTXMXPKTSZ_TSN 0x19/* 1600 bytes of max TX DMA packet size */ #define IGC_DTXMXPKTSZ_DEFAULT 0x98/* 9728-byte Jumbo frames */
@@ -422,10 +424,14 @@/* Transmit Scheduling */ #define IGC_TQAVCTRL_TRANSMIT_MODE_TSN 0x00000001#define IGC_TQAVCTRL_ENHANCED_QAV 0x00000008 +#define IGC_TQAVCTRL_PREEMPT_ENA 0x00000002 +#define IGC_TQAVCTRL_MIN_FRAG_MASK 0x0000C000 +#define IGC_TQAVCTRL_MIN_FRAG_SHIFT 14#define IGC_TXQCTL_QUEUE_MODE_LAUNCHT 0x00000001#define IGC_TXQCTL_STRICT_CYCLE 0x00000002#define IGC_TXQCTL_STRICT_END 0x00000004 +#define IGC_TXQCTL_PREEMPTABLE 0x00000008/* Receive Checksum Control */ #define IGC_RXCSUM_CRCOFL 0x00000800/* CRC32 offload enable */
drivers/net/ethernet/intel/igc/igc_ethtool.c +68 -0
86 lines not on this page · raw message
drivers/net/ethernet/intel/igc/igc_tsn.c +41 -5
101 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [next-queue RFC 3/4] igc: Add support for configuring frame preemption
Hi Vinicius,
Quoted 2 lines · Vinicius Costa Gomes
WIP
- cut -
Quoted 19 lines
/* forward declaration */ struct igc_stats {drivers/net/ethernet/intel/igc/igc_ethtool.c +12
-0@@ -1549,6 +1550,71 @@ static int igc_ethtool_set_priv_flags(struct net_device *netdev, u32 priv_flags)return 0 ; } +static int igc_ethtool_get_preempt (struct net_device *netdev, +struct ethtool_fp *fpcmd) +{ +struct igc_adapter *adapter =netdev_priv (netdev); +int i; + + fpcmd->fp_supported =1 ; + fpcmd->fp_enabled = adapter->frame_preemption_active; + fpcmd->min_frag_size = adapter->min_frag_size; + +for (i =0 ; i < adapter->num_tx_queues; i++) { +struct igc_ring *ring = adapter->tx_ring[i];
drivers/net/ethernet/intel/igc/igc_ethtool.c static int igc_ethtool_set_priv_flags(struct net_device *netdev, u32 priv_flags) · line 1574
Quoted 10 lines
drivers/net/ethernet/intel/igc/igc_ethtool.c +10
-0+ + fpcmd->supported_queues_mask |=BIT (i); + +if (ring->preemptible) + fpcmd->preemptible_queues_mask |=BIT (i); + } + +return 0 ; +} +
Is this something that can be provided by the driver when netdevice is registered so that a common function at net core layer be implemented instead of duplicating this in individual device drivers?
drivers/net/ethernet/intel/igc/igc_ethtool.c static int igc_ethtool_set_priv_flags(struct net_device *netdev, u32 priv_flags) · line 1587
Quoted 13 lines
drivers/net/ethernet/intel/igc/igc_ethtool.c +13
-0+static int igc_ethtool_set_preempt (struct net_device *netdev, +struct ethtool_fp *fpcmd) +{ +struct igc_adapter *adapter =netdev_priv (netdev); +int i; + +/* The formula is (Section 8.12.4 of the datasheet): +* MIN_FRAG_SIZE = 4 + (1 + MIN_FRAG)*64 +* MIN_FRAG is represented by two bits, so we can only have +* min_frag_size between 68 and 260. +*/ +if (fpcmd->min_frag_size <68 || fpcmd->min_frag_size >260 ) +return -EINVAL ;
- cut-
Signature · 2 lines
Murali Karicheri Texas Instruments
RFC 4/4 igc: Add support for exposing frame preemption stats registers
In reply to Vinicius Costa Gomes[WIP]
--- drivers/net/ethernet/intel/igc/igc_ethtool.c | 9 +++++++++ drivers/net/ethernet/intel/igc/igc_regs.h | 10 ++++++++++ 2 files changed, 19 insertions(+)
drivers/net/ethernet/intel/igc/igc_ethtool.c +9 -0
@@ -322,6 +322,15 @@ static void igc_ethtool_get_regs(struct net_device *netdev,for (i =0 ; i <8 ; i++) regs_buff[205 + i] =rd32 (IGC_ETQF (i)); + + regs_buff[214 ] =rd32 (IGC_PRMPTDTCNT ); + regs_buff[215 ] =rd32 (IGC_PRMEVNTTCNT ); + regs_buff[216 ] =rd32 (IGC_PRMPTDRCNT ); + regs_buff[217 ] =rd32 (IGC_PRMEVNTRCNT ); + regs_buff[218 ] =rd32 (IGC_PRMPBLTCNT ); + regs_buff[219 ] =rd32 (IGC_PRMPBLRCNT ); + regs_buff[220 ] =rd32 (IGC_PRMEXPTCNT ); + regs_buff[221 ] =rd32 (IGC_PRMEXPRCNT ); } static void igc_ethtool_get_wol(struct net_device *netdev,
drivers/net/ethernet/intel/igc/igc_regs.h +10 -0
@@ -211,6 +211,16 @@#define IGC_FTQF (_n) (0x059E0 + (4 * (_n)))/* 5-tuple Queue Fltr */ +/* Time sync registers - preemption statistics */ +#define IGC_PRMPTDTCNT 0x04280/* Good TX Preempted Packets */ +#define IGC_PRMEVNTTCNT 0x04298/* TX Preemption event counter */ +#define IGC_PRMPTDRCNT 0x04284/* Good RX Preempted Packets */ +#define IGC_PRMEVNTRCNT 0x0429C/* RX Preemption event counter */ +#define IGC_PRMPBLTCNT 0x04288/* Good TX Preemptable Packets */ +#define IGC_PRMPBLRCNT 0x0428C/* Good RX Preemptable Packets */ +#define IGC_PRMEXPTCNT 0x04290/* Good TX Express Packets */ +#define IGC_PRMEXPRCNT 0x042A0/* Preemption Exception Counter */ +/* Transmit Scheduling Registers */ #define IGC_TQAVCTRL 0x3570#define IGC_TXQCTL (_n) (0x3344 + 0x4 * (_n))
Signature · 1 line
2.26.2
Re: [next-queue RFC 4/4] igc: Add support for exposing frame preemption stats registers
Hi Vinicious,
Quoted 40 lines · Vinicius Costa Gomes
[WIP]
Signed-off-by Vinicius Costa Gomes <[email protected]>--- drivers/net/ethernet/intel/igc/igc_ethtool.c | 9 +++++++++ drivers/net/ethernet/intel/igc/igc_regs.h | 10 ++++++++++ 2 files changed, 19 insertions(+)drivers/net/ethernet/intel/igc/igc_ethtool.c +9
-0@@ -322,6 +322,15 @@ static void igc_ethtool_get_regs(struct net_device *netdev,for (i =0 ; i <8 ; i++) regs_buff[205 + i] =rd32 (IGC_ETQF (i)); + + regs_buff[214 ] =rd32 (IGC_PRMPTDTCNT ); + regs_buff[215 ] =rd32 (IGC_PRMEVNTTCNT ); + regs_buff[216 ] =rd32 (IGC_PRMPTDRCNT ); + regs_buff[217 ] =rd32 (IGC_PRMEVNTRCNT ); + regs_buff[218 ] =rd32 (IGC_PRMPBLTCNT ); + regs_buff[219 ] =rd32 (IGC_PRMPBLRCNT ); + regs_buff[220 ] =rd32 (IGC_PRMEXPTCNT ); + regs_buff[221 ] =rd32 (IGC_PRMEXPRCNT ); } static void igc_ethtool_get_wol(struct net_device *netdev,drivers/net/ethernet/intel/igc/igc_regs.h +3
-0@@ -211,6 +211,16 @@#define IGC_FTQF (_n) (0x059E0 + (4 * (_n)))/* 5-tuple Queue Fltr */ +/* Time sync registers - preemption statistics */ +#define IGC_PRMPTDTCNT 0x04280/* Good TX Preempted Packets */ +#define IGC_PRMEVNTTCNT 0x04298/* TX Preemption event counter */
drivers/net/ethernet/intel/igc/igc_regs.h line 226
Quoted 10 lines
drivers/net/ethernet/intel/igc/igc_regs.h +7
-0+#define IGC_PRMPTDRCNT 0x04284/* Good RX Preempted Packets */ +#define IGC_PRMEVNTRCNT 0x0429C/* RX Preemption event counter */ +#define IGC_PRMPBLTCNT 0x04288/* Good TX Preemptable Packets */ +#define IGC_PRMPBLRCNT 0x0428C/* Good RX Preemptable Packets */ +#define IGC_PRMEXPTCNT 0x04290/* Good TX Express Packets */ +#define IGC_PRMEXPRCNT 0x042A0/* Preemption Exception Counter */ +/* Transmit Scheduling Registers */ #define IGC_TQAVCTRL 0x3570#define IGC_TXQCTL (_n) (0x3344 + 0x4 * (_n))
There are some statistics supported by AM65 CPSW as well. So do you plan to add this to ethtool stats that is dumped using ethtool -S option?
Thanks and regards,
Signature · 2 lines
Murali Karicheri Texas Instruments
Quoted 15 lines · Vinicius Costa Gomes
Hi,
This series adds support for configuring frame preemption, as defined by IEEE 802.1Q-2018 (previously IEEE 802.1Qbu) and IEEE 802.3br.
Frame preemption allows a packet from a higher priority queue marked as "express" to preempt a packet from lower priority queue marked as "preemptible". The idea is that this can help reduce the latency for higher priority traffic.
Previously, the proposed interface for configuring these features was using the qdisc layer. But as this is very hardware dependent and all that qdisc did was pass the information to the driver, it makes sense to have this in ethtool.
Quoted 5 lines
One example, for retrieving and setting the configuration:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported
IMHO we don't need a special bool for this. IIUC this is not a state flag that would change value for a particular device; either the device supports the feature or it does not. If it does not, the ethtool_ops callbacks would return -EOPNOTSUPP (or would not even exist if the driver has no support) and ethtool would say so.
Quoted 8 lines
active: active supported queues: 0xf supported queues: 0xe minimum fragment size: 68$ ethtool --set-frame-preemption enp3s0 fp on min-frag-size 68 preemptible-queues-mask 0xe
Quoted 8 lines
This is a RFC because I wanted to have feedback on some points:
- The parameters added are enough for the hardware I have, is it enough in general?
- even with the ethtool via netlink effort, I chose to keep the
ioctl()way, in case someone wants to backport this to an older kernel, is there a problem with this?
I would prefer not extending ioctl interface with new features, with obvious exceptions like adding new link modes or so. Not only because having new features only available through netlink will motivate authors of userspace tools to support netlink but mostly because the lack of flexibility and extensibility of ioctl interface inevitably leads to compromises you wouldn't have to do if you only implement netlink requests.
One example I can see is the use of u32 for queue bitmaps. Perhaps you don't expect this feature to be supported on devices with more than 32 queues (and I don't have enough expertise to tell if it's justified at the moment) but can you be sure it will be the case in 10 or 20 years? As long as these hardcoded u32 bitmaps are only part of internal kernel API (ethtool_ops), extending the support for bigger devices will mean some code churn (possibly large if many drivers implement the feature) but it's something that can be done. But if you have this limit in userspace API, you are in a much bigger trouble. The same can be said for adding new attributes - easy with netlink but with ioctl you never know if those reserved fields will suffice.
Quoted 7 lines
- Some space for bikeshedding the names and location (for example, does it make sense for these settings to be per-queue?), as I am not quite happy with them, one example, is the use of preemptible vs. preemptable;
Quoted 7 lines
About the patches, should be quite straightforward:
Patch 1, adds the
ETHTOOL_GFPandETHOOL_SFPcommands and the associated data structures;Patch 2, adds the
ETHTOOL_MSG_PREEMPT_GETandETHTOOL_MSG_PREEMPT_SETnetlink messages and the associated attributes;
I didn't look too deeply but one thing I noticed is that setting the parameters using ioctl() does not trigger netlink notification. If we decide to implement ioctl support (and I'm not a fan of that), the notifications should be sent even when ioctl is used.
Michal
Hi,
Quoted 22 lines · Michal Kubecek
Quoted 15 lines · Vinicius Costa Gomes
Hi,
This series adds support for configuring frame preemption, as defined by IEEE 802.1Q-2018 (previously IEEE 802.1Qbu) and IEEE 802.3br.
Frame preemption allows a packet from a higher priority queue marked as "express" to preempt a packet from lower priority queue marked as "preemptible". The idea is that this can help reduce the latency for higher priority traffic.
Previously, the proposed interface for configuring these features was using the qdisc layer. But as this is very hardware dependent and all that qdisc did was pass the information to the driver, it makes sense to have this in ethtool.
Quoted 5 lines
One example, for retrieving and setting the configuration:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported
Quoted 5 lines
IMHO we don't need a special bool for this. IIUC this is not a state flag that would change value for a particular device; either the device supports the feature or it does not. If it does not, the
ethtool_opscallbacks would return-EOPNOTSUPP(or would not even exist if the driver has no support) and ethtool would say so.
(I know that the comments below only apply if "ethtool-way" is what's decided)
Cool. Will remove the supported bit.
Quoted 18 lines
Quoted 9 lines
active: active supported queues: 0xf supported queues: 0xe minimum fragment size: 68$ ethtool --set-frame-preemption enp3s0 fp on min-frag-size 68 preemptible-queues-mask 0xeQuoted 8 lines
This is a RFC because I wanted to have feedback on some points:
- The parameters added are enough for the hardware I have, is it enough in general?
- even with the ethtool via netlink effort, I chose to keep the
ioctl()way, in case someone wants to backport this to an older kernel, is there a problem with this?
Quoted 7 lines
I would prefer not extending ioctl interface with new features, with obvious exceptions like adding new link modes or so. Not only because having new features only available through netlink will motivate authors of userspace tools to support netlink but mostly because the lack of flexibility and extensibility of ioctl interface inevitably leads to compromises you wouldn't have to do if you only implement netlink requests.
Agreed. Will send the next version with only the netlink interface, and let's see who complains.
Quoted 12 lines
One example I can see is the use of u32 for queue bitmaps. Perhaps you don't expect this feature to be supported on devices with more than 32 queues (and I don't have enough expertise to tell if it's justified at the moment) but can you be sure it will be the case in 10 or 20 years? As long as these hardcoded u32 bitmaps are only part of internal kernel API (
ethtool_ops), extending the support for bigger devices will mean some code churn (possibly large if many drivers implement the feature) but it's something that can be done. But if you have this limit in userspace API, you are in a much bigger trouble. The same can be said for adding new attributes - easy with netlink but with ioctl you never know if those reserved fields will suffice.
A bit of background for this decision (using u32), frame preemption has dimishing returns in relation to link speeds, my gut feeling is that for links faster than 2.5G it stops making sense, at least in Linux, the measurement noise will hide any latency improvement brought by frame preemption. And I don't see many 2.5G NICs supporting more than 32 queues.
But I agree that keeping the interface future proof is better. Will change to expose the queues configuration as bitset.
Quoted 13 lines
Quoted 13 lines
- Some space for bikeshedding the names and location (for example, does it make sense for these settings to be per-queue?), as I am not quite happy with them, one example, is the use of preemptible vs. preemptable;
About the patches, should be quite straightforward:
Patch 1, adds the
ETHTOOL_GFPandETHOOL_SFPcommands and the associated data structures;
Quoted 7 lines
Quoted 2 lines
Patch 2, adds the
ETHTOOL_MSG_PREEMPT_GETandETHTOOL_MSG_PREEMPT_SETnetlink messages and the associated attributes;I didn't look too deeply but one thing I noticed is that setting the parameters using
ioctl()does not trigger netlink notification. If we decide to implement ioctl support (and I'm not a fan of that), the notifications should be sent even when ioctl is used.
Oh, yeah, that's right. Nice catch.
Cheers,
Signature · 1 line
Vinicius
Hi,
Quoting Vinicius Costa Gomes (2020-05-18 12:34:22)
Quoted 32 lines
Hi,
Quoted 22 lines · Michal Kubecek
Quoted 15 lines · Vinicius Costa Gomes
Hi,
This series adds support for configuring frame preemption, as defined by IEEE 802.1Q-2018 (previously IEEE 802.1Qbu) and IEEE 802.3br.
Frame preemption allows a packet from a higher priority queue marked as "express" to preempt a packet from lower priority queue marked as "preemptible". The idea is that this can help reduce the latency for higher priority traffic.
Previously, the proposed interface for configuring these features was using the qdisc layer. But as this is very hardware dependent and all that qdisc did was pass the information to the driver, it makes sense to have this in ethtool.
Quoted 5 lines
One example, for retrieving and setting the configuration:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supportedQuoted 5 lines
IMHO we don't need a special bool for this. IIUC this is not a state flag that would change value for a particular device; either the device supports the feature or it does not. If it does not, the
ethtool_opscallbacks would return-EOPNOTSUPP(or would not even exist if the driver has no support) and ethtool would say so.
Quoted 8 lines
(I know that the comments below only apply if "ethtool-way" is what's decided)
Cool. Will remove the supported bit.
Quoted 3 lines
Quoted 3 lines
active: active supported queues: 0xf
Following the same rationale, is this 'supported queue' going aways as well?
- Andre
Quoted 6 lines · Andre Guedes
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
active: active supported queues: 0xfFollowing the same rationale, is this 'supported queue' going aways as well?
I think so, with good error messages, when trying to set an express-only queue as preemptible, no need to expose this information.
Signature · 1 line
Vinicius
David Miller
maintainerFrom: Vinicius Costa Gomes <[email protected]>
Date: Fri, 15 May 2020 18:29:44 -0700
Quoted 7 lines
This series adds support for configuring frame preemption, as defined by IEEE 802.1Q-2018 (previously IEEE 802.1Qbu) and IEEE 802.3br.
Frame preemption allows a packet from a higher priority queue marked as "express" to preempt a packet from lower priority queue marked as "preemptible". The idea is that this can help reduce the latency for higher priority traffic.
Why do we need yet another name for something which is just basic traffic prioritization and why is this configured via ethtool instead of the "traffic classifier" which is where all of this stuff should be done?
Hi David,
Quoted 12 lines · David Miller
From: Vinicius Costa Gomes <[email protected]>
Date: Fri, 15 May 2020 18:29:44 -0700Quoted 7 lines
This series adds support for configuring frame preemption, as defined by IEEE 802.1Q-2018 (previously IEEE 802.1Qbu) and IEEE 802.3br.
Frame preemption allows a packet from a higher priority queue marked as "express" to preempt a packet from lower priority queue marked as "preemptible". The idea is that this can help reduce the latency for higher priority traffic.
Quoted 4 lines
Why do we need yet another name for something which is just basic traffic prioritization and why is this configured via ethtool instead of the "traffic classifier" which is where all of this stuff should be done?
It is not 'just another name for basic traffic prioritization'. With basic traffic prioritization only, a high-priority packet still has to wait in the egress queue of a switch until a (potentially large) low-priority packet has finished transmission and has freed the medium. Frame preemption changes that. Actually it requires hardware support on both ends, because the way it is transmitted on the wire is not compatible with regular Ethernet frames (it uses a special Start Of Frame Delimiter to encode preemptible traffic).
I know we are talking about ridiculously low improvements in latency, but the background is that Ethernet is making its way into the industrial and process control fields, and for that type of application you need to ensure minimally low and maximally consistent end-to-end latencies. Frame preemption helps with the "minimally low" part. The way it works is that typically there are 2 MACs per interface (1 is "express" - equivalent to the legacy type, and the other is "preemptible" - the new type) and this new IEEE 802.1Q clause thing allows some arbitration/preemption event to happen between the two MACs. When a preemption event happens, the preemptible MAC quickly wraps up and aborts the frame it's currently transmitting (to come back and continue later), making room for the express MAC to do its thing because it's time-constrained. Then, after the express MAC finishes, the preemptible MAC continues with the rest of the frame fragments from where it was preempted.
As to why this doesn't go to tc but to ethtool: why would it go to tc? You can't emulate such behavior in software. It's a hardware feature. You only* (more or less) need to specify which traffic classes on a port go to the preemptible MAC and which go to the express MAC. We discussed about the possibility of extending tc-taprio to configure frame preemption through it, but the consensus was that somebody might want to use frame preemption as a standalone feature, without scheduled traffic, and that inventing another qdisc for frame preemption alone would be too much of a formalism. (I hope I didn't omit anything important from the previous discussion on the topic)
Thanks,
-Vladimir
David Miller
maintainerFrom: Vladimir Oltean <[email protected]>
Date: Sun, 17 May 2020 00:03:39 +0300
Quoted 1 line
As to why this doesn't go to tc but to ethtool: why would it go to tc?
Maybe you can't %100 duplicate the on-the-wire special format and whatever, but the queueing behavior ABSOLUTELY you can emulate in software.
And then you have the proper hooks added for HW offload which can do the on-the-wire stuff.
That's how we do these things, not with bolted on ethtool stuff.
Quoted 6 lines · David Miller
From: Vladimir Oltean <[email protected]>
Date: Sun, 17 May 2020 00:03:39 +0300Quoted 1 line
As to why this doesn't go to tc but to ethtool: why would it go to tc?
Quoted 8 lines
Maybe you can't %100 duplicate the on-the-wire special format and whatever, but the queueing behavior ABSOLUTELY you can emulate in software.
And then you have the proper hooks added for HW offload which can do the on-the-wire stuff.
That's how we do these things, not with bolted on ethtool stuff.
When talking about frame preemption in the way that it is defined in 802.1Qbu and 802.3br, it says or assumes nothing about queuing. It describes the fact that there are 2 MACs per interface, 1 MAC dealing with some traffic classes and the other dealing with the rest. Queuing sits completely above the layer where frame preemption happens:
- The queuing layer does not care if packets go to a traffic class that is serviced by a preemptible MAC or an express MAC
- The MAC does not care by what means have packets been classified to one traffic class or another.
I have no idea what emulation of queuing behavior you are talking about. Frame preemption is a MAC hardware feature. Your NIC supports it or it doesn't. Which means it can talk to a link partner that supports frame preemption, or it can't.
Thanks,
-Vladimir
Andrew Lunn
maintainerQuoted 14 lines · Vladimir Oltean
Quoted 13 lines · David Miller
From: Vladimir Oltean <[email protected]>
Date: Sun, 17 May 2020 00:03:39 +0300Quoted 1 line
As to why this doesn't go to tc but to ethtool: why would it go to tc?
Maybe you can't %100 duplicate the on-the-wire special format and whatever, but the queueing behavior ABSOLUTELY you can emulate in software.
And then you have the proper hooks added for HW offload which can do the on-the-wire stuff.
Quoted 6 lines
Quoted 1 line
That's how we do these things, not with bolted on ethtool stuff.
When talking about frame preemption in the way that it is defined in 802.1Qbu and 802.3br, it says or assumes nothing about queuing. It describes the fact that there are 2 MACs per interface, 1 MAC dealing with some traffic classes and the other dealing with the rest.
I did not follow the previous discussion, but i assume you talked about modelling it in Linux as two MACs? Why was that approach not followed?
Andrew
Hi Andrew,
Quoted 23 lines · Andrew Lunn
Quoted 14 lines · Vladimir Oltean
Quoted 13 lines · David Miller
From: Vladimir Oltean <[email protected]>
Date: Sun, 17 May 2020 00:03:39 +0300Quoted 1 line
As to why this doesn't go to tc but to ethtool: why would it go to tc?
Maybe you can't %100 duplicate the on-the-wire special format and whatever, but the queueing behavior ABSOLUTELY you can emulate in software.
And then you have the proper hooks added for HW offload which can do the on-the-wire stuff.
Quoted 6 lines
Quoted 1 line
That's how we do these things, not with bolted on ethtool stuff.
When talking about frame preemption in the way that it is defined in 802.1Qbu and 802.3br, it says or assumes nothing about queuing. It describes the fact that there are 2 MACs per interface, 1 MAC dealing with some traffic classes and the other dealing with the rest.
Quoted 5 lines
I did not follow the previous discussion, but i assume you talked about modelling it in Linux as two MACs? Why was that approach not followed?
Andrew
I don't recall having discussed that option, but I guess that you can see why, if I quote this portion from IEEE 802.1Q-2018:
6.7.1 Support of the ISS by IEEE Std 802.3 (Ethernet) Mapping between M_CONTROL.requests/indications and IEEE 802.3 MA_CONTROL.requests/indications is performed as specified in IEEE Std 802.1AC. If the MAC supports IEEE 802.3br (TM) Interspersing Express Traffic, then PFC M_CONTROL.requests are mapped onto the MAC control interface associated with the express MAC (eMAC).
If frame preemption (6.7.2) is supported on a Port, then the IEEE 802.3 MAC provides the following two MAC service interfaces (99.4 of IEEE Std 802.3br (TM)-2016 [B21]):
a) A preemptible MAC (pMAC) service interface
b) An express MAC (eMAC) service interface
For priority values that are identified in the frame preemption status table (6.7.2) as preemptible, frames that are selected for transmission shall be transmitted using the pMAC service instance, and for priority values that are identified in the frame preemption status table as express, frames that are selected for transmission shall be transmitted using the eMAC service instance.
_In all other respects, the Port behaves as if it is supported by a single MAC service interface_ (emphasis mine). In particular, all frames received by the Port are treated as if they were received on a single MAC service interface regardless of whether they were received on the eMAC service interface or the pMAC service interface, except with respect to frame preemption.
Thanks,
-Vladimir
Hi,
Quoted 8 lines · David Miller
From: Vladimir Oltean <[email protected]>
Date: Sun, 17 May 2020 00:03:39 +0300Quoted 1 line
As to why this doesn't go to tc but to ethtool: why would it go to tc?
Maybe you can't %100 duplicate the on-the-wire special format and whatever, but the queueing behavior ABSOLUTELY you can emulate in software.
Just saying what Vladimir said in different words: the queueing behavior is already implemented in software, by mqprio or taprio, for example.
That is to say, if we add frame preemption support to those qdiscs all they will do is pass the information to the driver, and that's it. They won't be able to use that information at all.
The mental model I have for this feature is that is more similar to the segmentation offloads, energy efficient ethernet or auto-negotiation than it is to a traffic shaper like CBS.
Quoted 5 lines
And then you have the proper hooks added for HW offload which can do the on-the-wire stuff.
That's how we do these things, not with bolted on ethtool stuff.
Cheers,
Signature · 1 line
Vinicius
Jakub Kicinski
maintainerQuoted 13 lines · On Mon, 18 May 2020 12:05:04 -0700 Vinicius Costa Gomes wrote:
Quoted 8 lines · David Miller
From: Vladimir Oltean <[email protected]>
Date: Sun, 17 May 2020 00:03:39 +0300Quoted 1 line
As to why this doesn't go to tc but to ethtool: why would it go to tc?
Maybe you can't %100 duplicate the on-the-wire special format and whatever, but the queueing behavior ABSOLUTELY you can emulate in software.
Just saying what Vladimir said in different words: the queueing behavior is already implemented in software, by mqprio or taprio, for example.
Quoted 7 lines
That is to say, if we add frame preemption support to those qdiscs all they will do is pass the information to the driver, and that's it. They won't be able to use that information at all.
The mental model I have for this feature is that is more similar to the segmentation offloads, energy efficient ethernet or auto-negotiation than it is to a traffic shaper like CBS.
Please take a look at the example from the cover letter:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: active supported queues: 0xf supported queues: 0xe minimum fragment size: 68
Reading this I have no idea what 0xe is. I have to go and query TC API to see what priorities and queues that will be. Which IMHO is a strong argument that this information belongs there in the first place.
Hi,
Quoted 11 lines · Jakub Kicinski
Please take a look at the example from the cover letter:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: active supported queues: 0xf supported queues: 0xe minimum fragment size: 68
Quoted 3 lines
Reading this I have no idea what
0xeis. I have to go and query TC API to see what priorities and queues that will be. Which IMHO is a strong argument that this information belongs there in the first place.
That was the (only?) strong argument in favor of having frame preemption in the TC side when this was last discussed.
We can have a hybrid solution, we can move the express/preemptible per queue map to mqprio/taprio/whatever. And have the more specific configuration knobs, minimum fragment size, etc, in ethtool.
What do you think?
Cheers,
Signature · 1 line
Vinicius
Jakub Kicinski
maintainerQuoted 16 lines · On Mon, 18 May 2020 15:06:26 -0700 Vinicius Costa Gomes wrote:
Quoted 11 lines · Jakub Kicinski
Please take a look at the example from the cover letter:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: active supported queues: 0xf supported queues: 0xe minimum fragment size: 68Quoted 3 lines
Reading this I have no idea what
0xeis. I have to go and query TC API to see what priorities and queues that will be. Which IMHO is a strong argument that this information belongs there in the first place.
Quoted 8 lines
That was the (only?) strong argument in favor of having frame preemption in the TC side when this was last discussed.
We can have a hybrid solution, we can move the express/preemptible per queue map to mqprio/taprio/whatever. And have the more specific configuration knobs, minimum fragment size, etc, in ethtool.
What do you think?
Does the standard specify minimum fragment size as a global MAC setting?
Quoted 10 lines · Jakub Kicinski
Quoted 8 lines
That was the (only?) strong argument in favor of having frame preemption in the TC side when this was last discussed.
We can have a hybrid solution, we can move the express/preemptible per queue map to mqprio/taprio/whatever. And have the more specific configuration knobs, minimum fragment size, etc, in ethtool.
What do you think?
Does the standard specify minimum fragment size as a global MAC setting?
Yes, it's a per-MAC setting, not per-queue.
Signature · 1 line
Vinicius
Jakub Kicinski
maintainerQuoted 8 lines · On Mon, 18 May 2020 16:05:08 -0700 Vinicius Costa Gomes wrote:
Quoted 7 lines · Jakub Kicinski
Quoted 7 lines
That was the (only?) strong argument in favor of having frame preemption in the TC side when this was last discussed.
We can have a hybrid solution, we can move the express/preemptible per queue map to mqprio/taprio/whatever. And have the more specific configuration knobs, minimum fragment size, etc, in ethtool.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
What do you think?
Does the standard specify minimum fragment size as a global MAC setting?
Yes, it's a per-MAC setting, not per-queue.
If standard defines it as per-MAC and we can reasonably expect vendors won't try to "add value" and make it per queue (unlikely here AFAIU), then for this part ethtool configuration seems okay to me.
Hi,
Quoting Jakub Kicinski (2020-05-18 16:09:06)
Quoted 11 lines · On Mon, 18 May 2020 16:05:08 -0700 Vinicius Costa Gomes wrote:
Quoted 10 lines
Quoted 9 lines · Jakub Kicinski
Quoted 8 lines
That was the (only?) strong argument in favor of having frame preemption in the TC side when this was last discussed.
We can have a hybrid solution, we can move the express/preemptible per queue map to mqprio/taprio/whatever. And have the more specific configuration knobs, minimum fragment size, etc, in ethtool.
What do you think?
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Does the standard specify minimum fragment size as a global MAC setting?
Yes, it's a per-MAC setting, not per-queue.
If standard defines it as per-MAC and we can reasonably expect vendors won't try to "add value" and make it per queue (unlikely here AFAIU), then for this part ethtool configuration seems okay to me.
Before we move forward with this hybrid approach, let's recap a few points that we discussed in the previous thread and make sure it addresses them properly.
1) Frame Preemption (FP) can be enabled without EST, as described in IEEE 802.1Q. In this case, the user has to create a dummy EST schedule in taprio just to be able to enable FP, which doesn't look natural.
2) Mpqrio already looks overloaded. Besides mapping traffic classes into hardware queues, it also supports different modes and traffic shaping. Do we want to add yet another setting to it?
Regards,
Andre
Quoted 7 lines · Andre Guedes
Quoted 3 lines
If standard defines it as per-MAC and we can reasonably expect vendors won't try to "add value" and make it per queue (unlikely here AFAIU), then for this part ethtool configuration seems okay to me.
Before we move forward with this hybrid approach, let's recap a few points that we discussed in the previous thread and make sure it addresses them properly.
Thanks for bringing them up.
Quoted 4 lines
1) Frame Preemption (FP) can be enabled without EST, as described in IEEE 802.1Q. In this case, the user has to create a dummy EST schedule in taprio just to be able to enable FP, which doesn't look natural.
What I meant by "dummy" schedule, is to configure taprio without specifying any "sched-entry". And since we have support for adding schedules during runtime, this might be even useful in general.
Quoted 4 lines
2) Mpqrio already looks overloaded. Besides mapping traffic classes into hardware queues, it also supports different modes and traffic shaping. Do we want to add yet another setting to it?
I also don't see this as a problem. The parameters that mqprio has carry a lot of information, but the number of them is not that big.
Cheers,
Signature · 1 line
Vinicius
Quoted 18 lines · Vinicius Costa Gomes
Hi,
Quoted 11 lines · Jakub Kicinski
Please take a look at the example from the cover letter:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: active supported queues: 0xf supported queues: 0xe minimum fragment size: 68Quoted 3 lines
Reading this I have no idea what
0xeis. I have to go and query TC API to see what priorities and queues that will be. Which IMHO is a strong argument that this information belongs there in the first place.
Quoted 6 lines
That was the (only?) strong argument in favor of having frame preemption in the TC side when this was last discussed.
We can have a hybrid solution, we can move the express/preemptible per queue map to mqprio/taprio/whatever. And have the more specific configuration knobs, minimum fragment size, etc, in ethtool.
Isn't this a pure h/w feature? FPE is implemented at L2 and involves fragments that are only seen by h/w and never at Linux network core unlike IP fragments and is transparent to network stack. However it enhances priority handling at h/w to the next level by pre-empting existing lower priority traffic to give way to express queue traffic and improve latency. So everything happens in h/w. So ethtool makes perfect sense here as it is a queue configuration. I agree with Vinicius and Vladmir to support this in ethtool instead of TC.
Murali
Quoted 6 lines
What do you think?
Cheers,
Signature · 2 lines
Murali Karicheri Texas Instruments
Quoted 7 lines · Murali Karicheri
Quoted 6 lines
That was the (only?) strong argument in favor of having frame preemption in the TC side when this was last discussed.
We can have a hybrid solution, we can move the express/preemptible per queue map to mqprio/taprio/whatever. And have the more specific configuration knobs, minimum fragment size, etc, in ethtool.
Quoted 8 lines
Isn't this a pure h/w feature? FPE is implemented at L2 and involves fragments that are only seen by h/w and never at Linux network core unlike IP fragments and is transparent to network stack. However it enhances priority handling at h/w to the next level by pre-empting existing lower priority traffic to give way to express queue traffic and improve latency. So everything happens in h/w. So ethtool makes perfect sense here as it is a queue configuration. I agree with Vinicius and Vladmir to support this in ethtool instead of TC.
The way I see, the issue that Jakub is pointing here is more of usability/understandability.
By having the express/preemptible queue mapping in TC, we have the configuration near where the "priority to queue" mapping happens. That improves the ease of configuration, makes it easier to spot mistakes, that kind of thing, all of which are a big plus.
Right now, I am seeing this hybrid approach as a good compromise, we have the queue settings near to where the kinds of traffic are mapped to queues, and we have the rest of the hardware configuration in ethtool.
Cheers,
Signature · 1 line
Vinicius
Am Fri, 15 May 2020 18:29:44 -0700
schrieb Vinicius Costa Gomes <[email protected]>:
Quoted 26 lines
Hi,
This series adds support for configuring frame preemption, as defined by IEEE 802.1Q-2018 (previously IEEE 802.1Qbu) and IEEE 802.3br.
Frame preemption allows a packet from a higher priority queue marked as "express" to preempt a packet from lower priority queue marked as "preemptible". The idea is that this can help reduce the latency for higher priority traffic.
Previously, the proposed interface for configuring these features was using the qdisc layer. But as this is very hardware dependent and all that qdisc did was pass the information to the driver, it makes sense to have this in ethtool.
One example, for retrieving and setting the configuration:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: active supported queues: 0xf supported queues: 0xe minimum fragment size: 68
Quoted 7 lines
$ ethtool --set-frame-preemption enp3s0 fp on min-frag-size 68 preemptible-queues-mask 0xeThis is a RFC because I wanted to have feedback on some points:
- The parameters added are enough for the hardware I have, is it enough in general?
What about the Qbu handshake state? And some NICs support overriding this. I.e. enable frame preemption even if the handshake wasn't successful.
-michael
Quoted 51 lines
- even with the ethtool via netlink effort, I chose to keep the
ioctl()way, in case someone wants to backport this to an older kernel, is there a problem with this?- Some space for bikeshedding the names and location (for example, does it make sense for these settings to be per-queue?), as I am not quite happy with them, one example, is the use of preemptible vs. preemptable;
About the patches, should be quite straightforward:
Patch 1, adds the
ETHTOOL_GFPandETHOOL_SFPcommands and the associated data structures;Patch 2, adds the
ETHTOOL_MSG_PREEMPT_GETandETHTOOL_MSG_PREEMPT_SETnetlink messages and the associated attributes;Patch 3, is the example implementation for the igc driver, the catch here is that frame preemption in igc is dependent on the TSN "mode" being enabled;
Patch 4, adds some registers that helped during implementation.
Another thing is that because igc is still under development, to avoid conflicts, I think it might be easier for the PATCH version of this series to go through Jeff Kirsher's tree.
Vinicius Costa Gomes (4):
ethtool: Add support for configuring frame preemption
ethtool: Add support for configuring frame preemption via netlink
igc: Add support for configuring frame preemption
igc: Add support for exposing frame preemption stats registersdrivers/net/ethernet/intel/igc/igc.h | 3 + drivers/net/ethernet/intel/igc/igc_defines.h | 6 + drivers/net/ethernet/intel/igc/igc_ethtool.c | 77 ++++++++ drivers/net/ethernet/intel/igc/igc_regs.h | 10 + drivers/net/ethernet/intel/igc/igc_tsn.c | 46 ++++- include/linux/ethtool.h | 6 + include/uapi/linux/ethtool.h | 25 +++ include/uapi/linux/ethtool_netlink.h | 19 ++ net/ethtool/Makefile | 3 +- net/ethtool/ioctl.c | 36 ++++ net/ethtool/netlink.c | 15 ++ net/ethtool/netlink.h | 2 + net/ethtool/preempt.c | 181+++++++++++++++++++ 13 files changed, 423 insertions(+), 6 deletions(-) create mode 100644
net/ethtool/preempt.c
Hi,
Quoted 3 lines · Michael Walle
What about the Qbu handshake state? And some NICs support overriding this. I.e. enable frame preemption even if the handshake wasn't successful.
You are talking about Verify procedure to hand shake with peer to know if remote support IET fragmentation and re-assembly? If yes, this manual mode of provisioning is required as well. So one optional parameter needed is enable-verify. If that is not enabled then device assumes the remote is capable of fragmentation and re-assembly.
Signature · 2 lines
Murali Karicheri Texas Instruments
Hi,
Am 2020-05-18 15:36, schrieb Murali Karicheri:
Quoted 9 lines
Hi,
Quoted 3 lines · Michael Walle
What about the Qbu handshake state? And some NICs support overriding this. I.e. enable frame preemption even if the handshake wasn't successful.
You are talking about Verify procedure to hand shake with peer to know if remote support IET fragmentation and re-assembly?
yes
Quoted 4 lines
If yes, this manual mode of provisioning is required as well. So one optional parameter needed is enable-verify. If that is not enabled then device assumes the remote is capable of fragmentation and re-assembly.
sounds good.
-michael
Hi Vinicius,
Quoted 15 lines · Vinicius Costa Gomes
Hi,
This series adds support for configuring frame preemption, as defined by IEEE 802.1Q-2018 (previously IEEE 802.1Qbu) and IEEE 802.3br.
Frame preemption allows a packet from a higher priority queue marked as "express" to preempt a packet from lower priority queue marked as "preemptible". The idea is that this can help reduce the latency for higher priority traffic.
Previously, the proposed interface for configuring these features was using the qdisc layer. But as this is very hardware dependent and all that qdisc did was pass the information to the driver, it makes sense to have this in ethtool.
Quoted 7 lines
One example, for retrieving and setting the configuration:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: active supported queues: 0xf
I assume this is will be in sync with ethtool -L output which indicates how many tx h/w queues present? I mean if there are 8 h/w queues, supported queues will show 0xff.
Quoted 1 line
supported queues: 0xe
From the command below, it appears this is the preemptible queue mask.
bit 0 is Q0, bit 1 Q1 and so forth. Right? In that case isn't it more clear to display
preemptible queues : 0xef
In the above Q7 is express queue and Q6-Q0 are preemptible.
Also there is a handshake called verify that happens which initiated by the h/w to check the capability of peer. It looks like not all vendor's hardware supports it and good to have it displayed something like
Verify supported/{not supported}If Verify is supported, FPE is enabled only if it succeeds. So may be good to show a status of Verify if it is supported something like
Verify success/Failed
Quoted 9 lines
minimum fragment size: 68
$ ethtool --set-frame-preemption enp3s0 fp on min-frag-size 68 preemptible-queues-mask 0xeThis is a RFC because I wanted to have feedback on some points:
- The parameters added are enough for the hardware I have, is it enough in general?
As described above, it would be good to add an optional parameter for verify
ethtool --set-frame-preemption enp3s0 fp on min-frag-size 68 preemptible-queues-mask 0xe verify on
Quoted 51 lines
- even with the ethtool via netlink effort, I chose to keep the
ioctl()way, in case someone wants to backport this to an older kernel, is there a problem with this?- Some space for bikeshedding the names and location (for example, does it make sense for these settings to be per-queue?), as I am not quite happy with them, one example, is the use of preemptible vs. preemptable;
About the patches, should be quite straightforward:
Patch 1, adds the
ETHTOOL_GFPandETHOOL_SFPcommands and the associated data structures;Patch 2, adds the
ETHTOOL_MSG_PREEMPT_GETandETHTOOL_MSG_PREEMPT_SETnetlink messages and the associated attributes;Patch 3, is the example implementation for the igc driver, the catch here is that frame preemption in igc is dependent on the TSN "mode" being enabled;
Patch 4, adds some registers that helped during implementation.
Another thing is that because igc is still under development, to avoid conflicts, I think it might be easier for the PATCH version of this series to go through Jeff Kirsher's tree.
Vinicius Costa Gomes (4):
ethtool: Add support for configuring frame preemption
ethtool: Add support for configuring frame preemption via netlink
igc: Add support for configuring frame preemption
igc: Add support for exposing frame preemption stats registersdrivers/net/ethernet/intel/igc/igc.h | 3 + drivers/net/ethernet/intel/igc/igc_defines.h | 6 + drivers/net/ethernet/intel/igc/igc_ethtool.c | 77 ++++++++ drivers/net/ethernet/intel/igc/igc_regs.h | 10 + drivers/net/ethernet/intel/igc/igc_tsn.c | 46 ++++- include/linux/ethtool.h | 6 + include/uapi/linux/ethtool.h | 25 +++ include/uapi/linux/ethtool_netlink.h | 19 ++ net/ethtool/Makefile | 3 +- net/ethtool/ioctl.c | 36 ++++ net/ethtool/netlink.c | 15 ++ net/ethtool/netlink.h | 2 + net/ethtool/preempt.c | 181 +++++++++++++++++++ 13 files changed, 423 insertions(+), 6 deletions(-)create mode 100644
net/ethtool/preempt.c
Signature · 2 lines
Murali Karicheri Texas Instruments
Quoted 9 lines · Murali Karicheri
Quoted 5 lines
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: active supported queues: 0xfI assume this is will be in sync with ethtool -L output which indicates how many tx h/w queues present? I mean if there are 8 h/w queues, supported queues will show
0xff.
In this approach, the driver builds these bitmasks, so it's responsible to keep it consistent with the rest of the stuff that's exposed in ethtool.
Quoted 8 lines
Quoted 2 lines
supported queues: 0xeFrom the command below, it appears this is the preemptible queue mask.
bit 0 is Q0, bit 1 Q1 and so forth. Right? In that case isn't it more clear to displaypreemptible queues : 0xefIn the above Q7 is express queue and Q6-Q0 are preemptible.
In my case, the controller I have here only has 4 queues, and Queue 0 is the highest priority one, and it's marked as express.
Quoted 19 lines
Also there is a handshake called verify that happens which initiated by the h/w to check the capability of peer. It looks like not all vendor's hardware supports it and good to have it displayed something like
Verify supported/{not supported}If Verify is supported, FPE is enabled only if it succeeds. So may be good to show a status of Verify if it is supported something like
Verify success/FailedQuoted 7 lines
minimum fragment size: 68
$ ethtool --set-frame-preemption enp3s0 fp on min-frag-size 68 preemptible-queues-mask 0xeThis is a RFC because I wanted to have feedback on some points:
Quoted 9 lines
Quoted 2 lines
- The parameters added are enough for the hardware I have, is it enough in general?
As described above, it would be good to add an optional parameter for verify
ethtool --set-frame-preemption enp3s0 fp on min-frag-size 68 preemptible-queues-mask
0xeverify on
The hardware I have do not support this, but this makes sense.
Cheers,
Signature · 1 line
Vinicius
Hi Vinicius,
Quoted 8 lines · Vinicius Costa Gomes
Quoted 6 lines · Murali Karicheri
Quoted 5 lines
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: active supported queues: 0xf
Quoted 7 lines
Quoted 3 lines
I assume this is will be in sync with ethtool -L output which indicates how many tx h/w queues present? I mean if there are 8 h/w queues, supported queues will show
0xff.In this approach, the driver builds these bitmasks, so it's responsible to keep it consistent with the rest of the stuff that's exposed in ethtool.
Agree
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
supported queues: 0xeFrom the command below, it appears this is the preemptible queue mask.
bit 0 is Q0, bit 1 Q1 and so forth. Right? In that case isn't it more clear to displaypreemptible queues : 0xef
Quoted 5 lines
Quoted 1 line
In the above Q7 is express queue and Q6-Q0 are preemptible.
In my case, the controller I have here only has 4 queues, and Queue 0 is the highest priority one, and it's marked as express.
Ok. So it is up to the device driver to manage this.
Quoted 22 lines
Quoted 22 lines
Also there is a handshake called verify that happens which initiated by the h/w to check the capability of peer. It looks like not all vendor's hardware supports it and good to have it displayed something like
Verify supported/{not supported}If Verify is supported, FPE is enabled only if it succeeds. So may be good to show a status of Verify if it is supported something like
Verify success/FailedQuoted 9 lines
minimum fragment size: 68
$ ethtool --set-frame-preemption enp3s0 fp on min-frag-size 68 preemptible-queues-mask 0xeThis is a RFC because I wanted to have feedback on some points:
- The parameters added are enough for the hardware I have, is it enough in general?
Quoted 8 lines
Quoted 6 lines
As described above, it would be good to add an optional parameter for verify
ethtool --set-frame-preemption enp3s0 fp on min-frag-size 68 preemptible-queues-mask
0xeverify onThe hardware I have do not support this, but this makes sense.
I can work to support this for TI AM65 CPSW once a formal patch is available. AM65 CPSW supports Verify as an optional feature. H/w works also in manual mode where it is assumed that it is connected to a IET FPE capable partner, but can't do Verify.
Thanks for sending the RFC.
regards,
Murali
Quoted 4 lines
Cheers,
Signature · 2 lines
Murali Karicheri Texas Instruments
Hi,
Quoting Vinicius Costa Gomes (2020-05-15 18:29:44)
Quoted 6 lines
One example, for retrieving and setting the configuration:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: active
IIUC the code in patch 2, 'active' is the actual configuration knob that enables or disables the FP functionality on the NIC.
That sounded a bit confusing to me since the spec uses the term 'active' to indicate FP is currently enabled at both ends, and it is a read-only information (see 12.30.1.4 from IEEE 802.1Q-2018). Maybe if we called this 'enabled' it would be more clear.
Quoted 3 lines
supported queues: 0xf
supported queues: 0xe
minimum fragment size: 68I'm assuming this is the configuration knob for the minimal non-final fragment defined in 802.3br.
My understanding from the specs is that this value must be a multiple from 64 and cannot assume arbitrary values like shown here. See 99.4.7.3 from IEEE 802.3 and Note 1 in S.2 from IEEE 802.1Q. In the previous discussion about FP, we had this as a multiplier factor, not absolute value.
Regards,
Andre
Quoted 10 lines · Andre Guedes
Hi,
Quoting Vinicius Costa Gomes (2020-05-15 18:29:44)
Quoted 6 lines
One example, for retrieving and setting the configuration:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: active
Quoted 7 lines
IIUC the code in patch 2, 'active' is the actual configuration knob that enables or disables the FP functionality on the NIC.
That sounded a bit confusing to me since the spec uses the term 'active' to indicate FP is currently enabled at both ends, and it is a read-only information (see 12.30.1.4 from IEEE 802.1Q-2018). Maybe if we called this 'enabled' it would be more clear.
Good point. Will rename this to "enabled".
Quoted 5 lines
Quoted 4 lines
supported queues: 0xf supported queues: 0xe minimum fragment size: 68
Quoted 7 lines
I'm assuming this is the configuration knob for the minimal non-final fragment defined in 802.3br.
My understanding from the specs is that this value must be a multiple from 64 and cannot assume arbitrary values like shown here. See 99.4.7.3 from IEEE 802.3 and Note 1 in S.2 from IEEE 802.1Q. In the previous discussion about FP, we had this as a multiplier factor, not absolute value.
I thought that exposing this as "(1 + N)*64" (with 0 <= N <= 3) that it was more related to what's exposed via LLDP than the actual capabilities of the hardware. And for the hardware I have actually the values supported are: (1 + N)*64 + 4 (for N = 0, 1, 2, 3).
So I thought I was better to let the driver decide what values are acceptable.
This is a good question for people working with other hardware.
Signature · 1 line
Vinicius
Hi Vinicius,
Quoted 40 lines · Vinicius Costa Gomes
Quoted 10 lines · Andre Guedes
Hi,
Quoting Vinicius Costa Gomes (2020-05-15 18:29:44)
Quoted 6 lines
One example, for retrieving and setting the configuration:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: activeQuoted 7 lines
IIUC the code in patch 2, 'active' is the actual configuration knob that enables or disables the FP functionality on the NIC.
That sounded a bit confusing to me since the spec uses the term 'active' to indicate FP is currently enabled at both ends, and it is a read-only information (see 12.30.1.4 from IEEE 802.1Q-2018). Maybe if we called this 'enabled' it would be more clear.
Good point. Will rename this to "enabled".
Quoted 5 lines
Quoted 4 lines
supported queues: 0xf supported queues: 0xe minimum fragment size: 68Quoted 7 lines
I'm assuming this is the configuration knob for the minimal non-final fragment defined in 802.3br.
My understanding from the specs is that this value must be a multiple from 64 and cannot assume arbitrary values like shown here. See 99.4.7.3 from IEEE 802.3 and Note 1 in S.2 from IEEE 802.1Q. In the previous discussion about FP, we had this as a multiplier factor, not absolute value.
I thought that exposing this as "(1 + N)*64" (with 0 <= N <= 3) that it was more related to what's exposed via LLDP than the actual capabilities of the hardware. And for the hardware I have actually the values supported are: (1 + N)*64 + 4 (for N = 0, 1, 2, 3).
Quoted 6 lines
So I thought I was better to let the driver decide what values are acceptable.
This is a good question for people working with other hardware.
AM65 CPSW supports this as a multiple of 64. Since this ethtool configuration is for the hardware, might want to make it as a multiple of 64.
Murali
Signature · 2 lines
Murali Karicheri Texas Instruments
Quoted 41 lines · Vinicius Costa Gomes
Quoted 10 lines · Andre Guedes
Hi,
Quoting Vinicius Costa Gomes (2020-05-15 18:29:44)
Quoted 6 lines
One example, for retrieving and setting the configuration:
$ ethtool $ sudo ./ethtool --show-frame-preemption enp3s0 Frame preemption settings for enp3s0: support: supported active: activeQuoted 7 lines
IIUC the code in patch 2, 'active' is the actual configuration knob that enables or disables the FP functionality on the NIC.
That sounded a bit confusing to me since the spec uses the term 'active' to indicate FP is currently enabled at both ends, and it is a read-only information (see 12.30.1.4 from IEEE 802.1Q-2018). Maybe if we called this 'enabled' it would be more clear.
Good point. Will rename this to "enabled".
Quoted 5 lines
Quoted 4 lines
supported queues: 0xf supported queues: 0xe minimum fragment size: 68Quoted 7 lines
I'm assuming this is the configuration knob for the minimal non-final fragment defined in 802.3br.
My understanding from the specs is that this value must be a multiple from 64 and cannot assume arbitrary values like shown here. See 99.4.7.3 from IEEE 802.3 and Note 1 in S.2 from IEEE 802.1Q. In the previous discussion about FP, we had this as a multiplier factor, not absolute value.
I thought that exposing this as "(1 + N)*64" (with 0 <= N <= 3) that it was more related to what's exposed via LLDP than the actual capabilities of the hardware. And for the hardware I have actually the values supported are: (1 + N)*64 + 4 (for N = 0, 1, 2, 3).
Quoted 5 lines
So I thought I was better to let the driver decide what values are acceptable.
This is a good question for people working with other hardware.
I think it's most intuitive to use the values for AddFragSize as described in 802.3br (N = 0, 1, 2, 3).
You will anyway have to use one of these values when you want to expose the requirements of your receiver through LLDP.
Quoted 3 lines
--
Vinicius
Signature · 1 line
Joergen Andreasen, Microchip
Quoted 11 lines · Joergen Andreasen
Quoted 5 lines
So I thought I was better to let the driver decide what values are acceptable.
This is a good question for people working with other hardware.
I think it's most intuitive to use the values for AddFragSize as described in 802.3br (N = 0, 1, 2, 3).
You will anyway have to use one of these values when you want to expose the requirements of your receiver through LLDP.
Thanks. Seems that keeping this value restricted to multiples of 64 is the way to go. Will fix for the next version of the series.
Cheers,
Signature · 1 line
Vinicius