Skip to contentSkip to conversation

devlink: Support setting max_io_eqs

PATCH In review Parav Pandit 4 participants 12 messages Apr 10, 2024 – Apr 13, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover devlink: Support setting max_io_eqs

Devices send event notifications for the IO queues, such as tx and rx queues, through event queues.

Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.

example:
Get maximum IO event queues of the VF device::

  $ devlink port show pci/0000:06:00.0/2
  pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
      function:
          hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10

Set maximum IO event queues of the VF device::

  $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32
  $ devlink port show pci/0000:06:00.0/2
  pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
      function:
          hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32

patch summary:
patch-1 updates devlink uapi
patch-2 adds print, get and set routines for max_io_eqs field

changelog:
v1->v2:
- addressed comments from Jiri
- updated man page for the new parameter
- corrected print to not have EQs value as optional
- replaced 'value' with 'EQs'

Parav Pandit (2):
uapi: Update devlink kernel headers
devlink: Support setting max_io_eqs

 devlink/devlink.c            | 29 ++++++++++++++++++++++++++++-
 include/uapi/linux/devlink.h |  1 +
 man/man8/devlink-port.8      | 12 ++++++++++++
 3 files changed, 41 insertions(+), 1 deletion(-)
Signature · 1 line
2.26.2
#1 lore raw

PATCH v2 1/2 uapi: Update devlink kernel headers

Update devlink kernel header to commit:

    92de776d2090 ("Merge tag 'mlx5-updates-2023-12-20' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux")

---
note:
This patch can be dropped by first syncing all the uapi files.

---
 include/uapi/linux/devlink.h | 1 +
 1 file changed, 1 insertion(+)
include/uapi/linux/devlink.h +1 -0
index aaac2438..80051b8c 100644
@@ -686,6 +686,7 @@ enum devlink_port_function_attr {
 	DEVLINK_PORT_FN_ATTR_OPSTATE,	/* u8 */
 	DEVLINK_PORT_FN_ATTR_CAPS,	/* bitfield32 */
 	DEVLINK_PORT_FN_ATTR_DEVLINK,	/* nested */
+	DEVLINK_PORT_FN_ATTR_MAX_IO_EQS,	/* u32 */
 
 	__DEVLINK_PORT_FUNCTION_ATTR_MAX,
 	DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX - 1
Signature · 1 line
2.26.2
#2 lore raw

PATCH v2 2/2 devlink: Support setting max_io_eqs

In reply to Parav Pandit

Devices send event notifications for the IO queues, such as tx and rx queues, through event queues.

Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.

example:
Get maximum IO event queues of the VF device::

  $ devlink port show pci/0000:06:00.0/2
  pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
      function:
          hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10

Set maximum IO event queues of the VF device::

  $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32
  $ devlink port show pci/0000:06:00.0/2
  pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
      function:
          hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32

---
changelog:
v1->v2:
- addressed comments from Jiri
- updated man page for the new parameter
- corrected print to not have EQs value as optional
- replaced 'value' with 'EQs'

---
 devlink/devlink.c       | 29 ++++++++++++++++++++++++++++-
 man/man8/devlink-port.8 | 12 ++++++++++++
 2 files changed, 40 insertions(+), 1 deletion(-)
devlink/devlink.c +28 -1
index dbeb6e39..03d27202 100644
@@ -309,6 +309,7 @@ static int ifname_map_update(struct ifname_map *ifname_map, const char *ifname)
 #define DL_OPT_PORT_FN_RATE_TX_PRIORITY	BIT(55)
 #define DL_OPT_PORT_FN_RATE_TX_WEIGHT	BIT(56)
 #define DL_OPT_PORT_FN_CAPS	BIT(57)
+#define DL_OPT_PORT_FN_MAX_IO_EQS	BIT(58)
 
 struct dl_opts {
 	uint64_t present; /* flags of present items */
@@ -375,6 +376,7 @@ struct dl_opts {
 	const char *linecard_type;
 	bool selftests_opt[DEVLINK_ATTR_SELFTEST_ID_MAX + 1];
 	struct nla_bitfield32 port_fn_caps;
+	uint32_t port_fn_max_io_eqs;
 };
 
 struct dl {
@@ -773,6 +775,7 @@ devlink_function_policy[DEVLINK_PORT_FUNCTION_ATTR_MAX + 1] = {
 	[DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR ] = MNL_TYPE_BINARY,
 	[DEVLINK_PORT_FN_ATTR_STATE] = MNL_TYPE_U8,
 	[DEVLINK_PORT_FN_ATTR_DEVLINK] = MNL_TYPE_NESTED,
+	[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS] = MNL_TYPE_U32,
 };
 
 static int function_attr_cb(const struct nlattr *attr, void *data)
@@ -2298,6 +2301,17 @@ static int dl_argv_parse(struct dl *dl, uint64_t o_required,
 			if (ipsec_packet)
 				opts->port_fn_caps.value |= DEVLINK_PORT_FN_CAP_IPSEC_PACKET;
 			o_found |= DL_OPT_PORT_FN_CAPS;
+		} else if (dl_argv_match(dl, "max_io_eqs") &&
+			   (o_all & DL_OPT_PORT_FN_MAX_IO_EQS)) {
+			uint32_t max_io_eqs;
+
+			dl_arg_inc(dl);
+			err = dl_argv_uint32_t(dl, &max_io_eqs);
+			if (err)
+				return err;
+			opts->port_fn_max_io_eqs = max_io_eqs;
+			o_found |= DL_OPT_PORT_FN_MAX_IO_EQS;
+
 		} else {
 			pr_err("Unknown option \"%s\"\n", dl_argv(dl));
 			return -EINVAL;
@@ -2428,6 +2442,9 @@ dl_function_attr_put(struct nlmsghdr *nlh, const struct dl_opts *opts)
 	if (opts->present & DL_OPT_PORT_FN_CAPS)
 		mnl_attr_put(nlh, DEVLINK_PORT_FN_ATTR_CAPS,
 			     sizeof(opts->port_fn_caps), &opts->port_fn_caps);
+	if (opts->present & DL_OPT_PORT_FN_MAX_IO_EQS)
+		mnl_attr_put_u32(nlh, DEVLINK_PORT_FN_ATTR_MAX_IO_EQS,
+				opts->port_fn_max_io_eqs);
 
 	mnl_attr_nest_end(nlh, nest);
 }
@@ -4744,6 +4761,7 @@ static void cmd_port_help(void)
 	pr_err("       devlink port function set DEV/PORT_INDEX [ hw_addr ADDR ] [ state { active | inactive } ]\n");
 	pr_err("                      [ roce { enable | disable } ] [ migratable { enable | disable } ]\n");
 	pr_err("                      [ ipsec_crypto { enable | disable } ] [ ipsec_packet { enable | disable } ]\n");
+	pr_err("                      [ max_io_eqs EQS\n");
 	pr_err("       devlink port function rate { help | show | add | del | set }\n");
 	pr_err("       devlink port param set DEV/PORT_INDEX name PARAMETER value VALUE cmode { permanent | driverinit | runtime }\n");
 	pr_err("       devlink port param show [DEV/PORT_INDEX name PARAMETER]\n");
@@ -4878,6 +4896,15 @@ static void pr_out_port_function(struct dl *dl, struct nlattr **tb_port)
 				     port_fn_caps->value & DEVLINK_PORT_FN_CAP_IPSEC_PACKET ?
 				     "enable" : "disable");
 	}
+	if (tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS]) {
+		uint32_t max_io_eqs;
+
+		max_io_eqs = mnl_attr_get_u32(tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS]);
+
+		print_uint(PRINT_ANY, "max_io_eqs", " max_io_eqs %u",
+			   max_io_eqs);
+	}
+
 	if (tb[DEVLINK_PORT_FN_ATTR_DEVLINK])
 		pr_out_nested_handle_obj(dl, tb[DEVLINK_PORT_FN_ATTR_DEVLINK],
 					 true, true);
@@ -5086,7 +5113,7 @@ static int cmd_port_function_set(struct dl *dl)
 	}
 	err = dl_argv_parse(dl, DL_OPT_HANDLEP,
 			    DL_OPT_PORT_FUNCTION_HW_ADDR | DL_OPT_PORT_FUNCTION_STATE |
-			    DL_OPT_PORT_FN_CAPS);
+			    DL_OPT_PORT_FN_CAPS | DL_OPT_PORT_FN_MAX_IO_EQS);
 	if (err)
 		return err;
 
man/man8/devlink-port.8 +12 -0
index 70d8837e..6f582260 100644
@@ -83,6 +83,9 @@ devlink-port \- devlink port configuration
 .RI "[ "
 .BR ipsec_packet " { " enable " | " disable " }"
 .RI "]"
+.RI "[ "
+.BR max_io_eqs " EQS"
+.RI "]"
 
 .ti -8
 .BR "devlink port function rate "
@@ -238,6 +241,10 @@ crypto operation (Encrypt/Decrypt) offload.
 Set the IPsec packet offload capability of the function. Controls XFRM state
 and policy offload (Encrypt/Decrypt operation and IPsec encapsulation).
 
+.TP
+.BR max_io_eqs " EQS"
+Set the maximum number of IO event queues of the function.
+
 .ti -8
 .SS devlink port del - delete a devlink port
 .PP
@@ -377,6 +384,11 @@ devlink port function set pci/0000:01:00.0/1 ipsec_packet enable
 This will enable the IPsec packet offload functionality of the function.
 .RE
 .PP
+devlink port function set pci/0000:01:00.0/1 max_io_eqs 4
+.RS 4
+This will set the maximum number of IO event queues of the function to 4.
+.RE
+.PP
 devlink port function set pci/0000:01:00.0/1 hw_addr 00:00:00:11:22:33 state active
 .RS 4
 Configure hardware address and also active the function. When a function is
Signature · 1 line
2.26.2
#3 lore raw
In reply to Parav Pandit
Quoted 5 lines · Parav Pandit

Devices send event notifications for the IO queues, such as tx and rx queues, through event queues.

Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.

How do you provision tx/rx queues for VFs & SFs? Don't you need similar mechanism to setup max tx/rx queues too?

Quoted 38 lines

example:
Get maximum IO event queues of the VF device::

   $ devlink port show pci/0000:06:00.0/2
   pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
       function:
           hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10

Set maximum IO event queues of the VF device::

   $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32
   $ devlink port show pci/0000:06:00.0/2
   pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
       function:
           hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32

patch summary:
patch-1 updates devlink uapi
patch-2 adds print, get and set routines for max_io_eqs field

changelog:
v1->v2:
- addressed comments from Jiri
- updated man page for the new parameter
- corrected print to not have EQs value as optional
- replaced 'value' with 'EQs'

Parav Pandit (2):
uapi: Update devlink kernel headers
devlink: Support setting max_io_eqs

  devlink/devlink.c            | 29 ++++++++++++++++++++++++++++-
  include/uapi/linux/devlink.h |  1 +
  man/man8/devlink-port.8      | 12 ++++++++++++
  3 files changed, 41 insertions(+), 1 deletion(-)
#4 lore raw

Hi Sridhar,

Quoted 5 lines

From: Samudrala, Sridhar <[email protected]>
Sent: Thursday, April 11, 2024 4:53 AM

On 4/10/2024 6:58 AM, Parav Pandit wrote:

Quoted 8 lines
Quoted 5 lines

Devices send event notifications for the IO queues, such as tx and rx queues, through event queues.

Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.

How do you provision tx/rx queues for VFs & SFs? Don't you need similar mechanism to setup max tx/rx queues too?

Currently we don’t. They are derived from the IO event queues.
As you know, sometimes more txqs than IO event queues needed for XDP, timestamp, multiple TCs. If needed, probably additional knob for txq, rxq can be added to restrict device resources.

Quoted 44 lines
Quoted 8 lines

example:
Get maximum IO event queues of the VF device::

   $ devlink port show pci/0000:06:00.0/2
   pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0

vfnum 1

Quoted 10 lines
       function:
           hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs

10

Set maximum IO event queues of the VF device::

   $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32
   $ devlink port show pci/0000:06:00.0/2
   pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0

vfnum 1

Quoted 24 lines
       function:
           hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs

32

patch summary:
patch-1 updates devlink uapi
patch-2 adds print, get and set routines for max_io_eqs field

changelog:
v1->v2:
- addressed comments from Jiri
- updated man page for the new parameter
- corrected print to not have EQs value as optional
- replaced 'value' with 'EQs'

Parav Pandit (2):
uapi: Update devlink kernel headers
devlink: Support setting max_io_eqs

  devlink/devlink.c            | 29 ++++++++++++++++++++++++++++-
  include/uapi/linux/devlink.h |  1 +
  man/man8/devlink-port.8      | 12 ++++++++++++
  3 files changed, 41 insertions(+), 1 deletion(-)
#5 lore raw
Quoted 13 lines · Parav Pandit

Hi Sridhar,

Quoted 11 lines

From: Samudrala, Sridhar <[email protected]>
Sent: Thursday, April 11, 2024 4:53 AM

Quoted 5 lines · Parav Pandit

Devices send event notifications for the IO queues, such as tx and rx queues, through event queues.

Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.

Quoted 6 lines
Quoted 2 lines

How do you provision tx/rx queues for VFs & SFs? Don't you need similar mechanism to setup max tx/rx queues too?

Currently we don’t. They are derived from the IO event queues.
As you know, sometimes more txqs than IO event queues needed for XDP, timestamp, multiple TCs. If needed, probably additional knob for txq, rxq can be added to restrict device resources.

Rather than deriving tx and rx queues from IO event queues, isn't it more user friendly to do the other way. Let the host admin set the max number of tx and rx queues allowed and the driver derive the number of ioevent queues based on those values. This will be consistent with what ethtool reports as pre-set maximum values for the corresponding VF/SF.

Quoted 45 lines
Quoted 45 lines
Quoted 9 lines

example:
Get maximum IO event queues of the VF device::

    $ devlink port show pci/0000:06:00.0/2
    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0

vfnum 1

Quoted 10 lines
        function:
            hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs

10

Set maximum IO event queues of the VF device::

    $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32
    $ devlink port show pci/0000:06:00.0/2
    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0

vfnum 1

Quoted 24 lines
        function:
            hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs

32

patch summary:
patch-1 updates devlink uapi
patch-2 adds print, get and set routines for max_io_eqs field

changelog:
v1->v2:
- addressed comments from Jiri
- updated man page for the new parameter
- corrected print to not have EQs value as optional
- replaced 'value' with 'EQs'

Parav Pandit (2):
uapi: Update devlink kernel headers
devlink: Support setting max_io_eqs

   devlink/devlink.c            | 29 ++++++++++++++++++++++++++++-
   include/uapi/linux/devlink.h |  1 +
   man/man8/devlink-port.8      | 12 ++++++++++++
   3 files changed, 41 insertions(+), 1 deletion(-)
#6 lore raw
Quoted 25 lines · Sridhar
Quoted 13 lines · Parav Pandit

Hi Sridhar,

Quoted 11 lines

From: Samudrala, Sridhar <[email protected]>
Sent: Thursday, April 11, 2024 4:53 AM

Quoted 5 lines · Parav Pandit

Devices send event notifications for the IO queues, such as tx and rx queues, through event queues.

Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.

Quoted 8 lines
Quoted 2 lines

How do you provision tx/rx queues for VFs & SFs? Don't you need similar mechanism to setup max tx/rx queues too?

Currently we don’t. They are derived from the IO event queues. As you know, sometimes more txqs than IO event queues needed for XDP, timestamp, multiple TCs.
If needed, probably additional knob for txq, rxq can be added to restrict device resources.

Quoted 6 lines

Rather than deriving tx and rx queues from IO event queues, isn't it more user friendly to do the other way. Let the host admin set the max number of tx and rx queues allowed and the driver derive the number of ioevent queues based on those values. This will be consistent with what ethtool reports as pre-set maximum values for the corresponding VF/SF.

I agree with this point: IO EQ seems to be a mlx5 thing (or maybe I have not reviewed enough of the other drivers). Rx and Tx queues are already part of the ethtool API. This devlink feature is allowing resource limits to be configured, and a consistent API across tools would be better for users.

#7 lore raw

Hi David, Sridhar,

Quoted 35 lines

From: David Ahern <[email protected]>
Sent: Friday, April 12, 2024 7:36 AM

Quoted 9 lines · Sridhar
Quoted 6 lines · Parav Pandit

Hi Sridhar,

Quoted 4 lines

From: Samudrala, Sridhar <[email protected]>
Sent: Thursday, April 11, 2024 4:53 AM

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines · Parav Pandit
Quoted 5 lines

Devices send event notifications for the IO queues, such as tx and rx queues, through event queues.

Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning

stage.

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

How do you provision tx/rx queues for VFs & SFs? Don't you need similar mechanism to setup max tx/rx queues too?

Currently we don’t. They are derived from the IO event queues. As you know, sometimes more txqs than IO event queues needed for XDP, timestamp, multiple TCs.
If needed, probably additional knob for txq, rxq can be added to restrict device resources.

Quoted 5 lines

Rather than deriving tx and rx queues from IO event queues, isn't it more user friendly to do the other way. Let the host admin set the max number of tx and rx queues allowed and the driver derive the number of ioevent queues based on those values. This will be consistent with what ethtool reports as pre-set maximum values for the corresponding

Quoted 5 lines

VF/SF.

Quoted 1 line

I agree with this point: IO EQ seems to be a mlx5 thing (or maybe I have not reviewed enough of the other drivers).

IO EQs are used by hns3, mana, mlx5, mlxsw, be2net. They might not yet have the need to provision them.

Quoted 3 lines

Rx and Tx queues are already part of
the ethtool API. This devlink feature is allowing resource limits to be configured, and a consistent API across tools would be better for users.

IO Eqs of a function are utilized by the non netdev stack as well for a multi-functionality function like rdma completion vectors. Txq and rxq are yet another separate resource, so it is not mutually exclusive with IO EQs.

I can additionally add txq and rxq provisioning knob too if this is useful, yes?

Sridhar,
I didn’t lately check other drivers how usable is it, will you also implement the txq, rxq callbacks? Please let me know I can start the work later next week for those additional knobs.

#8 lore raw
Quoted 61 lines

From: Parav Pandit <[email protected]>
Sent: Friday, April 12, 2024 9:02 AM

Hi David, Sridhar,

Quoted 37 lines

From: David Ahern <[email protected]>
Sent: Friday, April 12, 2024 7:36 AM

Quoted 9 lines · Sridhar
Quoted 6 lines · Parav Pandit

Hi Sridhar,

Quoted 4 lines

From: Samudrala, Sridhar <[email protected]>
Sent: Thursday, April 11, 2024 4:53 AM

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Parav Pandit
Quoted 6 lines

Devices send event notifications for the IO queues, such as tx and rx queues, through event queues.

Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning

stage.

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

How do you provision tx/rx queues for VFs & SFs? Don't you need similar mechanism to setup max tx/rx queues too?

Currently we don’t. They are derived from the IO event queues. As you know, sometimes more txqs than IO event queues needed for XDP, timestamp, multiple TCs.
If needed, probably additional knob for txq, rxq can be added to restrict device resources.

Quoted 6 lines

Rather than deriving tx and rx queues from IO event queues, isn't it more user friendly to do the other way. Let the host admin set the max number of tx and rx queues allowed and the driver derive the number of ioevent queues based on those values. This will be consistent with what ethtool reports as pre-set maximum values for the corresponding

Quoted 5 lines

VF/SF.

Quoted 1 line

I agree with this point: IO EQ seems to be a mlx5 thing (or maybe I have not reviewed enough of the other drivers).

IO EQs are used by hns3, mana, mlx5, mlxsw, be2net. They might not yet have the need to provision them.

Quoted 3 lines

Rx and Tx queues are already part of
the ethtool API. This devlink feature is allowing resource limits to be configured, and a consistent API across tools would be better for users.

IO Eqs of a function are utilized by the non netdev stack as well for a multi- functionality function like rdma completion vectors.
Txq and rxq are yet another separate resource, so it is not mutually exclusive with IO EQs.

Quoted 7 lines

I can additionally add txq and rxq provisioning knob too if this is useful, yes?

Sridhar,
I didn’t lately check other drivers how usable is it, will you also implement the txq, rxq callbacks?
Please let me know I can start the work later next week for those additional knobs.

I also forgot to describe in above reply that some driver like mlx5 creates internal tx and rxqs not directly visible in channels, for xdp, timestamp, for traffic classes, dropping certain packets on rx, etc. So exact derivation of io queues is also hard there.

Regardless to me, both knobs are useful, and driver will create min() resource based on both the device limits.

#9 lore raw
Quoted 56 lines · Parav Pandit
Quoted 56 lines

From: Parav Pandit <[email protected]>
Sent: Friday, April 12, 2024 9:02 AM

Hi David, Sridhar,

Quoted 37 lines

From: David Ahern <[email protected]>
Sent: Friday, April 12, 2024 7:36 AM

Quoted 9 lines · Sridhar
Quoted 6 lines · Parav Pandit

Hi Sridhar,

Quoted 4 lines

From: Samudrala, Sridhar <[email protected]>
Sent: Thursday, April 11, 2024 4:53 AM

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Parav Pandit
Quoted 6 lines

Devices send event notifications for the IO queues, such as tx and rx queues, through event queues.

Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning

stage.

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

How do you provision tx/rx queues for VFs & SFs? Don't you need similar mechanism to setup max tx/rx queues too?

Currently we don’t. They are derived from the IO event queues. As you know, sometimes more txqs than IO event queues needed for XDP, timestamp, multiple TCs.
If needed, probably additional knob for txq, rxq can be added to restrict device resources.

Quoted 6 lines

Rather than deriving tx and rx queues from IO event queues, isn't it more user friendly to do the other way. Let the host admin set the max number of tx and rx queues allowed and the driver derive the number of ioevent queues based on those values. This will be consistent with what ethtool reports as pre-set maximum values for the corresponding

Quoted 5 lines

VF/SF.

Quoted 1 line

I agree with this point: IO EQ seems to be a mlx5 thing (or maybe I have not reviewed enough of the other drivers).

IO EQs are used by hns3, mana, mlx5, mlxsw, be2net. They might not yet have the need to provision them.

Quoted 3 lines

Rx and Tx queues are already part of
the ethtool API. This devlink feature is allowing resource limits to be configured, and a consistent API across tools would be better for users.

Quoted 6 lines
Quoted 6 lines

IO Eqs of a function are utilized by the non netdev stack as well for a multi- functionality function like rdma completion vectors.
Txq and rxq are yet another separate resource, so it is not mutually exclusive with IO EQs.

I can additionally add txq and rxq provisioning knob too if this is useful, yes?

Yes. We need knobs for txq and rxq too.
IO Eq looks like a completion queue. We don't need them for ice driver at this time, but for our idpf based control/switchdev driver we need a way to setup max number of txqueues, rxqueues, rxbuffer queues and tx completion queues.

Quoted 6 lines
Quoted 6 lines

Sridhar,
I didn’t lately check other drivers how usable is it, will you also implement the txq, rxq callbacks?
Please let me know I can start the work later next week for those additional knobs.

Sure. Our subfunction support for ice is currently under review and we are defaulting to 1 rx/tx queue for now. These knobs would be required and useful when we enable more than 1 queue for each SF.

Quoted 5 lines

I also forgot to describe in above reply that some driver like mlx5 creates internal tx and rxqs not directly visible in channels, for xdp, timestamp, for traffic classes, dropping certain packets on rx, etc. So exact derivation of io queues is also hard there >
Regardless to me, both knobs are useful, and driver will create min() resource based on both the device limits.

#10 lore raw
Quoted 67 lines

From: Samudrala, Sridhar <[email protected]>
Sent: Saturday, April 13, 2024 3:33 AM

Quoted 61 lines · Parav Pandit
Quoted 61 lines

From: Parav Pandit <[email protected]>
Sent: Friday, April 12, 2024 9:02 AM

Hi David, Sridhar,

Quoted 37 lines

From: David Ahern <[email protected]>
Sent: Friday, April 12, 2024 7:36 AM

Quoted 9 lines · Sridhar
Quoted 6 lines · Parav Pandit

Hi Sridhar,

Quoted 4 lines

From: Samudrala, Sridhar <[email protected]>
Sent: Thursday, April 11, 2024 4:53 AM

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Parav Pandit
Quoted 6 lines

Devices send event notifications for the IO queues, such as tx and rx queues, through event queues.

Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning

stage.

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

How do you provision tx/rx queues for VFs & SFs? Don't you need similar mechanism to setup max tx/rx queues too?

Currently we don’t. They are derived from the IO event queues. As you know, sometimes more txqs than IO event queues needed for XDP, timestamp, multiple TCs.
If needed, probably additional knob for txq, rxq can be added to restrict device resources.

Quoted 6 lines

Rather than deriving tx and rx queues from IO event queues, isn't it more user friendly to do the other way. Let the host admin set the max number of tx and rx queues allowed and the driver derive the number of ioevent queues based on those values. This will be consistent with what ethtool reports as pre-set maximum values for the corresponding

Quoted 5 lines

VF/SF.

Quoted 1 line

I agree with this point: IO EQ seems to be a mlx5 thing (or maybe I have not reviewed enough of the other drivers).

IO EQs are used by hns3, mana, mlx5, mlxsw, be2net. They might not yet have the need to provision them.

Quoted 3 lines

Rx and Tx queues are already part of the ethtool API. This devlink feature is allowing resource limits to be configured, and a consistent API across tools would be better for users.

IO Eqs of a function are utilized by the non netdev stack as well for a multi- functionality function like rdma completion vectors. Txq and rxq are yet another separate resource, so it is not mutually exclusive with IO EQs.

Quoted 9 lines
Quoted 1 line
Quoted 1 line

I can additionally add txq and rxq provisioning knob too if this is useful,

yes?

Yes. We need knobs for txq and rxq too.
IO Eq looks like a completion queue. We don't need them for ice driver at this time, but for our idpf based control/switchdev driver we need a way to setup max number of txqueues, rxqueues, rxbuffer queues and tx completion queues.

Understood. Make sense.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Sridhar,
I didn’t lately check other drivers how usable is it, will you also implement the txq, rxq callbacks?
Please let me know I can start the work later next week for those

Quoted 7 lines

additional

Quoted 1 line
Quoted 1 line

knobs.

Sure. Our subfunction support for ice is currently under review and we are defaulting to 1 rx/tx queue for now. These knobs would be required and useful when we enable more than 1 queue for each SF.

Got it.
I will start the kernel side patches and CC you for reviews after completing this iproute2 patch. It would be good if you can help verify on your device.

Quoted 8 lines
Quoted 2 lines

I also forgot to describe in above reply that some driver like mlx5 creates

internal tx and rxqs not directly visible in channels, for xdp, timestamp, for traffic classes, dropping certain packets on rx, etc.

Quoted 2 lines

So exact derivation of io queues is also hard there > Regardless to me, both knobs are useful, and driver will create min()

resource based on both the device limits.

Quoted 1 line
#11 lore raw
In reply to Parav Pandit
This series was applied to iproute2/iproute2-next.git (main) by David Ahern <[email protected]>:

Hello:

This series was applied to iproute2/iproute2-next.git (main) by David Ahern <[email protected]>:

Quoted 10 lines · On Wed, 10 Apr 2024 14:58:06 +0300 you wrote:

Devices send event notifications for the IO queues, such as tx and rx queues, through event queues.

Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.

example:
Get maximum IO event queues of the VF device::

[...]

Here is the summary with links:
- [v2,1/2] uapi: Update devlink kernel headers
(no matching commit)
- [v2,2/2] devlink: Support setting max_io_eqs

    https://git.kernel.org/pub/scm/network/iproute2/iproute2-next.git/commit/?id=e8add23c59b7

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html