devlink: Add unique identifier to devlink port function
Subsystems (8)
cover devlink: Add unique identifier to devlink port function
A function unique identifier (UID) is a vendor defined string of arbitrary length that universally identifies a function. The function UID can be reported by device drivers via devlink dev info command.
This patch set adds UID attribute to devlink port function that reports the UID of the function that pertains to the devlink port. Code is also added to mlx5 as the first user to implement this attribute.
The main purpose of adding this attribute is to allow users to unambiguously map between a function and the devlink port that manages it, which might be on another host.
For example, one can retrieve the UID of a function using the "devlink dev info" command and then search for the same UID in the output of "devlink port show" command.
The "devlink dev info" support for UID of a function is added by a separate patchset [1]. This patchset is submitted as an RFC to illustrate the other side of the solution.
Other existing identifiers such as serial_number or board.serial_number are not good enough as they don't guarantee uniqueness per function. For example, in a multi-host NIC all PFs report the same value.
Example output:
$ devlink port show pci/0000:03:00.0/327680 -jp
{
"port": {
"pci/0000:03:00.0/327680": {
"type": "eth",
"netdev": "pf0hpf",
"flavour": "pcipf",
"controller": 1,
"pfnum": 0,
"external": true,
"splittable": false,
"function": {
"hw_addr": "5c:25:73:37:70:5a",
"roce": "enable",
"max_io_eqs": 120,
"uid":
"C6A76AD20605BE026D23C14E70B90704F4A5F5B3F304D83B37000732BF861D48MLNXS0D0F0"
}
}
}
}[1] https://lore.kernel.org/netdev/[email protected]/
Avihai Horon (5):
devlink: Add unique identifier to devlink port function net/mlx5: Move mlx5_cmd_query_vuid() from IB to core net/mlx5: Add vhca_id argument to mlx5_core_query_vuid() net/mlx5: Add define for max VUID string size net/mlx5: Expose unique identifier in devlink port function
Documentation/netlink/specs/devlink.yaml | 3 ++ .../networking/devlink/devlink-port.rst | 12 +++++++ drivers/infiniband/hw/mlx5/cmd.c | 21 ------------ drivers/infiniband/hw/mlx5/cmd.h | 2 -- drivers/infiniband/hw/mlx5/main.c | 5 +-- .../mellanox/mlx5/core/esw/devlink_port.c | 2 ++ .../net/ethernet/mellanox/mlx5/core/eswitch.h | 2 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 34 +++++++++++++++++++ drivers/net/ethernet/mellanox/mlx5/core/fw.c | 22 ++++++++++++ include/linux/mlx5/driver.h | 3 ++ include/net/devlink.h | 8 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 32 +++++++++++++++++ 13 files changed, 122 insertions(+), 25 deletions(-)
Signature · 1 line
2.27.0
RFC 1/5 devlink: Add unique identifier to devlink port function
From: Avihai Horon <[email protected]>
A function unique identifier (UID) is a vendor defined string of arbitrary length that universally identifies a function. The function UID can be reported via devlink dev info.
Add UID attribute to devlink port function that reports the UID of the function that pertains to the devlink port.
This can be used to unambiguously map between a function and the devlink port that manages it, and vice versa.
Example output:
$ devlink port show pci/0000:03:00.0/327680 -jp
{
"port": {
"pci/0000:03:00.0/327680": {
"type": "eth",
"netdev": "pf0hpf",
"flavour": "pcipf",
"controller": 1,
"pfnum": 0,
"external": true,
"splittable": false,
"function": {
"hw_addr": "5c:25:73:37:70:5a",
"roce": "enable",
"max_io_eqs": 120,
"uid": "C6A76AD20605BE026D23C14E70B90704F4A5F5B3F304D83B37000732BF861D48MLNXS0D0F0"
}
}
}
}--- Documentation/netlink/specs/devlink.yaml | 3 ++ .../networking/devlink/devlink-port.rst | 12 +++++++ include/net/devlink.h | 8 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 32 +++++++++++++++++++ 5 files changed, 56 insertions(+)
Documentation/netlink/specs/devlink.yaml +3 -0
@@ -894,6 +894,9 @@ attribute-sets: type: bitfield32 enum: port-fn-attr-cap enum-as-flags: True + - + name: uid + type: string - name: dl-dpipe-tables
Documentation/networking/devlink/devlink-port.rst +12 -0
@@ -328,6 +328,18 @@ interrupt vector. function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32 +Function unique identifier +-------------------------- +A function unique identifier (UID) is a vendor defined string of arbitrary +length that universally identifies a function. The function UID can be reported +via devlink dev info. + +The devlink port function UID attribute reports the UID of the function that +pertains to the devlink port. + +This can be used to unambiguously map between a function and the devlink port +that manages it, and vice versa. + Subfunction ============
include/net/devlink.h +8 -0
@@ -1627,6 +1627,11 @@ void devlink_free(struct devlink *devlink);* of event queues. Should be used by device drivers to * configure maximum number of event queues * of a function managed by the devlink port. +* @port_fn_uid_get: Callback used to get port function's uid. Should be used by +* device drivers to report the uid of the function managed by +* the devlink port. +* @port_fn_uid_max_size: The maximum size of the port function's uid including +* the null terminating byte. * * Note: Driver should return -EOPNOTSUPP if it doesn't support * port function (@port_fn_*) handling for a particular port.
@@ -1682,6 +1687,9 @@ struct devlink_port_ops {int (*port_fn_max_io_eqs_set)(struct devlink_port *devlink_port,u32 max_eqs,struct netlink_ext_ack *extack); +int (*port_fn_uid_get)(struct devlink_port *devlink_port,char *fuid, +struct netlink_ext_ack *extack); +size_t port_fn_uid_max_size; }; void devlink_port_init(struct devlink *devlink,
include/uapi/linux/devlink.h +1 -0
@@ -687,6 +687,7 @@ enum devlink_port_function_attr {DEVLINK_PORT_FN_ATTR_CAPS ,/* bitfield32 */ DEVLINK_PORT_FN_ATTR_DEVLINK ,/* nested */ DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ,/* u32 */ +DEVLINK_PORT_FN_ATTR_UID ,/* string */ __DEVLINK_PORT_FUNCTION_ATTR_MAX,DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX -1
net/devlink/port.c +32 -0
@@ -207,6 +207,35 @@ static int devlink_port_fn_max_io_eqs_fill(struct devlink_port *port,return 0 ; } +static int devlink_port_fn_uid_fill (struct devlink_port *port, +struct sk_buff *msg, +struct netlink_ext_ack *extack, +bool *msg_updated) +{ +char *fuid; +int err; + +if (!port->ops->port_fn_uid_get) +return 0 ; + + fuid =kzalloc (port->ops->port_fn_uid_max_size,GFP_KERNEL ); +if (!fuid) +return -ENOMEM ; + + err = port->ops->port_fn_uid_get (port, fuid, extack); +if (err) { +kfree (fuid); +return err == -EOPNOTSUPP ?0 : err; + } + + err =nla_put_string (msg,DEVLINK_PORT_FN_ATTR_UID , fuid); +if (!err) + *msg_updated = true; + +kfree (fuid); +return err; +} +int devlink_nl_port_handle_fill (struct sk_buff *msg,struct devlink_port *devlink_port) {if (devlink_nl_put_handle (msg, devlink_port->devlink))
@@ -468,6 +497,9 @@ devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *porif (err) goto out; err =devlink_port_fn_max_io_eqs_fill (port, msg, extack, &msg_updated); +if (err) + goto out; + err =devlink_port_fn_uid_fill (port, msg, extack, &msg_updated);if (err) goto out; err = devlink_rel_devlink_handle_put(msg, port->devlink,
Signature · 1 line
2.27.0
Simon Horman
reviewerQuoted 49 lines · Moshe Shemesh
From: Avihai Horon <[email protected]>
A function unique identifier (UID) is a vendor defined string of arbitrary length that universally identifies a function. The function UID can be reported via devlink dev info.
Add UID attribute to devlink port function that reports the UID of the function that pertains to the devlink port.
This can be used to unambiguously map between a function and the devlink port that manages it, and vice versa.
Example output:
$ devlink port show pci/0000:03:00.0/327680 -jp { "port": { "pci/0000:03:00.0/327680": { "type": "eth", "netdev": "pf0hpf", "flavour": "pcipf", "controller": 1, "pfnum": 0, "external": true, "splittable": false, "function": { "hw_addr": "5c:25:73:37:70:5a", "roce": "enable", "max_io_eqs": 120, "uid": "C6A76AD20605BE026D23C14E70B90704F4A5F5B3F304D83B37000732BF861D48MLNXS0D0F0" } } } }Signed-off-by Avihai Horon <[email protected]>--- Documentation/netlink/specs/devlink.yaml | 3 ++ .../networking/devlink/devlink-port.rst | 12 +++++++ include/net/devlink.h | 8 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 32 +++++++++++++++++++ 5 files changed, 56 insertions(+)Documentation/netlink/specs/devlink.yaml +0
-0
Documentation/netlink/specs/devlink.yaml attribute-sets: · line 902
Quoted 9 lines
Documentation/netlink/specs/devlink.yaml +3
-0@@ -894,6 +894,9 @@ attribute-sets: type: bitfield32 enum: port-fn-attr-cap enum-as-flags: True + - + name: uid + type: string - name: dl-dpipe-tables
Hi Avihai,
With this patch, after running tools/net/ynl/ynl-regen.sh -f, I see the following when I run git diff. So I think this patch needs these changes too.
net/devlink/netlink_gen.c +2 -1
@@ -11,11 +11,12 @@#include <uapi/linux/devlink.h> /* Common nested types */ -const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_CAPS +1 ] = { +const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_UID +1 ] = { [DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR ] = { .type =NLA_BINARY , }, [DEVLINK_PORT_FN_ATTR_STATE ] =NLA_POLICY_MAX (NLA_U8 ,1 ), [DEVLINK_PORT_FN_ATTR_OPSTATE ] =NLA_POLICY_MAX (NLA_U8 ,1 ), [DEVLINK_PORT_FN_ATTR_CAPS ] =NLA_POLICY_BITFIELD32 (15 ), + [DEVLINK_PORT_FN_ATTR_UID ] = { .type =NLA_NUL_STRING , }, };const struct nla_policy devlink_dl_selftest_id_nl_policy[DEVLINK_ATTR_SELFTEST_ID_FLASH +1 ] = {
net/devlink/netlink_gen.h +1 -1
@@ -12,7 +12,7 @@#include <uapi/linux/devlink.h> /* Common nested types */ -extern const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_CAPS +1 ]; +extern const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_UID +1 ];extern const struct nla_policy devlink_dl_selftest_id_nl_policy[DEVLINK_ATTR_SELFTEST_ID_FLASH +1 ];/* Ops table for devlink */
...
Quoted 60 lines · Simon Horman
External email: Use caution opening links or attachments
Quoted 56 lines · Moshe Shemesh
From: Avihai Horon <[email protected]>
A function unique identifier (UID) is a vendor defined string of arbitrary length that universally identifies a function. The function UID can be reported via devlink dev info.
Add UID attribute to devlink port function that reports the UID of the function that pertains to the devlink port.
This can be used to unambiguously map between a function and the devlink port that manages it, and vice versa.
Example output:
$ devlink port show pci/0000:03:00.0/327680 -jp { "port": { "pci/0000:03:00.0/327680": { "type": "eth", "netdev": "pf0hpf", "flavour": "pcipf", "controller": 1, "pfnum": 0, "external": true, "splittable": false, "function": { "hw_addr": "5c:25:73:37:70:5a", "roce": "enable", "max_io_eqs": 120, "uid": "C6A76AD20605BE026D23C14E70B90704F4A5F5B3F304D83B37000732BF861D48MLNXS0D0F0" } } } }Signed-off-by Avihai Horon <[email protected]>--- Documentation/netlink/specs/devlink.yaml | 3 ++ .../networking/devlink/devlink-port.rst | 12 +++++++ include/net/devlink.h | 8 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 32 +++++++++++++++++++ 5 files changed, 56 insertions(+)Documentation/netlink/specs/devlink.yaml +3
-0@@ -894,6 +894,9 @@ attribute-sets: type: bitfield32 enum: port-fn-attr-cap enum-as-flags: True + - + name: uid + type: string
Quoted 7 lines
Documentation/netlink/specs/devlink.yaml attribute-sets: · line 902
Quoted 2 lines
Documentation/netlink/specs/devlink.yaml +0
-0- name: dl-dpipe-tablesHi Avihai,
With this patch, after running
tools/net/ynl/ynl-regen.sh-f, I see the following when I run git diff. So I think this patch needs these changes too.
Oh, right, I will add these in next version.
Thanks!
Quoted 34 lines
net/devlink/netlink_gen.c +2
-1@@ -11,11 +11,12 @@#include <uapi/linux/devlink.h> /* Common nested types */ -const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_CAPS +1 ] = { +const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_UID +1 ] = { [DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR ] = { .type =NLA_BINARY , }, [DEVLINK_PORT_FN_ATTR_STATE ] =NLA_POLICY_MAX (NLA_U8 ,1 ), [DEVLINK_PORT_FN_ATTR_OPSTATE ] =NLA_POLICY_MAX (NLA_U8 ,1 ), [DEVLINK_PORT_FN_ATTR_CAPS ] =NLA_POLICY_BITFIELD32 (15 ), + [DEVLINK_PORT_FN_ATTR_UID ] = { .type =NLA_NUL_STRING , }, };const struct nla_policy devlink_dl_selftest_id_nl_policy[DEVLINK_ATTR_SELFTEST_ID_FLASH +1 ] = {net/devlink/netlink_gen.h +1
-1@@ -12,7 +12,7 @@#include <uapi/linux/devlink.h> /* Common nested types */ -extern const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_CAPS +1 ]; +extern const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_UID +1 ];extern const struct nla_policy devlink_dl_selftest_id_nl_policy[DEVLINK_ATTR_SELFTEST_ID_FLASH +1 ];/* Ops table for devlink */
RFC 2/5 net/mlx5: Move mlx5_cmd_query_vuid() from IB to core
In reply to Moshe ShemeshFrom: Avihai Horon <[email protected]>
Querying of VUID will be needed in the following patches to get the function unique identifier of a devlink port function.
Move the existing function mlx5_cmd_query_vuid() to fw.c so it can be used in both core and IB. Rename it to mlx5_core_query_vuid().
No functional changes intended.
--- drivers/infiniband/hw/mlx5/cmd.c | 21 ------------------- drivers/infiniband/hw/mlx5/cmd.h | 2 -- drivers/infiniband/hw/mlx5/main.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/fw.c | 22 ++++++++++++++++++++ include/linux/mlx5/driver.h | 2 ++ 5 files changed, 25 insertions(+), 24 deletions(-)
drivers/infiniband/hw/mlx5/cmd.c +0 -21
@@ -245,24 +245,3 @@ int mlx5_cmd_uar_dealloc(struct mlx5_core_dev *dev, u32 uarn, u16 uid)MLX5_SET (dealloc_uar_in, in, uid, uid);return mlx5_cmd_exec_in (dev, dealloc_uar, in); } - -int mlx5_cmd_query_vuid (struct mlx5_core_dev *dev,bool data_direct, -char *out_vuid) -{ -u8 out[MLX5_ST_SZ_BYTES (query_vuid_out) + -MLX5_ST_SZ_BYTES (array1024_auto)] = {}; -u8 in[MLX5_ST_SZ_BYTES (query_vuid_in)] = {}; -char *vuid; -int err; - -MLX5_SET (query_vuid_in, in, opcode,MLX5_CMD_OPCODE_QUERY_VUID ); -MLX5_SET (query_vuid_in, in, vhca_id,MLX5_CAP_GEN (dev, vhca_id)); -MLX5_SET (query_vuid_in, in, data_direct, data_direct); - err =mlx5_cmd_exec (dev, in,sizeof (in), out,sizeof (out)); -if (err) -return err; - - vuid =MLX5_ADDR_OF (query_vuid_out, out, vuid); -memcpy (out_vuid, vuid,MLX5_ST_SZ_BYTES (array1024_auto)); -return 0 ; -}
drivers/infiniband/hw/mlx5/cmd.h +0 -2
@@ -58,6 +58,4 @@ int mlx5_cmd_mad_ifc(struct mlx5_ib_dev *dev, const void *inb, void *outb, u16 opmod, u8 port);int mlx5_cmd_uar_alloc (struct mlx5_core_dev *dev,u32 *uarn,u16 uid);int mlx5_cmd_uar_dealloc (struct mlx5_core_dev *dev,u32 uarn,u16 uid); -int mlx5_cmd_query_vuid (struct mlx5_core_dev *dev,bool data_direct, -char *out_vuid);#endif /* MLX5_IB_CMD_H */
drivers/infiniband/hw/mlx5/main.c +1 -1
@@ -3594,7 +3594,7 @@ static int mlx5_ib_data_direct_init(struct mlx5_ib_dev *dev) !MLX5_CAP_GEN_2(dev->mdev, query_vuid))return 0 ; - ret =mlx5_cmd_query_vuid (dev->mdev, true, vuid); + ret =mlx5_core_query_vuid (dev->mdev, true, vuid);if (ret)return ret;
drivers/net/ethernet/mellanox/mlx5/core/fw.c +22 -0
@@ -118,6 +118,28 @@ int mlx5_core_query_vendor_id(struct mlx5_core_dev *mdev, u32 *vendor_id) }EXPORT_SYMBOL (mlx5_core_query_vendor_id); +int mlx5_core_query_vuid (struct mlx5_core_dev *dev,bool data_direct, +char *out_vuid) +{ +u8 out[MLX5_ST_SZ_BYTES (query_vuid_out) + +MLX5_ST_SZ_BYTES (array1024_auto)] = {}; +u8 in[MLX5_ST_SZ_BYTES (query_vuid_in)] = {}; +char *vuid; +int err; + +MLX5_SET (query_vuid_in, in, opcode,MLX5_CMD_OPCODE_QUERY_VUID ); +MLX5_SET (query_vuid_in, in, vhca_id,MLX5_CAP_GEN (dev, vhca_id)); +MLX5_SET (query_vuid_in, in, data_direct, data_direct); + err =mlx5_cmd_exec (dev, in,sizeof (in), out,sizeof (out)); +if (err) +return err; + + vuid =MLX5_ADDR_OF (query_vuid_out, out, vuid); +memcpy (out_vuid, vuid,MLX5_ST_SZ_BYTES (array1024_auto)); +return 0 ; +} +EXPORT_SYMBOL (mlx5_core_query_vuid); +static int mlx5_get_pcam_reg (struct mlx5_core_dev *dev) { return mlx5_query_pcam_reg(dev, dev->caps.pcam,
include/linux/mlx5/driver.h +2 -0
@@ -1127,6 +1127,8 @@ int mlx5_blocking_notifier_call_chain(struct mlx5_core_dev *dev, unsigned int ev void *data);int mlx5_core_query_vendor_id (struct mlx5_core_dev *mdev,u32 *vendor_id); +int mlx5_core_query_vuid (struct mlx5_core_dev *dev,bool data_direct, +char *out_vuid);int mlx5_cmd_create_vport_lag (struct mlx5_core_dev *dev);int mlx5_cmd_destroy_vport_lag (struct mlx5_core_dev *dev);
Signature · 1 line
2.27.0
RFC 3/5 net/mlx5: Add vhca_id argument to mlx5_core_query_vuid()
In reply to Moshe ShemeshFrom: Avihai Horon <[email protected]>
Querying VUID of a specific vhca_id will be needed in the following patches. To accommodate it, add vhca_id argument to mlx5_core_query_vuid().
--- drivers/infiniband/hw/mlx5/main.c | 3 ++- drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 +++--- include/linux/mlx5/driver.h | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-)
drivers/infiniband/hw/mlx5/main.c +2 -1
@@ -3594,7 +3594,8 @@ static int mlx5_ib_data_direct_init(struct mlx5_ib_dev *dev) !MLX5_CAP_GEN_2(dev->mdev, query_vuid))return 0 ; - ret =mlx5_core_query_vuid (dev->mdev, true, vuid); + ret =mlx5_core_query_vuid (dev->mdev,MLX5_CAP_GEN (dev->mdev, vhca_id), + true, vuid);if (ret)return ret;
drivers/net/ethernet/mellanox/mlx5/core/fw.c +3 -3
@@ -118,8 +118,8 @@ int mlx5_core_query_vendor_id(struct mlx5_core_dev *mdev, u32 *vendor_id) }EXPORT_SYMBOL (mlx5_core_query_vendor_id); -int mlx5_core_query_vuid (struct mlx5_core_dev *dev,bool data_direct, -char *out_vuid) +int mlx5_core_query_vuid (struct mlx5_core_dev *dev,u16 vhca_id, +bool data_direct,char *out_vuid) {u8 out[MLX5_ST_SZ_BYTES (query_vuid_out) +MLX5_ST_SZ_BYTES (array1024_auto)] = {};
@@ -128,7 +128,7 @@ int mlx5_core_query_vuid(struct mlx5_core_dev *dev, bool data_direct,int err;MLX5_SET (query_vuid_in, in, opcode,MLX5_CMD_OPCODE_QUERY_VUID ); -MLX5_SET (query_vuid_in, in, vhca_id,MLX5_CAP_GEN (dev, vhca_id)); +MLX5_SET (query_vuid_in, in, vhca_id, vhca_id);MLX5_SET (query_vuid_in, in, data_direct, data_direct); err =mlx5_cmd_exec (dev, in,sizeof (in), out,sizeof (out));if (err)
include/linux/mlx5/driver.h +2 -2
@@ -1127,8 +1127,8 @@ int mlx5_blocking_notifier_call_chain(struct mlx5_core_dev *dev, unsigned int ev void *data);int mlx5_core_query_vendor_id (struct mlx5_core_dev *mdev,u32 *vendor_id); -int mlx5_core_query_vuid (struct mlx5_core_dev *dev,bool data_direct, -char *out_vuid); +int mlx5_core_query_vuid (struct mlx5_core_dev *dev,u16 vhca_id, +bool data_direct,char *out_vuid);int mlx5_cmd_create_vport_lag (struct mlx5_core_dev *dev);int mlx5_cmd_destroy_vport_lag (struct mlx5_core_dev *dev);
Signature · 1 line
2.27.0
RFC 4/5 net/mlx5: Add define for max VUID string size
In reply to Moshe ShemeshFrom: Avihai Horon <[email protected]>
mlx5_core_query_vuid() puts the queried VUID in a user provided buffer without setting a null terminating byte at the end. Thus, users that use the VUID as a string calculate the extra byte themselves.
To make it clearer, add a define for the max VUID string size that includes the null terminating byte.
--- drivers/infiniband/hw/mlx5/main.c | 2 +- include/linux/mlx5/driver.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mlx5/main.c +1 -1
@@ -3587,7 +3587,7 @@ static bool mlx5_ib_bind_slave_port(struct mlx5_ib_dev *ibdev,static int mlx5_ib_data_direct_init (struct mlx5_ib_dev *dev) { -char vuid[MLX5_ST_SZ_BYTES (array1024_auto) +1 ] = {}; +char vuid[MLX5_VUID_STR_MAX_SIZE ] = {};int ret; if (!MLX5_CAP_GEN(dev->mdev, data_direct) ||
include/linux/mlx5/driver.h +1 -0
@@ -1127,6 +1127,7 @@ int mlx5_blocking_notifier_call_chain(struct mlx5_core_dev *dev, unsigned int ev void *data);int mlx5_core_query_vendor_id (struct mlx5_core_dev *mdev,u32 *vendor_id); +#define MLX5_VUID_STR_MAX_SIZE (MLX5_ST_SZ_BYTES(array1024_auto) + 1)int mlx5_core_query_vuid (struct mlx5_core_dev *dev,u16 vhca_id,bool data_direct,char *out_vuid);
Signature · 1 line
2.27.0
RFC 5/5 net/mlx5: Expose unique identifier in devlink port function
In reply to Moshe ShemeshFrom: Avihai Horon <[email protected]>
The devlink port function unique identifier (UID) attribute allows to report the UID of the function that pertains to the devlink port.
Get the port function's VUID and report it as its unique identifier.
--- .../mellanox/mlx5/core/esw/devlink_port.c | 2 ++ .../net/ethernet/mellanox/mlx5/core/eswitch.h | 2 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 34 +++++++++++++++++++ 3 files changed, 38 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +2 -0
@@ -100,6 +100,8 @@ static const struct devlink_port_ops mlx5_esw_pf_vf_dl_port_ops = {#endif /* CONFIG_XFRM_OFFLOAD */ .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, + .port_fn_uid_get = mlx5_devlink_port_fn_uid_get, + .port_fn_uid_max_size = MLX5_VUID_STR_MAX_SIZE, }; static void mlx5_esw_offloads_sf_devlink_port_attrs_set(struct mlx5_eswitch *esw,
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +2 -0
@@ -580,6 +580,8 @@ int mlx5_devlink_port_fn_max_io_eqs_set(struct devlink_port *port, struct netlink_ext_ack *extack);int mlx5_devlink_port_fn_max_io_eqs_set_sf_default (struct devlink_port *port,struct netlink_ext_ack *extack); +int mlx5_devlink_port_fn_uid_get (struct devlink_port *port,char *fuid, +struct netlink_ext_ack *extack);void *mlx5_eswitch_get_uplink_priv (struct mlx5_eswitch *esw,u8 rep_type);
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +34 -0
@@ -4701,3 +4701,37 @@ mlx5_devlink_port_fn_max_io_eqs_set_sf_default(struct devlink_port *port,MLX5_ESW_DEFAULT_SF_COMP_EQS , extack); } + +int mlx5_devlink_port_fn_uid_get (struct devlink_port *port,char *fuid, +struct netlink_ext_ack *extack) +{ +struct mlx5_vport *vport =mlx5_devlink_port_vport_get (port); +u16 vport_num = vport->vport; +struct mlx5_eswitch *esw; +u16 vhca_id; +int err; + +if (vport_num !=MLX5_VPORT_PF ) +return -EOPNOTSUPP ; + + esw =mlx5_devlink_eswitch_nocheck_get (port->devlink); +if (!MLX5_CAP_GEN (esw->dev, vhca_resource_manager)) +return -EOPNOTSUPP ; + +if (!MLX5_CAP_GEN_2 (esw->dev, query_vuid)) +return -EOPNOTSUPP ; + + err =mlx5_vport_get_vhca_id (esw->dev, vport_num, &vhca_id); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed getting vhca_id of vport" ); +return err; + } + + err =mlx5_core_query_vuid (esw->dev, vhca_id, false, fuid); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed querying vuid" ); +return err; + } + +return 0 ; +}
Signature · 1 line
2.27.0
Jakub Kicinski
maintainerQuoted 16 lines · On Wed, 23 Apr 2025 16:50:37 +0300 Moshe Shemesh wrote:
A function unique identifier (UID) is a vendor defined string of arbitrary length that universally identifies a function. The function UID can be reported by device drivers via devlink dev info command.
This patch set adds UID attribute to devlink port function that reports the UID of the function that pertains to the devlink port. Code is also added to mlx5 as the first user to implement this attribute.
The main purpose of adding this attribute is to allow users to unambiguously map between a function and the devlink port that manages it, which might be on another host.
For example, one can retrieve the UID of a function using the "devlink dev info" command and then search for the same UID in the output of "devlink port show" command.
Quoted 7 lines
The "devlink dev info" support for UID of a function is added by a separate patchset [1]. This patchset is submitted as an RFC to illustrate the other side of the solution.
Other existing identifiers such as
serial_numberor board.serial_number are not good enough as they don't guarantee uniqueness per function. For example, in a multi-host NIC all PFs report the same value.
Makes sense, tho, could you please use UUID?
Let's use industry standards when possible, not "arbitrary strings".
Jiri Pirko
maintainerQuoted 21 lines · Fri, Apr 25, 2025 at 01:24:25AM +0200, [email protected] wrote:
Quoted 20 lines · On Wed, 23 Apr 2025 16:50:37 +0300 Moshe Shemesh wrote:
A function unique identifier (UID) is a vendor defined string of arbitrary length that universally identifies a function. The function UID can be reported by device drivers via devlink dev info command.
This patch set adds UID attribute to devlink port function that reports the UID of the function that pertains to the devlink port. Code is also added to mlx5 as the first user to implement this attribute.
The main purpose of adding this attribute is to allow users to unambiguously map between a function and the devlink port that manages it, which might be on another host.
For example, one can retrieve the UID of a function using the "devlink dev info" command and then search for the same UID in the output of "devlink port show" command.
The "devlink dev info" support for UID of a function is added by a separate patchset [1]. This patchset is submitted as an RFC to illustrate the other side of the solution.
Quoted 6 lines
Quoted 3 lines
Other existing identifiers such as
serial_numberor board.serial_number are not good enough as they don't guarantee uniqueness per function. For example, in a multi-host NIC all PFs report the same value.Makes sense, tho, could you please use UUID?
Let's use industry standards when possible, not "arbitrary strings".
Well, you could make the same request for serial number of asic and board. Could be uuids too, but they aren't. I mean, it makes sense to have all uids as uuid, but since the fw already exposes couple of uids as arbitrary strings, why this one should be treated differently all of the sudden?
Jakub Kicinski
maintainerQuoted 8 lines · On Fri, 25 Apr 2025 13:26:01 +0200 Jiri Pirko wrote:
Quoted 2 lines
Makes sense, tho, could you please use UUID?
Let's use industry standards when possible, not "arbitrary strings".Well, you could make the same request for serial number of asic and board. Could be uuids too, but they aren't. I mean, it makes sense to have all uids as uuid, but since the fw already exposes couple of uids as arbitrary strings, why this one should be treated differently all of the sudden?
Are you asking me what the difference is here, or you're just telling me that I'm wrong and inconsistent?
Jiri Pirko
maintainerQuoted 4 lines · Fri, Apr 25, 2025 at 07:51:45PM +0200, [email protected] wrote:
Quoted 3 lines · On Fri, 25 Apr 2025 13:26:01 +0200 Jiri Pirko wrote:
Quoted 2 lines
Makes sense, tho, could you please use UUID?
Let's use industry standards when possible, not "arbitrary strings".
Quoted 8 lines
Quoted 5 lines
Well, you could make the same request for serial number of asic and board. Could be uuids too, but they aren't. I mean, it makes sense to have all uids as uuid, but since the fw already exposes couple of uids as arbitrary strings, why this one should be treated differently all of the sudden?
Are you asking me what the difference is here, or you're just telling me that I'm wrong and inconsistent?
Both I guess? I'm just trying to understand the rationale behind the request, that's all.
Quoted 24 lines · Jakub Kicinski
External email: Use caution opening links or attachments
Quoted 20 lines · On Wed, 23 Apr 2025 16:50:37 +0300 Moshe Shemesh wrote:
A function unique identifier (UID) is a vendor defined string of arbitrary length that universally identifies a function. The function UID can be reported by device drivers via devlink dev info command.
This patch set adds UID attribute to devlink port function that reports the UID of the function that pertains to the devlink port. Code is also added to mlx5 as the first user to implement this attribute.
The main purpose of adding this attribute is to allow users to unambiguously map between a function and the devlink port that manages it, which might be on another host.
For example, one can retrieve the UID of a function using the "devlink dev info" command and then search for the same UID in the output of "devlink port show" command.
The "devlink dev info" support for UID of a function is added by a separate patchset [1]. This patchset is submitted as an RFC to illustrate the other side of the solution.
Quoted 6 lines
Quoted 3 lines
Other existing identifiers such as
serial_numberor board.serial_number are not good enough as they don't guarantee uniqueness per function. For example, in a multi-host NIC all PFs report the same value.Makes sense, tho, could you please use UUID?
Let's use industry standards when possible, not "arbitrary strings".
UUID is limited, like it has to be 128 bits, while here it is variable length up to the vendor. We would like to keep it flexible per vendor. If vendor wants to use UUID here, it will work too.
Jakub Kicinski
maintainerQuoted 7 lines · On Mon, 28 Apr 2025 15:11:04 +0300 Moshe Shemesh wrote:
Quoted 2 lines
Makes sense, tho, could you please use UUID?
Let's use industry standards when possible, not "arbitrary strings".UUID is limited, like it has to be 128 bits, while here it is variable length up to the vendor.
We would like to keep it flexible per vendor. If vendor wants to use UUID here, it will work too.
Could you please provide at least one clear user scenario for the discussion? Matching up the ports to function is presumably a means to an end for the user.
Quoted 5 lines · Jakub Kicinski
Quoted 3 lines · On Mon, 28 Apr 2025 15:11:04 +0300 Moshe Shemesh wrote:
Quoted 2 lines
Makes sense, tho, could you please use UUID?
Let's use industry standards when possible, not "arbitrary strings".
Quoted 8 lines
Quoted 4 lines
UUID is limited, like it has to be 128 bits, while here it is variable length up to the vendor.
We would like to keep it flexible per vendor. If vendor wants to use UUID here, it will work too.Could you please provide at least one clear user scenario for the discussion? Matching up the ports to function is presumably a means to an end for the user.
Sure. Multi-host system with smart-NIC, on the smart-NIC internal host we will see a representor for each PF on each of the external hosts.
However, we can't tell which representor belongs to which host. Actually, each host doesn't know about the others or where it is in the topology. The function uid can help the user match the host PF to the representor on the smart-NIC internal host and use the right representor to config the required host function.
Jakub Kicinski
maintainerQuoted 9 lines · On Tue, 29 Apr 2025 11:37:51 +0300 Moshe Shemesh wrote:
Quoted 8 lines
Quoted 4 lines
UUID is limited, like it has to be 128 bits, while here it is variable length up to the vendor.
We would like to keep it flexible per vendor. If vendor wants to use UUID here, it will work too.Could you please provide at least one clear user scenario for the discussion? Matching up the ports to function is presumably a means to an end for the user.
Quoted 7 lines
Sure. Multi-host system with smart-NIC, on the smart-NIC internal host we will see a representor for each PF on each of the external hosts. However, we can't tell which representor belongs to which host. Actually, each host doesn't know about the others or where it is in the topology. The function uid can help the user match the host PF to the representor on the smart-NIC internal host and use the right representor to config the required host function.
Insufficient information. There are many many hosts deployed with multi-host NICs which do not need this sort of matching. I'm not saying you don't have a use case. I'm saying you haven't explained it.
We exchanged so many emails on this topic, counting the emails with Jiri. And you still haven't explained to me the use case. This is ridiculous.
Mark Bloch
maintainerQuoted 18 lines · Jakub Kicinski
Quoted 9 lines · On Tue, 29 Apr 2025 11:37:51 +0300 Moshe Shemesh wrote:
Quoted 8 lines
Quoted 4 lines
UUID is limited, like it has to be 128 bits, while here it is variable length up to the vendor.
We would like to keep it flexible per vendor. If vendor wants to use UUID here, it will work too.Could you please provide at least one clear user scenario for the discussion? Matching up the ports to function is presumably a means to an end for the user.
Quoted 7 lines
Sure. Multi-host system with smart-NIC, on the smart-NIC internal host we will see a representor for each PF on each of the external hosts. However, we can't tell which representor belongs to which host. Actually, each host doesn't know about the others or where it is in the topology. The function uid can help the user match the host PF to the representor on the smart-NIC internal host and use the right representor to config the required host function.
Quoted 7 lines
Insufficient information. There are many many hosts deployed with multi-host NICs which do not need this sort of matching. I'm not saying you don't have a use case. I'm saying you haven't explained it.
We exchanged so many emails on this topic, counting the emails with Jiri. And you still haven't explained to me the use case. This is ridiculous.
Hi Jakub,
I'll try to explain the use case more clearly, I realize that some internal context at NVIDIA may not be obvious externally, and we sometimes take that for granted.
We're dealing with a multi-host system using a DPU (smart-NIC). In such a system, each external (x86) host has its own PFs/VFs/SFs, but the E-Switch manager for each PF resides on the DPU's ARM core (the internal host).
To illustrate, consider a system with two external hosts:
Host 1: PF0
VF0 on PF0
Host 2: PF0
VF0 on PF0
Each host is unaware that it's part of a multi-host system, internally, each sees its PF simply as PF0, with no notion of the global topology.
On the DPU (ARM), we see representors for each BDF. For simplicity, assume each BDF corresponds to a single devlink port. So the ARM would expose:
PF0_HOST0_REPUPLINK0_REPPF0_HOST1_REPUPLINK1_REP
In devlink terms, we're referring to the c argument in phys_port_name, which represents the controller, effectively indicating which host the BDF belongs to.
The problem we're addressing is matching the PF seen on a host to its corresponding representor on the DPU. From the ARM side, we know that this rep X belongs to pf0 on host y, but we don't which host is which. From within each host, you can't tell which host you are, because all see their PF as PF0.
With the proposed feature (along with Jiri's changes), this becomes trivial, you just match the function UID and you're done.
As a side note, I believe this feature has merit even beyond this specific use case. It makes the mapping between representors and what they represent more explicit and straightforward. Which is always a good thing from a usability and clarity standpoint.
Mark
Jakub Kicinski
maintainerQuoted 13 lines · On Sun, 4 May 2025 20:46:51 +0300 Mark Bloch wrote:
On the DPU (ARM), we see representors for each BDF. For simplicity, assume each BDF corresponds to a single devlink port. So the ARM would expose:
PF0_HOST0_REPUPLINK0_REPPF0_HOST1_REPUPLINK1_REPIn devlink terms, we're referring to the c argument in
phys_port_name, which represents the controller, effectively indicating which host the BDF belongs to.
Quoted 8 lines
The problem we're addressing is matching the PF seen on a host to its corresponding representor on the DPU. From the ARM side, we know that this rep X belongs to pf0 on host y, but we don't which host is which. From within each host, you can't tell which host you are, because all see their PF as PF0.
With the proposed feature (along with Jiri's changes), this becomes trivial, you just match the function UID and you're done.
Thanks for explaining the setup. Could you please explain the user scenario now? Perhaps thinking of it as a sequence diagram would be helpful, but whatever is easiest, just make it concrete.
Quoted 2 lines
As a side note, I believe this feature has merit even beyond this specific use case.
I also had that belief when I implemented something similar for the NFP long time ago. Jiri didn't like the solution / understand the problem at the time. But it turned out not to matter in practice.
Mark Bloch
maintainerQuoted 20 lines · Jakub Kicinski
Quoted 19 lines · On Sun, 4 May 2025 20:46:51 +0300 Mark Bloch wrote:
On the DPU (ARM), we see representors for each BDF. For simplicity, assume each BDF corresponds to a single devlink port. So the ARM would expose:
PF0_HOST0_REPUPLINK0_REPPF0_HOST1_REPUPLINK1_REPIn devlink terms, we're referring to the c argument in
phys_port_name, which represents the controller, effectively indicating which host the BDF belongs to.The problem we're addressing is matching the PF seen on a host to its corresponding representor on the DPU. From the ARM side, we know that this rep X belongs to pf0 on host y, but we don't which host is which. From within each host, you can't tell which host you are, because all see their PF as PF0.
Quoted 7 lines
Quoted 2 lines
With the proposed feature (along with Jiri's changes), this becomes trivial, you just match the function UID and you're done.
Thanks for explaining the setup. Could you please explain the user scenario now? Perhaps thinking of it as a sequence diagram would be helpful, but whatever is easiest, just make it concrete.
It's a rough flow, but I believe it clearly illustrates the use case we're targeting:
Some system configuration info:
- A static mapping file exists that defines the relationship between a host and the corresponding ARM/DPU host that manages it.
- OVN, OVS and Kubernetes are used to manage network connectivity and resource allocation.
Flow:
1. A user requests a container with networking connectivity.
2. Kubernetes allocates a VF on host X. An agent on the host handles VF configuration and sends the PF number and VF index to the central management software.
3. An agent on the DPU side detects the changes made on host X. Using the PF number and VF index, it identifies the corresponding representor, attaches it to an OVS bridge, and allows OVN to program the relevant steering rules.
This setup works well when the mapping file defines a one-to-one relationship between a host and a single ARM/DPU host. It's already supported in upstream today [1]
However, in a slightly more generic scenario like:
Control Host A: External host X
External host Y
A single ARM/DPU host manages multiple external hosts. In this case, step 2—where only the PF number and VF index are sent is insufficient. During step 3, the agent on the DPU reads the data but cannot determine which external host created the VF. As a result, it cannot correctly associate the representor with the appropriate OVS bridge.
To resolve this, we plan to modify step 2 to include the VUID along with the PF number and VF index. The DPU-side agent will use the VUID to match it with the FUID, identify the correct PF representor, and then use standard devlink mechanisms to locate the corresponding VF representor.
1: https://github.com/ovn-kubernetes/ovn-kubernetes
You can look at: go-controller/pkg/util/dpu_annotations.go for more info.
Mark
Quoted 6 lines
Quoted 2 lines
As a side note, I believe this feature has merit even beyond this specific use case.
I also had that belief when I implemented something similar for the NFP long time ago. Jiri didn't like the solution / understand the problem at the time. But it turned out not to matter in practice.
Jakub Kicinski
maintainerQuoted 13 lines · On Tue, 6 May 2025 14:25:10 +0300 Mark Bloch wrote:
Quoted 4 lines
Thanks for explaining the setup. Could you please explain the user scenario now? Perhaps thinking of it as a sequence diagram would be helpful, but whatever is easiest, just make it concrete.
It's a rough flow, but I believe it clearly illustrates the use case we're targeting:
Some system configuration info:
- A static mapping file exists that defines the relationship between a host and the corresponding ARM/DPU host that manages it.
Quoted 8 lines
- OVN, OVS and Kubernetes are used to manage network connectivity and resource allocation.
Flow:
1. A user requests a container with networking connectivity.
2. Kubernetes allocates a VF on host X. An agent on the host handles VF configuration and sends the PF number and VF index to the central management software.
What is "central management software" here? Deployment specific or some part of k8s?
Quoted 4 lines
3. An agent on the DPU side detects the changes made on host X. Using the PF number and VF index, it identifies the corresponding representor, attaches it to an OVS bridge, and allows OVN to program the relevant steering rules.
What does it mean that DPU "detects it", what's the source and mechanism of the notification?
Is it communicating with the central SW during the process?
Quoted 15 lines
This setup works well when the mapping file defines a one-to-one relationship between a host and a single ARM/DPU host. It's already supported in upstream today [1]
However, in a slightly more generic scenario like:
Control Host A: External host X
External host YA single ARM/DPU host manages multiple external hosts. In this case, step 2—where only the PF number and VF index are sent is insufficient. During step 3, the agent on the DPU reads the data but cannot determine which external host created the VF. As a result, it cannot correctly associate the representor with the appropriate OVS bridge.
Quoted 7 lines
To resolve this, we plan to modify step 2 to include the VUID along with the PF number and VF index. The DPU-side agent will use the VUID to match it with the FUID, identify the correct PF representor, and then use standard devlink mechanisms to locate the corresponding VF representor.
1: https://github.com/ovn-kubernetes/ovn-kubernetes
You can look at: go-controller/pkg/util/dpu_annotations.go for more info.
A link to the actual file / relevant code would be more helpful :(
Mark Bloch
maintainerQuoted 17 lines · Jakub Kicinski
Quoted 16 lines · On Tue, 6 May 2025 14:25:10 +0300 Mark Bloch wrote:
Quoted 4 lines
Thanks for explaining the setup. Could you please explain the user scenario now? Perhaps thinking of it as a sequence diagram would be helpful, but whatever is easiest, just make it concrete.
It's a rough flow, but I believe it clearly illustrates the use case we're targeting:
Some system configuration info:
- A static mapping file exists that defines the relationship between a host and the corresponding ARM/DPU host that manages it.
- OVN, OVS and Kubernetes are used to manage network connectivity and resource allocation.
Quoted 8 lines
Quoted 5 lines
Flow:
1. A user requests a container with networking connectivity.
2. Kubernetes allocates a VF on host X. An agent on the host handles VF configuration and sends the PF number and VF index to the central management software.What is "central management software" here? Deployment specific or some part of k8s?
It's the k8s API server.
Quoted 9 lines
Quoted 5 lines
3. An agent on the DPU side detects the changes made on host X. Using the PF number and VF index, it identifies the corresponding representor, attaches it to an OVS bridge, and allows OVN to program the relevant steering rules.
What does it mean that DPU "detects it", what's the source and mechanism of the notification?
Is it communicating with the central SW during the process?
The agent (running in the ARM/DPU) listens for events from the k8s API server.
Quoted 21 lines
Quoted 21 lines
This setup works well when the mapping file defines a one-to-one relationship between a host and a single ARM/DPU host. It's already supported in upstream today [1]
However, in a slightly more generic scenario like:
Control Host A: External host X
External host YA single ARM/DPU host manages multiple external hosts. In this case, step 2—where only the PF number and VF index are sent is insufficient. During step 3, the agent on the DPU reads the data but cannot determine which external host created the VF. As a result, it cannot correctly associate the representor with the appropriate OVS bridge.
To resolve this, we plan to modify step 2 to include the VUID along with the PF number and VF index. The DPU-side agent will use the VUID to match it with the FUID, identify the correct PF representor, and then use standard devlink mechanisms to locate the corresponding VF representor.
Quoted 4 lines
Quoted 2 lines
1: https://github.com/ovn-kubernetes/ovn-kubernetes
You can look at: go-controller/pkg/util/dpu_annotations.go for more info.A link to the actual file / relevant code would be more helpful :(
This code listens for events on the ARM/DPU from the Kubernetes API server:
https://github.com/ovn-kubernetes/ovn-kubernetes/blob/39e94d80c286f69c5416166d8acda5d1d2f9add5/go-controller/pkg/node/base_node_network_controller_dpu.go#L100
I’m not very familiar with this part of the code, so I asked our k8s team to help me identify the relevant function. Hopefully, this is what you were looking for.
Mark
Jakub Kicinski
maintainerQuoted 17 lines · On Tue, 6 May 2025 18:34:22 +0300 Mark Bloch wrote:
Quoted 8 lines
Quoted 5 lines
Flow:
1. A user requests a container with networking connectivity.
2. Kubernetes allocates a VF on host X. An agent on the host handles VF configuration and sends the PF number and VF index to the central management software.What is "central management software" here? Deployment specific or some part of k8s?
It's the k8s API server.
Quoted 6 lines
Quoted 5 lines
3. An agent on the DPU side detects the changes made on host X. Using the PF number and VF index, it identifies the corresponding representor, attaches it to an OVS bridge, and allows OVN to program the relevant steering rules.
Quoted 5 lines
Quoted 3 lines
What does it mean that DPU "detects it", what's the source and mechanism of the notification?
Is it communicating with the central SW during the process?The agent (running in the ARM/DPU) listens for events from the k8s API server.
Interesting. So a deployment with no security boundaries. The internals of the IPU and the k8s on the host are in the same domain of control.
So how does the user remotely power cycle the hosts?
What I'm getting at is that your mental model seems to be missing any sort of HW inventory database, which lists all the hosts and how they plug into the DC. The administrator of the system must already know where each machine is exactly in the chassis for basic DC ops. And that HW DB is normally queried in what you describe. If there is any security domain crossing in the picture it will require cross checking against that HW DB.
I don't think this is sufficiently well established to warrant new uAPI. You can use a UUID and pass it via ndo_get_phys_port_id.
Mark Bloch
maintainerQuoted 18 lines · Jakub Kicinski
Quoted 17 lines · On Tue, 6 May 2025 18:34:22 +0300 Mark Bloch wrote:
Quoted 8 lines
Quoted 5 lines
Flow:
1. A user requests a container with networking connectivity.
2. Kubernetes allocates a VF on host X. An agent on the host handles VF configuration and sends the PF number and VF index to the central management software.What is "central management software" here? Deployment specific or some part of k8s?
It's the k8s API server.
Quoted 6 lines
Quoted 5 lines
3. An agent on the DPU side detects the changes made on host X. Using the PF number and VF index, it identifies the corresponding representor, attaches it to an OVS bridge, and allows OVN to program the relevant steering rules.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
What does it mean that DPU "detects it", what's the source and mechanism of the notification?
Is it communicating with the central SW during the process?The agent (running in the ARM/DPU) listens for events from the k8s API server.
Interesting. So a deployment with no security boundaries. The internals of the IPU and the k8s on the host are in the same domain of control.
The VF is created on host X, but the corresponding representor appears on a different host, the IPU. Naturally, they need to be able to synchronize and exchange information for everything to work correctly.
Quoted 2 lines
So how does the user remotely power cycle the hosts?
Why should a user be able to power cycle the hosts? Are you are asking about the administrator?
Quoted 8 lines
What I'm getting at is that your mental model seems to be missing any sort of HW inventory database, which lists all the hosts and how they plug into the DC. The administrator of the system must already know where each machine is exactly in the chassis for basic DC ops. And that HW DB is normally queried in what you describe. If there is any security domain crossing in the picture it will require cross checking against that HW DB.
You're assuming that external host numbering and PCI enumeration are stable, also users can determine the mapping only after creating VFs. But even then, the mapping is indirect e.g: “I created a VF on this PF, and I see a single representor appear on the IPU, so they must be linked.” That approach is fragile and error prone.
Also, keep in mind: the external hosts and their kernels shouldn’t be aware they’re part of a multi-host system. With our current approach, you just need to provide a host-to-IPU mapping upfront, no guesswork involved.
Just thinking out loud, once this feature is in place, we might not even need a static mapping between external hosts and IPU hosts.
If VUID and FUID are globally unique, the following workflow becomes possible:
- A user requests a container with network connectivity.
- k8s allocates and configures a VF on one of the hosts. It then sends the VUID, PF number, and VF index for the new VF to the k8S API server.
- Somewhere in the network, a representor appears. An agent detects this and notifies the k8s API server, including its FUID, PF number, and VF index.
- The API server matches the VF and representor data based on the globally unique identifiers and sends the relevant information back to the agent that reported the representor creation.
- The agent attaches the representor to the OVS bridge, and with OVN configures the appropriate steering rules.
This would remove the need for pre defined host to IPU mappings and allow for a more dynamic and flexible setup.
Quoted 3 lines
I don't think this is sufficiently well established to warrant new uAPI. You can use a UUID and pass it via
ndo_get_phys_port_id.
phys_port_id only applies to netdev interfaces, whereas this use case is broader and more aligned with devlink. We believe devlink is a more appropriate place for this functionality.
Mark
Mark Bloch
maintainerQuoted 83 lines · Mark Bloch
Quoted 18 lines · Jakub Kicinski
Quoted 17 lines · On Tue, 6 May 2025 18:34:22 +0300 Mark Bloch wrote:
Quoted 8 lines
Quoted 5 lines
Flow:
1. A user requests a container with networking connectivity.
2. Kubernetes allocates a VF on host X. An agent on the host handles VF configuration and sends the PF number and VF index to the central management software.What is "central management software" here? Deployment specific or some part of k8s?
It's the k8s API server.
Quoted 6 lines
Quoted 5 lines
3. An agent on the DPU side detects the changes made on host X. Using the PF number and VF index, it identifies the corresponding representor, attaches it to an OVS bridge, and allows OVN to program the relevant steering rules.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
What does it mean that DPU "detects it", what's the source and mechanism of the notification?
Is it communicating with the central SW during the process?The agent (running in the ARM/DPU) listens for events from the k8s API server.
Interesting. So a deployment with no security boundaries. The internals of the IPU and the k8s on the host are in the same domain of control.
The VF is created on host X, but the corresponding representor appears on a different host, the IPU. Naturally, they need to be able to synchronize and exchange information for everything to work correctly.
Quoted 2 lines
So how does the user remotely power cycle the hosts?
Why should a user be able to power cycle the hosts? Are you are asking about the administrator?
Quoted 8 lines
What I'm getting at is that your mental model seems to be missing any sort of HW inventory database, which lists all the hosts and how they plug into the DC. The administrator of the system must already know where each machine is exactly in the chassis for basic DC ops. And that HW DB is normally queried in what you describe. If there is any security domain crossing in the picture it will require cross checking against that HW DB.
You're assuming that external host numbering and PCI enumeration are stable, also users can determine the mapping only after creating VFs. But even then, the mapping is indirect e.g: “I created a VF on this PF, and I see a single representor appear on the IPU, so they must be linked.” That approach is fragile and error prone.
Also, keep in mind: the external hosts and their kernels shouldn’t be aware they’re part of a multi-host system. With our current approach, you just need to provide a host-to-IPU mapping upfront, no guesswork involved.
Just thinking out loud, once this feature is in place, we might not even need a static mapping between external hosts and IPU hosts.
If VUID and FUID are globally unique, the following workflow becomes possible:
- A user requests a container with network connectivity.
- k8s allocates and configures a VF on one of the hosts. It then sends the VUID, PF number, and VF index for the new VF to the k8S API server.
- Somewhere in the network, a representor appears. An agent detects this and notifies the k8s API server, including its FUID, PF number, and VF index.
- The API server matches the VF and representor data based on the globally unique identifiers and sends the relevant information back to the agent that reported the representor creation.
- The agent attaches the representor to the OVS bridge, and with OVN configures the appropriate steering rules.This would remove the need for pre defined host to IPU mappings and allow for a more dynamic and flexible setup.
Quoted 1 line
Quoted 9 lines
Quoted 2 lines
I don't think this is sufficiently well established to warrant new uAPI. You can use a UUID and pass it via
ndo_get_phys_port_id.
phys_port_idonly applies to netdev interfaces, whereas this use case is broader and more aligned with devlink. We believe devlink is a more appropriate place for this functionality.Mark
Hi Jakub,
Just checking in, have you had a chance to review my earlier email? Would appreciate your thoughts or guidance on the right path forward.
Mark
Jakub Kicinski
maintainerQuoted 2 lines · On Wed, 14 May 2025 15:01:40 +0300 Mark Bloch wrote:
Just checking in, have you had a chance to review my earlier email? Would appreciate your thoughts or guidance on the right path forward.
Based on your previous reply I'm afraid you don't have sufficient understanding of real life deployments to be extending this uAPI. Or you're not telling me something, but I'll go with Hanlon's razor.