mlx5 ConnectX control misc driver
Subsystems (4)
v3 cover mlx5 ConnectX control misc driver
From: Saeed Mahameed <[email protected]>
V1: https://lore.kernel.org/all/[email protected]/#r
V2: https://lore.kernel.org/all/[email protected]/#r
V2->V3:
- Fix bad Sign-off line
- Fix kernel robot warnings, define a user ptr arg for umem_unreg ioctl instead of plain integer to simplify compat_ioctl usage added in V2
V1->V2:
- Provide legal statement and sign-off for dual license use
- Fix License clause to use: BSD-3-Clause OR GPL-2.0
- Fix kernel robot warnings
- Use dev_dbg directly instead of umem_dbg() local wrapper
- Implement .compat_ioctl for 32bit compatibility
- Fix mlx5ctl_info ABI structure size and alignment
- Local pointer to correct type instead of in-place cast
- Check unused fields and flags are 0 on ioctl path
- Use correct macro to declare scalar arg ioctl command
#define MLX5CTL_IOCTL_UMEM_UNREG \
_IO(MLX5CTL_IOCTL_MAGIC, 0x3)mlx5 ConnectX control misc driver
=================================
The ConnectX HW family supported by the mlx5 drivers uses an architecture where a FW component executes "mailbox RPCs" issued by the driver to make changes to the device. This results in a complex debugging environment where the FW component has information and low level configuration that needs to be accessed to userspace for debugging purposes.
Historically a userspace program was used that accessed the PCI register and config space directly through /sys/bus/pci/.../XXX and could operate these debugging interfaces in parallel with the running driver. This approach is incompatible with secure boot and kernel lockdown so this driver provides a secure and restricted interface to that.
1) The first patch in the series introduces the main driver file with the implementation of a new mlx5 auxiliary device driver to run on top mlx5_core device instances, on probe it creates a new misc device and in this patch we implement the open and release fops, On open the driver would allocate a special FW UID (user context ID) restricted to debug RPCs only, where all user debug rpcs will be executed under this UID, and on release the UID will be freed.
2) The second patch adds an info ioctl that will show the allocated UID and the available capability masks of the device and the current UID, and some other useful device information such as the underlying ConnectX
Example:
$ sudo ./mlx5ctlu mlx5_core.ctl.0
mlx5dev: 0000:00:04.0
UCTX UID: 1
UCTX CAP: 0x3
DEV UCTX CAP: 0x3
USER CAP: 0x1d3) Third patch will add the capability to execute debug RPCs under the special UID.
In the mlx5 architecture the FW RPC commands are of the format of inbox and outbox buffers. The inbox buffer contains the command rpc layout as described in the ConnectX Programmers Reference Manual (PRM) document and as defined in linux/include/mlx5/mlx5_ifc.h.
On success the user outbox buffer will be filled with the device's rpc response.
For example to query device capabilities:
a user fills out an inbox buffer with the inbox layout:
struct mlx5_ifc_query_hca_cap_in_bits
and expects an outbox buffer with the layout:
struct mlx5_ifc_cmd_hca_cap_bits
4) The fourth patch adds the ability to register user memory into the ConntectX device and create a umem object that points to that memory.
Command rpc outbox buffer is limited in size, which can be very annoying when trying to pull large traces out of the device. Many rpcs offer the ability to scatter output traces, contexts and logs directly into user space buffers in a single shot.
The registered memory will be described by a device UMEM object which has a unique umem_id, this umem_id can be later used in the rpc inbox to tell the device where to populate the response output, e.g HW traces and other debug object queries.
Example usecase, a ConnectX device coredump can be as large as 2MB. Using inline rpcs will take thousands of rpcs to get the full coredump which can consume multiple seconds.
With UMEM, it can be done in a single rpc, using 2MB of umem user buffer.
Other usecases with umem:
- dynamic HW and FW trace monitoring
- high frequency diagnostic counters sampling
- batched objects and resource dumps
See links below for information about user space tools that use this interface:
[1] https://github.com/saeedtx/mlx5ctl
[2] https://github.com/Mellanox/mstflint
see:
d) mstregdump utility
This utility dumps hardware registers from Mellanox hardware for later analysis by Mellanox.
g) mstconfig
This tool sets or queries non-volatile configurable options for Mellanox HCAs.
h) mstfwmanager
Mellanox firmware update and query utility which scans the system for available Mellanox devices (only mst PCI devices) and performs the necessary firmware updates.
i) mstreg
The mlxreg utility allows users to obtain information regarding supported access registers, such as their fields
License: BSD-3-Clause OR GPL-2.0
================================
After a review of this thread [3], and a conversation with the LF, Mellanox and NVIDIA legal continue to approve the use of a Dual GPL & Permissive License for mlx5 related driver contributions. This makes it clear to future contributors that this file may be adapted and reused under BSD-3-Clause terms on other operating systems. Contributions will be handled in the normal way and the dual license will apply automatically. If people wish to contribute significantly and opt out of a dual license they may separate their GPL only contributions in dedicated files.
Jason has a signing authority for NVIDIA and has gone through our internal process to get approval.
[3] https://lore.kernel.org/all/[email protected]/#r
================================
Saeed Mahameed (5):
mlx5: Add aux dev for ctl interface
misc: mlx5ctl: Add mlx5ctl misc driver
misc: mlx5ctl: Add info ioctl
misc: mlx5ctl: Add command rpc ioctl
misc: mlx5ctl: Add umem reg/unreg ioctl
.../userspace-api/ioctl/ioctl-number.rst | 1 + MAINTAINERS | 8 + drivers/misc/Kconfig | 1 + drivers/misc/Makefile | 1 + drivers/misc/mlx5ctl/Kconfig | 14 + drivers/misc/mlx5ctl/Makefile | 5 + drivers/misc/mlx5ctl/main.c | 579 ++++++++++++++++++ drivers/misc/mlx5ctl/umem.c | 322 ++++++++++ drivers/misc/mlx5ctl/umem.h | 17 + drivers/net/ethernet/mellanox/mlx5/core/dev.c | 8 + include/uapi/misc/mlx5ctl.h | 59 ++ 11 files changed, 1015 insertions(+) create mode 100644 drivers/misc/mlx5ctl/Kconfig create mode 100644 drivers/misc/mlx5ctl/Makefile create mode 100644 drivers/misc/mlx5ctl/main.c create mode 100644 drivers/misc/mlx5ctl/umem.c create mode 100644 drivers/misc/mlx5ctl/umem.h create mode 100644 include/uapi/misc/mlx5ctl.h
--
2.42.0
PATCH v3 1/5 mlx5: Add aux dev for ctl interface
From: Saeed Mahameed <[email protected]>
Allow ctl protocol interface auxiliary driver in mlx5.
--- drivers/net/ethernet/mellanox/mlx5/core/dev.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/dev.c +8 -0
@@ -228,8 +228,14 @@ enum {MLX5_INTERFACE_PROTOCOL_VNET ,MLX5_INTERFACE_PROTOCOL_DPLL , + MLX5_INTERFACE_PROTOCOL_CTL, }; +static bool is_ctl_supported (struct mlx5_core_dev *dev) +{ +return MLX5_CAP_GEN (dev, uctx_cap); +} + static const struct mlx5_adev_device { const char *suffix; bool (*is_supported)(struct mlx5_core_dev *dev);
@@ -252,6 +258,8 @@ static const struct mlx5_adev_device { .is_supported = &is_mp_supported }, [MLX5_INTERFACE_PROTOCOL_DPLL] = { .suffix = "dpll", .is_supported = &is_dpll_supported }, + [MLX5_INTERFACE_PROTOCOL_CTL] = { .suffix = "ctl", + .is_supported = &is_ctl_supported }, };int mlx5_adev_idx_alloc (void )
Signature · 1 line
2.42.0
PATCH v3 2/5 misc: mlx5ctl: Add mlx5ctl misc driver
In reply to Saeed MahameedFrom: Saeed Mahameed <[email protected]>
The ConnectX HW family supported by the mlx5 drivers uses an architecture where a FW component executes "mailbox RPCs" issued by the driver to make changes to the device. This results in a complex debugging environment where the FW component has information and low level configuration that needs to be accessed to userspace for debugging purposes.
Historically a userspace program was used that accessed the PCI register and config space directly through /sys/bus/pci/.../XXX and could operate these debugging interfaces in parallel with the running driver. This approach is incompatible with secure boot and kernel lockdown so this driver provides a secure and restricted interface to that same data.
On open the driver would allocate a special FW UID (user context ID) restrected to debug RPCs only, later in this series all user RPCs will be stamped with this UID.
--- MAINTAINERS | 8 + drivers/misc/Kconfig | 1 + drivers/misc/Makefile | 1 + drivers/misc/mlx5ctl/Kconfig | 14 ++ drivers/misc/mlx5ctl/Makefile | 4 + drivers/misc/mlx5ctl/main.c | 314 ++++++++++++++++++++++++++++++++++ 6 files changed, 342 insertions(+) create mode 100644 drivers/misc/mlx5ctl/Kconfig create mode 100644 drivers/misc/mlx5ctl/Makefile create mode 100644 drivers/misc/mlx5ctl/main.c
MAINTAINERS +8 -0
@@ -13835,6 +13835,14 @@ L: [email protected] S: Supported F: drivers/vdpa/mlx5/ +MELLANOX MLX5 ConnectX Diag DRIVER +M: Saeed Mahameed <[email protected]> +R: Itay Avraham <[email protected]> +L: [email protected] +S: Supported +F: drivers/misc/mlx5ctl/ +F: include/uapi/misc/mlx5ctl.h + MELLANOX MLXCPLD I2C AND MUX DRIVER M: Vadim Pasternak <[email protected]> M: Michael Shych <[email protected]>
drivers/misc/Kconfig +1 -0
@@ -579,4 +579,5 @@ source "drivers/misc/cardreader/Kconfig" source "drivers/misc/uacce/Kconfig" source "drivers/misc/pvpanic/Kconfig" source "drivers/misc/mchp_pci1xxxx/Kconfig" +source "drivers/misc/mlx5ctl/Kconfig" endmenu
drivers/misc/Makefile +1 -0
@@ -67,3 +67,4 @@ obj-$(CONFIG_TMR_MANAGER) += xilinx_tmr_manager.o obj-$(CONFIG_TMR_INJECT) += xilinx_tmr_inject.o obj-$(CONFIG_TPS6594_ESM) += tps6594-esm.o obj-$(CONFIG_TPS6594_PFSM) += tps6594-pfsm.o +obj-$(CONFIG_MLX5CTL) += mlx5ctl/
drivers/misc/mlx5ctl/Kconfig new file +14 -0
@@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0 +# + +config MLX5CTL + tristate "mlx5 ConnectX control misc driver" + depends on MLX5_CORE + help + MLX5CTL provides interface for the user process to access the debug and + configuration registers of the ConnectX hardware family + (NICs, PCI switches and SmartNIC SoCs). + This will allow configuration and debug tools to work out of the box on + mainstream kernel. + + If you don't know what to do here, say N.
drivers/misc/mlx5ctl/Makefile new file +4 -0
@@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_MLX5CTL) += mlx5ctl.o +mlx5ctl-y := main.o
drivers/misc/mlx5ctl/main.c new file +314 -0
314 lines not on this page · raw message
Signature · 1 line
2.42.0
Greg Kroah-Hartman
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
drivers/misc/mlx5ctl/Kconfig line 12
Quoted 9 lines · Saeed Mahameed
drivers/misc/mlx5ctl/Kconfig new file +9
-0+config MLX5CTL + tristate "mlx5 ConnectX control misc driver" + depends on MLX5_CORE + help + MLX5CTL provides interface for the user process to access the debug and + configuration registers of the ConnectX hardware family + (NICs, PCI switches and SmartNIC SoCs). + This will allow configuration and debug tools to work out of the box on + mainstream kernel.
Did you run checkpatch.pl on this? You lost the tabs :(
drivers/misc/mlx5ctl/main.c line 1
Quoted 4 lines
drivers/misc/mlx5ctl/main.c new file +1
-0@@ -0,0 +1,314 @@ +// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
Why is this dual licensed? Again, you are using a GPL-only api for this driver, how would that even be possible to make this code be BSD?
I thought we already discussed this, AND I talked to someone who discussed this with a nvidia lawyer already and I thought this was going to get changed. What happened to that?
thanks,
greg k-h
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 2 lines · Greg Kroah-Hartman
Why is this dual licensed? Again, you are using a GPL-only api for this driver, how would that even be possible to make this code be BSD?
The code to FreeBSD with mlx5 related dual licensed code integrated is freely available to inspect. I've never looked myself but I'm told FreeBSD changes the reference Linux code to remove the use of GPL code, and that FreeBSD has created BSD licensed versions of some kernel APIs to support that.
Quoted 3 lines
I thought we already discussed this, AND I talked to someone who discussed this with a nvidia lawyer already and I thought this was going to get changed. What happened to that?
It is in the cover letter. You asked for an approval and statement from our legal and we obtained it. Our lawyers did a review, discussed with a LF contact, and continue to instruct to use the dual license. We've done what you required us to do.
The summary I have of the call you refer to does not include a discussion or agreement about change in nvidia policy regarding mlx5 code.
Like Dave said, our lawyers are not your lawyers. Now that we have involved legal, and they have given an instruction, we must follow it.
Regards,
Jason
Greg Kroah-Hartman
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 10 lines · Jason Gunthorpe
Quoted 2 lines · Greg Kroah-Hartman
Why is this dual licensed? Again, you are using a GPL-only api for this driver, how would that even be possible to make this code be BSD?
The code to FreeBSD with mlx5 related dual licensed code integrated is freely available to inspect. I've never looked myself but I'm told FreeBSD changes the reference Linux code to remove the use of GPL code, and that FreeBSD has created BSD licensed versions of some kernel APIs to support that.
Yeah, I'm not going to get into the legality of "creating BSD licensed versions of gpl-only Linux apis" but note, lots of lawyers look longingly at those things when they consider early retirement :)
Quoted 8 lines
Quoted 3 lines
I thought we already discussed this, AND I talked to someone who discussed this with a nvidia lawyer already and I thought this was going to get changed. What happened to that?
It is in the cover letter. You asked for an approval and statement from our legal and we obtained it. Our lawyers did a review, discussed with a LF contact, and continue to instruct to use the dual license. We've done what you required us to do.
Ah, missed that, sorry, I didn't see it in the changes for this set of patches, it was in the previous submission.
Quoted 6 lines
The summary I have of the call you refer to does not include a discussion or agreement about change in nvidia policy regarding mlx5 code.
Like Dave said, our lawyers are not your lawyers. Now that we have involved legal, and they have given an instruction, we must follow it.
I think everyone involved is thankful that your lawyers are not mine :)
Ok, best of luck with this mess, I'll stop harping on it now and just point out all of the other issues here. First off, you all need to get the network maintainers to agree that this driver is ok to do this way, and I don't think that has happened yet, so I'll wait on reviewing the series until that is resolved.
thanks,
greg k-h
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 5 lines · Greg Kroah-Hartman
Ok, best of luck with this mess, I'll stop harping on it now and just point out all of the other issues here. First off, you all need to get the network maintainers to agree that this driver is ok to do this way, and I don't think that has happened yet, so I'll wait on reviewing the series until that is resolved.
As I said already, I strongly disagree with the idea that the netdev maintainers get a global veto on what happens with mlx5 devices just because they sometimes have an ethernet port on the back of the card.
This module is primarily (but not exclusively) for rdma related functionality, not netdev, and the RDMA maintainers Ack it.
Thanks,
Jason
Greg Kroah-Hartman
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 11 lines · Jason Gunthorpe
Quoted 5 lines · Greg Kroah-Hartman
Ok, best of luck with this mess, I'll stop harping on it now and just point out all of the other issues here. First off, you all need to get the network maintainers to agree that this driver is ok to do this way, and I don't think that has happened yet, so I'll wait on reviewing the series until that is resolved.
As I said already, I strongly disagree with the idea that the netdev maintainers get a global veto on what happens with mlx5 devices just because they sometimes have an ethernet port on the back of the card.
I understand you might disagree, however I hold their opinion in high regard and want to ensure that they agree that exposing device-specific debugging information for a device that deals with networking is ok to do so in a device-specific misc device node and not through some other way that other networking devices normally do (i.e. netlink or some-other-such-thing.)
Note, device-specific character devices have almost always proven to be a bad idea in the long run, I understand your immediate need to do something like this, but remember that keeping it alive for the next 20+ years is going to be tough.
Quoted 2 lines
This module is primarily (but not exclusively) for rdma related functionality, not netdev, and the RDMA maintainers Ack it.
In my mind, RDMA implies networking, as it's over a network connection, but hey, I might be wrong :)
thanks,
greg k-h
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 20 lines · Greg Kroah-Hartman
Quoted 11 lines · Jason Gunthorpe
Quoted 5 lines · Greg Kroah-Hartman
Ok, best of luck with this mess, I'll stop harping on it now and just point out all of the other issues here. First off, you all need to get the network maintainers to agree that this driver is ok to do this way, and I don't think that has happened yet, so I'll wait on reviewing the series until that is resolved.
As I said already, I strongly disagree with the idea that the netdev maintainers get a global veto on what happens with mlx5 devices just because they sometimes have an ethernet port on the back of the card.
I understand you might disagree, however I hold their opinion in high regard and want to ensure that they agree that exposing device-specific debugging information for a device that deals with networking is ok to do so in a device-specific misc device node and not through some other way that other networking devices normally do (i.e. netlink or some-other-such-thing.)
Quoted 5 lines
Note, device-specific character devices have almost always proven to be a bad idea in the long run, I understand your immediate need to do something like this, but remember that keeping it alive for the next 20+ years is going to be tough.
This driver is different as it doesn't replace existing mlx5 drivers, mlx5 functionality drivers are still there to expose the device features through the standard stacks, this is just a companion driver to access debug information, by driver and FW design mlx5ctl is not meant to manage or pilot the device like other device specific char drivers.
To be clear this debug driver (or at least an older version of it) has been already in use for over than 15 years, since the beginning of mlx5, we used to only provide it as external package called mft debug tools [1] which has the kernel parts as well. Now it's time to upstream it.
mlx5ctl will keep serving existing and future HW for the next few decades, I am pretty sure of that. as the cover-letter explains mlx5 architecture is set in stone and written in ink, the same mlx5 drivers work on any ConnectX chip since 2012, and the will keep working on the next generations of chips, mlx5ctl will be no different.
[1] https://network.nvidia.com/products/adapter-software/firmware-tools/
Quoted 3 lines
Quoted 2 lines
This module is primarily (but not exclusively) for rdma related functionality, not netdev, and the RDMA maintainers Ack it.
For Infiniband/virtio/vfio/vdpa/nvme/fpga ConnectX devices mlx5 netdev doesn't even exist, so it is not reasonable to ask that the debug interface should go via the netdev stack, mlx5ctl is needed to serve all users of mlx5 devices, not only netdev (networking).
So I really find this odd, that one stack maintainer gets a veto over all others.
Quoted 6 lines
In my mind, RDMA implies networking, as it's over a network connection, but hey, I might be wrong :)
thanks,
greg k-h
Jakub Kicinski
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 5 lines · On Mon, 27 Nov 2023 11:26:06 -0800 Saeed Mahameed wrote:
This driver is different as it doesn't replace existing mlx5 drivers, mlx5 functionality drivers are still there to expose the device features through the standard stacks, this is just a companion driver to access debug information, by driver and FW design mlx5ctl is not meant to manage or pilot the device like other device specific char drivers.
You keep saying "debug information" which is really underselling this driver. Are you not going to support mstreg?
The common development flow as far as netdev is concerned is:
- some customer is interested in a new feature of a chip
- vendor hacks the support out of tree, using oot module and/or user space tooling
- customer does a PoC with that hacked up, non-upstream solution
- if it works, vendor has to find out a proper upstream API, hopefully agreed on with other vendors
- if it doesn't match customer needs the whole thing lands in the bin
If the vendor specific PoC can be achieved with fully upstream software we lose leverage to force vendors to agree on common APIs.
This should all be self-evident for people familiar with netdev, but I thought I'd spell it out.
I understand that the device has other uses, but every modern NIC has other uses. I don't see how netdev can agree to this driver as long as there is potential for configuring random networking things thru it. All major netdev vendors have a set of out of tree tools / "expose everything misc drivers", "for debug". They will soon follow your example if we let this in.
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 19 lines · Jakub Kicinski
Quoted 5 lines · On Mon, 27 Nov 2023 11:26:06 -0800 Saeed Mahameed wrote:
This driver is different as it doesn't replace existing mlx5 drivers, mlx5 functionality drivers are still there to expose the device features through the standard stacks, this is just a companion driver to access debug information, by driver and FW design mlx5ctl is not meant to manage or pilot the device like other device specific char drivers.
You keep saying "debug information" which is really underselling this driver. Are you not going to support mstreg?
The common development flow as far as netdev is concerned is:
- some customer is interested in a new feature of a chip
- vendor hacks the support out of tree, using oot module and/or user space tooling
- customer does a PoC with that hacked up, non-upstream solution
- if it works, vendor has to find out a proper upstream API, hopefully agreed on with other vendors
- if it doesn't match customer needs the whole thing lands in the bin
Quoted 2 lines
If the vendor specific PoC can be achieved with fully upstream software we lose leverage to force vendors to agree on common APIs.
Please elaborate on what "common" API there is to create here?
Do you agree that each ASIC in the device is unique and hence will have made different trade offs - both features and nerd knobs to tune and affect the performance and runtime capabilities? If you do not agree, then we need to have a different discussion ...
If you do, please elaborate on the outline of some common API that could possibly be done here.
You said no to the devlink parameters as a way to tune an ASIC. This is a common, established tool, using a common, established message channel but in an open, free form way of letting a customer see what tunables there are for a specific H/W version and firmware version and then set them. That is about as common as can be for different vendors creating different ASICs with different goals and design objectives. Yet, you somehow expect all of them to have nerd knob X and tunable Y. That is not realistic.
Quoted 10 lines
This should all be self-evident for people familiar with netdev, but I thought I'd spell it out.
I understand that the device has other uses, but every modern NIC has other uses. I don't see how netdev can agree to this driver as long as there is potential for configuring random networking things thru it. All major netdev vendors have a set of out of tree tools / "expose everything misc drivers", "for debug". They will soon follow your example if we let this in.
Out of tree drivers are already ingrained into customers now. Mellanox (in this case) has tried many different angles at getting access to H/W unique tunables (i.e., the devlink comment) and now dumping huge amounts of data. Your response to the devlink parameters attempt is to basically abuse the firmware upgrade command as a way to push a binary blob that can contain said settings (and I still have not fully wrapped my head around the fact that you suggested that option).
What specifically are you looking for? There are different vendors and different h/w options for specific market based reasons. Your hard line stance against needs like this is what is pushing out of tree drivers and tools to continue.
Jakub Kicinski
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 12 lines · On Mon, 27 Nov 2023 21:46:28 -0700 David Ahern wrote:
Quoted 12 lines
You keep saying "debug information" which is really underselling this driver. Are you not going to support mstreg?
The common development flow as far as netdev is concerned is:
- some customer is interested in a new feature of a chip
- vendor hacks the support out of tree, using oot module and/or user space tooling
- customer does a PoC with that hacked up, non-upstream solution
- if it works, vendor has to find out a proper upstream API, hopefully agreed on with other vendors
- if it doesn't match customer needs the whole thing lands in the bin
Quoted 4 lines
Quoted 2 lines
If the vendor specific PoC can be achieved with fully upstream software we lose leverage to force vendors to agree on common APIs.
Please elaborate on what "common" API there is to create here?
Damn, am I so bad at explaining basic things or y'all are spending 5 seconds reading this and are not really paying attention? :|
Quoted 6 lines
Do you agree that each ASIC in the device is unique and hence will have made different trade offs - both features and nerd knobs to tune and affect the performance and runtime capabilities? If you do not agree, then we need to have a different discussion ...
If you do, please elaborate on the outline of some common API that could possibly be done here.
We have devlink params. If that doesn't scale we can look for other solutions but let's see them not scale _in practice_ first.
Quoted 1 line
You said no to the devlink parameters as a way to tune an ASIC.
What? When?
Quoted 7 lines
This is a common, established tool, using a common, established message channel but in an open, free form way of letting a customer see what tunables there are for a specific H/W version and firmware version and then set them. That is about as common as can be for different vendors creating different ASICs with different goals and design objectives. Yet, you somehow expect all of them to have nerd knob X and tunable Y. That is not realistic.
I don't know what you're talking about.
Quoted 18 lines
Quoted 9 lines
This should all be self-evident for people familiar with netdev, but I thought I'd spell it out.
I understand that the device has other uses, but every modern NIC has other uses. I don't see how netdev can agree to this driver as long as there is potential for configuring random networking things thru it. All major netdev vendors have a set of out of tree tools / "expose everything misc drivers", "for debug". They will soon follow your example if we let this in.
Out of tree drivers are already ingrained into customers now. Mellanox (in this case) has tried many different angles at getting access to H/W unique tunables (i.e., the devlink comment) and now dumping huge amounts of data. Your response to the devlink parameters attempt is to basically abuse the firmware upgrade command as a way to push a binary blob that can contain said settings (and I still have not fully wrapped my head around the fact that you suggested that option).
Quoted 4 lines
What specifically are you looking for? There are different vendors and different h/w options for specific market based reasons. Your hard line stance against needs like this is what is pushing out of tree drivers and tools to continue.
Sounds like you'd like a similar open-ended interface for your device.
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 3 lines · Jakub Kicinski
Quoted 1 line
You said no to the devlink parameters as a way to tune an ASIC.
What? When?
You said you already rejected it at the very start of this discussion and linked to the video recording of the rejection discussion:
https://lore.kernel.org/all/[email protected]/
This session was specifically on the 600 FW configuration parameters that mlx5 has. This is something that is done today on non-secure boot systems with direct PCI access on sysfs and would be absorbed into this driver on secure-boot systems. Ie nothing really changes from the broader ecosystem perspective.
The discussion starts at time index 22:11.
Dave (IIRC? sorry) is talking about unique things and suggesting devlink. Many people in the audiance seem to support this idea
27:00 -> 28:00 you repeated this thread's argument about said NO "occasionally you are allowed to use [devlink parameters] them"
At 29 you said just keep it all out of tree.
Oh, I chimed in around 30:00 saying it is not the kernel maintainers job to force standardization. This is a user problem.
31:25 you said again "nothing"
31:30 S390 teams would like this and pushed back against "keep it all out of tree" describing the situation as a "deadlock".
33:00 you said need two implementors for device specific parameters, which misses the point of the discussion, IMHO.
And this was at the Dublin LPC, so over a year ago.
I second Dave's question - if you do not like mlx5ctl, then what is your vision to solve all these user problems?
Jason
Jakub Kicinski
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 10 lines · On Tue, 28 Nov 2023 12:24:13 -0400 Jason Gunthorpe wrote:
You said you already rejected it at the very start of this discussion and linked to the video recording of the rejection discussion:
https://lore.kernel.org/all/[email protected]/
This session was specifically on the 600 FW configuration parameters that mlx5 has. This is something that is done today on non-secure boot systems with direct PCI access on sysfs and would be absorbed into this driver on secure-boot systems. Ie nothing really changes from the broader ecosystem perspective.
The question at LPC was about making devlink params completely transparent to the kernel. Basically added directly from FW. That what I was not happy about.
You can add as many params at the driver level as you want. In fact I asked Saeed repeatedly to start posting all those params instead of complaining.
Quoted 2 lines
I second Dave's question - if you do not like mlx5ctl, then what is your vision to solve all these user problems?
Let the users complain about the user problems. Also something I repeatedly told Saeed. His response was something along the lines of users are secret, they can't post on the list, blah, blah.
You know one user who is participating in this thread? *ME*
While the lot of you work for vendors.
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 12 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 28 Nov 2023 12:24:13 -0400 Jason Gunthorpe wrote:
You said you already rejected it at the very start of this discussion and linked to the video recording of the rejection discussion:
https://lore.kernel.org/all/[email protected]/
This session was specifically on the 600 FW configuration parameters that mlx5 has. This is something that is done today on non-secure boot systems with direct PCI access on sysfs and would be absorbed into this driver on secure-boot systems. Ie nothing really changes from the broader ecosystem perspective.
Quoted 3 lines
The question at LPC was about making devlink params completely transparent to the kernel. Basically added directly from FW. That what I was not happy about.
It is creating a back-porting nightmare for all the enterprise distributions.
Quoted 3 lines
You can add as many params at the driver level as you want. In fact I asked Saeed repeatedly to start posting all those params instead of complaining.
That really isn't what you said in the video.
Regardless, configurables are only one part of what mlx5ctl addresses, we still have all the debugability problems, which are arguably more important.
Quoted 6 lines
Quoted 2 lines
I second Dave's question - if you do not like mlx5ctl, then what is your vision to solve all these user problems?
Let the users complain about the user problems. Also something I repeatedly told Saeed. His response was something along the lines of users are secret, they can't post on the list, blah, blah.
You mean like the S390 team at IBM did in the video?
This is not a reasonable position. One of the jobs of the vendors is to aggregate the user requests. Even the giant hyperscale customers that do have the capacity to come on this list prefer to delegate these things to us.
If you want to get a direct user forum the kernel mailing list is not an appropriate place to do it.
Quoted 3 lines
You know one user who is participating in this thread? *ME*
While the lot of you work for vendors.
I'm sick of this vendor bashing. You work for *one* user. You know who talks to *every* user out there? *ME*.
User and vendors need debugging of this complex HW. I don't need to bring a parade of a dozen users to this thread to re-enforce that obvious truth. Indeed when debugging is required the vendor usually has to do it, so we are the user in this discussion.
You didn't answer the question, what is your alternative debug-ability vision here?
Jason
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 6 lines · On Tue, 28 Nov 2023 13:52:24 -0400 Jason Gunthorpe wrote:
Quoted 3 lines
The question at LPC was about making devlink params completely transparent to the kernel. Basically added directly from FW. That what I was not happy about.
It is creating a back-porting nightmare for all the enterprise distributions.
We don't care about enterprise distros, Jason, or stable kernel APIs.
Quoted 9 lines
Quoted 3 lines
You can add as many params at the driver level as you want. In fact I asked Saeed repeatedly to start posting all those params instead of complaining.
That really isn't what you said in the video.
Regardless, configurables are only one part of what mlx5ctl addresses, we still have all the debugability problems, which are arguably more important.
Read-only debug interfaces are "do whatever you want" in netdev. Params controlling them (ie. writing stuff) need to be reviewed but are also allowed.
Doesn't mlx5 have a pile of stuff in debugfs already?
Nobody bothered to answer my "are you not going support mstreg over this" question (arbitrary register writes).
Quoted 6 lines
Quoted 3 lines
Let the users complain about the user problems. Also something I repeatedly told Saeed. His response was something along the lines of users are secret, they can't post on the list, blah, blah.
You mean like the S390 team at IBM did in the video?
Quoted 7 lines
This is not a reasonable position. One of the jobs of the vendors is to aggregate the user requests. Even the giant hyperscale customers that do have the capacity to come on this list prefer to delegate these things to us.
If you want to get a direct user forum the kernel mailing list is not an appropriate place to do it.
Agree to disagree.
Quoted 7 lines
Quoted 3 lines
You know one user who is participating in this thread? *ME*
While the lot of you work for vendors.I'm sick of this vendor bashing. You work for *one* user. You know who talks to *every* user out there? *ME*.
Quoted 7 lines
User and vendors need debugging of this complex HW. I don't need to bring a parade of a dozen users to this thread to re-enforce that obvious truth. Indeed when debugging is required the vendor usually has to do it, so we are the user in this discussion.
You didn't answer the question, what is your alternative debug-ability vision here?
Covered above. And it's been discussed multiple times.
Honestly I don't want to spend any more time discussing this. Once you're ready to work together in good faith let me know.
On future revisions of this series please carry:
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 20 lines · Jakub Kicinski
Quoted 6 lines · On Tue, 28 Nov 2023 13:52:24 -0400 Jason Gunthorpe wrote:
Quoted 3 lines
The question at LPC was about making devlink params completely transparent to the kernel. Basically added directly from FW. That what I was not happy about.
It is creating a back-porting nightmare for all the enterprise distributions.
We don't care about enterprise distros, Jason, or stable kernel APIs.
Quoted 9 lines
Quoted 3 lines
You can add as many params at the driver level as you want. In fact I asked Saeed repeatedly to start posting all those params instead of complaining.
That really isn't what you said in the video.
Regardless, configurables are only one part of what mlx5ctl addresses, we still have all the debugability problems, which are arguably more important.
Quoted 6 lines
Read-only debug interfaces are "do whatever you want" in netdev. Params controlling them (ie. writing stuff) need to be reviewed but are also allowed.
Doesn't mlx5 have a pile of stuff in debugfs already?
not enough, not scalable and it's a backporting and maintenance nightmare as Jason already showed.
mlx5 supports creating millions of objects, tools need to selectively pick which objects to dump for a specific use case, if it's ok with you to do this in debugfs, then ioctl is much cleaner .. so what's your problem with mlx5ctl?
Quoted 34 lines
Nobody bothered to answer my "are you not going support mstreg over this" question (arbitrary register writes).
Quoted 6 lines
Quoted 3 lines
Let the users complain about the user problems. Also something I repeatedly told Saeed. His response was something along the lines of users are secret, they can't post on the list, blah, blah.
You mean like the S390 team at IBM did in the video?
Quoted 7 lines
This is not a reasonable position. One of the jobs of the vendors is to aggregate the user requests. Even the giant hyperscale customers that do have the capacity to come on this list prefer to delegate these things to us.
If you want to get a direct user forum the kernel mailing list is not an appropriate place to do it.
Agree to disagree.
Quoted 7 lines
Quoted 3 lines
You know one user who is participating in this thread? *ME*
While the lot of you work for vendors.I'm sick of this vendor bashing. You work for *one* user. You know who talks to *every* user out there? *ME*.
Quoted 7 lines
User and vendors need debugging of this complex HW. I don't need to bring a parade of a dozen users to this thread to re-enforce that obvious truth. Indeed when debugging is required the vendor usually has to do it, so we are the user in this discussion.
You didn't answer the question, what is your alternative debug-ability vision here?
Quoted 8 lines
Covered above. And it's been discussed multiple times.
Honestly I don't want to spend any more time discussing this. Once you're ready to work together in good faith let me know.
On future revisions of this series please carry:
Nacked-by Jakub Kicinski <[email protected]>
I asked before and I never got a technical answer, based on what?
All we got is just political views and complaints against vendors. What is your proposal for accessing every possible debug information from a vendor specific device ? devlink X Y Z, debugfs? won't work, sorry.
And I can't accept "do it out of tree" as an answer from a well established linux maintainer, the whole point of this is to have this available in every linux box with any mlx5 configuration (not only netdev) so we can start debugging on the spot.
For your claims that we need this for setting device parameters, it is simply not true, because we don't need this driver to do that, so please go back and read the cover-letter and code, and let me know what is wrong with our approach to get access to our device's debug info.
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
In reply to Jakub KicinskiQuoted 10 lines · Jakub Kicinski
Quoted 6 lines · On Tue, 28 Nov 2023 13:52:24 -0400 Jason Gunthorpe wrote:
Quoted 3 lines
The question at LPC was about making devlink params completely transparent to the kernel. Basically added directly from FW. That what I was not happy about.
It is creating a back-porting nightmare for all the enterprise distributions.
We don't care about enterprise distros, Jason, or stable kernel APIs.
Oh, I missed this one, so you don't care about users?
Users often pay to distros for support, and distros always turn to vendors for debug situations, in fact one of the high stakeholders for this is an enterprise distro..
Also Jason and I are users, and more than 300 engineers at nvidia and dozens of customers are users, deploying 100s of thousands of ConnectX chips in their fleets.
if it weren't important for us and our users, I wouldn't even fight this hard.. but it is important, and very useful, I can't express that hard enough.
So you don't care ? Then don't ask about who the users are, in the other email, apparently it's all about your personal opinion and views about vendors what drives your responses. So it is really hard to debate with you..
Greg Kroah-Hartman
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 6 lines · Saeed Mahameed
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · On Tue, 28 Nov 2023 13:52:24 -0400 Jason Gunthorpe wrote:
Quoted 3 lines
The question at LPC was about making devlink params completely transparent to the kernel. Basically added directly from FW. That what I was not happy about.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
It is creating a back-porting nightmare for all the enterprise distributions.
We don't care about enterprise distros, Jason, or stable kernel APIs.
Oh, I missed this one, so you don't care about users?
Ok, time out please. This isn't going anywhere fast except to make everyone else mad.
To Jakub's point, no, we don't care about enterprise distros, they are a consumer of our releases and while some of them pay the salaries of our developers, they really don't have much influence over our development rules as they are just so far behind in releases that their releases look nothing like what we do in places (i.e. Linux "like" just like many Android systems.)
If the enterprise distros pop in here and give us their opinions of the patchset, I would GREATLY appreciate it, as having more people review code at this point in time would be most helpful instead of having to hear about how the interfaces are broken 2 years from now.
And I think that's what is driving your work now, the "enterprise" distros finally picked up the "lock down the kernel from random PCI device access in userspace" which caused you to have to drop your userspace implementation to create a real kernel driver, correct?
And as for stable kernel apis, you all know that's just not a thing, and has nothing to do with users EXCEPT it benefits users because it keeps kernel code smaller and faster overall, that's well documented and users appreciate it.
Quoted 3 lines
Users often pay to distros for support, and distros always turn to vendors for debug situations, in fact one of the high stakeholders for this is an enterprise distro..
Hey, I was right, an enterprise distro wants this driver, great, can you get them to review it as well? I'm tired of being the only one to review patches like this and could use the help if they are going to rely on this (why do they pass that work to me?). They should be the ones helping us catch basic things like the reference count issues I pointed out, as they can test the code, I can't :)
But, let's step back a bit further.
It seems the network device normal interface for this is using devlink. And drivers are allowed to have driver-specific devlink interfaces, as you know, your driver has lots of them today. Why not just add more? What's wrong with 600+ more interfaces being added as that way they would be well documented and fit in with the existing infrastructure that you have today.
Is the issue that the firmware doesn't guarantee that these interfaces will be documented or stable or even known at this point in time? If so, how are your going to have a good userspace tool for this? What am I missing that requires a totally-new-and-custom user/kernel api from what all other network drivers are using today?
thanks,
greg k-h
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Hi,
Quoted 25 lines · Greg Kroah-Hartman
Quoted 6 lines · Saeed Mahameed
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · On Tue, 28 Nov 2023 13:52:24 -0400 Jason Gunthorpe wrote:
Quoted 3 lines
The question at LPC was about making devlink params completely transparent to the kernel. Basically added directly from FW. That what I was not happy about.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
It is creating a back-porting nightmare for all the enterprise distributions.
We don't care about enterprise distros, Jason, or stable kernel APIs.
Oh, I missed this one, so you don't care about users?
Ok, time out please. This isn't going anywhere fast except to make everyone else mad.
To Jakub's point, no, we don't care about enterprise distros, they are a consumer of our releases and while some of them pay the salaries of our developers, they really don't have much influence over our development rules as they are just so far behind in releases that their releases look nothing like what we do in places (i.e. Linux "like" just like many Android systems.)
Quoted 4 lines
If the enterprise distros pop in here and give us their opinions of the patchset, I would GREATLY appreciate it, as having more people review code at this point in time would be most helpful instead of having to hear about how the interfaces are broken 2 years from now.
We will be happy to test and review v4 of this series.
Fully interactive debugging has become essential to getting to the root cause of complex issues that arise between the types of devices being discussed and their interactions with various software layers. Turning knobs and dumping registers just isn't sufficient, and I wish we'd had this capability long ago.
Our customers have already benefited from the interactive debugging capability that these patches provide, but the full potential won't be realized until this is upstream.
Quoted 15 lines
And I think that's what is driving your work now, the "enterprise" distros finally picked up the "lock down the kernel from random PCI device access in userspace" which caused you to have to drop your userspace implementation to create a real kernel driver, correct?
And as for stable kernel apis, you all know that's just not a thing, and has nothing to do with users EXCEPT it benefits users because it keeps kernel code smaller and faster overall, that's well documented and users appreciate it.
Quoted 3 lines
Users often pay to distros for support, and distros always turn to vendors for debug situations, in fact one of the high stakeholders for this is an enterprise distro..
Quoted 6 lines
Hey, I was right, an enterprise distro wants this driver, great, can you get them to review it as well? I'm tired of being the only one to review patches like this and could use the help if they are going to rely on this (why do they pass that work to me?). They should be the ones helping us catch basic things like the reference count issues I pointed out, as they can test the code, I can't :)
Guilty as charged? But, I'm sure we are not the only ones. I'm sorry that we were not able to spot this and provide a review earlier, but we are happy to do so for v4.
Quoted 16 lines
But, let's step back a bit further.
It seems the network device normal interface for this is using devlink. And drivers are allowed to have driver-specific devlink interfaces, as you know, your driver has lots of them today. Why not just add more? What's wrong with 600+ more interfaces being added as that way they would be well documented and fit in with the existing infrastructure that you have today.
Is the issue that the firmware doesn't guarantee that these interfaces will be documented or stable or even known at this point in time? If so, how are your going to have a good userspace tool for this? What am I missing that requires a totally-new-and-custom user/kernel api from what all other network drivers are using today?
Quoted 3 lines
thanks,
greg k-h
Aron
Jakub Kicinski
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 12 lines · On Mon, 4 Dec 2023 15:37:56 -0600 Aron Silverton wrote:
Quoted 11 lines
To Jakub's point, no, we don't care about enterprise distros, they are a consumer of our releases and while some of them pay the salaries of our developers, they really don't have much influence over our development rules as they are just so far behind in releases that their releases look nothing like what we do in places (i.e. Linux "like" just like many Android systems.)
If the enterprise distros pop in here and give us their opinions of the patchset, I would GREATLY appreciate it, as having more people review code at this point in time would be most helpful instead of having to hear about how the interfaces are broken 2 years from now.
Quoted 7 lines
We will be happy to test and review v4 of this series.
Fully interactive debugging has become essential to getting to the root cause of complex issues that arise between the types of devices being discussed and their interactions with various software layers. Turning knobs and dumping registers just isn't sufficient, and I wish we'd had this capability long ago.
Could you shed some light on what issues you were debugging, broadly?
Quoted 3 lines
Our customers have already benefited from the interactive debugging capability that these patches provide, but the full potential won't be realized until this is upstream.
Can you elaborate on why "full potential won't be realized until this is upstream"? The driver looks very easy to ship as a standalone module.
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Hi Jakub,
Quoted 15 lines · Jakub Kicinski
Quoted 14 lines · On Mon, 4 Dec 2023 15:37:56 -0600 Aron Silverton wrote:
Quoted 11 lines
To Jakub's point, no, we don't care about enterprise distros, they are a consumer of our releases and while some of them pay the salaries of our developers, they really don't have much influence over our development rules as they are just so far behind in releases that their releases look nothing like what we do in places (i.e. Linux "like" just like many Android systems.)
If the enterprise distros pop in here and give us their opinions of the patchset, I would GREATLY appreciate it, as having more people review code at this point in time would be most helpful instead of having to hear about how the interfaces are broken 2 years from now.
We will be happy to test and review v4 of this series.
Quoted 8 lines
Quoted 5 lines
Fully interactive debugging has become essential to getting to the root cause of complex issues that arise between the types of devices being discussed and their interactions with various software layers. Turning knobs and dumping registers just isn't sufficient, and I wish we'd had this capability long ago.
Could you shed some light on what issues you were debugging, broadly?
I'll do my best, with two recent instances:
1. As mentioned already, we recently faced a complex problem with RDMA in KVM and were getting nowhere trying to debug using the usual methods. Mellanox support was able to use this debug interface to see what was happening on the PCI bus and prove that the issue was caused by corrupted PCIe transactions. This finally put the investigation on the correct path. The debug interface was used consistently and extensively to test theories about what was happening in the system and, ultimately, allowed the problem to be solved.
2. We've faced RDMA issues related to lost EQ doorbells, requiring complex debug, and ultimately root-caused as a defective CPU. Without interactive access to the device allowing us to test theories like, "what if we manually restart the EQ", we could not have proven this definitively.
Quoted 6 lines
Quoted 3 lines
Our customers have already benefited from the interactive debugging capability that these patches provide, but the full potential won't be realized until this is upstream.
Can you elaborate on why "full potential won't be realized until this is upstream"? The driver looks very easy to ship as a standalone module.
Firstly, We believe in working upstream and all of the advantages that that brings to all the distros as well as to us and our customers.
Secondly, Our cloud business offers many types of machine instances, some with bare metal/vfio mlx5 devices, that require customer driven debug and we want our customers to have the freedom to choose which OS they want to use.
Aron
Jakub Kicinski
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 8 lines · On Tue, 5 Dec 2023 11:11:00 -0600 Aron Silverton wrote:
1. As mentioned already, we recently faced a complex problem with RDMA in KVM and were getting nowhere trying to debug using the usual methods. Mellanox support was able to use this debug interface to see what was happening on the PCI bus and prove that the issue was caused by corrupted PCIe transactions. This finally put the investigation on the correct path. The debug interface was used consistently and extensively to test theories about what was happening in the system and, ultimately, allowed the problem to be solved.
You hit on an important point, and what is also my experience working at Meta. I may have even mentioned it in this thread already. If there is a serious issue with a complex device, there are two ways you can get support - dump all you can and send the dump to the vendor or get on a live debugging session with their engineers. Users' ability to debug those devices is practically non-existent. The idea that we need access to FW internals is predicated on the assumption that we have an ability to make sense of those internals.
Once you're on a support call with the vendor - just load a custom kernel, module, whatever, it's already extremely expensive manual labor.
Quoted 5 lines
2. We've faced RDMA issues related to lost EQ doorbells, requiring complex debug, and ultimately root-caused as a defective CPU. Without interactive access to the device allowing us to test theories like, "what if we manually restart the EQ", we could not have proven this definitively.
I'm not familiar with the RDMA debugging capabilities. Perhaps there are some gaps there. The more proprietary the implementation the harder it is to debug. An answer to that would be "try to keep as much as possible open".. and interfaces which let closed user space talk to closed FW take us in the opposite direction.
FWIW good netdevice drivers have a selftest which tests IRQ generation and EQ handling. I think that'd cover the case you're describing? IDK if mlx5 has them, but if it doesn't definitely worth adding. And I recommend running those on suspicious machines (ethtool -t, devlink has some selftests, too)
Quoted 7 lines
Firstly, We believe in working upstream and all of the advantages that that brings to all the distros as well as to us and our customers.
Secondly, Our cloud business offers many types of machine instances, some with bare metal/vfio mlx5 devices, that require customer driven debug and we want our customers to have the freedom to choose which OS they want to use.
I understand that having everything packaged and shipped together makes life easier.
If the point of the kernel at this stage of its evolution is to collect incompatible bits of vendor software, make sure they build cleanly and ship them to distros - someone should tell me, and I will relent.
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 19 lines · Jakub Kicinski
Quoted 8 lines · On Tue, 5 Dec 2023 11:11:00 -0600 Aron Silverton wrote:
1. As mentioned already, we recently faced a complex problem with RDMA in KVM and were getting nowhere trying to debug using the usual methods. Mellanox support was able to use this debug interface to see what was happening on the PCI bus and prove that the issue was caused by corrupted PCIe transactions. This finally put the investigation on the correct path. The debug interface was used consistently and extensively to test theories about what was happening in the system and, ultimately, allowed the problem to be solved.
You hit on an important point, and what is also my experience working at Meta. I may have even mentioned it in this thread already. If there is a serious issue with a complex device, there are two ways you can get support - dump all you can and send the dump to the vendor or get on a live debugging session with their engineers. Users' ability to debug those devices is practically non-existent. The idea that we need access to FW internals is predicated on the assumption that we have an ability to make sense of those internals.
Quoted 2 lines
Once you're on a support call with the vendor - just load a custom kernel, module, whatever, it's already extremely expensive manual labor.
You rail against out of tree drivers and vendor proprietary tools, and now you argue for just that. There is no reason debugging capabilities can not be built into the OS and used when needed. That means anything needed - from kernel modules to userspace tools.
The Meta data point is not representative of the world at large - different scale, different needs, different expertise on staff (OS and H/W). Getting S/W installed (especially anything requiring a compiler) in a production server (and VMs) is not an easy request and in many cases not even possible.
When a customer hits problem, the standard steps are to run a script, generate a tar file and ship it to the OS vendor. Engineers at the OS vendor go through it and may need other data - like getting detailed dumps from individual pieces of H/W. Every time those requests require going to a vendor web site to pull down vendor tools, get permission to install them, schedule the run of said tool ... it only serves to drag out the debugging process. ie., this open-ended stance only serves to hurt Linux users.
Jakub Kicinski
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
David, I agree with your points. I think you're misreading what I said.
Quoted 2 lines · On Thu, 7 Dec 2023 08:54:51 -0700 David Ahern wrote:
You rail against out of tree drivers and vendor proprietary tools, and now you argue for just that.
I don't rail against out of tree drivers, very much the opposite. Linux supports out of tree modules, and I agree that's 100% the correct thing to do. I'd encourage more people to take advantage of that. The problem is quite the opposite, all the "security hardening" is making it almost impossible for users to take advantage of OOT modules.
Quoted 9 lines
There is no reason debugging capabilities
can not be built into the OS and used when needed. That means anything needed - from kernel modules to userspace tools.The Meta data point is not representative of the world at large - different scale, different needs, different expertise on staff (OS and H/W). Getting S/W installed (especially anything requiring a compiler) in a production server (and VMs) is not an easy request and in many cases not even possible.
I did not say it's easy.
Quoted 4 lines
When a customer hits problem, the standard steps are to run a script, generate a tar file and ship it to the OS vendor. Engineers at the OS vendor go through it and may need other data - like getting detailed dumps from individual pieces of H/W.
You say that like this is not _exactly_ what I just said!?
Quoted 5 lines
Every time those requests require
going to a vendor web site to pull down vendor tools, get permission to install them, schedule the run of said tool ... it only serves to drag out the debugging process. ie., this open-ended stance only serves to hurt Linux users.
Right, exactly. What are you arguing with then? As I said - we have a very open / accommodating policy for extracting all sort of debug and state dumps. You can put whatever read only stuff you want in debugfs** Read-write interfaces must be constrained to a clear set of commands / settings but also very much allowed. As you said users need to be able to extract debug info to share with the vendors, no tools necessary.
** (this will surprise you but you can also put stats there, if they
are custom, I don't care if they go into ethtool -S or debugfs)
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
In reply to Jakub KicinskiQuoted 34 lines · Jakub Kicinski
Quoted 8 lines · On Tue, 5 Dec 2023 11:11:00 -0600 Aron Silverton wrote:
1. As mentioned already, we recently faced a complex problem with RDMA in KVM and were getting nowhere trying to debug using the usual methods. Mellanox support was able to use this debug interface to see what was happening on the PCI bus and prove that the issue was caused by corrupted PCIe transactions. This finally put the investigation on the correct path. The debug interface was used consistently and extensively to test theories about what was happening in the system and, ultimately, allowed the problem to be solved.
You hit on an important point, and what is also my experience working at Meta. I may have even mentioned it in this thread already. If there is a serious issue with a complex device, there are two ways you can get support - dump all you can and send the dump to the vendor or get on a live debugging session with their engineers. Users' ability to debug those devices is practically non-existent. The idea that we need access to FW internals is predicated on the assumption that we have an ability to make sense of those internals.
Once you're on a support call with the vendor - just load a custom kernel, module, whatever, it's already extremely expensive manual labor.
Quoted 5 lines
2. We've faced RDMA issues related to lost EQ doorbells, requiring complex debug, and ultimately root-caused as a defective CPU. Without interactive access to the device allowing us to test theories like, "what if we manually restart the EQ", we could not have proven this definitively.
I'm not familiar with the RDMA debugging capabilities. Perhaps there are some gaps there. The more proprietary the implementation the harder it is to debug. An answer to that would be "try to keep as much as possible open".. and interfaces which let closed user space talk to closed FW take us in the opposite direction.
Quoted 5 lines
FWIW good netdevice drivers have a selftest which tests IRQ generation and EQ handling. I think that'd cover the case you're describing? IDK if mlx5 has them, but if it doesn't definitely worth adding. And I recommend running those on suspicious machines (ethtool -t, devlink has some selftests, too)
Essentially, a warning light, and that doesn't solve the underlying problem. We still need experts (e.g., vendors) to investigate with their toolsets when and where the problem occurs.
I offered this as an example of one issue we solved. I cannot predict what kind of issues will pop up in the future, and writing a self-test for every possible situation is impossible by definition.
Quoted 11 lines
Quoted 8 lines
Firstly, We believe in working upstream and all of the advantages that that brings to all the distros as well as to us and our customers.
Secondly, Our cloud business offers many types of machine instances, some with bare metal/vfio mlx5 devices, that require customer driven debug and we want our customers to have the freedom to choose which OS they want to use.
I understand that having everything packaged and shipped together makes life easier.
I think it is a requirement. We operate with Secure Boot. The kernel is locked down. We don't have debugfs access, even if it were sufficient, and we cannot compile and load modules. Even without Secure Boot, there may not be a build environment available.
We really need the module ready-to-go when the debug calls for it - no building, no reboots, no months long attempts to reproduce in some lab - just immediate availability of the debug interface on the affected machine.
Quoted 4 lines
If the point of the kernel at this stage of its evolution is to collect incompatible bits of vendor software, make sure they build cleanly and ship them to distros - someone should tell me, and I will relent.
I'm not sure I follow you... The mlx5ctl driver seems very compatible with the mlx5 device driver. I may be misunderstanding.
Jakub Kicinski
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 7 lines · On Thu, 7 Dec 2023 10:41:25 -0600 Aron Silverton wrote:
Quoted 2 lines
I understand that having everything packaged and shipped together makes life easier.
I think it is a requirement. We operate with Secure Boot. The kernel is locked down. We don't have debugfs access, even if it were sufficient, and we cannot compile and load modules. Even without Secure Boot, there may not be a build environment available.
This 'no debugfs' requirement is a kernel lockdown thing, I presume? Are we expected to throw debugfs out the window and for all vendors to reimplement their debug functionality via a misc driver taking arbitrary ioctls? Not only does that sound like a complete waste of time and going backward in terms of quality of the interfaces, needing custom vendor tools etc. etc., but also you go from (hopefully somewhat) upstream reviewed debugfs interface to an interface where the only security assurance is vendor telling you "trust me, it's all good".
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 9 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 7 Dec 2023 10:41:25 -0600 Aron Silverton wrote:
Quoted 2 lines
I understand that having everything packaged and shipped together makes life easier.
I think it is a requirement. We operate with Secure Boot. The kernel is locked down. We don't have debugfs access, even if it were sufficient, and we cannot compile and load modules. Even without Secure Boot, there may not be a build environment available.
Quoted 8 lines
This 'no debugfs' requirement is a kernel lockdown thing, I presume? Are we expected to throw debugfs out the window and for all vendors to reimplement their debug functionality via a misc driver taking arbitrary ioctls? Not only does that sound like a complete waste of time and going backward in terms of quality of the interfaces, needing custom vendor tools etc. etc., but also you go from (hopefully somewhat) upstream reviewed debugfs interface to an interface where the only security assurance is vendor telling you "trust me, it's all good".
IIRC, with lockdown, we can read from debugfs IFF the entries' permissions are 0400. We cannot write. It's not suitable for implementing an interactive debug interface.
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 19 lines · Aron Silverton
Quoted 9 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 7 Dec 2023 10:41:25 -0600 Aron Silverton wrote:
Quoted 2 lines
I understand that having everything packaged and shipped together makes life easier.
I think it is a requirement. We operate with Secure Boot. The kernel is locked down. We don't have debugfs access, even if it were sufficient, and we cannot compile and load modules. Even without Secure Boot, there may not be a build environment available.
Quoted 8 lines
This 'no debugfs' requirement is a kernel lockdown thing, I presume? Are we expected to throw debugfs out the window and for all vendors to reimplement their debug functionality via a misc driver taking arbitrary ioctls? Not only does that sound like a complete waste of time and going backward in terms of quality of the interfaces, needing custom vendor tools etc. etc., but also you go from (hopefully somewhat) upstream reviewed debugfs interface to an interface where the only security assurance is vendor telling you "trust me, it's all good".
Quoted 3 lines
IIRC, with lockdown, we can read from debugfs IFF the entries' permissions are 0400. We cannot write. It's not suitable for implementing an interactive debug interface.
I would like to add that debugfs is usually used to expose the driver software states, as it evolves and changes with the driver code, but as I explained in the other email, it's clearly not a good solution to expose arbitrary objects of complex devices, that require interactive and selective debug interfaces tailored to the user use-case.
Greg Kroah-Hartman
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 5 lines · Saeed Mahameed
I would like to add that debugfs is usually used to expose the driver software states, as it evolves and changes with the driver code, but as I explained in the other email, it's clearly not a good solution to expose arbitrary objects of complex devices, that require interactive and selective debug interfaces tailored to the user use-case.
Why not? Remember, the only rule in debugfs is "there are no rules!"
Well, there is one practical one, "do not rely on debugfs for any functioning system properties", i.e. "if access to debugfs is not present, or something in debugfs breaks, the kernel should continue to work just fine with no change in operations". But that's an overall system-level rule, not a rule for what you can put into debugfs.
Have fun!
greg k-h
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 8 lines · Greg Kroah-Hartman
Quoted 5 lines · Saeed Mahameed
I would like to add that debugfs is usually used to expose the driver software states, as it evolves and changes with the driver code, but as I explained in the other email, it's clearly not a good solution to expose arbitrary objects of complex devices, that require interactive and selective debug interfaces tailored to the user use-case.
Why not? Remember, the only rule in debugfs is "there are no rules!"
We already have debugfs files to issue RPCs. They are not secure and not lockdown compatible. Few users have been interested in this, Aron does a good job explaining the general perspective I've seen in many places.
Users want an in-tree solution that is compatible with lockdown. A solution that works for all the mlx5 deployment modes (including Infiniband native without netdev) and covers most of the functionality they previously enjoyed with the /sys/../resource based tooling.
This series delivers that.
Nobody has offered an alterative vision that achieves the same thing. There have been lots of suggestions how to do small little parts, but not everything together as this does.
Quoted 2 lines
Well, there is one practical one, "do not rely on debugfs for any functioning system properties"
Jakub expressed additional "netdev only" rules for debugfs.
Read-write interfaces must be constrained to a clear set of commands / settings
Which I think is what Saeed is reacting to.
Jason
Greg Kroah-Hartman
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
In reply to Aron SilvertonQuoted 19 lines · Aron Silverton
Quoted 9 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 7 Dec 2023 10:41:25 -0600 Aron Silverton wrote:
Quoted 2 lines
I understand that having everything packaged and shipped together makes life easier.
I think it is a requirement. We operate with Secure Boot. The kernel is locked down. We don't have debugfs access, even if it were sufficient, and we cannot compile and load modules. Even without Secure Boot, there may not be a build environment available.
Quoted 8 lines
This 'no debugfs' requirement is a kernel lockdown thing, I presume? Are we expected to throw debugfs out the window and for all vendors to reimplement their debug functionality via a misc driver taking arbitrary ioctls? Not only does that sound like a complete waste of time and going backward in terms of quality of the interfaces, needing custom vendor tools etc. etc., but also you go from (hopefully somewhat) upstream reviewed debugfs interface to an interface where the only security assurance is vendor telling you "trust me, it's all good".
Quoted 3 lines
IIRC, with lockdown, we can read from debugfs IFF the entries' permissions are 0400. We cannot write. It's not suitable for implementing an interactive debug interface.
This is a policy decision that a distro decides to do, and I have seen it happen many times. The problem with this is then, as you have found out, vendors try to work around the removal of access to debugfs by creating new interfaces like misc drivers and sysfs files to emulate what they were previously exporting with debugfs.
When they do that, they break the reason why the distro/vendor decided to prevent access to debugfs in the first place, making the whole system insecure again!
I see this happen all the time in Android devices as Android restricted access to debugfs many years ago to try to solve the problem of drivers doing insecure things there. Those drivers then just moved those insecure operations to a different interface, making the system insecure again.
To stop this cat-and-mouse game, work with the vendors that are implementing these requirements to shut down access to these interfaces. That is a policy decision made by them, and does NOT mean that the kernel community needs to then start taking code to circumvent those decisions as that makes the whole thing pointless.
So in short, use debugfs, that is what it was designed for. If a vendor does not wish to enable access to debugfs, then that is their decision (probably a good one), don't circumvent it by making a new interface, as odds are, the vendor will eventually realize it and work to cut off that new interface as well, as they rightfully should.
thanks,
greg k-h
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 3 lines · Greg Kroah-Hartman
When they do that, they break the reason why the distro/vendor decided to prevent access to debugfs in the first place, making the whole system insecure again!
Maybe others have, be we did not!
In fact Saeed addresses this explicitly in the cover letter:
Historically a userspace program was used that accessed the PCI register and config space directly through /sys/bus/pci/.../XXX and could operate these debugging interfaces in parallel with the running driver. This approach is incompatible with secure boot and kernel lockdown so this driver provides a secure and restricted interface to that.
We did a lot of work here to build an interface that is compatible with the security principles of lockdown. This work is embodied by the new FW bit MLX5_UCTX_OBJECT_CAP_TOOLS_RESOURCES which causes the FW to run these RPCs in a security context that is compatible with lockdown.
The overall philosophical architecture is similar to the NVMe vendor command channel which is also a way to deliver opaque RPCs to a device FW and is considered lockdown compatible.
This series is not some insecure attempt to bypass lockdown like you may have seen in Android. mlx5 is the driver for the most popular high speed NIC in the world. Our customers take security seriously, and we take security seriously.
Jason
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
In reply to Aron SilvertonQuoted 72 lines · Aron Silverton
Quoted 34 lines · Jakub Kicinski
Quoted 8 lines · On Tue, 5 Dec 2023 11:11:00 -0600 Aron Silverton wrote:
1. As mentioned already, we recently faced a complex problem with RDMA in KVM and were getting nowhere trying to debug using the usual methods. Mellanox support was able to use this debug interface to see what was happening on the PCI bus and prove that the issue was caused by corrupted PCIe transactions. This finally put the investigation on the correct path. The debug interface was used consistently and extensively to test theories about what was happening in the system and, ultimately, allowed the problem to be solved.
You hit on an important point, and what is also my experience working at Meta. I may have even mentioned it in this thread already. If there is a serious issue with a complex device, there are two ways you can get support - dump all you can and send the dump to the vendor or get on a live debugging session with their engineers. Users' ability to debug those devices is practically non-existent. The idea that we need access to FW internals is predicated on the assumption that we have an ability to make sense of those internals.
Once you're on a support call with the vendor - just load a custom kernel, module, whatever, it's already extremely expensive manual labor.
Quoted 5 lines
2. We've faced RDMA issues related to lost EQ doorbells, requiring complex debug, and ultimately root-caused as a defective CPU. Without interactive access to the device allowing us to test theories like, "what if we manually restart the EQ", we could not have proven this definitively.
I'm not familiar with the RDMA debugging capabilities. Perhaps there are some gaps there. The more proprietary the implementation the harder it is to debug. An answer to that would be "try to keep as much as possible open".. and interfaces which let closed user space talk to closed FW take us in the opposite direction.
Quoted 5 lines
FWIW good netdevice drivers have a selftest which tests IRQ generation and EQ handling. I think that'd cover the case you're describing? IDK if mlx5 has them, but if it doesn't definitely worth adding. And I recommend running those on suspicious machines (ethtool -t, devlink has some selftests, too)
Essentially, a warning light, and that doesn't solve the underlying problem. We still need experts (e.g., vendors) to investigate with their toolsets when and where the problem occurs.
I offered this as an example of one issue we solved. I cannot predict what kind of issues will pop up in the future, and writing a self-test for every possible situation is impossible by definition.
Quoted 11 lines
Quoted 8 lines
Firstly, We believe in working upstream and all of the advantages that that brings to all the distros as well as to us and our customers.
Secondly, Our cloud business offers many types of machine instances, some with bare metal/vfio mlx5 devices, that require customer driven debug and we want our customers to have the freedom to choose which OS they want to use.
I understand that having everything packaged and shipped together makes life easier.
I think it is a requirement. We operate with Secure Boot. The kernel is locked down. We don't have debugfs access, even if it were sufficient, and we cannot compile and load modules. Even without Secure Boot, there may not be a build environment available.
We really need the module ready-to-go when the debug calls for it - no building, no reboots, no months long attempts to reproduce in some lab - just immediate availability of the debug interface on the affected machine.
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
If the point of the kernel at this stage of its evolution is to collect incompatible bits of vendor software, make sure they build cleanly and ship them to distros - someone should tell me, and I will relent.
I'm not sure I follow you... The mlx5ctl driver seems very compatible with the mlx5 device driver. I may be misunderstanding.
mlx5ctl is 100% compatible with mlx5 ConnectX open spec [1], and supports any mlx5 driven stacks, not only netdev, it is able to expose millions of objects and device states interactively, debugfs would explode if we even try to accommodate some of these objects or states via debugfs, not to mention it is also impossible to maintain a stable debugfs output for such a huge data set, when this mlx5ctl interface speaks out a clear and open ConnectX language, which is the hole point of the driver.
ConnectX is a highly programmable device for the enduser, and we have a very open / accommodating policy, an advanced user who can read the open spec [1], will also have the ability to do self-debug of their own RDMA/DPU/FPGA apps or similar usecases.
Also I would like to repeat, this is not touching netdev, netdev's policies do not apply to the greater kernel or RDMA, and we have use cases with pure-infiniband/DPU/FPGA cards that have no netdev at all, or other cases with pur virtio instances, and much more.
[1] https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 4 lines · Saeed Mahameed
Also I would like to repeat, this is not touching netdev, netdev's policies do not apply to the greater kernel or RDMA, and we have use cases with pure-infiniband/DPU/FPGA cards that have no netdev at all, or other cases with pur virtio instances, and much more.
Yes. I mean just about every complex block driver has some kind of vendor spcific tooling for debugging, statistics, etc. Trying to deny it just because one function expose by a device is a network device is even more silly than disallowing it for pure net devices (which already tend to be complex beasts).
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
In reply to Jakub KicinskiQuoted 12 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 28 Nov 2023 12:24:13 -0400 Jason Gunthorpe wrote:
You said you already rejected it at the very start of this discussion and linked to the video recording of the rejection discussion:
https://lore.kernel.org/all/[email protected]/
This session was specifically on the 600 FW configuration parameters that mlx5 has. This is something that is done today on non-secure boot systems with direct PCI access on sysfs and would be absorbed into this driver on secure-boot systems. Ie nothing really changes from the broader ecosystem perspective.
Quoted 8 lines
The question at LPC was about making devlink params completely transparent to the kernel. Basically added directly from FW. That what I was not happy about.
You can add as many params at the driver level as you want. In fact I asked Saeed repeatedly to start posting all those params instead of complaining.
We posted many params over the years the you shot down on the spot, do you really expect me to implement 600 of those knowing that you will nack 80% of them asking to have common knobs for all vendors, and you know that is impossible.
you nack patches then ask for a porpossal, we came up with many proposal and discussed them face to face on multiple occasions, LPC/netconf etc, then you ask for patches, then you nack again, we are just going in circles here..
Quoted 7 lines
Quoted 2 lines
I second Dave's question - if you do not like mlx5ctl, then what is your vision to solve all these user problems?
Let the users complain about the user problems. Also something I repeatedly told Saeed. His response was something along the lines of users are secret, they can't post on the list, blah, blah.
I never said it is a secret, but I can't publicly reveal who my customers are and what they want, you know very well who asked for the high frequency counter sampling.. So we came up with a very clear solution, that has nothing to do with netdev, since for that particular use-case it is not netdev specific, and netdev stack isn't even present.
Quoted 3 lines
You know one user who is participating in this thread? *ME*
While the lot of you work for vendors.
And how *YOU* expect the vendors to debug *YOUR* issues, if you don't allow them to access their HW?
Asking all vendors to use *YOUR* "devlink generic_dev generic_knob" is an insult to all vendors, how about you provide the ASIC design and RTLs to all vendors and we just manufacture it for you..
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
In reply to Jakub KicinskiQuoted 28 lines · Jakub Kicinski
Quoted 12 lines · On Mon, 27 Nov 2023 21:46:28 -0700 David Ahern wrote:
Quoted 12 lines
You keep saying "debug information" which is really underselling this driver. Are you not going to support mstreg?
The common development flow as far as netdev is concerned is:
- some customer is interested in a new feature of a chip
- vendor hacks the support out of tree, using oot module and/or user space tooling
- customer does a PoC with that hacked up, non-upstream solution
- if it works, vendor has to find out a proper upstream API, hopefully agreed on with other vendors
- if it doesn't match customer needs the whole thing lands in the binQuoted 4 lines
Quoted 2 lines
If the vendor specific PoC can be achieved with fully upstream software we lose leverage to force vendors to agree on common APIs.
Please elaborate on what "common" API there is to create here?
Damn, am I so bad at explaining basic things or y'all are spending 5 seconds reading this and are not really paying attention? :|
Quoted 6 lines
Do you agree that each ASIC in the device is unique and hence will have made different trade offs - both features and nerd knobs to tune and affect the performance and runtime capabilities? If you do not agree, then we need to have a different discussion ...
If you do, please elaborate on the outline of some common API that could possibly be done here.
Quoted 6 lines
We have devlink params. If that doesn't scale we can look for other solutions but let's see them not scale
_inpractice_first.Quoted 1 line
You said no to the devlink parameters as a way to tune an ASIC.
What? When?
Jason responded with the same LPC reference, so I will only add a reference to the slides for interested parties:
https://lpc.events/event/16/contributions/1359/attachments/1092/2094/FW-Centric-devices.pdf
Quoted 2 lines
Sounds like you'd like a similar open-ended interface for your device.
That's the danger of chiming in on threads like this - people drawing conclusions they should not.
My interest on this thread is along the same lines for commenting during both the LPC 2022 discussion and again at netconf this year - trying to understand and keep track of the strongly held opinions of maintainers and what options are deemed off limits for similar needs. No vendor wants to go in circles redesigning and rewriting s/w.
Greg Kroah-Hartman
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
In reply to Saeed Mahameeddrivers/misc/mlx5ctl/main.c line 25
Quoted 8 lines · Saeed Mahameed
drivers/misc/mlx5ctl/main.c new file +8
-0+struct mlx5ctl_dev { +struct mlx5_core_dev *mdev; +struct miscdevice miscdev; +struct auxiliary_device *adev; +struct list_head fd_list; +spinlock_t fd_list_lock;/* protect list add/del */ +struct rw_semaphore rw_lock; +struct kref refcount;
You now have 2 different things that control the lifespan of this structure. We really need some way to automatically check this so that people don't keep making this same mistake, it happens all the time :(
Please pick one structure (miscdevice) or the other (kref) to control the lifespan, having 2 will just not work.
Also, why a rw_semaphore? Only use those if you can prove with a benchmark that it is actually faster, otherwise it's simpler to just use a normal mutex (hint, you are changing the fields in the structure with the read lock held, which feels very wrong, and so needs a LOT of documentation, or just use a normal mutex...)
thanks,
greg k-h
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 14 lines · Greg Kroah-Hartman
drivers/misc/mlx5ctl/main.c line 25
Quoted 8 lines · Saeed Mahameed
drivers/misc/mlx5ctl/main.c new file +8
-0+struct mlx5ctl_dev { +struct mlx5_core_dev *mdev; +struct miscdevice miscdev; +struct auxiliary_device *adev; +struct list_head fd_list; +spinlock_t fd_list_lock;/* protect list add/del */ +struct rw_semaphore rw_lock; +struct kref refcount;You now have 2 different things that control the lifespan of this structure. We really need some way to automatically check this so that people don't keep making this same mistake, it happens all the time :(
Quoted 3 lines
Please pick one structure (miscdevice) or the other (kref) to control the lifespan, having 2 will just not work.
miscdevice doesn't handle the lifespan, open files will remain open even after the miscdevice was unregistered, hence we use the kref to defer the kfree until the last open file is closed.
Quoted 6 lines
Also, why a
rw_semaphore? Only use those if you can prove with a benchmark that it is actually faster, otherwise it's simpler to just use a normal mutex (hint, you are changing the fields in the structure with the read lock held, which feels very wrong, and so needs a LOT of documentation, or just use a normal mutex...)
It is needed so we can protect against underlaying device unloading while miscdevice is active, we use rw semaphore since we don't care about synchronization between any of the fops, but we want to protect current active ioctls and fops from sudden mlx5ctl_remove (auxiliary_driver.remove) which can happen dynamically due to underlaying device removal.
So here is the locking scheme:
write_lock() : only on mlx5_ctl remove and mark the device is down via assigning NULL to mcdev->dev, to let all new readers abort and to wait for current readers to finish their task.
read_lock(): used in all fops and ioctls, to make sure underlaying mlx5_core device is still active, and to prevent open files to access the device when miscdevice was already unregistered.
I agree, this should've been documented in the code, I will add documentation.
Quoted 3 lines
thanks,
greg k-h
Greg Kroah-Hartman
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 15 lines · Saeed Mahameed
Quoted 14 lines · Greg Kroah-Hartman
drivers/misc/mlx5ctl/main.c line 25
Quoted 8 lines · Saeed Mahameed
drivers/misc/mlx5ctl/main.c new file +8
-0+struct mlx5ctl_dev { +struct mlx5_core_dev *mdev; +struct miscdevice miscdev; +struct auxiliary_device *adev; +struct list_head fd_list; +spinlock_t fd_list_lock;/* protect list add/del */ +struct rw_semaphore rw_lock; +struct kref refcount;You now have 2 different things that control the lifespan of this structure. We really need some way to automatically check this so that people don't keep making this same mistake, it happens all the time :(
Quoted 7 lines
Quoted 3 lines
Please pick one structure (miscdevice) or the other (kref) to control the lifespan, having 2 will just not work.
miscdevice doesn't handle the lifespan, open files will remain open even after the miscdevice was unregistered, hence we use the kref to defer the kfree until the last open file is closed.
miscdevice has a reference counter and a lifecycle, you can not have two reference counted objects in the same structure and expect things to work well.
Quoted 7 lines
Quoted 6 lines
Also, why a
rw_semaphore? Only use those if you can prove with a benchmark that it is actually faster, otherwise it's simpler to just use a normal mutex (hint, you are changing the fields in the structure with the read lock held, which feels very wrong, and so needs a LOT of documentation, or just use a normal mutex...)
Quoted 5 lines
It is needed so we can protect against underlaying device unloading while miscdevice is active, we use rw semaphore since we don't care about synchronization between any of the fops, but we want to protect current active ioctls and fops from sudden
mlx5ctl_remove(auxiliary_driver.remove) which can happen dynamically due to underlaying device removal.
Then use a normal mutex. Only use a rw lock if you can prove the performance needs it as usually a rw lock is slower and more complex as you then have to document stuff like:
Quoted 6 lines
So here is the locking scheme:
write_lock(): only onmlx5_ctlremove and mark the device is down via assigning NULL tomcdev->dev, to let all new readers abort and to wait for current readers to finish their task.
Quoted 6 lines
read_lock(): used in all fops and ioctls, to make sure underlayingmlx5_coredevice is still active, and to prevent open files to access the device when miscdevice was already unregistered.I agree, this should've been documented in the code, I will add documentation.
Just make it simple and use a normal mutex please.
And fix up the reference counting, it shouldn't be this complex, it's just a "simple" misc device driver :)
But before you do that, please see my other email about why not using devlink for all of this instead.
thanks,
greg k-h
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 20 lines · Greg Kroah-Hartman
Quoted 19 lines · Saeed Mahameed
Quoted 17 lines · Greg Kroah-Hartman
drivers/misc/mlx5ctl/main.c line 25
Quoted 8 lines · Saeed Mahameed
drivers/misc/mlx5ctl/main.c new file +8
-0+struct mlx5ctl_dev { +struct mlx5_core_dev *mdev; +struct miscdevice miscdev; +struct auxiliary_device *adev; +struct list_head fd_list; +spinlock_t fd_list_lock;/* protect list add/del */ +struct rw_semaphore rw_lock; +struct kref refcount;You now have 2 different things that control the lifespan of this structure. We really need some way to automatically check this so that people don't keep making this same mistake, it happens all the time :(
Please pick one structure (miscdevice) or the other (kref) to control the lifespan, having 2 will just not work.
Quoted 7 lines
Quoted 3 lines
miscdevice doesn't handle the lifespan, open files will remain open even after the miscdevice was unregistered, hence we use the kref to defer the kfree until the last open file is closed.
miscdevice has a reference counter and a lifecycle, you can not have two reference counted objects in the same structure and expect things to work well.
This second refcount is hidden well:
struct miscdevice {
int minor;
const char *name;
const struct file_operations *fops;
struct list_head list;
struct device *parent;
struct device *this_device;
const struct attribute_group **groups;
const char *nodename;
umode_t mode;
};You have slammed us in the other thread for doing a poor review job here, but none of us see what you are seeing. Can you explain better?
Quoted 4 lines
Quoted 4 lines
write_lock(): only onmlx5_ctlremove and mark the device is down via assigning NULL tomcdev->dev, to let all new readers abort and to wait for current readers to finish their task.
Quoted 8 lines
Quoted 6 lines
read_lock(): used in all fops and ioctls, to make sure underlayingmlx5_coredevice is still active, and to prevent open files to access the device when miscdevice was already unregistered.I agree, this should've been documented in the code, I will add documentation.
Just make it simple and use a normal mutex please.
A normal mutex would make the entire ioctl interface single threaded, this is not desirable.
Quoted 2 lines
But before you do that, please see my other email about why not using devlink for all of this instead.
We've been over this already, the devlink discussion is about some configuration stuff. It has never been suggested to cover the debug interface. This series is primarily about debug, the devlink thing is a distraction to main point.
Jason
Greg Kroah-Hartman
maintainerRe: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 31 lines · Jason Gunthorpe
Quoted 20 lines · Greg Kroah-Hartman
Quoted 19 lines · Saeed Mahameed
Quoted 17 lines · Greg Kroah-Hartman
drivers/misc/mlx5ctl/main.c line 25
Quoted 8 lines · Saeed Mahameed
drivers/misc/mlx5ctl/main.c new file +8
-0+struct mlx5ctl_dev { +struct mlx5_core_dev *mdev; +struct miscdevice miscdev; +struct auxiliary_device *adev; +struct list_head fd_list; +spinlock_t fd_list_lock;/* protect list add/del */ +struct rw_semaphore rw_lock; +struct kref refcount;You now have 2 different things that control the lifespan of this structure. We really need some way to automatically check this so that people don't keep making this same mistake, it happens all the time :(
Please pick one structure (miscdevice) or the other (kref) to control the lifespan, having 2 will just not work.
Quoted 7 lines
Quoted 3 lines
miscdevice doesn't handle the lifespan, open files will remain open even after the miscdevice was unregistered, hence we use the kref to defer the kfree until the last open file is closed.
miscdevice has a reference counter and a lifecycle, you can not have two reference counted objects in the same structure and expect things to work well.
This second refcount is hidden well:
Quoted 11 lines
struct miscdevice {
int minor;
const char *name;
const struct file_operations *fops;
struct list_head list;
struct device *parent;
struct device *this_device;
const struct attribute_group **groups;
const char *nodename;
umode_t mode;
};Ugh, you are right, I was wrong, there is no reference count here, using a miscdevice _requires_ you to have a separate reference count, like you all did. My fault.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
write_lock(): only onmlx5_ctlremove and mark the device is down via assigning NULL tomcdev->dev, to let all new readers abort and to wait for current readers to finish their task.
read_lock(): used in all fops and ioctls, to make sure underlayingmlx5_coredevice is still active, and to prevent open files to access the device when miscdevice was already unregistered.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I agree, this should've been documented in the code, I will add documentation.
Just make it simple and use a normal mutex please.
A normal mutex would make the entire ioctl interface single threaded, this is not desirable.
Why not? It's an ioctl for a single device, surely this isn't performance criticial. And then only grab it when needed, on read/write/ioctl path it shouldn't be needed at all due to the proper reference counting of the structures. Only on open/close, right?
And again, for a rw semaphore, benchmarks matter, often, if not almost always, a normal mutex is faster for stuff like this. If not, then a benchmark will show it.
Quoted 5 lines
Quoted 2 lines
But before you do that, please see my other email about why not using devlink for all of this instead.
We've been over this already, the devlink discussion is about some configuration stuff.
It was? I see device-specific diagonostic data for the mlx5 driver being exported through devlink today, that's not configuration. Why not just add more?
Quoted 3 lines
It has never been suggested to cover the debug interface. This series is primarily about debug, the devlink thing is a distraction to main point.
For me it is the main point at the moment. Please explain why devlink does not work for the information that you have created a misc device where you want an ioctl api instead, as I honestly do not understand.
thanks,
greg k-h
Re: [PATCH V3 2/5] misc: mlx5ctl: Add mlx5ctl misc driver
Quoted 3 lines · Greg Kroah-Hartman
Ugh, you are right, I was wrong, there is no reference count here, using a miscdevice _requires_ you to have a separate reference count, like you all did. My fault.
I think we did not do a good job conveying that several experienced people have looked at this internally already. I see you noticed a few things we missed, but this is not unreviewed garbage code. We have different opionions on how some of these things should work but none of them are what I'd consider unlinuxy or wrong.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Just make it simple and use a normal mutex please.
A normal mutex would make the entire ioctl interface single threaded, this is not desirable.
Why not? It's an ioctl for a single device, surely this isn't performance criticial.
The RPCs to the FW can take a long time to execute and the "single device" is really a multi-threaded CPU complex on the other side. Being able to submit any commands for (say) 0.5s while one thread chews on something is not desirable.
Quoted 3 lines
And then only grab it when needed, on read/write/ioctl path it shouldn't be needed at all due to the proper reference counting of the structures. Only on open/close, right?
No - this lock is protecting the mcdev->mdev from being freed. If the read side is held then mcdev->mdev cannot be freed. This is not a fully refcounted object because the mdev is actually the device underlying the aux device and it will be logically destroyed after the aux device_driver remove() function returns.
The purpose of the rwlock is to ensure that the remove() function does not return until all threads have completed using the mdev. Once remove proceeds the read side of the lock will observe mdev==NULL and fail all operations until the FD is closed. The refcount keeps the memory underpinning the basic operation of the FD alive after remove() until the FD finally closes.
At least for the purposes of this driver it is how it has to be. I agree the mdev API surface that requires some of this may not be the best it can be.
Quoted 3 lines
And again, for a rw semaphore, benchmarks matter, often, if not almost always, a normal mutex is faster for stuff like this. If not, then a benchmark will show it.
We are not talking about micro differences in lock/unlock cost here. This is obviously better because of how much time the lock will stay locked while executing RPCs and sleeping waiting for interrupts back from FW. We are talking >> 100's of ms of time spent locked while executing FW RPCs.
Quoted 6 lines
Quoted 2 lines
We've been over this already, the devlink discussion is about some configuration stuff.
It was? I see device-specific diagonostic data for the mlx5 driver being exported through devlink today, that's not configuration. Why not just add more?
Today mlx5 has a bunch of debuggables:
- counters, even some device specific counters. Both in RDMA and netdev objects exposed through netlink and sysfs
- A "core dump" mechanism through devlink
- Some debugfs stuff, but we are now aware of serious problems with it
- Some "devlink parameters" which configure a limited set of things
These are not "interactive". Yes we can add more counters, more parameters and more stuff to the core dump, but that isn't what I mean by debugging.
This is a complex device, it has processors on the other side running a lot of software. A primary purpose of this design is to allow bi-directional communication with debug agents in the FW.
Consider, (I'm trying to draw a picture here without disclosing trade secrets about how the device operates), that one of the debug agents is a GDB stub. There are many CPUs inside these devices, field engineers may wish to enter a breakpoint and inspect something. With this scheme we would put the GDB stub protocol packets into RPC messages and execute them to the FW.
In the very worst nightmare debug scenario we might build a custom FW with a custom debug agent targetting the specific problem and use these generic FW RPCs to communicate however that very special one-off agent requires.
Notice what such a thing requires: a future compatible channel to exchange data bi-directionally with the SW components in the FW.
The design here of processing *generic RPCs* is not some dirty nod toward enterprise distros - it is a sane and logical design to put things in userspace that the kernel has no value-add to being part of. This is standard straight-from-Linus design values. It is how things like nvme built their very similar "vendor commands" system (see for instance the intel & wd commands in nvmecli).
So, can we run generic opaque RPCs over some TBD devlink interface, including with DMA? Technologically yes, of course. Practically? No. Jakub has clearly stated his position on philisophical limits in netdev and those include devlink. It excludes soemthing so open-ended and ill-defined. FWIW, I agree with this, devlink should be much more constrained. It is why we never suggested devlink for this kind of debugging.
Genericness is also the source of alot of the cross discussion in this thread.
A generic RPC interface can do *alot*. Actually, it can do *almost anything* to the device. That is the entire point. What it can't do is violate the integrity of the kernel - hence why it is a FW RPC interface and not direct access to device registers.
So when Jakub says we can send configuration values that could also be sent over devlink parameters, or read counters that could also be read over netlink, he is not wrong. But we already do all those duplicative things through the direct PCI access and have for at least 10 years.
I disagree that this duplication is a problem. Re-usable, cross vendor generic interfaces have merit on their own. mlx5 has long been implementing a wide selection of them as they are invented. We've been doing this even though those interfaces duplicate what the raw PCI tools already did.
This new driver isn't going to change that. We will still participate in netdev and still implement these other things. We are a big part of the community, I don't need someone holding a stick over the mlx5 team's head beating them to do the "right" thing.
Jason
PATCH v3 3/5 misc: mlx5ctl: Add info ioctl
In reply to Saeed MahameedFrom: Saeed Mahameed <[email protected]>
Implement INFO ioctl to return the allocated UID and the capability flags and some other useful device information such as the underlying ConnectX device.
Example:
$ sudo ./mlx5ctlu mlx5_core.ctl.0 mlx5dev: 0000:00:04.0 UCTX UID: 1 UCTX CAP: 0x3 DEV UCTX CAP: 0x3 USER CAP: 0x1d
--- .../userspace-api/ioctl/ioctl-number.rst | 1 + drivers/misc/mlx5ctl/main.c | 71 +++++++++++++++++++ include/uapi/misc/mlx5ctl.h | 24 +++++++ 3 files changed, 96 insertions(+) create mode 100644 include/uapi/misc/mlx5ctl.h
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
@@ -89,6 +89,7 @@ Code Seq# Include File Comments 0x20 all drivers/cdrom/cm206.h 0x22 all scsi/sg.h 0x3E 00-0F linux/counter.h <mailto:[email protected]> +0x5c all uapi/misc/mlx5ctl.h Nvidia ConnectX control '!' 00-1F uapi/linux/seccomp.h '#' 00-3F IEEE 1394 Subsystem Block for the entire subsystem
drivers/misc/mlx5ctl/main.c +71 -0
87 lines not on this page · raw message
include/uapi/misc/mlx5ctl.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 WITH Linux-syscall-note */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ + +#ifndef __MLX5CTL_IOCTL_H__ +#define __MLX5CTL_IOCTL_H__ + +struct mlx5ctl_info { +__aligned_u64 flags; +__u32 size; +__u8 devname[64 ];/* underlaying ConnectX device */ +__u16 uctx_uid;/* current process allocated UCTX UID */ +__u16 reserved1; +__u32 uctx_cap;/* current process effective UCTX cap */ +__u32 dev_uctx_cap;/* device's UCTX capabilities */ +__u32 ucap;/* process user capability */ +__u32 reserved2; +}; + +#define MLX5CTL_IOCTL_MAGIC 0x5c + +#define MLX5CTL_IOCTL_INFO \ + _IOR(MLX5CTL_IOCTL_MAGIC, 0x0, struct mlx5ctl_info) + +#endif /* __MLX5CTL_IOCTL_H__ */
Signature · 1 line
2.42.0
Greg Kroah-Hartman
maintainerRe: [PATCH V3 3/5] misc: mlx5ctl: Add info ioctl
drivers/misc/mlx5ctl/main.c static int mlx5ctl_release(struct inode *inode, struct file *file) · line 213
Quoted 12 lines · Saeed Mahameed
drivers/misc/mlx5ctl/main.c +12
-0+static int mlx5ctl_info_ioctl (struct file *file, +struct mlx5ctl_info __user *arg, +size_t usize) +{ +struct mlx5ctl_fd *mfd = file->private_data; +struct mlx5ctl_dev *mcdev = mfd->mcdev; +struct mlx5_core_dev *mdev = mcdev->mdev; +struct mlx5ctl_info *info; +size_t ksize =0 ; +int err =0 ; + + ksize =max (sizeof (struct mlx5ctl_info ), usize);
Why / How can usize be larger than the structure size and you still want to allocate a memory chunk that big? Shouldn't the size always match?
And what if it's too small?
drivers/misc/mlx5ctl/main.c static int mlx5ctl_release(struct inode *inode, struct file *file) · line 214
Quoted 1 line
drivers/misc/mlx5ctl/main.c +1
-0+ info =kzalloc (ksize,GFP_KERNEL_ACCOUNT );
Why account as it will go away almost instantly?
Quoted 5 lines
drivers/misc/mlx5ctl/main.c +5
-0+if (!info) +return -ENOMEM ; + + info->size =sizeof (struct mlx5ctl_info ); +
drivers/misc/mlx5ctl/main.c static int mlx5ctl_release(struct inode *inode, struct file *file) · line 229
Quoted 10 lines
drivers/misc/mlx5ctl/main.c +10
-0+ info->dev_uctx_cap =MLX5_CAP_GEN (mdev, uctx_cap); + info->uctx_cap = mfd->uctx_cap; + info->uctx_uid = mfd->uctx_uid; + info->ucap = mfd->ucap; + +strscpy (info->devname,dev_name (&mdev->pdev->dev), +sizeof (info->devname)); + +if (copy_to_user (arg, info, usize)) + err = -EFAULT ;
So if usize is smaller than the structure you don't copy it all?
What am I missing here?
Quoted 23 lines
drivers/misc/mlx5ctl/main.c +23
-0+ +kfree (info); +return err; +} + +static long mlx5ctl_ioctl (struct file *file,unsigned int cmd,unsigned long arg) +{ +struct mlx5ctl_fd *mfd = file->private_data; +struct mlx5ctl_dev *mcdev = mfd->mcdev; +void __user *argp = (void __user *)arg; +size_t size =_IOC_SIZE (cmd); +int err =0 ; + +if (!capable (CAP_SYS_ADMIN )) +return -EPERM ; + +mlx5ctl_dbg (mcdev,"ioctl 0x%x type/nr: %d/%d size: %d DIR:%d\n" , cmd, +_IOC_TYPE (cmd),_IOC_NR (cmd),_IOC_SIZE (cmd),_IOC_DIR (cmd)); + +down_read (&mcdev->rw_lock); +if (!mcdev->mdev) { + err = -ENODEV ; + goto unlock;
drivers/misc/mlx5ctl/main.c static int mlx5ctl_release(struct inode *inode, struct file *file) · line 262
Quoted 10 lines
drivers/misc/mlx5ctl/main.c +10
-0+ } + +switch (cmd) { +case MLX5CTL_IOCTL_INFO : + err =mlx5ctl_info_ioctl (file, argp, size); +break ; + +default : +mlx5ctl_dbg (mcdev,"Unknown ioctl %x\n" , cmd); + err = -ENOIOCTLCMD ;
-ENOTTY is the correct error.
include/uapi/misc/mlx5ctl.h line 8
Quoted 11 lines
include/uapi/misc/mlx5ctl.h new file +8
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 WITH Linux-syscall-note */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ + +#ifndef __MLX5CTL_IOCTL_H__ +#define __MLX5CTL_IOCTL_H__ + +struct mlx5ctl_info { + __aligned_u64 flags;
Is this used?
include/uapi/misc/mlx5ctl.h line 10
Quoted 2 lines
include/uapi/misc/mlx5ctl.h new file +2
-0+__u32 size; +__u8 devname[64 ];/* underlaying ConnectX device */
64 should be a define somewhere, right? And why 64?
include/uapi/misc/mlx5ctl.h line 12
Quoted 2 lines
include/uapi/misc/mlx5ctl.h new file +2
-0+__u16 uctx_uid;/* current process allocated UCTX UID */ +__u16 reserved1;
Where is this checked to be always 0? Well it's a read so I guess where is the documentation saying it will always be set to 0?
include/uapi/misc/mlx5ctl.h line 16
Quoted 4 lines
include/uapi/misc/mlx5ctl.h new file +4
-0+__u32 uctx_cap;/* current process effective UCTX cap */ +__u32 dev_uctx_cap;/* device's UCTX capabilities */ +__u32 ucap;/* process user capability */ +__u32 reserved2;
Same here.
And why reserve anything? What does that help with?
thanks,
greg k-h
Re: [PATCH V3 3/5] misc: mlx5ctl: Add info ioctl
Quoted 17 lines · Greg Kroah-Hartman
drivers/misc/mlx5ctl/main.c static int mlx5ctl_release(struct inode *inode, struct file *file) · line 213
Quoted 12 lines · Saeed Mahameed
drivers/misc/mlx5ctl/main.c +12
-0+static int mlx5ctl_info_ioctl (struct file *file, +struct mlx5ctl_info __user *arg, +size_t usize) +{ +struct mlx5ctl_fd *mfd = file->private_data; +struct mlx5ctl_dev *mcdev = mfd->mcdev; +struct mlx5_core_dev *mdev = mcdev->mdev; +struct mlx5ctl_info *info; +size_t ksize =0 ; +int err =0 ; + + ksize =max (sizeof (struct mlx5ctl_info ), usize);Why / How can usize be larger than the structure size and you still want to allocate a memory chunk that big? Shouldn't the size always match?
new user-space old kernel, the driver would allocate the usiae and make sure to clear all the buffer with 0's, then fill in what the kernel understands and send the whole buffer back to user with trailer always zeroed out.
Quoted 2 lines
And what if it's too small?
Too small means old user new kernel, the kernel will honor that and only fill in what the user understands.
Quoted 4 lines
drivers/misc/mlx5ctl/main.c static int mlx5ctl_release(struct inode *inode, struct file *file) · line 214
Quoted 1 line
drivers/misc/mlx5ctl/main.c +1
-0+ info =kzalloc (ksize,GFP_KERNEL_ACCOUNT );Why account as it will go away almost instantly?
Will change that back to GFP_KERNEL.
Quoted 18 lines
Quoted 5 lines
drivers/misc/mlx5ctl/main.c +5
-0+if (!info) +return -ENOMEM ; + + info->size =sizeof (struct mlx5ctl_info ); +drivers/misc/mlx5ctl/main.c static int mlx5ctl_release(struct inode *inode, struct file *file) · line 229
Quoted 10 lines
drivers/misc/mlx5ctl/main.c +10
-0+ info->dev_uctx_cap =MLX5_CAP_GEN (mdev, uctx_cap); + info->uctx_cap = mfd->uctx_cap; + info->uctx_uid = mfd->uctx_uid; + info->ucap = mfd->ucap; + +strscpy (info->devname,dev_name (&mdev->pdev->dev), +sizeof (info->devname)); + +if (copy_to_user (arg, info, usize)) + err = -EFAULT ;So if usize is smaller than the structure you don't copy it all?
Quoted 2 lines
What am I missing here?
We copy back to user as much data as they expect, newer kernel can have extended functionality and we want to allow expanding the ioctl structs in the future while keeping support for old user, so new kernel would honor old user and copy only the size the user expects.
Quoted 36 lines
Quoted 23 lines
drivers/misc/mlx5ctl/main.c +23
-0+ +kfree (info); +return err; +} + +static long mlx5ctl_ioctl (struct file *file,unsigned int cmd,unsigned long arg) +{ +struct mlx5ctl_fd *mfd = file->private_data; +struct mlx5ctl_dev *mcdev = mfd->mcdev; +void __user *argp = (void __user *)arg; +size_t size =_IOC_SIZE (cmd); +int err =0 ; + +if (!capable (CAP_SYS_ADMIN )) +return -EPERM ; + +mlx5ctl_dbg (mcdev,"ioctl 0x%x type/nr: %d/%d size: %d DIR:%d\n" , cmd, +_IOC_TYPE (cmd),_IOC_NR (cmd),_IOC_SIZE (cmd),_IOC_DIR (cmd)); + +down_read (&mcdev->rw_lock); +if (!mcdev->mdev) { + err = -ENODEV ; + goto unlock;drivers/misc/mlx5ctl/main.c static int mlx5ctl_release(struct inode *inode, struct file *file) · line 262
Quoted 10 lines
drivers/misc/mlx5ctl/main.c +10
-0+ } + +switch (cmd) { +case MLX5CTL_IOCTL_INFO : + err =mlx5ctl_info_ioctl (file, argp, size); +break ; + +default : +mlx5ctl_dbg (mcdev,"Unknown ioctl %x\n" , cmd); + err = -ENOIOCTLCMD ;
-ENOTTYis the correct error.
Quoted 14 lines
include/uapi/misc/mlx5ctl.h line 8
Quoted 11 lines
include/uapi/misc/mlx5ctl.h new file +8
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 WITH Linux-syscall-note */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ + +#ifndef __MLX5CTL_IOCTL_H__ +#define __MLX5CTL_IOCTL_H__ + +struct mlx5ctl_info { + __aligned_u64 flags;Is this used?
no, not yet, but it is good for future extendibility and compatibility checking.
Quoted 5 lines
include/uapi/misc/mlx5ctl.h line 10
Quoted 2 lines
include/uapi/misc/mlx5ctl.h new file +2
-0+__u32 size; +__u8 devname[64 ];/* underlaying ConnectX device */ 64 should be a define somewhere, right? And why 64?
It is usually the kobj->name of the underlying device, I will have to define this in the uAPI. 64 seemed large enough, any other suggestion ?
This field is informational only for the user to have an idea which is the underlying physical device, it's ok if in odd situation the name has to be truncated to fit into the uAPI buffer.
Quoted 6 lines
include/uapi/misc/mlx5ctl.h line 12
Quoted 2 lines
include/uapi/misc/mlx5ctl.h new file +2
-0+__u16 uctx_uid;/* current process allocated UCTX UID */ +__u16 reserved1;Where is this checked to be always 0? Well it's a read so I guess where is the documentation saying it will always be set to 0?
I forgot to add the checks in the info ioctl path, will add that. Isn't it an unwritten rule that reserved fields has to be always 0 ? Do I really need to document this ?
Quoted 9 lines
include/uapi/misc/mlx5ctl.h line 16
Quoted 4 lines
include/uapi/misc/mlx5ctl.h new file +4
-0+__u32 uctx_cap;/* current process effective UCTX cap */ +__u32 dev_uctx_cap;/* device's UCTX capabilities */ +__u32 ucap;/* process user capability */ +__u32 reserved2;Same here.
And why reserve anything? What does that help with?
struct alignment and explicit padding.
Quoted 3 lines
thanks,
greg k-h
Greg Kroah-Hartman
maintainerRe: [PATCH V3 3/5] misc: mlx5ctl: Add info ioctl
Quoted 19 lines · Saeed Mahameed
Quoted 17 lines · Greg Kroah-Hartman
drivers/misc/mlx5ctl/main.c static int mlx5ctl_release(struct inode *inode, struct file *file) · line 213
Quoted 12 lines · Saeed Mahameed
drivers/misc/mlx5ctl/main.c +12
-0+static int mlx5ctl_info_ioctl (struct file *file, +struct mlx5ctl_info __user *arg, +size_t usize) +{ +struct mlx5ctl_fd *mfd = file->private_data; +struct mlx5ctl_dev *mcdev = mfd->mcdev; +struct mlx5_core_dev *mdev = mcdev->mdev; +struct mlx5ctl_info *info; +size_t ksize =0 ; +int err =0 ; + + ksize =max (sizeof (struct mlx5ctl_info ), usize);Why / How can usize be larger than the structure size and you still want to allocate a memory chunk that big? Shouldn't the size always match?
Quoted 4 lines
new user-space old kernel, the driver would allocate the usiae and make sure to clear all the buffer with 0's, then fill in what the kernel understands and send the whole buffer back to user with trailer always zeroed out.
No, at that point you know something is wrong and you need to just abort and return -EINVAL as the structure sizes do not match.
If you need to "extend" the structure to include more information, do so in a new ioctl.
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
include/uapi/misc/mlx5ctl.h new file +2
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 WITH Linux-syscall-note */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ Quoted 9 lines
include/uapi/misc/mlx5ctl.h line 8
Quoted 6 lines
include/uapi/misc/mlx5ctl.h new file +6
-0+ +#ifndef __MLX5CTL_IOCTL_H__ +#define __MLX5CTL_IOCTL_H__ + +struct mlx5ctl_info { + __aligned_u64 flags;Is this used?
Quoted 2 lines
no, not yet, but it is good for future extendibility and compatibility checking.
But you are not checking anything now, so please don't include something that will not work in the future.
Quoted 8 lines
Quoted 5 lines
include/uapi/misc/mlx5ctl.h line 10
Quoted 2 lines
include/uapi/misc/mlx5ctl.h new file +2
-0+__u32 size; +__u8 devname[64 ];/* underlaying ConnectX device */ 64 should be a define somewhere, right? And why 64?
It is usually the
kobj->nameof the underlying device, I will have to define this in the uAPI. 64 seemed large enough, any other suggestion ?
What happens if the names get bigger?
Quoted 3 lines
This field is informational only for the user to have an idea which is the underlying physical device, it's ok if in odd situation the name has to be truncated to fit into the uAPI buffer.
As the truncation will happen on the right side of the string, usually the actual device id or unique identifier, that's not going to help out much to drop that portion :(
Quoted 10 lines
Quoted 6 lines
include/uapi/misc/mlx5ctl.h line 12
Quoted 2 lines
include/uapi/misc/mlx5ctl.h new file +2
-0+__u16 uctx_uid;/* current process allocated UCTX UID */ +__u16 reserved1;Where is this checked to be always 0? Well it's a read so I guess where is the documentation saying it will always be set to 0?
I forgot to add the checks in the info ioctl path, will add that. Isn't it an unwritten rule that reserved fields has to be always 0 ? Do I really need to document this ?
It is a written rule that reserved fields must be 0, please see the documentation for how to write an ioctl.
thanks,
greg k-h
Re: [PATCH V3 3/5] misc: mlx5ctl: Add info ioctl
Quoted 25 lines · Greg Kroah-Hartman
Quoted 19 lines · Saeed Mahameed
Quoted 17 lines · Greg Kroah-Hartman
drivers/misc/mlx5ctl/main.c static int mlx5ctl_release(struct inode *inode, struct file *file) · line 213
Quoted 12 lines · Saeed Mahameed
drivers/misc/mlx5ctl/main.c +12
-0+static int mlx5ctl_info_ioctl (struct file *file, +struct mlx5ctl_info __user *arg, +size_t usize) +{ +struct mlx5ctl_fd *mfd = file->private_data; +struct mlx5ctl_dev *mcdev = mfd->mcdev; +struct mlx5_core_dev *mdev = mcdev->mdev; +struct mlx5ctl_info *info; +size_t ksize =0 ; +int err =0 ; + + ksize =max (sizeof (struct mlx5ctl_info ), usize);Why / How can usize be larger than the structure size and you still want to allocate a memory chunk that big? Shouldn't the size always match?
Quoted 4 lines
new user-space old kernel, the driver would allocate the usiae and make sure to clear all the buffer with 0's, then fill in what the kernel understands and send the whole buffer back to user with trailer always zeroed out.
Quoted 6 lines
No, at that point you know something is wrong and you need to just abort and return
-EINVALas the structure sizes do not match.If you need to "extend" the structure to include more information, do so in a new ioctl.
Ack, will remove these fields.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
include/uapi/misc/mlx5ctl.h line 8
Quoted 11 lines
include/uapi/misc/mlx5ctl.h new file +8
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 WITH Linux-syscall-note */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ + +#ifndef __MLX5CTL_IOCTL_H__ +#define __MLX5CTL_IOCTL_H__ + +struct mlx5ctl_info { + __aligned_u64 flags;Is this used?
Quoted 6 lines
Quoted 2 lines
no, not yet, but it is good for future extendibility and compatibility checking.
But you are not checking anything now, so please don't include something that will not work in the future.
Ack, will remove.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
include/uapi/misc/mlx5ctl.h line 10
Quoted 2 lines
include/uapi/misc/mlx5ctl.h new file +2
-0+__u32 size; +__u8 devname[64 ];/* underlaying ConnectX device */ 64 should be a define somewhere, right? And why 64?
It is usually the
kobj->nameof the underlying device, I will have to define this in the uAPI. 64 seemed large enough, any other suggestion ?What happens if the names get bigger?
Quoted 8 lines
Quoted 3 lines
This field is informational only for the user to have an idea which is the underlying physical device, it's ok if in odd situation the name has to be truncated to fit into the uAPI buffer.
As the truncation will happen on the right side of the string, usually the actual device id or unique identifier, that's not going to help out much to drop that portion :(
Right :/, it's an assumption that mlx5 devices can either be a pci device or an auxiliary device in case of a mlx5-subfunction, so i don't expect the names to get larger and can easily fit in 64B string, but you are right, I shouldn't make such assumption in an IOCTL, I will figure out something or completely drop this field in V4.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
include/uapi/misc/mlx5ctl.h line 12
Quoted 2 lines
include/uapi/misc/mlx5ctl.h new file +2
-0+__u16 uctx_uid;/* current process allocated UCTX UID */ +__u16 reserved1;Where is this checked to be always 0? Well it's a read so I guess where is the documentation saying it will always be set to 0?
Quoted 7 lines
Quoted 3 lines
I forgot to add the checks in the info ioctl path, will add that. Isn't it an unwritten rule that reserved fields has to be always 0 ? Do I really need to document this ?
It is a written rule that reserved fields must be 0, please see the documentation for how to write an ioctl.
Ack, will document.
Quoted 3 lines
thanks,
greg k-h
PATCH v3 4/5 misc: mlx5ctl: Add command rpc ioctl
In reply to Saeed MahameedFrom: Saeed Mahameed <[email protected]>
Add new IOCTL to allow user space to send device debug rpcs and attach the user's uctx UID to each rpc.
In the mlx5 architecture the FW RPC commands are of the format of inbox and outbox buffers. The inbox buffer contains the command rpc layout as described in the ConnectX Programmers Reference Manual (PRM) document and as defined in include/linux/mlx5/mlx5_ifc.h.
On success the user outbox buffer will be filled with the device's rpc response.
For example to query device capabilities:
a user fills out an inbox buffer with the inbox layout:
struct mlx5_ifc_query_hca_cap_in_bits
and expects an outbox buffer with the layout:
struct mlx5_ifc_cmd_hca_cap_bits
--- drivers/misc/mlx5ctl/main.c | 95 +++++++++++++++++++++++++++++++++++++ include/uapi/misc/mlx5ctl.h | 13 +++++ 2 files changed, 108 insertions(+)
drivers/misc/mlx5ctl/main.c +95 -0
@@ -232,6 +232,97 @@ static int mlx5ctl_info_ioctl(struct file *file,return err; } +struct mlx5_ifc_mbox_in_hdr_bits { +u8 opcode[0x10 ]; +u8 uid[0x10 ]; + +u8 reserved_at_20[0x10 ]; +u8 op_mod[0x10 ]; + +u8 reserved_at_40[0x40 ]; +}; + +struct mlx5_ifc_mbox_out_hdr_bits { +u8 status[0x8 ]; +u8 reserved_at_8[0x18 ]; + +u8 syndrome[0x20 ]; + +u8 reserved_at_40[0x40 ]; +}; + +static int mlx5ctl_cmdrpc_ioctl (struct file *file, +struct mlx5ctl_cmdrpc __user *arg, +size_t usize) +{ +struct mlx5ctl_fd *mfd = file->private_data; +struct mlx5ctl_dev *mcdev = mfd->mcdev; +struct mlx5ctl_cmdrpc *rpc =NULL ; +void *in =NULL , *out =NULL ; +size_t ksize =0 ; +int err; + + ksize =max (sizeof (struct mlx5ctl_cmdrpc ), usize); + rpc =kzalloc (ksize,GFP_KERNEL_ACCOUNT ); +if (!rpc) +return -ENOMEM ; + + err =copy_from_user (rpc, arg, usize); +if (err) + goto out; + +mlx5ctl_dbg (mcdev,"[UID %d] cmdrpc: rpc->inlen %d rpc->outlen %d\n" , + mfd->uctx_uid, rpc->inlen, rpc->outlen); + +if (rpc->inlen <MLX5_ST_SZ_BYTES (mbox_in_hdr) || + rpc->outlen <MLX5_ST_SZ_BYTES (mbox_out_hdr) || + rpc->inlen >MLX5CTL_MAX_RPC_SIZE || + rpc->outlen >MLX5CTL_MAX_RPC_SIZE ) { + err = -EINVAL ; + goto out; + } + +if (rpc->flags) { + err = -EOPNOTSUPP ; + goto out; + } + + in =memdup_user (u64_to_user_ptr (rpc->in), rpc->inlen); +if (IS_ERR (in)) { + err =PTR_ERR (in); + goto out; + } + + out =kvzalloc (rpc->outlen,GFP_KERNEL_ACCOUNT ); +if (!out) { + err = -ENOMEM ; + goto out; + } + +mlx5ctl_dbg (mcdev,"[UID %d] cmdif: opcode 0x%x inlen %d outlen %d\n" , + mfd->uctx_uid, +MLX5_GET (mbox_in_hdr, in, opcode), rpc->inlen, rpc->outlen); + +MLX5_SET (mbox_in_hdr, in, uid, mfd->uctx_uid); + err =mlx5_cmd_do (mcdev->mdev, in, rpc->inlen, out, rpc->outlen); +mlx5ctl_dbg (mcdev,"[UID %d] cmdif: opcode 0x%x retval %d\n" , + mfd->uctx_uid, +MLX5_GET (mbox_in_hdr, in, opcode), err); + +/* -EREMOTEIO means outbox is valid, but out.status is not */ +if (!err || err == -EREMOTEIO ) { + err =0 ; +if (copy_to_user (u64_to_user_ptr (rpc->out), out, rpc->outlen)) + err = -EFAULT ; + } + +out: +kvfree (out); +kfree (in); +kfree (rpc); +return err; +} +static long mlx5ctl_ioctl (struct file *file,unsigned int cmd,unsigned long arg) {struct mlx5ctl_fd *mfd = file->private_data;
@@ -257,6 +348,10 @@ static long mlx5ctl_ioctl(struct file *file, unsigned int cmd, unsigned long arg err =mlx5ctl_info_ioctl (file, argp, size);break ; +case MLX5CTL_IOCTL_CMDRPC : + err =mlx5ctl_cmdrpc_ioctl (file, argp, size); +break ; +default :mlx5ctl_dbg (mcdev,"Unknown ioctl %x\n" , cmd); err = -ENOIOCTLCMD ;
include/uapi/misc/mlx5ctl.h +13 -0
@@ -16,9 +16,22 @@ struct mlx5ctl_info { __u32 reserved2; }; +struct mlx5ctl_cmdrpc { +__aligned_u64 in;/* RPC inbox buffer user address */ +__aligned_u64 out;/* RPC outbox buffer user address */ +__u32 inlen;/* inbox buffer length */ +__u32 outlen;/* outbox buffer length */ +__aligned_u64 flags; +}; + +#define MLX5CTL_MAX_RPC_SIZE 8192 +#define MLX5CTL_IOCTL_MAGIC 0x5c#define MLX5CTL_IOCTL_INFO \ _IOR(MLX5CTL_IOCTL_MAGIC, 0x0, struct mlx5ctl_info) +#define MLX5CTL_IOCTL_CMDRPC \ + _IOWR(MLX5CTL_IOCTL_MAGIC, 0x1, struct mlx5ctl_cmdrpc) +#endif /* __MLX5CTL_IOCTL_H__ */
Signature · 1 line
2.42.0
PATCH v3 5/5 misc: mlx5ctl: Add umem reg/unreg ioctl
In reply to Saeed MahameedFrom: Saeed Mahameed <[email protected]>
Command rpc outbox buffer is limited in size, which can be very annoying when trying to pull large traces out of the device. Many device rpcs offer the ability to scatter output traces, contexts and logs directly into user space buffers in a single shot.
Allow user to register user memory space, so the device may dump information directly into user memory space.
The registered memory will be described by a device UMEM object which has a unique umem_id, this umem_id can be later used in the rpc inbox to tell the device where to populate the response output, e.g HW traces and other debug object queries.
To do so this patch introduces two ioctls:
MLX5CTL_IOCTL_UMEM_REG(va_address, size):
- calculate page fragments from the user provided virtual address
- pin the pages, and allocate a sg list
- dma map the sg list
- create a UMEM device object that points to the dma addresses
- add a driver umem object to an xarray data base for bookkeeping
- return UMEM ID to user so it can be used in subsequent rpcs
MLX5CTL_IOCTL_UMEM_UNREG(umem_id):
- user provides a pre allocated umem ID
- unwinds the above
Example usecase, ConnectX device coredump can be as large as 2MB. Using inline rpcs will take thousands of rpcs to get the full coredump which can take multiple seconds.
With UMEM, it can be done in a single rpc, using 2MB of umem user buffer.
$ ./mlx5ctlu mlx5_core.ctl.0 coredump --umem_size=$(( 2 ** 20 ))
00 00 00 00 01 00 20 00 00 00 00 04 00 00 48 ec
00 00 00 08 00 00 00 00 00 00 00 0c 00 00 00 03
00 00 00 10 00 00 00 00 00 00 00 14 00 00 00 00
....
00 50 0b 3c 00 00 00 00 00 50 0b 40 00 00 00 00
00 50 0b 44 00 00 00 00 00 50 0b 48 00 00 00 00
00 50 0c 00 00 00 00 00
INFO : Core dump done
INFO : Core dump size 831304
INFO : Core dump address 0x0
INFO : Core dump cookie 0x500c04
INFO : More Dump 0
Other usecases are: dynamic HW and FW trace monitoring, high frequency diagnostic counters sampling and batched objects and resource dumps.
--- drivers/misc/mlx5ctl/Makefile | 1 + drivers/misc/mlx5ctl/main.c | 99 +++++++++++ drivers/misc/mlx5ctl/umem.c | 322 ++++++++++++++++++++++++++++++++++ drivers/misc/mlx5ctl/umem.h | 17 ++ include/uapi/misc/mlx5ctl.h | 22 +++ 5 files changed, 461 insertions(+) create mode 100644 drivers/misc/mlx5ctl/umem.c create mode 100644 drivers/misc/mlx5ctl/umem.h
drivers/misc/mlx5ctl/Makefile +1 -0
@@ -2,3 +2,4 @@ obj-$(CONFIG_MLX5CTL) += mlx5ctl.o mlx5ctl-y := main.o +mlx5ctl-y += umem.o
drivers/misc/mlx5ctl/main.c +99 -0
135 lines not on this page · raw message
drivers/misc/mlx5ctl/umem.c new file +322 -0
322 lines not on this page · raw message
drivers/misc/mlx5ctl/umem.h new file +17 -0
@@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ + +#ifndef __MLX5CTL_UMEM_H__ +#define __MLX5CTL_UMEM_H__ + +#include <linux/types.h> +#include <linux/mlx5/driver.h> + +struct mlx5ctl_umem_db ; + +struct mlx5ctl_umem_db *mlx5ctl_umem_db_create (struct mlx5_core_dev *mdev,u32 uctx_uid); +void mlx5ctl_umem_db_destroy (struct mlx5ctl_umem_db *umem_db); +int mlx5ctl_umem_reg (struct mlx5ctl_umem_db *umem_db,unsigned long addr,size_t size); +int mlx5ctl_umem_unreg (struct mlx5ctl_umem_db *umem_db,u32 umem_id); + +#endif /* __MLX5CTL_UMEM_H__ */
include/uapi/misc/mlx5ctl.h +22 -0
@@ -24,6 +24,22 @@ struct mlx5ctl_cmdrpc {__aligned_u64 flags; }; +struct mlx5ctl_umem_reg { +__aligned_u64 flags; +__u32 size; +__u32 reserved1; +__aligned_u64 addr;/* user address */ +__aligned_u64 len;/* user buffer length */ +__u32 umem_id;/* returned device's umem ID */ +__u32 reserved2; +}; + +struct mlx5ctl_umem_unreg { +__aligned_u64 flags; +__u32 size; +__u32 umem_id; +}; +#define MLX5CTL_MAX_RPC_SIZE 8192#define MLX5CTL_IOCTL_MAGIC 0x5c
@@ -34,4 +50,10 @@ struct mlx5ctl_cmdrpc {#define MLX5CTL_IOCTL_CMDRPC \ _IOWR(MLX5CTL_IOCTL_MAGIC, 0x1, struct mlx5ctl_cmdrpc) +#define MLX5CTL_IOCTL_UMEM_REG \ + _IOWR(MLX5CTL_IOCTL_MAGIC, 0x2, struct mlx5ctl_umem_reg) + +#define MLX5CTL_IOCTL_UMEM_UNREG \ + _IOWR(MLX5CTL_IOCTL_MAGIC, 0x3, struct mlx5ctl_umem_unreg) +#endif /* __MLX5CTL_IOCTL_H__ */
Signature · 1 line
2.42.0
Jakub Kicinski
maintainerRe: [PATCH V3 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl
Quoted 1 line · On Mon, 20 Nov 2023 23:06:19 -0800 Saeed Mahameed wrote:
high frequency diagnostic counters
So is it a debug driver or not a debug driver?
Because I'm pretty sure some people want to have access to high freq counters in production, across their fleet. What's worse David Ahern has been pitching a way of exposing device counters which would be common across netdev.
Definite nack on this patch.
Saeed Mahameed
maintainerRe: [PATCH V3 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl
Quoted 5 lines · Jakub Kicinski
Quoted 1 line · On Mon, 20 Nov 2023 23:06:19 -0800 Saeed Mahameed wrote:
high frequency diagnostic counters
So is it a debug driver or not a debug driver?
High frequency _diagnostic_ counters are a very useful tool for debugging a high performance chip. So yes this is for diagnostics/debug.
Quoted 5 lines
Because I'm pretty sure some people want to have access to high freq counters in production, across their fleet. What's worse David Ahern has been pitching a way of exposing device counters which would be common across netdev.
This is not netdev, this driver is to support ConnectX chips and SoCs with any stack, netdev/rdma/vdpa/virtio and internal chip units and acceleration engines, add to that ARM core diagnostics in case of Blue-Field DPUs.
I am not looking for counting netdev ethernet packets in this driver.
I am also pretty sure David will also want an interface to access other than netdev counters, to get more visibility on how a specific chip is behaving.
Quoted 1 line
Definite nack on this patch.
Based on what ?
Jakub Kicinski
maintainerRe: [PATCH V3 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl
Quoted 8 lines · On Tue, 21 Nov 2023 13:04:06 -0800 Saeed Mahameed wrote:
Quoted 4 lines · Jakub Kicinski
Quoted 1 line · On Mon, 20 Nov 2023 23:06:19 -0800 Saeed Mahameed wrote:
high frequency diagnostic counters
So is it a debug driver or not a debug driver?
High frequency _diagnostic_ counters are a very useful tool for debugging a high performance chip. So yes this is for diagnostics/debug.
You keep saying debugging but if it's expected to run on all servers in the fleet _monitoring_ performance, then it's a very different thing. To me it certainly moves this driver from "debug thing loaded when things fail" to the "always loaded in production" category.
Re: [PATCH V3 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Tue, 21 Nov 2023 13:04:06 -0800 Saeed Mahameed wrote:
Quoted 4 lines · Jakub Kicinski
Quoted 1 line · On Mon, 20 Nov 2023 23:06:19 -0800 Saeed Mahameed wrote:
high frequency diagnostic counters
So is it a debug driver or not a debug driver?
Quoted 7 lines
Quoted 2 lines
High frequency _diagnostic_ counters are a very useful tool for debugging a high performance chip. So yes this is for diagnostics/debug.
You keep saying debugging but if it's expected to run on all servers in the fleet _monitoring_ performance, then it's a very different thing. To me it certainly moves this driver from "debug thing loaded when things fail" to the "always loaded in production" category.
Exactly, only when things fails or the user want to debug something.
For your example, you can monitor network performance via standard netdev tools, once you start experiencing hiccups, you can use this driver and the corresponding tools to quickly grab HW debug information, useful to further root cause and analyze the network hiccups.
Again this is only one use-case, the driver is intended to provide any debug information, not only diagnostic counters or monitoring tools. The goal of this driver is not the one use case you have in mind.
Re: [PATCH V3 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl
In reply to Saeed MahameedQuoted 14 lines · Saeed Mahameed
Quoted 5 lines · Jakub Kicinski
Quoted 1 line · On Mon, 20 Nov 2023 23:06:19 -0800 Saeed Mahameed wrote:
high frequency diagnostic counters
So is it a debug driver or not a debug driver?
High frequency _diagnostic_ counters are a very useful tool for debugging a high performance chip. So yes this is for diagnostics/debug.
Quoted 4 lines
Because I'm pretty sure some people want to have access to high freq counters in production, across their fleet. What's worse David Ahern has been pitching a way of exposing device counters which would be common across netdev.
.
For context on the what's worse ... comment for those who have not seen the netconf slides:
https://netdev.bots.linux.dev/netconf/2023/david.pdf
and I am having a hard time parsing Kuba's intent with that comment here (knowing you did not like the pitch I made at netconf :-))
Quoted 10 lines
This is not netdev, this driver is to support ConnectX chips and SoCs with any stack, netdev/rdma/vdpa/virtio and internal chip units and acceleration engines, add to that ARM core diagnostics in case of Blue-Field DPUs.
I am not looking for counting netdev ethernet packets in this driver.I am also pretty sure David will also want an interface to access other than netdev counters, to get more visibility on how a specific chip is behaving.
yes, and h/w counters were part of the proposal. One thought is to leverage userspace registered memory with the device vs mapping bar space, but we have not moved beyond a theoretical discussion at this point.
Quoted 4 lines
Quoted 2 lines
Definite nack on this patch.
Based on what ?
It's a generic interface argument?
Re: [PATCH V3 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl
Quoted 41 lines · David Ahern
Quoted 14 lines · Saeed Mahameed
Quoted 5 lines · Jakub Kicinski
Quoted 1 line · On Mon, 20 Nov 2023 23:06:19 -0800 Saeed Mahameed wrote:
high frequency diagnostic counters
So is it a debug driver or not a debug driver?
High frequency _diagnostic_ counters are a very useful tool for debugging a high performance chip. So yes this is for diagnostics/debug.
Quoted 4 lines
Because I'm pretty sure some people want to have access to high freq counters in production, across their fleet. What's worse David Ahern has been pitching a way of exposing device counters which would be common across netdev.
.
For context on the
what's worse ...comment for those who have not seen the netconf slides:
https://netdev.bots.linux.dev/netconf/2023/david.pdfand I am having a hard time parsing Kuba's intent with that comment here (knowing you did not like the pitch I made at netconf :-))
Quoted 10 lines
This is not netdev, this driver is to support ConnectX chips and SoCs with any stack, netdev/rdma/vdpa/virtio and internal chip units and acceleration engines, add to that ARM core diagnostics in case of Blue-Field DPUs.
I am not looking for counting netdev ethernet packets in this driver.I am also pretty sure David will also want an interface to access other than netdev counters, to get more visibility on how a specific chip is behaving.
yes, and h/w counters were part of the proposal. One thought is to leverage userspace registered memory with the device vs mapping bar space, but we have not moved beyond a theoretical discussion at this point.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Definite nack on this patch.
Based on what ?
It's a generic interface argument?
For this driver the diagnostic counters is only a small part of the debug utilities the driver provides, so it is not fair to nak this patch based on one use-case, we need this driver to also dump other stuff like core dumps, FW contexts, internal objects, register dumps, resource dumps, etc ..
This patch original purpose was to allow core dumps, since core dump can go up to 2MB of memory, without this patch we won't have core dump ability which is more important for debugging than diagnostic counters.
You can find more here:
https://github.com/saeedtx/mlx5ctl#mlx5ctl-userspace-linux-debug-utilities-for-mlx5-connectx-devices
For diagnostic counters we can continue the discussion to have a generic interface I am all for it, but it's irrelevant for this submission.
Re: [PATCH V3 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl
In reply to Jakub KicinskiQuoted 4 lines · Jakub Kicinski
Quoted 1 line · On Mon, 20 Nov 2023 23:06:19 -0800 Saeed Mahameed wrote:
high frequency diagnostic counters
So is it a debug driver or not a debug driver?
In the part you decided not to quote Saeed explained how the main purpose of the generic DMA to userspace mechanism is to transfer FW trace, FW memory copies and other large data dumps.
The thing with generic stuff is you can use it for lots of things if you are so inclined. Saeed gave many examples. I think you took it in the wrong way as I am not aware of any plan for actual high speed netdev relavent counters in a performance monitor application. It isn't that kind of "high speed".
The main interest is for micro-architectural debugging information. The kind that are opaque unless you can reference the RTL to understand what it means. It is "high speed" in the sense that executing a FW command per register/counter would be offensively slow compared to executing a FW command to bulk DMA a cluster of micro-architecture registers/etc in the device.
The design is so generic because it is a debug interface that we want to be always available and always fully functional. Mellanox ships new FW and new chips at a rapid rate, we do not want to be changing the kernel driver every time we do anything. That will never get backported into production kernels across all our supported customers fast enough. Debug features that a field support engineer cannot access simply do not exist.
Debugs are challenging. mlx5 is the most popular datacenter NIC in the world. We have so many insane problems, you wouldn't belive it. I just spent 8 months leading a debug that turned out to be a qemu defect (thanks Paolo for all the help!!). This debug data and flexibility is critical to making these hugely complex systems work.
Jason