Skip to contentSkip to conversation

mlx5 ConnectX diagnostic misc driver

PATCH Superseded Saeed Mahameed 4 participants 11 messages Nov 19, 2023 – Nov 21, 2023

Subsystems (4)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover mlx5 ConnectX diagnostic misc driver

From: Saeed Mahameed <[email protected]>

V1: https://lore.kernel.org/all/[email protected]/T/#rf22c2070325b7da020ee29242dc25ffbd286d40d

Following Greg's and Arnd's comments on V1 here are the V2 changes:

V1->V2:
- Provide legal statement and sign-off for dual license use
- Fix License clause to use: BSD-3-Clause OR GPL-2.0
- Fix kernel robot warnings
- Use dev_dbg directly instead of umem_dbg() local wrapper
- Implement .compat_ioctl for 32bit compatibility
- Fix mlx5ctl_info ABI structure size and alignment
- Local pointer to correct type instead of in-place cast
- Check unused fields and flags are 0 on ioctl path
- Use correct macro to declare scalar arg ioctl command

    #define MLX5CTL_IOCTL_UMEM_UNREG \
           _IO(MLX5CTL_IOCTL_MAGIC, 0x3)

The ConnectX HW family supported by the mlx5 drivers uses an architecture where a FW component executes "mailbox RPCs" issued by the driver to make changes to the device. This results in a complex debugging environment where the FW component has information and low level configuration that needs to be accessed to userspace for debugging purposes.

Historically a userspace program was used that accessed the PCI register and config space directly through /sys/bus/pci/.../XXX and could operate these debugging interfaces in parallel with the running driver. This approach is incompatible with secure boot and kernel lockdown so this driver provides a secure and restricted interface to that.

1) The first patch in the series introduces the main driver file with the implementation of a new mlx5 auxiliary device driver to run on top mlx5_core device instances, on probe it creates a new misc device and in this patch we implement the open and release fops, On open the driver would allocate a special FW UID (user context ID) restricted to debug RPCs only, where all user debug rpcs will be executed under this UID, and on release the UID will be freed.

2) The second patch adds an info ioctl that will show the allocated UID and the available capability masks of the device and the current UID, and some other useful device information such as the underlying ConnectX

Example:

    $ sudo ./mlx5ctlu mlx5_core.ctl.0
    mlx5dev: 0000:00:04.0
    UCTX UID: 1
    UCTX CAP: 0x3
    DEV UCTX CAP: 0x3
    USER CAP: 0x1d

3) Third patch will add the capability to execute debug RPCs under the special UID.

In the mlx5 architecture the FW RPC commands are of the format of inbox and outbox buffers. The inbox buffer contains the command rpc layout as described in the ConnectX Programmers Reference Manual (PRM) document and as defined in linux/include/mlx5/mlx5_ifc.h.

On success the user outbox buffer will be filled with the device's rpc response.

For example to query device capabilities:
a user fills out an inbox buffer with the inbox layout:
struct mlx5_ifc_query_hca_cap_in_bits
and expects an outbox buffer with the layout:
struct mlx5_ifc_cmd_hca_cap_bits

4) The fourth patch adds the ability to register user memory into the ConntectX device and create a umem object that points to that memory.

Command rpc outbox buffer is limited in size, which can be very annoying when trying to pull large traces out of the device. Many rpcs offer the ability to scatter output traces, contexts and logs directly into user space buffers in a single shot.

The registered memory will be described by a device UMEM object which has a unique umem_id, this umem_id can be later used in the rpc inbox to tell the device where to populate the response output, e.g HW traces and other debug object queries.

Example usecase, a ConnectX device coredump can be as large as 2MB. Using inline rpcs will take thousands of rpcs to get the full coredump which can consume multiple seconds.

With UMEM, it can be done in a single rpc, using 2MB of umem user buffer.

Other usecases with umem:
- dynamic HW and FW trace monitoring
- high frequency diagnostic counters sampling
- batched objects and resource dumps

See links below for information about user space tools that use this interface:

[1] https://github.com/saeedtx/mlx5ctl

[2] https://github.com/Mellanox/mstflint
see:

d) mstregdump utility
This utility dumps hardware registers from Mellanox hardware for later analysis by Mellanox.

g) mstconfig
This tool sets or queries non-volatile configurable options for Mellanox HCAs.

h) mstfwmanager
Mellanox firmware update and query utility which scans the system for available Mellanox devices (only mst PCI devices) and performs the necessary firmware updates.

i) mstreg
The mlxreg utility allows users to obtain information regarding supported access registers, such as their fields

License: BSD-3-Clause OR GPL-2.0
================================
After a review of this thread [3], and a conversation with the LF, Mellanox and NVIDIA legal continue to approve the use of a Dual GPL & Permissive License for mlx5 related driver contributions. This makes it clear to future contributors that this file may be adapted and reused under BSD-3-Clause terms on other operating systems. Contributions will be handled in the normal way and the dual license will apply automatically. If people wish to contribute significantly and opt out of a dual license they may separate their GPL only contributions in dedicated files.

Jason has a signing authority for NVIDIA and has gone through our internal process to get approval.

[3] https://lore.kernel.org/all/[email protected]/#r

================================

Saeed Mahameed (5):
mlx5: Add aux dev for ctl interface
misc: mlx5ctl: Add mlx5ctl misc driver
misc: mlx5ctl: Add info ioctl
misc: mlx5ctl: Add command rpc ioctl
misc: mlx5ctl: Add umem reg/unreg ioctl

 .../userspace-api/ioctl/ioctl-number.rst      |   1 +
 MAINTAINERS                                   |   8 +
 drivers/misc/Kconfig                          |   1 +
 drivers/misc/Makefile                         |   1 +
 drivers/misc/mlx5ctl/Kconfig                  |  14 +
 drivers/misc/mlx5ctl/Makefile                 |   5 +
 drivers/misc/mlx5ctl/main.c                   | 559 ++++++++++++++++++
 drivers/misc/mlx5ctl/umem.c                   | 320 ++++++++++
 drivers/misc/mlx5ctl/umem.h                   |  17 +
 drivers/net/ethernet/mellanox/mlx5/core/dev.c |   8 +
 include/uapi/misc/mlx5ctl.h                   |  53 ++
 11 files changed, 987 insertions(+)
 create mode 100644 drivers/misc/mlx5ctl/Kconfig
 create mode 100644 drivers/misc/mlx5ctl/Makefile
 create mode 100644 drivers/misc/mlx5ctl/main.c
 create mode 100644 drivers/misc/mlx5ctl/umem.c
 create mode 100644 drivers/misc/mlx5ctl/umem.h
 create mode 100644 include/uapi/misc/mlx5ctl.h

--
2.41.0

#1 lore raw

PATCH v2 1/5 mlx5: Add aux dev for ctl interface

From: Saeed Mahameed <[email protected]>

Allow ctl protocol interface auxiliary driver in mlx5.

---
 drivers/net/ethernet/mellanox/mlx5/core/dev.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/dev.c +8 -0
index cf0477f53dc4..753c11f7050d 100644
@@ -228,8 +228,14 @@ enum {
 	MLX5_INTERFACE_PROTOCOL_VNET,
 
 	MLX5_INTERFACE_PROTOCOL_DPLL,
+	MLX5_INTERFACE_PROTOCOL_CTL,
 };
 
+static bool is_ctl_supported(struct mlx5_core_dev *dev)
+{
+	return MLX5_CAP_GEN(dev, uctx_cap);
+}
+
 static const struct mlx5_adev_device {
 	const char *suffix;
 	bool (*is_supported)(struct mlx5_core_dev *dev);
@@ -252,6 +258,8 @@ static const struct mlx5_adev_device {
 					   .is_supported = &is_mp_supported },
 	[MLX5_INTERFACE_PROTOCOL_DPLL] = { .suffix = "dpll",
 					   .is_supported = &is_dpll_supported },
+	[MLX5_INTERFACE_PROTOCOL_CTL] = { .suffix = "ctl",
+					  .is_supported = &is_ctl_supported },
 };
 
 int mlx5_adev_idx_alloc(void)
Signature · 1 line
2.41.0
#2 lore raw

PATCH v2 2/5 misc: mlx5ctl: Add mlx5ctl misc driver

In reply to Saeed Mahameed

From: Saeed Mahameed <[email protected]>

The ConnectX HW family supported by the mlx5 drivers uses an architecture where a FW component executes "mailbox RPCs" issued by the driver to make changes to the device. This results in a complex debugging environment where the FW component has information and low level configuration that needs to be accessed to userspace for debugging purposes.

Historically a userspace program was used that accessed the PCI register and config space directly through /sys/bus/pci/.../XXX and could operate these debugging interfaces in parallel with the running driver. This approach is incompatible with secure boot and kernel lockdown so this driver provides a secure and restricted interface to that same data.

On open the driver would allocate a special FW UID (user context ID) restrected to debug RPCs only, later in this series all user RPCs will be stamped with this UID.

---
 MAINTAINERS                   |   8 +
 drivers/misc/Kconfig          |   1 +
 drivers/misc/Makefile         |   1 +
 drivers/misc/mlx5ctl/Kconfig  |  14 ++
 drivers/misc/mlx5ctl/Makefile |   4 +
 drivers/misc/mlx5ctl/main.c   | 314 ++++++++++++++++++++++++++++++++++
 6 files changed, 342 insertions(+)
 create mode 100644 drivers/misc/mlx5ctl/Kconfig
 create mode 100644 drivers/misc/mlx5ctl/Makefile
 create mode 100644 drivers/misc/mlx5ctl/main.c
MAINTAINERS +8 -0
index 97f51d5ec1cf..4b532df16b19 100644
@@ -13835,6 +13835,14 @@ L:	[email protected]
 S:	Supported
 F:	drivers/vdpa/mlx5/
 
+MELLANOX MLX5 ConnectX Diag DRIVER
+M:	Saeed Mahameed <[email protected]>
+R:	Itay Avraham <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	drivers/misc/mlx5ctl/
+F:	include/uapi/misc/mlx5ctl.h
+
 MELLANOX MLXCPLD I2C AND MUX DRIVER
 M:	Vadim Pasternak <[email protected]>
 M:	Michael Shych <[email protected]>
drivers/misc/Kconfig +1 -0
index f37c4b8380ae..c0e4823648ed 100644
@@ -579,4 +579,5 @@ source "drivers/misc/cardreader/Kconfig"
 source "drivers/misc/uacce/Kconfig"
 source "drivers/misc/pvpanic/Kconfig"
 source "drivers/misc/mchp_pci1xxxx/Kconfig"
+source "drivers/misc/mlx5ctl/Kconfig"
 endmenu
drivers/misc/Makefile +1 -0
index f2a4d1ff65d4..49bc4697f498 100644
@@ -67,3 +67,4 @@ obj-$(CONFIG_TMR_MANAGER)      += xilinx_tmr_manager.o
 obj-$(CONFIG_TMR_INJECT)	+= xilinx_tmr_inject.o
 obj-$(CONFIG_TPS6594_ESM)	+= tps6594-esm.o
 obj-$(CONFIG_TPS6594_PFSM)	+= tps6594-pfsm.o
+obj-$(CONFIG_MLX5CTL)		+= mlx5ctl/
drivers/misc/mlx5ctl/Kconfig new file +14 -0
new file mode 100644
index 000000000000..faaa1dba2cc2
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+
+config MLX5CTL
+	tristate "mlx5 ConnectX control misc driver"
+	depends on MLX5_CORE
+	help
+	  MLX5CTL provides interface for the user process to access the debug and
+          configuration registers of the ConnectX hardware family
+          (NICs, PCI switches and SmartNIC SoCs).
+          This will allow configuration and debug tools to work out of the box on
+          mainstream kernel.
+
+	  If you don't know what to do here, say N.
drivers/misc/mlx5ctl/Makefile new file +4 -0
new file mode 100644
index 000000000000..b5c7f99e0ab6
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_MLX5CTL) += mlx5ctl.o
+mlx5ctl-y := main.o
drivers/misc/mlx5ctl/main.c new file +314 -0

314 lines not on this page · raw message

Signature · 1 line
2.41.0
maintainer
#4 lore raw

Re: [PATCH V2 2/5] misc: mlx5ctl: Add mlx5ctl misc driver

Quoted 4 lines · Greg Kroah-Hartman
Quoted 1 line · Saeed Mahameed

Something went wrong here :(

:(, Just sent V3 now, I needed to fix two new kernel robot warnings anyways.

#5 lore raw

PATCH v2 3/5 misc: mlx5ctl: Add info ioctl

In reply to Saeed Mahameed

From: Saeed Mahameed <[email protected]>

Implement INFO ioctl to return the allocated UID and the capability flags and some other useful device information such as the underlying ConnectX device.

Example:

$ sudo ./mlx5ctlu mlx5_core.ctl.0
mlx5dev: 0000:00:04.0
UCTX UID: 1
UCTX CAP: 0x3
DEV UCTX CAP: 0x3
USER CAP: 0x1d
---
 .../userspace-api/ioctl/ioctl-number.rst      |  1 +
 drivers/misc/mlx5ctl/main.c                   | 71 +++++++++++++++++++
 include/uapi/misc/mlx5ctl.h                   | 24 +++++++
 3 files changed, 96 insertions(+)
 create mode 100644 include/uapi/misc/mlx5ctl.h
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index 4ea5b837399a..9faf91ffefff 100644
@@ -89,6 +89,7 @@ Code  Seq#    Include File                                           Comments
 0x20  all    drivers/cdrom/cm206.h
 0x22  all    scsi/sg.h
 0x3E  00-0F  linux/counter.h                                         <mailto:[email protected]>
+0x5c  all    uapi/misc/mlx5ctl.h                                     Nvidia ConnectX control
 '!'   00-1F  uapi/linux/seccomp.h
 '#'   00-3F                                                          IEEE 1394 Subsystem
                                                                      Block for the entire subsystem
drivers/misc/mlx5ctl/main.c +71 -0

87 lines not on this page · raw message

include/uapi/misc/mlx5ctl.h new file +24 -0
new file mode 100644
index 000000000000..37153cc0fc6e
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
+
+#ifndef __MLX5CTL_IOCTL_H__
+#define __MLX5CTL_IOCTL_H__
+
+struct mlx5ctl_info {
+	__aligned_u64 flags;
+	__u32 size;
+	__u8 devname[64]; /* underlaying ConnectX device */
+	__u16 uctx_uid; /* current process allocated UCTX UID */
+	__u16 reserved1;
+	__u32 uctx_cap; /* current process effective UCTX cap */
+	__u32 dev_uctx_cap; /* device's UCTX capabilities */
+	__u32 ucap; /* process user capability */
+	__u32 reserved2;
+};
+
+#define MLX5CTL_IOCTL_MAGIC 0x5c
+
+#define MLX5CTL_IOCTL_INFO \
+	_IOR(MLX5CTL_IOCTL_MAGIC, 0x0, struct mlx5ctl_info)
+
+#endif /* __MLX5CTL_IOCTL_H__ */
Signature · 1 line
2.41.0
#6 lore raw

PATCH v2 4/5 misc: mlx5ctl: Add command rpc ioctl

In reply to Saeed Mahameed

From: Saeed Mahameed <[email protected]>

Add new IOCTL to allow user space to send device debug rpcs and attach the user's uctx UID to each rpc.

In the mlx5 architecture the FW RPC commands are of the format of inbox and outbox buffers. The inbox buffer contains the command rpc layout as described in the ConnectX Programmers Reference Manual (PRM) document and as defined in include/linux/mlx5/mlx5_ifc.h.

On success the user outbox buffer will be filled with the device's rpc response.

For example to query device capabilities:
a user fills out an inbox buffer with the inbox layout:
struct mlx5_ifc_query_hca_cap_in_bits
and expects an outbox buffer with the layout:
struct mlx5_ifc_cmd_hca_cap_bits

---
 drivers/misc/mlx5ctl/main.c | 95 +++++++++++++++++++++++++++++++++++++
 include/uapi/misc/mlx5ctl.h | 13 +++++
 2 files changed, 108 insertions(+)
drivers/misc/mlx5ctl/main.c +95 -0
index 6a98b40e4300..e7776ea4bfca 100644
@@ -232,6 +232,97 @@ static int mlx5ctl_info_ioctl(struct file *file,
 	return err;
 }
 
+struct mlx5_ifc_mbox_in_hdr_bits {
+	u8         opcode[0x10];
+	u8         uid[0x10];
+
+	u8         reserved_at_20[0x10];
+	u8         op_mod[0x10];
+
+	u8         reserved_at_40[0x40];
+};
+
+struct mlx5_ifc_mbox_out_hdr_bits {
+	u8         status[0x8];
+	u8         reserved_at_8[0x18];
+
+	u8         syndrome[0x20];
+
+	u8         reserved_at_40[0x40];
+};
+
+static int mlx5ctl_cmdrpc_ioctl(struct file *file,
+				struct mlx5ctl_cmdrpc __user *arg,
+				size_t usize)
+{
+	struct mlx5ctl_fd *mfd = file->private_data;
+	struct mlx5ctl_dev *mcdev = mfd->mcdev;
+	struct mlx5ctl_cmdrpc *rpc = NULL;
+	void *in = NULL, *out = NULL;
+	size_t ksize = 0;
+	int err;
+
+	ksize = max(sizeof(struct mlx5ctl_cmdrpc), usize);
+	rpc = kzalloc(ksize, GFP_KERNEL_ACCOUNT);
+	if (!rpc)
+		return -ENOMEM;
+
+	err = copy_from_user(rpc, arg, usize);
+	if (err)
+		goto out;
+
+	mlx5ctl_dbg(mcdev, "[UID %d] cmdrpc: rpc->inlen %d rpc->outlen %d\n",
+		    mfd->uctx_uid, rpc->inlen, rpc->outlen);
+
+	if (rpc->inlen < MLX5_ST_SZ_BYTES(mbox_in_hdr) ||
+	    rpc->outlen < MLX5_ST_SZ_BYTES(mbox_out_hdr) ||
+	    rpc->inlen > MLX5CTL_MAX_RPC_SIZE ||
+	    rpc->outlen > MLX5CTL_MAX_RPC_SIZE) {
+		err = -EINVAL;
+		goto out;
+	}
+
+	if (rpc->flags) {
+		err = -EOPNOTSUPP;
+		goto out;
+	}
+
+	in = memdup_user(u64_to_user_ptr(rpc->in), rpc->inlen);
+	if (IS_ERR(in)) {
+		err = PTR_ERR(in);
+		goto out;
+	}
+
+	out = kvzalloc(rpc->outlen, GFP_KERNEL_ACCOUNT);
+	if (!out) {
+		err = -ENOMEM;
+		goto out;
+	}
+
+	mlx5ctl_dbg(mcdev, "[UID %d] cmdif: opcode 0x%x inlen %d outlen %d\n",
+		    mfd->uctx_uid,
+		    MLX5_GET(mbox_in_hdr, in, opcode), rpc->inlen, rpc->outlen);
+
+	MLX5_SET(mbox_in_hdr, in, uid, mfd->uctx_uid);
+	err = mlx5_cmd_do(mcdev->mdev, in, rpc->inlen, out, rpc->outlen);
+	mlx5ctl_dbg(mcdev, "[UID %d] cmdif: opcode 0x%x retval %d\n",
+		    mfd->uctx_uid,
+		    MLX5_GET(mbox_in_hdr, in, opcode), err);
+
+	/* -EREMOTEIO means outbox is valid, but out.status is not */
+	if (!err || err == -EREMOTEIO) {
+		err = 0;
+		if (copy_to_user(u64_to_user_ptr(rpc->out), out, rpc->outlen))
+			err = -EFAULT;
+	}
+
+out:
+	kvfree(out);
+	kfree(in);
+	kfree(rpc);
+	return err;
+}
+
 static long mlx5ctl_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 {
 	struct mlx5ctl_fd *mfd = file->private_data;
@@ -257,6 +348,10 @@ static long mlx5ctl_ioctl(struct file *file, unsigned int cmd, unsigned long arg
 		err = mlx5ctl_info_ioctl(file, argp, size);
 		break;
 
+	case MLX5CTL_IOCTL_CMDRPC:
+		err = mlx5ctl_cmdrpc_ioctl(file, argp, size);
+		break;
+
 	default:
 		mlx5ctl_dbg(mcdev, "Unknown ioctl %x\n", cmd);
 		err = -ENOIOCTLCMD;
include/uapi/misc/mlx5ctl.h +13 -0
index 37153cc0fc6e..3277eaf78a37 100644
@@ -16,9 +16,22 @@ struct mlx5ctl_info {
 	__u32 reserved2;
 };
 
+struct mlx5ctl_cmdrpc {
+	__aligned_u64 in; /* RPC inbox buffer user address */
+	__aligned_u64 out; /* RPC outbox buffer user address */
+	__u32 inlen; /* inbox buffer length */
+	__u32 outlen; /* outbox buffer length */
+	__aligned_u64 flags;
+};
+
+#define MLX5CTL_MAX_RPC_SIZE 8192
+
 #define MLX5CTL_IOCTL_MAGIC 0x5c
 
 #define MLX5CTL_IOCTL_INFO \
 	_IOR(MLX5CTL_IOCTL_MAGIC, 0x0, struct mlx5ctl_info)
 
+#define MLX5CTL_IOCTL_CMDRPC \
+	_IOWR(MLX5CTL_IOCTL_MAGIC, 0x1, struct mlx5ctl_cmdrpc)
+
 #endif /* __MLX5CTL_IOCTL_H__ */
Signature · 1 line
2.41.0
#7 lore raw

PATCH v2 5/5 misc: mlx5ctl: Add umem reg/unreg ioctl

In reply to Saeed Mahameed

From: Saeed Mahameed <[email protected]>

Command rpc outbox buffer is limited in size, which can be very annoying when trying to pull large traces out of the device. Many device rpcs offer the ability to scatter output traces, contexts and logs directly into user space buffers in a single shot.

Allow user to register user memory space, so the device may dump information directly into user memory space.

The registered memory will be described by a device UMEM object which has a unique umem_id, this umem_id can be later used in the rpc inbox to tell the device where to populate the response output, e.g HW traces and other debug object queries.

To do so this patch introduces two ioctls:

MLX5CTL_IOCTL_UMEM_REG(va_address, size):
- calculate page fragments from the user provided virtual address
- pin the pages, and allocate a sg list
- dma map the sg list
- create a UMEM device object that points to the dma addresses
- add a driver umem object to an xarray data base for bookkeeping
- return UMEM ID to user so it can be used in subsequent rpcs

MLX5CTL_IOCTL_UMEM_UNREG(umem_id):
- user provides a pre allocated umem ID
- unwinds the above

Example usecase, ConnectX device coredump can be as large as 2MB. Using inline rpcs will take thousands of rpcs to get the full coredump which can take multiple seconds.

With UMEM, it can be done in a single rpc, using 2MB of umem user buffer.

$ ./mlx5ctlu mlx5_core.ctl.0 coredump --umem_size=$(( 2 ** 20 ))

00 00 00 00 01 00 20 00 00 00 00 04 00 00 48 ec
00 00 00 08 00 00 00 00 00 00 00 0c 00 00 00 03
00 00 00 10 00 00 00 00 00 00 00 14 00 00 00 00
....
00 50 0b 3c 00 00 00 00 00 50 0b 40 00 00 00 00
00 50 0b 44 00 00 00 00 00 50 0b 48 00 00 00 00
00 50 0c 00 00 00 00 00

INFO : Core dump done
INFO : Core dump size 831304
INFO : Core dump address 0x0
INFO : Core dump cookie 0x500c04
INFO : More Dump 0

Other usecases are: dynamic HW and FW trace monitoring, high frequency diagnostic counters sampling and batched objects and resource dumps.

---
 drivers/misc/mlx5ctl/Makefile |   1 +
 drivers/misc/mlx5ctl/main.c   |  81 ++++++++-
 drivers/misc/mlx5ctl/umem.c   | 320 ++++++++++++++++++++++++++++++++++
 drivers/misc/mlx5ctl/umem.h   |  17 ++
 include/uapi/misc/mlx5ctl.h   |  16 ++
 5 files changed, 434 insertions(+), 1 deletion(-)
 create mode 100644 drivers/misc/mlx5ctl/umem.c
 create mode 100644 drivers/misc/mlx5ctl/umem.h
drivers/misc/mlx5ctl/Makefile +1 -0
index b5c7f99e0ab6..f35234e931a8 100644
@@ -2,3 +2,4 @@
 
 obj-$(CONFIG_MLX5CTL) += mlx5ctl.o
 mlx5ctl-y := main.o
+mlx5ctl-y += umem.o
drivers/misc/mlx5ctl/main.c +80 -1

128 lines not on this page · raw message

drivers/misc/mlx5ctl/umem.c new file +320 -0

320 lines not on this page · raw message

drivers/misc/mlx5ctl/umem.h new file +17 -0
new file mode 100644
index 000000000000..9cf62e5e775e
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 */
+/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
+
+#ifndef __MLX5CTL_UMEM_H__
+#define __MLX5CTL_UMEM_H__
+
+#include <linux/types.h>
+#include <linux/mlx5/driver.h>
+
+struct mlx5ctl_umem_db;
+
+struct mlx5ctl_umem_db *mlx5ctl_umem_db_create(struct mlx5_core_dev *mdev, u32 uctx_uid);
+void mlx5ctl_umem_db_destroy(struct mlx5ctl_umem_db *umem_db);
+int mlx5ctl_umem_reg(struct mlx5ctl_umem_db *umem_db, unsigned long addr, size_t size);
+int mlx5ctl_umem_unreg(struct mlx5ctl_umem_db *umem_db, u32 umem_id);
+
+#endif /* __MLX5CTL_UMEM_H__ */
include/uapi/misc/mlx5ctl.h +16 -0
index 3277eaf78a37..eaf6c954a1ff 100644
@@ -24,6 +24,16 @@ struct mlx5ctl_cmdrpc {
 	__aligned_u64 flags;
 };
 
+struct mlx5ctl_umem_reg {
+	__aligned_u64 flags;
+	__u32 size;
+	__u32 reserved1;
+	__aligned_u64 addr; /* user address */
+	__aligned_u64 len; /* user buffer length */
+	__u32 umem_id; /* returned device's umem ID */
+	__u32 reserved2;
+};
+
 #define MLX5CTL_MAX_RPC_SIZE 8192
 
 #define MLX5CTL_IOCTL_MAGIC 0x5c
@@ -34,4 +44,10 @@ struct mlx5ctl_cmdrpc {
 #define MLX5CTL_IOCTL_CMDRPC \
 	_IOWR(MLX5CTL_IOCTL_MAGIC, 0x1, struct mlx5ctl_cmdrpc)
 
+#define MLX5CTL_IOCTL_UMEM_REG \
+	_IOWR(MLX5CTL_IOCTL_MAGIC, 0x2, struct mlx5ctl_umem_reg)
+
+#define MLX5CTL_IOCTL_UMEM_UNREG \
+	_IO(MLX5CTL_IOCTL_MAGIC, 0x3)
+
 #endif /* __MLX5CTL_IOCTL_H__ */
Signature · 1 line
2.41.0
#8 lore raw

Re: [PATCH v2 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl

kernel test robot noticed the following build warnings:

Hi Saeed,

kernel test robot noticed the following build warnings:

[auto build test WARNING on char-misc/char-misc-testing]
[also build test WARNING on rdma/for-next linus/master v6.7-rc1 next-20231117] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Saeed-Mahameed/misc-mlx5ctl-Add-mlx5ctl-misc-driver/20231119-215311
base:   char-misc/char-misc-testing
patch link:    https://lore.kernel.org/r/20231119092450.164996-6-saeed%40kernel.org
patch subject: [PATCH v2 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl
config: alpha-allyesconfig (https://download.01.org/0day-ci/archive/20231120/[email protected]/config)
compiler: alpha-linux-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231120/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

In file included from include/uapi/linux/posix_types.h:5,
from include/uapi/linux/types.h:14,
from include/linux/types.h:6,
from include/linux/kcsan-checks.h:14,
from include/asm-generic/barrier.h:17,
from arch/alpha/include/asm/barrier.h:21,
from arch/alpha/include/asm/rwonce.h:10,
from include/linux/compiler.h:251,
from include/linux/build_bug.h:5,
from include/linux/container_of.h:5,
from include/linux/list.h:5,
from include/linux/miscdevice.h:5,
from drivers/misc/mlx5ctl/main.c:4:
drivers/misc/mlx5ctl/main.c: In function 'mlx5ctl_compat_ioctl':
include/linux/stddef.h:8:14: error: called object is not a function or function pointer

       8 | #define NULL ((void *)0)
         |              ^
   include/linux/fs.h:1837:26: note: in expansion of macro 'NULL'
    1837 | #define compat_ptr_ioctl NULL
         |                          ^~~~
   drivers/misc/mlx5ctl/main.c:441:16: note: in expansion of macro 'compat_ptr_ioctl'
     441 |         return compat_ptr_ioctl(file, cmd, arg);
         |                ^~~~~~~~~~~~~~~~
>> drivers/misc/mlx5ctl/main.c:442:1: warning: control reaches end of non-void function [-Wreturn-type]
     442 | }
         | ^

vim +442 drivers/misc/mlx5ctl/main.c

434
435 static long mlx5ctl_compat_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
436 {
437 if (cmd == MLX5CTL_IOCTL_UMEM_UNREG) /* arg is a scalar */
438 return mlx5ctl_ioctl(file, cmd, arg);
439
440 /* arg is a pointer */
441 return compat_ptr_ioctl(file, cmd, arg);

 > 442	}

443

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#9 lore raw

Re: [PATCH v2 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl

In reply to Saeed Mahameed
kernel test robot noticed the following build errors:

Hi Saeed,

kernel test robot noticed the following build errors:

[auto build test ERROR on char-misc/char-misc-testing]
[also build test ERROR on rdma/for-next linus/master v6.7-rc1 next-20231117] [cannot apply to char-misc/char-misc-next char-misc/char-misc-linus soc/for-next] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Saeed-Mahameed/misc-mlx5ctl-Add-mlx5ctl-misc-driver/20231119-215311
base:   char-misc/char-misc-testing
patch link:    https://lore.kernel.org/r/20231119092450.164996-6-saeed%40kernel.org
patch subject: [PATCH v2 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl
config: i386-allmodconfig (https://download.01.org/0day-ci/archive/20231120/[email protected]/config)
compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231120/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> drivers/misc/mlx5ctl/main.c:441:25: error: called object type 'void *' is not a function or function pointer
           return compat_ptr_ioctl(file, cmd, arg);
                  ~~~~~~~~~~~~~~~~^

1 error generated.

vim +441 drivers/misc/mlx5ctl/main.c

434
435 static long mlx5ctl_compat_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
436 {
437 if (cmd == MLX5CTL_IOCTL_UMEM_UNREG) /* arg is a scalar */
438 return mlx5ctl_ioctl(file, cmd, arg);
439
440 /* arg is a pointer */

 > 441		return compat_ptr_ioctl(file, cmd, arg);

442 }
443

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#10 lore raw

Re: [PATCH v2 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl

In reply to Saeed Mahameed
kernel test robot noticed the following build errors:

Hi Saeed,

kernel test robot noticed the following build errors:

[auto build test ERROR on char-misc/char-misc-testing]
[also build test ERROR on rdma/for-next linus/master v6.7-rc1 next-20231117] [cannot apply to char-misc/char-misc-next char-misc/char-misc-linus soc/for-next] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Saeed-Mahameed/misc-mlx5ctl-Add-mlx5ctl-misc-driver/20231119-215311
base:   char-misc/char-misc-testing
patch link:    https://lore.kernel.org/r/20231119092450.164996-6-saeed%40kernel.org
patch subject: [PATCH v2 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl
config: powerpc-allmodconfig (https://download.01.org/0day-ci/archive/20231120/[email protected]/config)
compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231120/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> drivers/misc/mlx5ctl/umem.c:79:21: error: call to undeclared function 'pages_to_mb'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
      79 |         if (npages == 0 || pages_to_mb(npages) > MLX5CTL_UMEM_MAX_MB)
         |                            ^

1 error generated.

vim +/pages_to_mb +79 drivers/misc/mlx5ctl/umem.c

    60	
    61	static struct mlx5ctl_umem *mlx5ctl_umem_pin(struct mlx5ctl_umem_db *umem_db,
    62						     unsigned long addr, size_t size)
    63	{
    64		size_t npages = umem_num_pages(addr, size);
    65		struct mlx5_core_dev *mdev = umem_db->mdev;
    66		unsigned long endaddr = addr + size;
    67		struct mlx5ctl_umem *umem;
    68		struct page **page_list;
    69		int err = -EINVAL;
    70		int pinned = 0;
    71	
    72		dev_dbg(mdev->device, "%s: addr %p size %zu npages %zu\n",
    73			__func__, (void __user *)addr, size, npages);
    74	
    75		/* Avoid integer overflow */
    76		if (endaddr < addr || PAGE_ALIGN(endaddr) < endaddr)
    77			return ERR_PTR(-EINVAL);
    78	
  > 79		if (npages == 0 || pages_to_mb(npages) > MLX5CTL_UMEM_MAX_MB)
    80			return ERR_PTR(-EINVAL);
    81	
    82		page_list = kvmalloc_array(npages, sizeof(struct page *), GFP_KERNEL_ACCOUNT);
    83		if (!page_list)
    84			return ERR_PTR(-ENOMEM);
    85	
    86		umem = kzalloc(sizeof(*umem), GFP_KERNEL_ACCOUNT);
    87		if (!umem) {
    88			kvfree(page_list);
    89			return ERR_PTR(-ENOMEM);
    90		}
    91	
    92		umem->addr = addr;
    93		umem->size = size;
    94		umem->offset = addr & ~PAGE_MASK;
    95		umem->npages = npages;
    96	
    97		umem->page_list = page_list;
    98		umem->source_mm = current->mm;
    99		umem->source_task = current->group_leader;

100 get_task_struct(current->group_leader);
101 umem->source_user = get_uid(current_user());
102
103 /* mm and RLIMIT_MEMLOCK user task accounting similar to what is
104 * being done in iopt_alloc_pages() and do_update_pinned()
105 * for IOPT_PAGES_ACCOUNT_USER @drivers/iommu/iommufd/pages.c
106 */
107 mmgrab(umem->source_mm);
108
109 pinned = pin_user_pages_fast(addr, npages, FOLL_WRITE, page_list);
110 if (pinned != npages) {
111 dev_dbg(mdev->device, "pin_user_pages_fast failed %d\n", pinned);
112 err = pinned < 0 ? pinned : -ENOMEM;
113 goto pin_failed;
114 }
115
116 err = inc_user_locked_vm(umem, npages);
117 if (err)
118 goto pin_failed;
119
120 atomic64_add(npages, &umem->source_mm->pinned_vm);
121
122 err = sg_alloc_table_from_pages(&umem->sgt, page_list, npages, 0,
123 npages << PAGE_SHIFT, GFP_KERNEL_ACCOUNT);
124 if (err) {
125 dev_dbg(mdev->device, "sg_alloc_table failed: %d\n", err);
126 goto sgt_failed;
127 }
128
129 dev_dbg(mdev->device, "\tsgt: size %zu npages %zu sgt.nents (%d)\n",
130 size, npages, umem->sgt.nents);
131
132 err = dma_map_sgtable(mdev->device, &umem->sgt, DMA_BIDIRECTIONAL, 0);
133 if (err) {
134 dev_dbg(mdev->device, "dma_map_sgtable failed: %d\n", err);
135 goto dma_failed;
136 }
137
138 dev_dbg(mdev->device, "\tsgt: dma_nents %d\n", umem->sgt.nents);
139 return umem;
140
141 dma_failed:
142 sgt_failed:
143 sg_free_table(&umem->sgt);
144 atomic64_sub(npages, &umem->source_mm->pinned_vm);
145 dec_user_locked_vm(umem, npages);
146 pin_failed:
147 if (pinned > 0)
148 unpin_user_pages(page_list, pinned);
149 mmdrop(umem->source_mm);
150 free_uid(umem->source_user);
151 put_task_struct(umem->source_task);
152
153 kfree(umem);
154 kvfree(page_list);
155 return ERR_PTR(err);
156 }
157

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki