Skip to contentSkip to conversation

Introduce UCAP API and usage in mlx5

PATCH In review Leon Romanovsky 3 participants 15 messages Feb 26, 2025 – Mar 5, 2025

Subsystems (3)
Documentation InfiniBand Mellanox MLX5 IB driver
lore
Share & tools
lore

Leon Romanovsky

author
#0 lore raw

cover Introduce UCAP API and usage in mlx5

This series introduces the User CAPability (UCAP) API that allows creating user contexts with various firmware privileges.

The UCAP API provides fine-grained control over specific firmware features by representing each capability as a character device with root read-write access. Root processes can grant users special privileges by allowing access to these character devices. User contexts created using a file descriptor of a UCAP will have specific UCAP privileges.

Two UCAP character devices are created for mlx5, and user contexts opened with at least one of these UCAPs are considered privileged. To ensure that privileged commands can always proceed, non-privileged commands are limited when a privileged user is present on the device.

Thanks

Chiara Meiohas (5):
RDMA/uverbs: Introduce UCAP (User CAPabilities) API RDMA/mlx5: Create UCAP char devices for supported device capabilities RDMA/uverbs: Add support for UCAPs in context creation RDMA/mlx5: Check enabled UCAPs when creating ucontext
docs: infiniband: document the UCAP API

Patrisious Haddad (1):
RDMA/mlx5: Expose RDMA TRANSPORT flow table types to userspace

 Documentation/infiniband/index.rst            |   1 +
 Documentation/infiniband/ucaps.rst            |  71 +++++
 drivers/infiniband/core/Makefile              |   3 +-
 drivers/infiniband/core/ucaps.c               | 255 ++++++++++++++++++
 drivers/infiniband/core/uverbs_cmd.c          |  19 ++
 drivers/infiniband/core/uverbs_main.c         |   2 +
 .../infiniband/core/uverbs_std_types_device.c |   4 +
 drivers/infiniband/hw/mlx5/devx.c             |  31 ++-
 drivers/infiniband/hw/mlx5/devx.h             |   5 +-
 drivers/infiniband/hw/mlx5/fs.c               | 154 ++++++++++-
 drivers/infiniband/hw/mlx5/fs.h               |   2 +
 drivers/infiniband/hw/mlx5/main.c             |  77 +++++-
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |   3 +
 include/rdma/ib_ucaps.h                       |  25 ++
 include/rdma/ib_verbs.h                       |   1 +
 include/uapi/rdma/ib_user_ioctl_cmds.h        |   1 +
 include/uapi/rdma/mlx5_user_ioctl_cmds.h      |   1 +
 include/uapi/rdma/mlx5_user_ioctl_verbs.h     |   2 +
 18 files changed, 635 insertions(+), 22 deletions(-)
 create mode 100644 Documentation/infiniband/ucaps.rst
 create mode 100644 drivers/infiniband/core/ucaps.c
 create mode 100644 include/rdma/ib_ucaps.h
Signature · 1 line
2.48.1
author
#1 lore raw

PATCH 1/6 RDMA/uverbs: Introduce UCAP (User CAPabilities) API

From: Chiara Meiohas <[email protected]>

Implement a new User CAPabilities (UCAP) API to provide fine-grained control over specific firmware features.

This approach offers more granular capabilities than the existing Linux capabilities, which may be too generic for certain FW features.

This mechanism represents each capability as a character device with root read-write access. Root processes can grant users special privileges by allowing access to these character devices (e.g., using chown).

UCAP character devices are located in /dev/infiniband and the class path is /sys/class/infiniband_ucaps.

Signed-off-by Chiara Meiohas <[email protected]>
Reviewed-by Yishai Hadas <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/infiniband/core/Makefile      |   3 +-
 drivers/infiniband/core/ucaps.c       | 255 ++++++++++++++++++++++++++
 drivers/infiniband/core/uverbs_main.c |   2 +
 include/rdma/ib_ucaps.h               |  25 +++
 4 files changed, 284 insertions(+), 1 deletion(-)
 create mode 100644 drivers/infiniband/core/ucaps.c
 create mode 100644 include/rdma/ib_ucaps.h
drivers/infiniband/core/Makefile +2 -1
index 8ab4eea5a0a5..d49ded7e95f0 100644
@@ -39,6 +39,7 @@ ib_uverbs-y :=			uverbs_main.o uverbs_cmd.o uverbs_marshall.o \
 				uverbs_std_types_async_fd.o \
 				uverbs_std_types_srq.o \
 				uverbs_std_types_wq.o \
-				uverbs_std_types_qp.o
+				uverbs_std_types_qp.o \
+				ucaps.o
 ib_uverbs-$(CONFIG_INFINIBAND_USER_MEM) += umem.o umem_dmabuf.o
 ib_uverbs-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += umem_odp.o
drivers/infiniband/core/ucaps.c new file +255 -0

255 lines not on this page · raw message

drivers/infiniband/core/uverbs_main.c +2 -0
index 85cfc790a7bb..973fe2c7ef53 100644
@@ -52,6 +52,7 @@
 #include <rdma/ib.h>
 #include <rdma/uverbs_std_types.h>
 #include <rdma/rdma_netlink.h>
+#include <rdma/ib_ucaps.h>
 
 #include "uverbs.h"
 #include "core_priv.h"

@@ -1345,6 +1346,7 @@ static void __exit ib_uverbs_cleanup(void)
 				 IB_UVERBS_NUM_FIXED_MINOR);
 	unregister_chrdev_region(dynamic_uverbs_dev,
 				 IB_UVERBS_NUM_DYNAMIC_MINOR);
+	ib_cleanup_ucaps();
 	mmu_notifier_synchronize();
 }
include/rdma/ib_ucaps.h new file +25 -0
new file mode 100644
index 000000000000..d044d02f087f
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/*
+ * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+
+#ifndef _IB_UCAPS_H_
+#define _IB_UCAPS_H_
+
+#define UCAP_ENABLED(ucaps, type) (!!((ucaps) & (1U << type)))
+
+enum rdma_user_cap {
+	RDMA_UCAP_MLX5_CTRL_LOCAL,
+	RDMA_UCAP_MLX5_CTRL_OTHER_VHCA,
+	RDMA_UCAP_MAX
+};
+
+void ib_cleanup_ucaps(void);
+
+int ib_create_ucap(enum rdma_user_cap type);
+
+void ib_remove_ucap(enum rdma_user_cap type);
+
+int ib_get_ucaps(int *fds, int fd_count, uint64_t *idx_mask);
+
+#endif /* _IB_UCAPS_H_ */
Signature · 1 line
2.48.1
maintainer
#2 lore raw

Re: [PATCH rdma-next 1/6] RDMA/uverbs: Introduce UCAP (User CAPabilities) API

Quoted 21 lines · Leon Romanovsky
drivers/infiniband/core/ucaps.c new file +21 -0
+int ib_create_ucap(enum rdma_user_cap type)
+{
+	struct ib_ucap *ucap;
+	int ret;
+
+	if (type >= RDMA_UCAP_MAX)
+		return -EINVAL;
+
+	mutex_lock(&ucaps_mutex);
+	ret = ib_ucaps_init();
+	if (ret)
+		goto unlock;
+
+	ucap = ucaps_list[type];
+	if (ucap) {
+		ucap->refcount++;
+		mutex_unlock(&ucaps_mutex);
+		return 0;
+	}
+
+	ucap = kzalloc(sizeof(*ucap), GFP_KERNEL);

drivers/infiniband/core/ucaps.c line 172

Quoted 10 lines
drivers/infiniband/core/ucaps.c new file +10 -0
+	if (!ucap) {
+		ret = -ENOMEM;
+		goto unlock;
+	}
+
+	device_initialize(&ucap->dev);
+	ucap->dev.class = &ucaps_class;
+	ucap->dev.devt = MKDEV(MAJOR(ucaps_base_dev), type);
+	ucap->dev.release = ucap_dev_release;
+	dev_set_name(&ucap->dev, ucap_names[type]);

Missing error handling on dev_set_name()

Quoted 1 line

+#define UCAP_ENABLED(ucaps, type) (!!((ucaps) & (1U << type)))

Missing () around type

Why not use a static inline?

Jason

author
#3 lore raw

Re: [PATCH rdma-next 1/6] RDMA/uverbs: Introduce UCAP (User CAPabilities) API

Quoted 26 lines · Jason Gunthorpe
Quoted 25 lines · Leon Romanovsky
drivers/infiniband/core/ucaps.c new file +25 -0
+int ib_create_ucap(enum rdma_user_cap type)
+{
+	struct ib_ucap *ucap;
+	int ret;
+
+	if (type >= RDMA_UCAP_MAX)
+		return -EINVAL;
+
+	mutex_lock(&ucaps_mutex);
+	ret = ib_ucaps_init();
+	if (ret)
+		goto unlock;
+
+	ucap = ucaps_list[type];
+	if (ucap) {
+		ucap->refcount++;
+		mutex_unlock(&ucaps_mutex);
+		return 0;
+	}
+
+	ucap = kzalloc(sizeof(*ucap), GFP_KERNEL);
+	if (!ucap) {
+		ret = -ENOMEM;
+		goto unlock;
+	}
Quoted 8 lines

drivers/infiniband/core/ucaps.c line 172

Quoted 6 lines
drivers/infiniband/core/ucaps.c new file +6 -0
+
+	device_initialize(&ucap->dev);
+	ucap->dev.class = &ucaps_class;
+	ucap->dev.devt = MKDEV(MAJOR(ucaps_base_dev), type);
+	ucap->dev.release = ucap_dev_release;
+	dev_set_name(&ucap->dev, ucap_names[type]);

Missing error handling on dev_set_name()

Most of the kernel users don't check dev_set_name(). It can't fail in reality.

Quoted 6 lines
Quoted 2 lines

+#define UCAP_ENABLED(ucaps, type) (!!((ucaps) & (1U << type)))

Missing () around type

Why not use a static inline?

I don't think that static inline gives any benefit here.

Thanks

Quoted 2 lines

Jason

maintainer
#4 lore raw

Re: [PATCH rdma-next 1/6] RDMA/uverbs: Introduce UCAP (User CAPabilities) API

Quoted 10 lines · Leon Romanovsky
Quoted 7 lines

drivers/infiniband/core/ucaps.c line 172

Quoted 5 lines
drivers/infiniband/core/ucaps.c new file +5 -0
+	device_initialize(&ucap->dev);
+	ucap->dev.class = &ucaps_class;
+	ucap->dev.devt = MKDEV(MAJOR(ucaps_base_dev), type);
+	ucap->dev.release = ucap_dev_release;
+	dev_set_name(&ucap->dev, ucap_names[type]);

Missing error handling on dev_set_name()

Most of the kernel users don't check dev_set_name(). It can't fail in reality.

I'd still add the missing check

Jason

author
#5 lore raw

Re: [PATCH rdma-next 1/6] RDMA/uverbs: Introduce UCAP (User CAPabilities) API

Quoted 10 lines · Jason Gunthorpe
Quoted 8 lines · Leon Romanovsky
Quoted 7 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+	device_initialize(&ucap->dev);
+	ucap->dev.class = &ucaps_class;
+	ucap->dev.devt = MKDEV(MAJOR(ucaps_base_dev), type);
+	ucap->dev.release = ucap_dev_release;
+	dev_set_name(&ucap->dev, ucap_names[type]);

Missing error handling on dev_set_name()

Quoted 4 lines
Quoted 2 lines

Most of the kernel users don't check dev_set_name(). It can't fail in reality.

I'd still add the missing check

I can add this hunk while applying the patch.

drivers/infiniband/core/ucaps.c +3 -1
index 82b1184891ed..e958bd6e34d7 100644
@@ -169,7 +169,9 @@ int ib_create_ucap(enum rdma_user_cap type)
        ucap->dev.class = &ucaps_class;
        ucap->dev.devt = MKDEV(MAJOR(ucaps_base_dev), type);
        ucap->dev.release = ucap_dev_release;
-       dev_set_name(&ucap->dev, ucap_names[type]);
+       ret = dev_set_name(&ucap->dev, ucap_names[type]);
+       if (ret)
+               goto err_device;
 
        cdev_init(&ucap->cdev, &ucaps_cdev_fops);
        ucap->cdev.owner = THIS_MODULE;
Quoted 2 lines

Jason

#6 lore raw

Re: [PATCH rdma-next 1/6] RDMA/uverbs: Introduce UCAP (User CAPabilities) API

In reply to Leon Romanovsky
Quoted 217 lines · Leon Romanovsky

From: Chiara Meiohas <[email protected]>

Implement a new User CAPabilities (UCAP) API to provide fine-grained control over specific firmware features.

This approach offers more granular capabilities than the existing Linux capabilities, which may be too generic for certain FW features.

This mechanism represents each capability as a character device with root read-write access. Root processes can grant users special privileges by allowing access to these character devices (e.g., using chown).

UCAP character devices are located in /dev/infiniband and the class path is /sys/class/infiniband_ucaps.

Signed-off-by Chiara Meiohas <[email protected]>
Reviewed-by Yishai Hadas <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/infiniband/core/Makefile      |   3 +-
 drivers/infiniband/core/ucaps.c       | 255 ++++++++++++++++++++++++++
 drivers/infiniband/core/uverbs_main.c |   2 +
 include/rdma/ib_ucaps.h               |  25 +++
 4 files changed, 284 insertions(+), 1 deletion(-)
 create mode 100644 drivers/infiniband/core/ucaps.c
 create mode 100644 include/rdma/ib_ucaps.h
drivers/infiniband/core/Makefile +2 -1
index 8ab4eea5a0a5..d49ded7e95f0 100644
@@ -39,6 +39,7 @@ ib_uverbs-y :=                        uverbs_main.o uverbs_cmd.o uverbs_marshall.o \
                                uverbs_std_types_async_fd.o \
                                uverbs_std_types_srq.o \
                                uverbs_std_types_wq.o \
-                               uverbs_std_types_qp.o
+                               uverbs_std_types_qp.o \
+                               ucaps.o
 ib_uverbs-$(CONFIG_INFINIBAND_USER_MEM) += umem.o umem_dmabuf.o
 ib_uverbs-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += umem_odp.o
drivers/infiniband/core/ucaps.c new file +169 -0
new file mode 100644
index 000000000000..82b1184891ed
@@ -0,0 +1,255 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/*
+ * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+
+#include <linux/cdev.h>
+#include <linux/mutex.h>
+#include <linux/file.h>
+#include <linux/fs.h>
+#include <rdma/ib_ucaps.h>
+
+#define RDMA_UCAP_FIRST RDMA_UCAP_MLX5_CTRL_LOCAL
+
+static DEFINE_MUTEX(ucaps_mutex);
+static struct ib_ucap *ucaps_list[RDMA_UCAP_MAX];
+static bool ucaps_class_is_registered;
+static dev_t ucaps_base_dev;
+
+struct ib_ucap {
+       struct cdev cdev;
+       struct device dev;
+       int refcount;
+};
+
+static const char *ucap_names[RDMA_UCAP_MAX] = {
+       [RDMA_UCAP_MLX5_CTRL_LOCAL] = "mlx5_perm_ctrl_local",
+       [RDMA_UCAP_MLX5_CTRL_OTHER_VHCA] = "mlx5_perm_ctrl_other_vhca"
+};
+
+static char *ucaps_devnode(const struct device *dev, umode_t *mode)
+{
+       if (mode)
+               *mode = 0600;
+
+       return kasprintf(GFP_KERNEL, "infiniband/%s", dev_name(dev));
+}
+
+static const struct class ucaps_class = {
+       .name = "infiniband_ucaps",
+       .devnode = ucaps_devnode,
+};
+
+static const struct file_operations ucaps_cdev_fops = {
+       .owner = THIS_MODULE,
+       .open = simple_open,
+};
+
+/**
+ * ib_cleanup_ucaps - cleanup all API resources and class.
+ *
+ * This is called once, when removing the ib_uverbs module.
+ */
+void ib_cleanup_ucaps(void)
+{
+       mutex_lock(&ucaps_mutex);
+       if (!ucaps_class_is_registered) {
+               mutex_unlock(&ucaps_mutex);
+               return;
+       }
+
+       for (int i = RDMA_UCAP_FIRST; i < RDMA_UCAP_MAX; i++)
+               WARN_ON(ucaps_list[i]);
+
+       class_unregister(&ucaps_class);
+       ucaps_class_is_registered = false;
+       unregister_chrdev_region(ucaps_base_dev, RDMA_UCAP_MAX);
+       mutex_unlock(&ucaps_mutex);
+}
+
+static int get_ucap_from_devt(dev_t devt, u64 *idx_mask)
+{
+       for (int type = RDMA_UCAP_FIRST; type < RDMA_UCAP_MAX; type++) {
+               if (ucaps_list[type] && ucaps_list[type]->dev.devt == devt) {
+                       *idx_mask |= 1 << type;
+                       return 0;
+               }
+       }
+
+       return -EINVAL;
+}
+
+static int get_devt_from_fd(unsigned int fd, dev_t *ret_dev)
+{
+       struct file *file;
+
+       file = fget(fd);
+       if (!file)
+               return -EBADF;
+
+       *ret_dev = file_inode(file)->i_rdev;
+       fput(file);
+       return 0;
+}
+
+/**
+ * ib_ucaps_init - Initialization required before ucap creation.
+ *
+ * Return: 0 on success, or a negative errno value on failure
+ */
+static int ib_ucaps_init(void)
+{
+       int ret = 0;
+
+       if (ucaps_class_is_registered)
+               return ret;
+
+       ret = class_register(&ucaps_class);
+       if (ret)
+               return ret;
+
+       ret = alloc_chrdev_region(&ucaps_base_dev, 0, RDMA_UCAP_MAX,
+                                 ucaps_class.name);
+       if (ret < 0) {
+               class_unregister(&ucaps_class);
+               return ret;
+       }
+
+       ucaps_class_is_registered = true;
+
+       return 0;
+}
+
+static void ucap_dev_release(struct device *device)
+{
+       struct ib_ucap *ucap = container_of(device, struct ib_ucap, dev);
+
+       kfree(ucap);
+}
+
+/**
+ * ib_create_ucap - Add a ucap character device
+ * @type: UCAP type
+ *
+ * Creates a ucap character device in the /dev/infiniband directory. By default,
+ * the device has root-only read-write access.
+ *
+ * A driver may call this multiple times with the same UCAP type. A reference
+ * count tracks creations and deletions.
+ *
+ * Return: 0 on success, or a negative errno value on failure
+ */
+int ib_create_ucap(enum rdma_user_cap type)
+{
+       struct ib_ucap *ucap;
+       int ret;
+
+       if (type >= RDMA_UCAP_MAX)
+               return -EINVAL;
+
+       mutex_lock(&ucaps_mutex);
+       ret = ib_ucaps_init();
+       if (ret)
+               goto unlock;
+
+       ucap = ucaps_list[type];
+       if (ucap) {
+               ucap->refcount++;
+               mutex_unlock(&ucaps_mutex);
+               return 0;
+       }
+
+       ucap = kzalloc(sizeof(*ucap), GFP_KERNEL);
+       if (!ucap) {
+               ret = -ENOMEM;
+               goto unlock;
+       }
+
+       device_initialize(&ucap->dev);
+       ucap->dev.class = &ucaps_class;

drivers/infiniband/core/ucaps.c line 179

Quoted 10 lines
drivers/infiniband/core/ucaps.c new file +10 -0
+       ucap->dev.devt = MKDEV(MAJOR(ucaps_base_dev), type);
+       ucap->dev.release = ucap_dev_release;
+       dev_set_name(&ucap->dev, ucap_names[type]);
+
+       cdev_init(&ucap->cdev, &ucaps_cdev_fops);
+       ucap->cdev.owner = THIS_MODULE;
+
+       ret = cdev_device_add(&ucap->cdev, &ucap->dev);
+       if (ret)
+               goto err_device;

Memory leak in the error path, need to free ucap here?

Quoted 28 lines
drivers/infiniband/core/ucaps.c new file +28 -0
+
+       ucap->refcount = 1;
+       ucaps_list[type] = ucap;
+       mutex_unlock(&ucaps_mutex);
+
+       return 0;
+
+err_device:
+       put_device(&ucap->dev);
+unlock:
+       mutex_unlock(&ucaps_mutex);
+       return ret;
+}
+EXPORT_SYMBOL(ib_create_ucap);
+
+/**
+ * ib_remove_ucap - Remove a ucap character device
+ * @type: User cap type
+ *
+ * Removes the ucap character device according to type. The device is completely
+ * removed from the filesystem when its reference count reaches 0.
+ */
+void ib_remove_ucap(enum rdma_user_cap type)
+{
+       struct ib_ucap *ucap;
+
+       mutex_lock(&ucaps_mutex);
+       ucap = ucaps_list[type];

drivers/infiniband/core/ucaps.c line 217

Quoted 10 lines
drivers/infiniband/core/ucaps.c new file +10 -0
+       if (WARN_ON(!ucap))
+               goto end;
+
+       ucap->refcount--;
+       if (ucap->refcount)
+               goto end;
+
+       ucaps_list[type] = NULL;
+       cdev_device_del(&ucap->cdev, &ucap->dev);
+       put_device(&ucap->dev);

need to free ucap here

Quoted 93 lines
drivers/infiniband/core/ucaps.c new file +38 -0
+
+end:
+       mutex_unlock(&ucaps_mutex);
+}
+EXPORT_SYMBOL(ib_remove_ucap);
+
+/**
+ * ib_get_ucaps - Get bitmask of ucap types from file descriptors
+ * @fds: Array of file descriptors
+ * @fd_count: Number of file descriptors in the array
+ * @idx_mask: Bitmask to be updated based on the ucaps in the fd list
+ *
+ * Given an array of file descriptors, this function returns a bitmask of
+ * the ucaps where a bit is set if an FD for that ucap type was in the array.
+ *
+ * Return: 0 on success, or a negative errno value on failure
+ */
+int ib_get_ucaps(int *fds, int fd_count, uint64_t *idx_mask)
+{
+       int ret = 0;
+       dev_t dev;
+
+       *idx_mask = 0;
+       mutex_lock(&ucaps_mutex);
+       for (int i = 0; i < fd_count; i++) {
+               ret = get_devt_from_fd(fds[i], &dev);
+               if (ret)
+                       goto end;
+
+               ret = get_ucap_from_devt(dev, idx_mask);
+               if (ret)
+                       goto end;
+       }
+
+end:
+       mutex_unlock(&ucaps_mutex);
+       return ret;
+}
drivers/infiniband/core/uverbs_main.c +2 -0
index 85cfc790a7bb..973fe2c7ef53 100644
@@ -52,6 +52,7 @@
 #include <rdma/ib.h>
 #include <rdma/uverbs_std_types.h>
 #include <rdma/rdma_netlink.h>
+#include <rdma/ib_ucaps.h>
 
 #include "uverbs.h"
 #include "core_priv.h"

@@ -1345,6 +1346,7 @@ static void __exit ib_uverbs_cleanup(void)
                                 IB_UVERBS_NUM_FIXED_MINOR);
        unregister_chrdev_region(dynamic_uverbs_dev,
                                 IB_UVERBS_NUM_DYNAMIC_MINOR);
+       ib_cleanup_ucaps();
        mmu_notifier_synchronize();
 }
include/rdma/ib_ucaps.h new file +25 -1
new file mode 100644
index 000000000000..d044d02f087f
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/*
+ * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+
+#ifndef _IB_UCAPS_H_
+#define _IB_UCAPS_H_
+
+#define UCAP_ENABLED(ucaps, type) (!!((ucaps) & (1U << type)))
+
+enum rdma_user_cap {
+       RDMA_UCAP_MLX5_CTRL_LOCAL,
+       RDMA_UCAP_MLX5_CTRL_OTHER_VHCA,
+       RDMA_UCAP_MAX
+};
+
+void ib_cleanup_ucaps(void);
+
+int ib_create_ucap(enum rdma_user_cap type);
+
+void ib_remove_ucap(enum rdma_user_cap type);
+
+int ib_get_ucaps(int *fds, int fd_count, uint64_t *idx_mask);
+
+#endif /* _IB_UCAPS_H_ */
--

2.48.1

Signature · 2 lines
Regards,
Kalesh AP
author
#7 lore raw

Re: [PATCH rdma-next 1/6] RDMA/uverbs: Introduce UCAP (User CAPabilities) API

Quoted 29 lines · Kalesh Anakkur Purayil
Quoted 28 lines · Leon Romanovsky

From: Chiara Meiohas <[email protected]>

Implement a new User CAPabilities (UCAP) API to provide fine-grained control over specific firmware features.

This approach offers more granular capabilities than the existing Linux capabilities, which may be too generic for certain FW features.

This mechanism represents each capability as a character device with root read-write access. Root processes can grant users special privileges by allowing access to these character devices (e.g., using chown).

UCAP character devices are located in /dev/infiniband and the class path is /sys/class/infiniband_ucaps.

Signed-off-by Chiara Meiohas <[email protected]>
Reviewed-by Yishai Hadas <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/infiniband/core/Makefile      |   3 +-
 drivers/infiniband/core/ucaps.c       | 255 ++++++++++++++++++++++++++
 drivers/infiniband/core/uverbs_main.c |   2 +
 include/rdma/ib_ucaps.h               |  25 +++
 4 files changed, 284 insertions(+), 1 deletion(-)
 create mode 100644 drivers/infiniband/core/ucaps.c
 create mode 100644 include/rdma/ib_ucaps.h

<...>

Quoted 5 lines
Quoted 5 lines
drivers/infiniband/core/ucaps.c new file +5 -0
+       device_initialize(&ucap->dev);
+       ucap->dev.class = &ucaps_class;
+       ucap->dev.devt = MKDEV(MAJOR(ucaps_base_dev), type);
+       ucap->dev.release = ucap_dev_release;
+       dev_set_name(&ucap->dev, ucap_names[type]);
Quoted 8 lines

drivers/infiniband/core/ucaps.c line 179

Quoted 7 lines
drivers/infiniband/core/ucaps.c new file +7 -0
+
+       cdev_init(&ucap->cdev, &ucaps_cdev_fops);
+       ucap->cdev.owner = THIS_MODULE;
+
+       ret = cdev_device_add(&ucap->cdev, &ucap->dev);
+       if (ret)
+               goto err_device;

Memory leak in the error path, need to free ucap here?

It is done through call to put_device(&ucap->dev) below. This is how device is freed after device_initialize().

<...>

Quoted 7 lines
Quoted 7 lines
drivers/infiniband/core/ucaps.c new file +7 -0
+err_device:
+       put_device(&ucap->dev);
+unlock:
+       mutex_unlock(&ucaps_mutex);
+       return ret;
+}
+EXPORT_SYMBOL(ib_create_ucap);

<...>

Quoted 4 lines

drivers/infiniband/core/ucaps.c line 217

Quoted 3 lines
drivers/infiniband/core/ucaps.c new file +3 -0
+       ucaps_list[type] = NULL;
+       cdev_device_del(&ucap->cdev, &ucap->dev);
+       put_device(&ucap->dev);

need to free ucap here

Same as above.

Thanks

#8 lore raw

Re: [PATCH rdma-next 1/6] RDMA/uverbs: Introduce UCAP (User CAPabilities) API

Quoted 48 lines · Leon Romanovsky
Quoted 29 lines · Kalesh Anakkur Purayil
Quoted 28 lines · Leon Romanovsky

From: Chiara Meiohas <[email protected]>

Implement a new User CAPabilities (UCAP) API to provide fine-grained control over specific firmware features.

This approach offers more granular capabilities than the existing Linux capabilities, which may be too generic for certain FW features.

This mechanism represents each capability as a character device with root read-write access. Root processes can grant users special privileges by allowing access to these character devices (e.g., using chown).

UCAP character devices are located in /dev/infiniband and the class path is /sys/class/infiniband_ucaps.

Signed-off-by Chiara Meiohas <[email protected]>
Reviewed-by Yishai Hadas <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/infiniband/core/Makefile      |   3 +-
 drivers/infiniband/core/ucaps.c       | 255 ++++++++++++++++++++++++++
 drivers/infiniband/core/uverbs_main.c |   2 +
 include/rdma/ib_ucaps.h               |  25 +++
 4 files changed, 284 insertions(+), 1 deletion(-)
 create mode 100644 drivers/infiniband/core/ucaps.c
 create mode 100644 include/rdma/ib_ucaps.h

<...>

Quoted 5 lines
Quoted 5 lines
drivers/infiniband/core/ucaps.c new file +5 -0
+       device_initialize(&ucap->dev);
+       ucap->dev.class = &ucaps_class;
+       ucap->dev.devt = MKDEV(MAJOR(ucaps_base_dev), type);
+       ucap->dev.release = ucap_dev_release;
+       dev_set_name(&ucap->dev, ucap_names[type]);
Quoted 8 lines

drivers/infiniband/core/ucaps.c line 179

Quoted 7 lines
drivers/infiniband/core/ucaps.c new file +7 -0
+
+       cdev_init(&ucap->cdev, &ucaps_cdev_fops);
+       ucap->cdev.owner = THIS_MODULE;
+
+       ret = cdev_device_add(&ucap->cdev, &ucap->dev);
+       if (ret)
+               goto err_device;

Memory leak in the error path, need to free ucap here?

Quoted 2 lines

It is done through call to put_device(&ucap->dev) below. This is how device is freed after device_initialize().

Got it, missed the ucap_dev_release() part.
Sorry for the confusion.

Quoted 21 lines

<...>

Quoted 7 lines
Quoted 7 lines
drivers/infiniband/core/ucaps.c new file +7 -0
+err_device:
+       put_device(&ucap->dev);
+unlock:
+       mutex_unlock(&ucaps_mutex);
+       return ret;
+}
+EXPORT_SYMBOL(ib_create_ucap);

<...>

Quoted 4 lines

drivers/infiniband/core/ucaps.c line 217

Quoted 3 lines
drivers/infiniband/core/ucaps.c new file +3 -0
+       ucaps_list[type] = NULL;
+       cdev_device_del(&ucap->cdev, &ucap->dev);
+       put_device(&ucap->dev);

need to free ucap here

Same as above.

Thanks

Signature · 2 lines
Regards,
Kalesh AP
author
#9 lore raw

Re: [PATCH rdma-next 1/6] RDMA/uverbs: Introduce UCAP (User CAPabilities) API

In reply to Leon Romanovsky
Quoted 27 lines · Leon Romanovsky

From: Chiara Meiohas <[email protected]>

Implement a new User CAPabilities (UCAP) API to provide fine-grained control over specific firmware features.

This approach offers more granular capabilities than the existing Linux capabilities, which may be too generic for certain FW features.

This mechanism represents each capability as a character device with root read-write access. Root processes can grant users special privileges by allowing access to these character devices (e.g., using chown).

UCAP character devices are located in /dev/infiniband and the class path is /sys/class/infiniband_ucaps.

Signed-off-by Chiara Meiohas <[email protected]>
Reviewed-by Yishai Hadas <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/infiniband/core/Makefile      |   3 +-
 drivers/infiniband/core/ucaps.c       | 255 ++++++++++++++++++++++++++
 drivers/infiniband/core/uverbs_main.c |   2 +
 include/rdma/ib_ucaps.h               |  25 +++
 4 files changed, 284 insertions(+), 1 deletion(-)
 create mode 100644 drivers/infiniband/core/ucaps.c
 create mode 100644 include/rdma/ib_ucaps.h

<...>

Quoted 13 lines
drivers/infiniband/core/ucaps.c new file +13 -0
+/**
+ * ib_remove_ucap - Remove a ucap character device
+ * @type: User cap type
+ *
+ * Removes the ucap character device according to type. The device is completely
+ * removed from the filesystem when its reference count reaches 0.
+ */
+void ib_remove_ucap(enum rdma_user_cap type)
+{
+	struct ib_ucap *ucap;
+
+	mutex_lock(&ucaps_mutex);
+	ucap = ucaps_list[type];

drivers/infiniband/core/ucaps.c line 217

Quoted 10 lines
drivers/infiniband/core/ucaps.c new file +10 -0
+	if (WARN_ON(!ucap))
+		goto end;
+
+	ucap->refcount--;
+	if (ucap->refcount)
+		goto end;
+
+	ucaps_list[type] = NULL;
+	cdev_device_del(&ucap->cdev, &ucap->dev);
+	put_device(&ucap->dev);

This is kref release pattern, please use it.

drivers/infiniband/core/ucaps.c line 222

Quoted 5 lines
drivers/infiniband/core/ucaps.c new file +5 -0
+
+end:
+	mutex_unlock(&ucaps_mutex);
+}
+EXPORT_SYMBOL(ib_remove_ucap);

Thanks

author
#10 lore raw

PATCH 2/6 RDMA/mlx5: Create UCAP char devices for supported device capabilities

In reply to Leon Romanovsky

From: Chiara Meiohas <[email protected]>

Create UCAP character devices when probing an IB device with supported firmware capabilities.

If the RDMA_CTRL general object type is supported, check for specific UCTX capabilities:
Create /dev/infiniband/mlx5_perm_ctrl_local for RDMA_UCAP_MLX5_CTRL_LOCAL Create /dev/infiniband/mlx5_perm_ctrl_other_vhca for RDMA_UCAP_MLX5_CTRL_OTHER_VHCA

Signed-off-by Chiara Meiohas <[email protected]>
Reviewed-by Yishai Hadas <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/infiniband/hw/mlx5/main.c | 47 +++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
drivers/infiniband/hw/mlx5/main.c +47 -0
index 81849eb671a1..04b489a6a449 100644
@@ -47,6 +47,7 @@
 #include <rdma/uverbs_ioctl.h>
 #include <rdma/mlx5_user_ioctl_verbs.h>
 #include <rdma/mlx5_user_ioctl_cmds.h>
+#include <rdma/ib_ucaps.h>
 #include "macsec.h"
 #include "data_direct.h"
 

@@ -4201,8 +4202,47 @@ static int mlx5_ib_init_var_table(struct mlx5_ib_dev *dev)
 	return (var_table->bitmap) ? 0 : -ENOMEM;
 }
 
+static void mlx5_ib_cleanup_ucaps(struct mlx5_ib_dev *dev)
+{
+	if (MLX5_CAP_GEN(dev->mdev, uctx_cap) & MLX5_UCTX_CAP_RDMA_CTRL)
+		ib_remove_ucap(RDMA_UCAP_MLX5_CTRL_LOCAL);
+
+	if (MLX5_CAP_GEN(dev->mdev, uctx_cap) &
+	    MLX5_UCTX_CAP_RDMA_CTRL_OTHER_VHCA)
+		ib_remove_ucap(RDMA_UCAP_MLX5_CTRL_OTHER_VHCA);
+}
+
+static int mlx5_ib_init_ucaps(struct mlx5_ib_dev *dev)
+{
+	int ret;
+
+	if (MLX5_CAP_GEN(dev->mdev, uctx_cap) & MLX5_UCTX_CAP_RDMA_CTRL) {
+		ret = ib_create_ucap(RDMA_UCAP_MLX5_CTRL_LOCAL);
+		if (ret)
+			return ret;
+	}
+
+	if (MLX5_CAP_GEN(dev->mdev, uctx_cap) &
+	    MLX5_UCTX_CAP_RDMA_CTRL_OTHER_VHCA) {
+		ret = ib_create_ucap(RDMA_UCAP_MLX5_CTRL_OTHER_VHCA);
+		if (ret)
+			goto remove_local;
+	}
+
+	return 0;
+
+remove_local:
+	if (MLX5_CAP_GEN(dev->mdev, uctx_cap) & MLX5_UCTX_CAP_RDMA_CTRL)
+		ib_remove_ucap(RDMA_UCAP_MLX5_CTRL_LOCAL);
+	return ret;
+}
+
 static void mlx5_ib_stage_caps_cleanup(struct mlx5_ib_dev *dev)
 {
+	if (MLX5_CAP_GEN_2_64(dev->mdev, general_obj_types_127_64) &
+	    MLX5_HCA_CAP_2_GENERAL_OBJECT_TYPES_RDMA_CTRL)
+		mlx5_ib_cleanup_ucaps(dev);
+
 	bitmap_free(dev->var_table.bitmap);
 }
 

@@ -4253,6 +4293,13 @@ static int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev)
 			return err;
 	}
 
+	if (MLX5_CAP_GEN_2_64(dev->mdev, general_obj_types_127_64) &
+	    MLX5_HCA_CAP_2_GENERAL_OBJECT_TYPES_RDMA_CTRL) {
+		err = mlx5_ib_init_ucaps(dev);
+		if (err)
+			return err;
+	}
+
 	dev->ib_dev.use_cq_dim = true;
 
 	return 0;
Signature · 1 line
2.48.1
author
#11 lore raw

PATCH 3/6 RDMA/uverbs: Add support for UCAPs in context creation

In reply to Leon Romanovsky

From: Chiara Meiohas <[email protected]>

Add support for file descriptor array attribute for GET_CONTEXT commands.

Check that the file descriptor (fd) array represents fds for valid UCAPs. Store the enabled UCAPs from the fd array as a bitmask in ib_ucontext.

Signed-off-by Chiara Meiohas <[email protected]>
Reviewed-by Yishai Hadas <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/infiniband/core/uverbs_cmd.c          | 19 +++++++++++++++++++
 .../infiniband/core/uverbs_std_types_device.c |  4 ++++
 include/rdma/ib_verbs.h                       |  1 +
 include/uapi/rdma/ib_user_ioctl_cmds.h        |  1 +
 4 files changed, 25 insertions(+)
drivers/infiniband/core/uverbs_cmd.c +19 -0
index 5ad14c39d48c..96d639e1ffa0 100644
@@ -42,6 +42,7 @@
 
 #include <rdma/uverbs_types.h>
 #include <rdma/uverbs_std_types.h>
+#include <rdma/ib_ucaps.h>
 #include "rdma_core.h"
 
 #include "uverbs.h"

@@ -232,6 +233,8 @@ int ib_init_ucontext(struct uverbs_attr_bundle *attrs)
 {
 	struct ib_ucontext *ucontext = attrs->context;
 	struct ib_uverbs_file *file = attrs->ufile;
+	int *fd_array;
+	int fd_count;
 	int ret;
 
 	if (!down_read_trylock(&file->hw_destroy_rwsem))

@@ -247,6 +250,22 @@ int ib_init_ucontext(struct uverbs_attr_bundle *attrs)
 	if (ret)
 		goto err;
 
+	if (uverbs_attr_is_valid(attrs, UVERBS_ATTR_GET_CONTEXT_FD_ARR)) {
+		fd_count = uverbs_attr_ptr_get_array_size(attrs,
+							  UVERBS_ATTR_GET_CONTEXT_FD_ARR,
+							  sizeof(int));
+		if (fd_count < 0) {
+			ret = fd_count;
+			goto err_uncharge;
+		}
+
+		fd_array = uverbs_attr_get_alloced_ptr(attrs,
+						       UVERBS_ATTR_GET_CONTEXT_FD_ARR);
+		ret = ib_get_ucaps(fd_array, fd_count, &ucontext->enabled_caps);
+		if (ret)
+			goto err_uncharge;
+	}
+
 	ret = ucontext->device->ops.alloc_ucontext(ucontext,
 						   &attrs->driver_udata);
 	if (ret)
drivers/infiniband/core/uverbs_std_types_device.c +4 -0
index fb0555647336..c0fd283d9d6c 100644
@@ -437,6 +437,10 @@ DECLARE_UVERBS_NAMED_METHOD(
 			    UVERBS_ATTR_TYPE(u32), UA_OPTIONAL),
 	UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_GET_CONTEXT_CORE_SUPPORT,
 			    UVERBS_ATTR_TYPE(u64), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(UVERBS_ATTR_GET_CONTEXT_FD_ARR,
+			   UVERBS_ATTR_MIN_SIZE(sizeof(int)),
+			   UA_OPTIONAL,
+			   UA_ALLOC_AND_COPY),
 	UVERBS_ATTR_UHW());
 
 DECLARE_UVERBS_NAMED_METHOD(
include/rdma/ib_verbs.h +1 -0
index b59bf30de430..84b6d64c9e3a 100644
@@ -1530,6 +1530,7 @@ struct ib_ucontext {
 	struct ib_uverbs_file  *ufile;
 
 	struct ib_rdmacg_object	cg_obj;
+	u64 enabled_caps;
 	/*
 	 * Implementation details of the RDMA core, don't use in drivers:
 	 */
include/uapi/rdma/ib_user_ioctl_cmds.h +1 -0
index ec719053aab9..ac7b162611ed 100644
@@ -88,6 +88,7 @@ enum uverbs_attrs_query_port_cmd_attr_ids {
 enum uverbs_attrs_get_context_attr_ids {
 	UVERBS_ATTR_GET_CONTEXT_NUM_COMP_VECTORS,
 	UVERBS_ATTR_GET_CONTEXT_CORE_SUPPORT,
+	UVERBS_ATTR_GET_CONTEXT_FD_ARR,
 };
 
 enum uverbs_attrs_query_context_attr_ids {
Signature · 1 line
2.48.1
author
#12 lore raw

PATCH 4/6 RDMA/mlx5: Check enabled UCAPs when creating ucontext

In reply to Leon Romanovsky

From: Chiara Meiohas <[email protected]>

Verify that the enabled UCAPs are supported by the device before creating the ucontext.
If supported, create the ucontext with the associated capabilities.

Store the privileged ucontext UID on creation and remove it when destroying the privileged ucontext. This allows the command interface to recognize privileged commands through its UID.

Signed-off-by Chiara Meiohas <[email protected]>
Reviewed-by Yishai Hadas <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/infiniband/hw/mlx5/devx.c | 31 +++++++++++++++++++++++++++++--
 drivers/infiniband/hw/mlx5/devx.h |  5 +++--
 drivers/infiniband/hw/mlx5/main.c | 30 ++++++++++++++++++++++++++----
 3 files changed, 58 insertions(+), 8 deletions(-)
drivers/infiniband/hw/mlx5/devx.c +29 -2
index 4186884c66e1..5d42087c7777 100644
@@ -13,6 +13,7 @@
 #include <rdma/uverbs_std_types.h>
 #include <linux/mlx5/driver.h>
 #include <linux/mlx5/fs.h>
+#include <rdma/ib_ucaps.h>
 #include "mlx5_ib.h"
 #include "devx.h"
 #include "qp.h"

@@ -122,7 +123,27 @@ devx_ufile2uctx(const struct uverbs_attr_bundle *attrs)
 	return to_mucontext(ib_uverbs_get_ucontext(attrs));
 }
 
-int mlx5_ib_devx_create(struct mlx5_ib_dev *dev, bool is_user)
+static int set_uctx_ucaps(struct mlx5_ib_dev *dev, u64 req_ucaps, u32 *cap)
+{
+	if (UCAP_ENABLED(req_ucaps, RDMA_UCAP_MLX5_CTRL_LOCAL)) {
+		if (MLX5_CAP_GEN(dev->mdev, uctx_cap) & MLX5_UCTX_CAP_RDMA_CTRL)
+			*cap |= MLX5_UCTX_CAP_RDMA_CTRL;
+		else
+			return -EOPNOTSUPP;
+	}
+
+	if (UCAP_ENABLED(req_ucaps, RDMA_UCAP_MLX5_CTRL_OTHER_VHCA)) {
+		if (MLX5_CAP_GEN(dev->mdev, uctx_cap) &
+		    MLX5_UCTX_CAP_RDMA_CTRL_OTHER_VHCA)
+			*cap |= MLX5_UCTX_CAP_RDMA_CTRL_OTHER_VHCA;
+		else
+			return -EOPNOTSUPP;
+	}
+
+	return 0;
+}
+
+int mlx5_ib_devx_create(struct mlx5_ib_dev *dev, bool is_user, u64 req_ucaps)
 {
 	u32 in[MLX5_ST_SZ_DW(create_uctx_in)] = {};
 	u32 out[MLX5_ST_SZ_DW(create_uctx_out)] = {};

@@ -144,6 +165,12 @@ int mlx5_ib_devx_create(struct mlx5_ib_dev *dev, bool is_user)
 	     MLX5_UCTX_CAP_INTERNAL_DEV_RES))
 		cap |= MLX5_UCTX_CAP_INTERNAL_DEV_RES;
 
+	if (req_ucaps) {
+		err = set_uctx_ucaps(dev, req_ucaps, &cap);
+		if (err)
+			return err;
+	}
+
 	MLX5_SET(create_uctx_in, in, opcode, MLX5_CMD_OP_CREATE_UCTX);
 	MLX5_SET(uctx, uctx, cap, cap);
 

@@ -2573,7 +2600,7 @@ int mlx5_ib_devx_init(struct mlx5_ib_dev *dev)
 	struct mlx5_devx_event_table *table = &dev->devx_event_table;
 	int uid;
 
-	uid = mlx5_ib_devx_create(dev, false);
+	uid = mlx5_ib_devx_create(dev, false, 0);
 	if (uid > 0) {
 		dev->devx_whitelist_uid = uid;
 		xa_init(&table->event_xa);
drivers/infiniband/hw/mlx5/devx.h +3 -2
index 1344bf4c9d21..ee9e7d3af93f 100644
@@ -24,13 +24,14 @@ struct devx_obj {
 	struct list_head event_sub; /* holds devx_event_subscription entries */
 };
 #if IS_ENABLED(CONFIG_INFINIBAND_USER_ACCESS)
-int mlx5_ib_devx_create(struct mlx5_ib_dev *dev, bool is_user);
+int mlx5_ib_devx_create(struct mlx5_ib_dev *dev, bool is_user, u64 req_ucaps);
 void mlx5_ib_devx_destroy(struct mlx5_ib_dev *dev, u16 uid);
 int mlx5_ib_devx_init(struct mlx5_ib_dev *dev);
 void mlx5_ib_devx_cleanup(struct mlx5_ib_dev *dev);
 void mlx5_ib_ufile_hw_cleanup(struct ib_uverbs_file *ufile);
 #else
-static inline int mlx5_ib_devx_create(struct mlx5_ib_dev *dev, bool is_user)
+static inline int mlx5_ib_devx_create(struct mlx5_ib_dev *dev, bool is_user,
+				      u64 req_ucaps)
 {
 	return -EOPNOTSUPP;
 }
drivers/infiniband/hw/mlx5/main.c +26 -4
index 04b489a6a449..d07cacaa0abd 100644
@@ -1935,6 +1935,12 @@ static int set_ucontext_resp(struct ib_ucontext *uctx,
 	return 0;
 }
 
+static bool uctx_rdma_ctrl_is_enabled(u64 enabled_caps)
+{
+	return UCAP_ENABLED(enabled_caps, RDMA_UCAP_MLX5_CTRL_LOCAL) ||
+	       UCAP_ENABLED(enabled_caps, RDMA_UCAP_MLX5_CTRL_OTHER_VHCA);
+}
+
 static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,
 				  struct ib_udata *udata)
 {

@@ -1977,10 +1983,17 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,
 		return -EINVAL;
 
 	if (req.flags & MLX5_IB_ALLOC_UCTX_DEVX) {
-		err = mlx5_ib_devx_create(dev, true);
+		err = mlx5_ib_devx_create(dev, true, uctx->enabled_caps);
 		if (err < 0)
 			goto out_ctx;
 		context->devx_uid = err;
+
+		if (uctx_rdma_ctrl_is_enabled(uctx->enabled_caps)) {
+			err = mlx5_cmd_add_privileged_uid(dev->mdev,
+							  context->devx_uid);
+			if (err)
+				goto out_devx;
+		}
 	}
 
 	lib_uar_4k = req.lib_caps & MLX5_LIB_CAP_4K_UAR;

@@ -1995,7 +2008,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,
 	/* updates req->total_num_bfregs */
 	err = calc_total_bfregs(dev, lib_uar_4k, &req, bfregi);
 	if (err)
-		goto out_devx;
+		goto out_ucap;
 
 	mutex_init(&bfregi->lock);
 	bfregi->lib_uar_4k = lib_uar_4k;

@@ -2003,7 +2016,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,
 				GFP_KERNEL);
 	if (!bfregi->count) {
 		err = -ENOMEM;
-		goto out_devx;
+		goto out_ucap;
 	}
 
 	bfregi->sys_pages = kcalloc(bfregi->num_sys_pages,

@@ -2067,6 +2080,11 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,
 out_count:
 	kfree(bfregi->count);
 
+out_ucap:
+	if (req.flags & MLX5_IB_ALLOC_UCTX_DEVX &&
+	    uctx_rdma_ctrl_is_enabled(uctx->enabled_caps))
+		mlx5_cmd_remove_privileged_uid(dev->mdev, context->devx_uid);
+
 out_devx:
 	if (req.flags & MLX5_IB_ALLOC_UCTX_DEVX)
 		mlx5_ib_devx_destroy(dev, context->devx_uid);

@@ -2111,8 +2129,12 @@ static void mlx5_ib_dealloc_ucontext(struct ib_ucontext *ibcontext)
 	kfree(bfregi->sys_pages);
 	kfree(bfregi->count);
 
-	if (context->devx_uid)
+	if (context->devx_uid) {
+		if (uctx_rdma_ctrl_is_enabled(ibcontext->enabled_caps))
+			mlx5_cmd_remove_privileged_uid(dev->mdev,
+						       context->devx_uid);
 		mlx5_ib_devx_destroy(dev, context->devx_uid);
+	}
 }
 
 static phys_addr_t uar_index2pfn(struct mlx5_ib_dev *dev,
Signature · 1 line
2.48.1
author
#13 lore raw

PATCH 5/6 RDMA/mlx5: Expose RDMA TRANSPORT flow table types to userspace

In reply to Leon Romanovsky

From: Patrisious Haddad <[email protected]>

This patch adds RDMA_TRANSPORT_RX and RDMA_TRANSPORT_TX as a new flow table type for matcher creation.

Signed-off-by Patrisious Haddad <[email protected]>
Reviewed-by Mark Bloch <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/infiniband/hw/mlx5/fs.c           | 154 ++++++++++++++++++++--
 drivers/infiniband/hw/mlx5/fs.h           |   2 +
 drivers/infiniband/hw/mlx5/mlx5_ib.h      |   3 +
 include/uapi/rdma/mlx5_user_ioctl_cmds.h  |   1 +
 include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
 5 files changed, 149 insertions(+), 13 deletions(-)
drivers/infiniband/hw/mlx5/fs.c +141 -13
index 162814ae8cb4..6ae2801fa13f 100644
@@ -12,6 +12,7 @@
 #include <rdma/mlx5_user_ioctl_verbs.h>
 #include <rdma/ib_hdrs.h>
 #include <rdma/ib_umem.h>
+#include <rdma/ib_ucaps.h>
 #include <linux/mlx5/driver.h>
 #include <linux/mlx5/fs.h>
 #include <linux/mlx5/fs_helpers.h>

@@ -690,7 +691,7 @@ static struct mlx5_ib_flow_prio *_get_prio(struct mlx5_ib_dev *dev,
 					   struct mlx5_ib_flow_prio *prio,
 					   int priority,
 					   int num_entries, int num_groups,
-					   u32 flags)
+					   u32 flags, u16 vport)
 {
 	struct mlx5_flow_table_attr ft_attr = {};
 	struct mlx5_flow_table *ft;

@@ -698,6 +699,7 @@ static struct mlx5_ib_flow_prio *_get_prio(struct mlx5_ib_dev *dev,
 	ft_attr.prio = priority;
 	ft_attr.max_fte = num_entries;
 	ft_attr.flags = flags;
+	ft_attr.vport = vport;
 	ft_attr.autogroup.max_num_groups = num_groups;
 	ft = mlx5_create_auto_grouped_flow_table(ns, &ft_attr);
 	if (IS_ERR(ft))

@@ -792,7 +794,7 @@ static struct mlx5_ib_flow_prio *get_flow_table(struct mlx5_ib_dev *dev,
 	ft = prio->flow_table;
 	if (!ft)
 		return _get_prio(dev, ns, prio, priority, max_table_size,
-				 num_groups, flags);
+				 num_groups, flags, 0);
 
 	return prio;
 }

@@ -935,7 +937,7 @@ int mlx5_ib_fs_add_op_fc(struct mlx5_ib_dev *dev, u32 port_num,
 	prio = &dev->flow_db->opfcs[type];
 	if (!prio->flow_table) {
 		prio = _get_prio(dev, ns, prio, priority,
-				 dev->num_ports * MAX_OPFC_RULES, 1, 0);
+				 dev->num_ports * MAX_OPFC_RULES, 1, 0, 0);
 		if (IS_ERR(prio)) {
 			err = PTR_ERR(prio);
 			goto free;

@@ -1413,17 +1415,51 @@ static struct ib_flow *mlx5_ib_create_flow(struct ib_qp *qp,
 	return ERR_PTR(err);
 }
 
+static int mlx5_ib_fill_transport_ns_info(struct mlx5_ib_dev *dev,
+					  enum mlx5_flow_namespace_type type,
+					  u32 *flags, u16 *vport_idx,
+					  u16 *vport,
+					  struct mlx5_core_dev **ft_mdev,
+					  u32 ib_port)
+{
+	struct mlx5_core_dev *esw_mdev;
+
+	if (!is_mdev_switchdev_mode(dev->mdev))
+		return 0;
+
+	if (!MLX5_CAP_ADV_RDMA(dev->mdev, rdma_transport_manager))
+		return -EOPNOTSUPP;
+
+	if (!dev->port[ib_port - 1].rep)
+		return -EINVAL;
+
+	esw_mdev = mlx5_eswitch_get_core_dev(dev->port[ib_port - 1].rep->esw);
+	if (esw_mdev != dev->mdev)
+		return -EOPNOTSUPP;
+
+	*flags |= MLX5_FLOW_TABLE_OTHER_VPORT;
+	*ft_mdev = esw_mdev;
+	*vport = dev->port[ib_port - 1].rep->vport;
+	*vport_idx = dev->port[ib_port - 1].rep->vport_index;
+
+	return 0;
+}
+
 static struct mlx5_ib_flow_prio *
 _get_flow_table(struct mlx5_ib_dev *dev, u16 user_priority,
 		enum mlx5_flow_namespace_type ns_type,
-		bool mcast)
+		bool mcast, u32 ib_port)
 {
+	struct mlx5_core_dev *ft_mdev = dev->mdev;
 	struct mlx5_flow_namespace *ns = NULL;
 	struct mlx5_ib_flow_prio *prio = NULL;
 	int max_table_size = 0;
+	u16 vport_idx = 0;
 	bool esw_encap;
 	u32 flags = 0;
+	u16 vport = 0;
 	int priority;
+	int ret;
 
 	if (mcast)
 		priority = MLX5_IB_FLOW_MCAST_PRIO;

@@ -1471,13 +1507,38 @@ _get_flow_table(struct mlx5_ib_dev *dev, u16 user_priority,
 			MLX5_CAP_FLOWTABLE_RDMA_TX(dev->mdev, log_max_ft_size));
 		priority = user_priority;
 		break;
+	case MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_RX:
+	case MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_TX:
+		if (ib_port == 0 || user_priority > MLX5_RDMA_TRANSPORT_BYPASS_PRIO)
+			return ERR_PTR(-EINVAL);
+		ret = mlx5_ib_fill_transport_ns_info(dev, ns_type, &flags,
+						     &vport_idx, &vport,
+						     &ft_mdev, ib_port);
+		if (ret)
+			return ERR_PTR(ret);
+
+		if (ns_type == MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_RX)
+			max_table_size =
+				BIT(MLX5_CAP_FLOWTABLE_RDMA_TRANSPORT_RX(
+					ft_mdev, log_max_ft_size));
+		else
+			max_table_size =
+				BIT(MLX5_CAP_FLOWTABLE_RDMA_TRANSPORT_TX(
+					ft_mdev, log_max_ft_size));
+		priority = user_priority;
+		break;
 	default:
 		break;
 	}
 
 	max_table_size = min_t(int, max_table_size, MLX5_FS_MAX_ENTRIES);
 
-	ns = mlx5_get_flow_namespace(dev->mdev, ns_type);
+	if (ns_type == MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_RX ||
+	    ns_type == MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_TX)
+		ns = mlx5_get_flow_vport_namespace(ft_mdev, ns_type, vport_idx);
+	else
+		ns = mlx5_get_flow_namespace(ft_mdev, ns_type);
+
 	if (!ns)
 		return ERR_PTR(-EOPNOTSUPP);
 

@@ -1497,6 +1558,12 @@ _get_flow_table(struct mlx5_ib_dev *dev, u16 user_priority,
 	case MLX5_FLOW_NAMESPACE_RDMA_TX:
 		prio = &dev->flow_db->rdma_tx[priority];
 		break;
+	case MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_RX:
+		prio = &dev->flow_db->rdma_transport_rx[ib_port - 1];
+		break;
+	case MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_TX:
+		prio = &dev->flow_db->rdma_transport_tx[ib_port - 1];
+		break;
 	default: return ERR_PTR(-EINVAL);
 	}
 

@@ -1507,7 +1574,7 @@ _get_flow_table(struct mlx5_ib_dev *dev, u16 user_priority,
 		return prio;
 
 	return _get_prio(dev, ns, prio, priority, max_table_size,
-			 MLX5_FS_MAX_TYPES, flags);
+			 MLX5_FS_MAX_TYPES, flags, vport);
 }
 
 static struct mlx5_ib_flow_handler *

@@ -1626,7 +1693,8 @@ static struct mlx5_ib_flow_handler *raw_fs_rule_add(
 	mutex_lock(&dev->flow_db->lock);
 
 	ft_prio = _get_flow_table(dev, fs_matcher->priority,
-				  fs_matcher->ns_type, mcast);
+				  fs_matcher->ns_type, mcast,
+				  fs_matcher->ib_port);
 	if (IS_ERR(ft_prio)) {
 		err = PTR_ERR(ft_prio);
 		goto unlock;

@@ -1742,6 +1810,12 @@ mlx5_ib_ft_type_to_namespace(enum mlx5_ib_uapi_flow_table_type table_type,
 	case MLX5_IB_UAPI_FLOW_TABLE_TYPE_RDMA_TX:
 		*namespace = MLX5_FLOW_NAMESPACE_RDMA_TX;
 		break;
+	case MLX5_IB_UAPI_FLOW_TABLE_TYPE_RDMA_TRANSPORT_RX:
+		*namespace = MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_RX;
+		break;
+	case MLX5_IB_UAPI_FLOW_TABLE_TYPE_RDMA_TRANSPORT_TX:
+		*namespace = MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_TX;
+		break;
 	default:
 		return -EINVAL;
 	}

@@ -1831,7 +1905,8 @@ static int get_dests(struct uverbs_attr_bundle *attrs,
 		return -EINVAL;
 
 	/* Allow only DEVX object or QP as dest when inserting to RDMA_RX */
-	if ((fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_RX) &&
+	if ((fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_RX ||
+	     fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_RX) &&
 	    ((!dest_devx && !dest_qp) || (dest_devx && dest_qp)))
 		return -EINVAL;
 

@@ -1848,7 +1923,8 @@ static int get_dests(struct uverbs_attr_bundle *attrs,
 			return -EINVAL;
 		/* Allow only flow table as dest when inserting to FDB or RDMA_RX */
 		if ((fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_FDB_BYPASS ||
-		     fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_RX) &&
+		     fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_RX ||
+		     fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_RX) &&
 		    *dest_type != MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE)
 			return -EINVAL;
 	} else if (dest_qp) {

@@ -1869,14 +1945,16 @@ static int get_dests(struct uverbs_attr_bundle *attrs,
 			*dest_id = mqp->raw_packet_qp.rq.tirn;
 		*dest_type = MLX5_FLOW_DESTINATION_TYPE_TIR;
 	} else if ((fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_EGRESS ||
-		    fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_TX) &&
+		    fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_TX ||
+		    fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_TX) &&
 		   !(*flags & MLX5_IB_ATTR_CREATE_FLOW_FLAGS_DROP)) {
 		*dest_type = MLX5_FLOW_DESTINATION_TYPE_PORT;
 	}
 
 	if (*dest_type == MLX5_FLOW_DESTINATION_TYPE_TIR &&
 	    (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_EGRESS ||
-	     fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_TX))
+	     fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_TX ||
+	     fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_TX))
 		return -EINVAL;
 
 	return 0;

@@ -2353,6 +2431,15 @@ static int mlx5_ib_matcher_ns(struct uverbs_attr_bundle *attrs,
 	return 0;
 }
 
+static bool verify_context_caps(struct mlx5_ib_dev *dev, u64 enabled_caps)
+{
+	if (is_mdev_switchdev_mode(dev->mdev))
+		return UCAP_ENABLED(enabled_caps,
+				    RDMA_UCAP_MLX5_CTRL_OTHER_VHCA);
+
+	return UCAP_ENABLED(enabled_caps, RDMA_UCAP_MLX5_CTRL_LOCAL);
+}
+
 static int UVERBS_HANDLER(MLX5_IB_METHOD_FLOW_MATCHER_CREATE)(
 	struct uverbs_attr_bundle *attrs)
 {

@@ -2401,6 +2488,26 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_FLOW_MATCHER_CREATE)(
 		goto end;
 	}
 
+	if (uverbs_attr_is_valid(attrs, MLX5_IB_ATTR_FLOW_MATCHER_IB_PORT)) {
+		err = uverbs_copy_from(&obj->ib_port, attrs,
+				       MLX5_IB_ATTR_FLOW_MATCHER_IB_PORT);
+		if (err)
+			goto end;
+		if (!rdma_is_port_valid(&dev->ib_dev, obj->ib_port)) {
+			err = -EINVAL;
+			goto end;
+		}
+		if (obj->ns_type != MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_RX &&
+		    obj->ns_type != MLX5_FLOW_NAMESPACE_RDMA_TRANSPORT_TX) {
+			err = -EINVAL;
+			goto end;
+		}
+		if (!verify_context_caps(dev, uobj->context->enabled_caps)) {
+			err = -EOPNOTSUPP;
+			goto end;
+		}
+	}
+
 	uobj->object = obj;
 	obj->mdev = dev->mdev;
 	atomic_set(&obj->usecnt, 0);

@@ -2448,7 +2555,7 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_STEERING_ANCHOR_CREATE)(
 
 	mutex_lock(&dev->flow_db->lock);
 
-	ft_prio = _get_flow_table(dev, priority, ns_type, 0);
+	ft_prio = _get_flow_table(dev, priority, ns_type, 0, 0);
 	if (IS_ERR(ft_prio)) {
 		err = PTR_ERR(ft_prio);
 		goto free_obj;

@@ -2834,7 +2941,10 @@ DECLARE_UVERBS_NAMED_METHOD(
 			     UA_OPTIONAL),
 	UVERBS_ATTR_CONST_IN(MLX5_IB_ATTR_FLOW_MATCHER_FT_TYPE,
 			     enum mlx5_ib_uapi_flow_table_type,
-			     UA_OPTIONAL));
+			     UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(MLX5_IB_ATTR_FLOW_MATCHER_IB_PORT,
+			   UVERBS_ATTR_TYPE(u32),
+			   UA_OPTIONAL));
 
 DECLARE_UVERBS_NAMED_METHOD_DESTROY(
 	MLX5_IB_METHOD_FLOW_MATCHER_DESTROY,

@@ -2904,8 +3014,26 @@ int mlx5_ib_fs_init(struct mlx5_ib_dev *dev)
 	if (!dev->flow_db)
 		return -ENOMEM;
 
+	dev->flow_db->rdma_transport_rx = kcalloc(dev->num_ports,
+					sizeof(struct mlx5_ib_flow_prio),
+					GFP_KERNEL);
+	if (!dev->flow_db->rdma_transport_rx)
+		goto free_flow_db;
+
+	dev->flow_db->rdma_transport_tx = kcalloc(dev->num_ports,
+					sizeof(struct mlx5_ib_flow_prio),
+					GFP_KERNEL);
+	if (!dev->flow_db->rdma_transport_tx)
+		goto free_rdma_transport_rx;
+
 	mutex_init(&dev->flow_db->lock);
 
 	ib_set_device_ops(&dev->ib_dev, &flow_ops);
 	return 0;
+
+free_rdma_transport_rx:
+	kfree(dev->flow_db->rdma_transport_rx);
+free_flow_db:
+	kfree(dev->flow_db);
+	return -ENOMEM;
 }
drivers/infiniband/hw/mlx5/fs.h +2 -0
index b9734904f5f0..0516555eb1c1 100644
@@ -40,6 +40,8 @@ static inline void mlx5_ib_fs_cleanup(struct mlx5_ib_dev *dev)
 	 * is a safe assumption that all references are gone.
 	 */
 	mlx5_ib_fs_cleanup_anchor(dev);
+	kfree(dev->flow_db->rdma_transport_tx);
+	kfree(dev->flow_db->rdma_transport_rx);
 	kfree(dev->flow_db);
 }
 #endif /* _MLX5_IB_FS_H */
drivers/infiniband/hw/mlx5/mlx5_ib.h +3 -0
index 974a45c92fbb..ccaaef20f50d 100644
@@ -276,6 +276,7 @@ struct mlx5_ib_flow_matcher {
 	struct mlx5_core_dev	*mdev;
 	atomic_t		usecnt;
 	u8			match_criteria_enable;
+	u32			ib_port;
 };
 
 struct mlx5_ib_steering_anchor {

@@ -307,6 +308,8 @@ struct mlx5_ib_flow_db {
 	struct mlx5_ib_flow_prio	rdma_tx[MLX5_IB_NUM_FLOW_FT];
 	struct mlx5_ib_flow_prio	opfcs[MLX5_IB_OPCOUNTER_MAX];
 	struct mlx5_flow_table		*lag_demux_ft;
+	struct mlx5_ib_flow_prio        *rdma_transport_rx;
+	struct mlx5_ib_flow_prio        *rdma_transport_tx;
 	/* Protect flow steering bypass flow tables
 	 * when add/del flow rules.
 	 * only single add/removal of flow steering rule could be done
include/uapi/rdma/mlx5_user_ioctl_cmds.h +1 -0
index fd2e4a3a56b3..18f9fe070213 100644
@@ -239,6 +239,7 @@ enum mlx5_ib_flow_matcher_create_attrs {
 	MLX5_IB_ATTR_FLOW_MATCHER_MATCH_CRITERIA,
 	MLX5_IB_ATTR_FLOW_MATCHER_FLOW_FLAGS,
 	MLX5_IB_ATTR_FLOW_MATCHER_FT_TYPE,
+	MLX5_IB_ATTR_FLOW_MATCHER_IB_PORT,
 };
 
 enum mlx5_ib_flow_matcher_destroy_attrs {
include/uapi/rdma/mlx5_user_ioctl_verbs.h +2 -0
index 7c233df475e7..8f86e79d78a5 100644
@@ -45,6 +45,8 @@ enum mlx5_ib_uapi_flow_table_type {
 	MLX5_IB_UAPI_FLOW_TABLE_TYPE_FDB	= 0x2,
 	MLX5_IB_UAPI_FLOW_TABLE_TYPE_RDMA_RX	= 0x3,
 	MLX5_IB_UAPI_FLOW_TABLE_TYPE_RDMA_TX	= 0x4,
+	MLX5_IB_UAPI_FLOW_TABLE_TYPE_RDMA_TRANSPORT_RX	= 0x5,
+	MLX5_IB_UAPI_FLOW_TABLE_TYPE_RDMA_TRANSPORT_TX	= 0x6,
 };
 
 enum mlx5_ib_uapi_flow_action_packet_reformat_type {
Signature · 1 line
2.48.1
author
#14 lore raw

PATCH 6/6 docs: infiniband: document the UCAP API

In reply to Leon Romanovsky

From: Chiara Meiohas <[email protected]>

Add an explanation on the newly added UCAP API.

Signed-off-by Chiara Meiohas <[email protected]>
Reviewed-by Yishai Hadas <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 Documentation/infiniband/index.rst |  1 +
 Documentation/infiniband/ucaps.rst | 71 ++++++++++++++++++++++++++++++
 2 files changed, 72 insertions(+)
 create mode 100644 Documentation/infiniband/ucaps.rst
Documentation/infiniband/index.rst +1 -0
index 9cd7615438b9..5b4c24125f66 100644
@@ -12,6 +12,7 @@ InfiniBand
    opa_vnic
    sysfs
    tag_matching
+   ucaps
    user_mad
    user_verbs
Documentation/infiniband/ucaps.rst new file +71 -0

71 lines not on this page · raw message

Signature · 1 line
2.48.1