Skip to contentSkip to conversation

vfio-mdev: support mediated device creation in kernel

PATCH In review Yonghyun Hwang 3 participants 9 messages Mar 20, 2020 – Mar 26, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vfio-mdev: support mediated device creation in kernel

To enable a mediated device, a device driver registers its device to VFIO MDev framework. Once the mediated device gets enabled, UUID gets fed onto the sysfs attribute, "create", to create the mediated device. This additional step happens after boot-up gets complete. If the driver knows how many mediated devices need to be created during probing time, the additional step becomes cumbersome. This commit implements a new function to allow the driver to create a mediated device in kernel.

---
 drivers/vfio/mdev/mdev_core.c | 45 +++++++++++++++++++++++++++++++++++
 include/linux/mdev.h          |  3 +++
 2 files changed, 48 insertions(+)
drivers/vfio/mdev/mdev_core.c +45 -0
index b558d4cfd082..a6d32516de42 100644
@@ -350,6 +350,51 @@ int mdev_device_create(struct kobject *kobj,
 	return ret;
 }
 
+/*
+ * mdev_create_device : Create a mdev device
+ * @dev: device structure representing parent device.
+ * @uuid: uuid char string for a mdev device.
+ * @group: index to supported type groups for a mdev device.
+ *
+ * Create a mdev device in kernel.
+ * Returns a negative value on error, otherwise 0.
+ */
+int mdev_create_device(struct device *dev,
+			const char *uuid, int group)
+{
+	struct mdev_parent *parent = NULL;
+	struct mdev_type *type = NULL;
+	guid_t guid;
+	int i = 1;
+	int ret;
+
+	ret = guid_parse(uuid, &guid);
+	if (ret) {
+		dev_err(dev, "Failed to parse UUID");
+		return ret;
+	}
+
+	parent = __find_parent_device(dev);
+	if (!parent) {
+		dev_err(dev, "Failed to find parent mdev device");
+		return -ENODEV;
+	}
+
+	list_for_each_entry(type, &parent->type_list, next) {
+		if (i == group)
+			break;
+		i++;
+	}
+
+	if (!type || i != group) {
+		dev_err(dev, "Failed to find mdev device");
+		return -ENODEV;
+	}
+
+	return mdev_device_create(&type->kobj, parent->dev, &guid);
+}
+EXPORT_SYMBOL(mdev_create_device);
+
 int mdev_device_remove(struct device *dev)
 {
 	struct mdev_device *mdev, *tmp;
include/linux/mdev.h +3 -0
index 0ce30ca78db0..b66f67998916 100644
@@ -145,4 +145,7 @@ struct device *mdev_parent_dev(struct mdev_device *mdev);
 struct device *mdev_dev(struct mdev_device *mdev);
 struct mdev_device *mdev_from_dev(struct device *dev);
 
+extern int mdev_create_device(struct device *dev,
+			const char *uuid, int group_idx);
+
 #endif /* MDEV_H */
Signature · 1 line
2.25.1.696.g5e7596f4ac-goog
#1 lore raw
Quoted 7 lines · On Fri, 20 Mar 2020 10:59:10 -0700 Yonghyun Hwang <[email protected]> wrote:

To enable a mediated device, a device driver registers its device to VFIO MDev framework. Once the mediated device gets enabled, UUID gets fed onto the sysfs attribute, "create", to create the mediated device. This additional step happens after boot-up gets complete. If the driver knows how many mediated devices need to be created during probing time, the additional step becomes cumbersome. This commit implements a new function to allow the driver to create a mediated device in kernel.

But pre-creating mdev devices seems like a policy decision. Why can't userspace make such a policy decision, and do so with persistent uuids, via something like mdevctl? Thanks,

Alex

Quoted 74 lines
---
 drivers/vfio/mdev/mdev_core.c | 45 +++++++++++++++++++++++++++++++++++
 include/linux/mdev.h          |  3 +++
 2 files changed, 48 insertions(+)
drivers/vfio/mdev/mdev_core.c +45 -0
index b558d4cfd082..a6d32516de42 100644
@@ -350,6 +350,51 @@ int mdev_device_create(struct kobject *kobj,
 	return ret;
 }
 
+/*
+ * mdev_create_device : Create a mdev device
+ * @dev: device structure representing parent device.
+ * @uuid: uuid char string for a mdev device.
+ * @group: index to supported type groups for a mdev device.
+ *
+ * Create a mdev device in kernel.
+ * Returns a negative value on error, otherwise 0.
+ */
+int mdev_create_device(struct device *dev,
+			const char *uuid, int group)
+{
+	struct mdev_parent *parent = NULL;
+	struct mdev_type *type = NULL;
+	guid_t guid;
+	int i = 1;
+	int ret;
+
+	ret = guid_parse(uuid, &guid);
+	if (ret) {
+		dev_err(dev, "Failed to parse UUID");
+		return ret;
+	}
+
+	parent = __find_parent_device(dev);
+	if (!parent) {
+		dev_err(dev, "Failed to find parent mdev device");
+		return -ENODEV;
+	}
+
+	list_for_each_entry(type, &parent->type_list, next) {
+		if (i == group)
+			break;
+		i++;
+	}
+
+	if (!type || i != group) {
+		dev_err(dev, "Failed to find mdev device");
+		return -ENODEV;
+	}
+
+	return mdev_device_create(&type->kobj, parent->dev, &guid);
+}
+EXPORT_SYMBOL(mdev_create_device);
+
 int mdev_device_remove(struct device *dev)
 {
 	struct mdev_device *mdev, *tmp;
include/linux/mdev.h +3 -0
index 0ce30ca78db0..b66f67998916 100644
@@ -145,4 +145,7 @@ struct device *mdev_parent_dev(struct mdev_device *mdev);
 struct device *mdev_dev(struct mdev_device *mdev);
 struct mdev_device *mdev_from_dev(struct device *dev);
 
+extern int mdev_create_device(struct device *dev,
+			const char *uuid, int group_idx);
+
 #endif /* MDEV_H */
#2 lore raw
Quoted 12 lines · Alex Williamson
Quoted 7 lines · On Fri, 20 Mar 2020 10:59:10 -0700 Yonghyun Hwang <[email protected]> wrote:

To enable a mediated device, a device driver registers its device to VFIO MDev framework. Once the mediated device gets enabled, UUID gets fed onto the sysfs attribute, "create", to create the mediated device. This additional step happens after boot-up gets complete. If the driver knows how many mediated devices need to be created during probing time, the additional step becomes cumbersome. This commit implements a new function to allow the driver to create a mediated device in kernel.

Quoted 5 lines

But pre-creating mdev devices seems like a policy decision. Why can't userspace make such a policy decision, and do so with persistent uuids, via something like mdevctl? Thanks,

Alex

Yep, it can be viewed as the policy decision and userspace can make the decision. However, it would be handy and plausible, if a device driver can pre-create "fixed or default" # of mdev devices, while allowing the device policy to come into play after bootup gets complete. Without this patch, a device driver should release the policy and the policy should be aligned with the driver, which would be cumbersome (sometimes painful) in a cloud environment. My use case with mdev is to enable a subset of vfio-pci features without losing my device driver.

Thank you,
Yonghyun

Quoted 77 lines
Quoted 76 lines
---
 drivers/vfio/mdev/mdev_core.c | 45 +++++++++++++++++++++++++++++++++++
 include/linux/mdev.h          |  3 +++
 2 files changed, 48 insertions(+)
drivers/vfio/mdev/mdev_core.c +45 -0
index b558d4cfd082..a6d32516de42 100644
@@ -350,6 +350,51 @@ int mdev_device_create(struct kobject *kobj,
      return ret;
 }
 
+/*
+ * mdev_create_device : Create a mdev device
+ * @dev: device structure representing parent device.
+ * @uuid: uuid char string for a mdev device.
+ * @group: index to supported type groups for a mdev device.
+ *
+ * Create a mdev device in kernel.
+ * Returns a negative value on error, otherwise 0.
+ */
+int mdev_create_device(struct device *dev,
+                     const char *uuid, int group)
+{
+     struct mdev_parent *parent = NULL;
+     struct mdev_type *type = NULL;
+     guid_t guid;
+     int i = 1;
+     int ret;
+
+     ret = guid_parse(uuid, &guid);
+     if (ret) {
+             dev_err(dev, "Failed to parse UUID");
+             return ret;
+     }
+
+     parent = __find_parent_device(dev);
+     if (!parent) {
+             dev_err(dev, "Failed to find parent mdev device");
+             return -ENODEV;
+     }
+
+     list_for_each_entry(type, &parent->type_list, next) {
+             if (i == group)
+                     break;
+             i++;
+     }
+
+     if (!type || i != group) {
+             dev_err(dev, "Failed to find mdev device");
+             return -ENODEV;
+     }
+
+     return mdev_device_create(&type->kobj, parent->dev, &guid);
+}
+EXPORT_SYMBOL(mdev_create_device);
+
 int mdev_device_remove(struct device *dev)
 {
      struct mdev_device *mdev, *tmp;
include/linux/mdev.h +3 -0
index 0ce30ca78db0..b66f67998916 100644
@@ -145,4 +145,7 @@ struct device *mdev_parent_dev(struct mdev_device *mdev);
 struct device *mdev_dev(struct mdev_device *mdev);
 struct mdev_device *mdev_from_dev(struct device *dev);
 
+extern int mdev_create_device(struct device *dev,
+                     const char *uuid, int group_idx);
+
 #endif /* MDEV_H */
#3 lore raw
Quoted 20 lines · On Fri, 20 Mar 2020 13:46:04 -0700 Yonghyun Hwang <[email protected]> wrote:
Quoted 12 lines · Alex Williamson
Quoted 7 lines · On Fri, 20 Mar 2020 10:59:10 -0700 Yonghyun Hwang <[email protected]> wrote:

To enable a mediated device, a device driver registers its device to VFIO MDev framework. Once the mediated device gets enabled, UUID gets fed onto the sysfs attribute, "create", to create the mediated device. This additional step happens after boot-up gets complete. If the driver knows how many mediated devices need to be created during probing time, the additional step becomes cumbersome. This commit implements a new function to allow the driver to create a mediated device in kernel.

Quoted 5 lines

But pre-creating mdev devices seems like a policy decision. Why can't userspace make such a policy decision, and do so with persistent uuids, via something like mdevctl? Thanks,

Alex

Quoted 9 lines

Yep, it can be viewed as the policy decision and userspace can make the decision. However, it would be handy and plausible, if a device driver can pre-create "fixed or default" # of mdev devices, while allowing the device policy to come into play after bootup gets complete. Without this patch, a device driver should release the policy and the policy should be aligned with the driver, which would be cumbersome (sometimes painful) in a cloud environment. My use case with mdev is to enable a subset of vfio-pci features without losing my device driver.

Does this last comment suggest the parent device is not being multiplexed through mdev, but only mediated? If so, would something like Yan's vendor-ops approach[1] be better? Without a multiplexed device, the lifecycle management of an mdev device doesn't make a lot of sense, and I wonder if that's what you're trying to bypass here. Even SR-IOV devices have moved to userspace enablement with most module options to enable a default number of VFs being deprecated. I do see that that transition left a gap, but I'm not sure that heading in the opposite direction with mdevs is a good idea either. Thanks,

Alex

[1]https://lore.kernel.org/kvm/[email protected]/

Quoted 76 lines
Quoted 75 lines
Quoted 74 lines
---
 drivers/vfio/mdev/mdev_core.c | 45 +++++++++++++++++++++++++++++++++++
 include/linux/mdev.h          |  3 +++
 2 files changed, 48 insertions(+)
drivers/vfio/mdev/mdev_core.c +45 -0
index b558d4cfd082..a6d32516de42 100644
@@ -350,6 +350,51 @@ int mdev_device_create(struct kobject *kobj,
      return ret;
 }
 
+/*
+ * mdev_create_device : Create a mdev device
+ * @dev: device structure representing parent device.
+ * @uuid: uuid char string for a mdev device.
+ * @group: index to supported type groups for a mdev device.
+ *
+ * Create a mdev device in kernel.
+ * Returns a negative value on error, otherwise 0.
+ */
+int mdev_create_device(struct device *dev,
+                     const char *uuid, int group)
+{
+     struct mdev_parent *parent = NULL;
+     struct mdev_type *type = NULL;
+     guid_t guid;
+     int i = 1;
+     int ret;
+
+     ret = guid_parse(uuid, &guid);
+     if (ret) {
+             dev_err(dev, "Failed to parse UUID");
+             return ret;
+     }
+
+     parent = __find_parent_device(dev);
+     if (!parent) {
+             dev_err(dev, "Failed to find parent mdev device");
+             return -ENODEV;
+     }
+
+     list_for_each_entry(type, &parent->type_list, next) {
+             if (i == group)
+                     break;
+             i++;
+     }
+
+     if (!type || i != group) {
+             dev_err(dev, "Failed to find mdev device");
+             return -ENODEV;
+     }
+
+     return mdev_device_create(&type->kobj, parent->dev, &guid);
+}
+EXPORT_SYMBOL(mdev_create_device);
+
 int mdev_device_remove(struct device *dev)
 {
      struct mdev_device *mdev, *tmp;
include/linux/mdev.h +3 -0
index 0ce30ca78db0..b66f67998916 100644
@@ -145,4 +145,7 @@ struct device *mdev_parent_dev(struct mdev_device *mdev);
 struct device *mdev_dev(struct mdev_device *mdev);
 struct mdev_device *mdev_from_dev(struct device *dev);
 
+extern int mdev_create_device(struct device *dev,
+                     const char *uuid, int group_idx);
+
 #endif /* MDEV_H */  
#4 lore raw

If my patch is not aligned with the direction, it results in a maintenance burden for the kernel. Thanks for sharing the direction and Yan's vendor-ops approach[1].

Thank you,
Yonghyun

Quoted 126 lines · Alex Williamson
Quoted 20 lines · On Fri, 20 Mar 2020 13:46:04 -0700 Yonghyun Hwang <[email protected]> wrote:
Quoted 12 lines · Alex Williamson
Quoted 7 lines · On Fri, 20 Mar 2020 10:59:10 -0700 Yonghyun Hwang <[email protected]> wrote:

To enable a mediated device, a device driver registers its device to VFIO MDev framework. Once the mediated device gets enabled, UUID gets fed onto the sysfs attribute, "create", to create the mediated device. This additional step happens after boot-up gets complete. If the driver knows how many mediated devices need to be created during probing time, the additional step becomes cumbersome. This commit implements a new function to allow the driver to create a mediated device in kernel.

Quoted 5 lines

But pre-creating mdev devices seems like a policy decision. Why can't userspace make such a policy decision, and do so with persistent uuids, via something like mdevctl? Thanks,

Alex

Quoted 9 lines

Yep, it can be viewed as the policy decision and userspace can make the decision. However, it would be handy and plausible, if a device driver can pre-create "fixed or default" # of mdev devices, while allowing the device policy to come into play after bootup gets complete. Without this patch, a device driver should release the policy and the policy should be aligned with the driver, which would be cumbersome (sometimes painful) in a cloud environment. My use case with mdev is to enable a subset of vfio-pci features without losing my device driver.

Does this last comment suggest the parent device is not being multiplexed through mdev, but only mediated? If so, would something like Yan's vendor-ops approach[1] be better? Without a multiplexed device, the lifecycle management of an mdev device doesn't make a lot of sense, and I wonder if that's what you're trying to bypass here. Even SR-IOV devices have moved to userspace enablement with most module options to enable a default number of VFs being deprecated. I do see that that transition left a gap, but I'm not sure that heading in the opposite direction with mdevs is a good idea either. Thanks,

Alex

[1]https://lore.kernel.org/kvm/[email protected]/

Quoted 76 lines
Quoted 75 lines
Quoted 74 lines
---
 drivers/vfio/mdev/mdev_core.c | 45 +++++++++++++++++++++++++++++++++++
 include/linux/mdev.h          |  3 +++
 2 files changed, 48 insertions(+)
drivers/vfio/mdev/mdev_core.c +45 -0
index b558d4cfd082..a6d32516de42 100644
@@ -350,6 +350,51 @@ int mdev_device_create(struct kobject *kobj,
      return ret;
 }
 
+/*
+ * mdev_create_device : Create a mdev device
+ * @dev: device structure representing parent device.
+ * @uuid: uuid char string for a mdev device.
+ * @group: index to supported type groups for a mdev device.
+ *
+ * Create a mdev device in kernel.
+ * Returns a negative value on error, otherwise 0.
+ */
+int mdev_create_device(struct device *dev,
+                     const char *uuid, int group)
+{
+     struct mdev_parent *parent = NULL;
+     struct mdev_type *type = NULL;
+     guid_t guid;
+     int i = 1;
+     int ret;
+
+     ret = guid_parse(uuid, &guid);
+     if (ret) {
+             dev_err(dev, "Failed to parse UUID");
+             return ret;
+     }
+
+     parent = __find_parent_device(dev);
+     if (!parent) {
+             dev_err(dev, "Failed to find parent mdev device");
+             return -ENODEV;
+     }
+
+     list_for_each_entry(type, &parent->type_list, next) {
+             if (i == group)
+                     break;
+             i++;
+     }
+
+     if (!type || i != group) {
+             dev_err(dev, "Failed to find mdev device");
+             return -ENODEV;
+     }
+
+     return mdev_device_create(&type->kobj, parent->dev, &guid);
+}
+EXPORT_SYMBOL(mdev_create_device);
+
 int mdev_device_remove(struct device *dev)
 {
      struct mdev_device *mdev, *tmp;
include/linux/mdev.h +3 -0
index 0ce30ca78db0..b66f67998916 100644
@@ -145,4 +145,7 @@ struct device *mdev_parent_dev(struct mdev_device *mdev);
 struct device *mdev_dev(struct mdev_device *mdev);
 struct mdev_device *mdev_from_dev(struct device *dev);
 
+extern int mdev_create_device(struct device *dev,
+                     const char *uuid, int group_idx);
+
 #endif /* MDEV_H */
#5 lore raw
In reply to Yonghyun Hwang
Quoted 7 lines · Yonghyun Hwang

To enable a mediated device, a device driver registers its device to VFIO MDev framework. Once the mediated device gets enabled, UUID gets fed onto the sysfs attribute, "create", to create the mediated device. This additional step happens after boot-up gets complete. If the driver knows how many mediated devices need to be created during probing time, the additional step becomes cumbersome. This commit implements a new function to allow the driver to create a mediated device in kernel.

Please send this along with your proposed user so that we can understand the use. Without that new exports have no chance of going in anyway.

#6 lore raw
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Yonghyun Hwang

To enable a mediated device, a device driver registers its device to VFIO MDev framework. Once the mediated device gets enabled, UUID gets fed onto the sysfs attribute, "create", to create the mediated device. This additional step happens after boot-up gets complete. If the driver knows how many mediated devices need to be created during probing time, the additional step becomes cumbersome. This commit implements a new function to allow the driver to create a mediated device in kernel.

Quoted 2 lines

Please send this along with your proposed user so that we can understand the use. Without that new exports have no chance of going in anyway.

My driver is still under development. Do you recommend me to implement an example code for the new exports and re-submit the commit?

Thank you,
Yonghyun

#7 lore raw
Quoted 11 lines · Yonghyun Hwang
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Yonghyun Hwang

To enable a mediated device, a device driver registers its device to VFIO MDev framework. Once the mediated device gets enabled, UUID gets fed onto the sysfs attribute, "create", to create the mediated device. This additional step happens after boot-up gets complete. If the driver knows how many mediated devices need to be created during probing time, the additional step becomes cumbersome. This commit implements a new function to allow the driver to create a mediated device in kernel.

Quoted 5 lines
Quoted 2 lines

Please send this along with your proposed user so that we can understand the use. Without that new exports have no chance of going in anyway.

My driver is still under development. Do you recommend me to implement an example code for the new exports and re-submit the commit?

Hell no. The point is that we don't add new APIs unless we have actual users (not example code!). And as Alex mentioned the use case is rather questionable anyway, so without a user that actually shows a good use case which would remove those doubts it is a complete no-go.

#8 lore raw
Quoted 16 lines · Christoph Hellwig
Quoted 14 lines · Yonghyun Hwang
Quoted 12 lines · Christoph Hellwig
Quoted 7 lines · Yonghyun Hwang

To enable a mediated device, a device driver registers its device to VFIO MDev framework. Once the mediated device gets enabled, UUID gets fed onto the sysfs attribute, "create", to create the mediated device. This additional step happens after boot-up gets complete. If the driver knows how many mediated devices need to be created during probing time, the additional step becomes cumbersome. This commit implements a new function to allow the driver to create a mediated device in kernel.

Please send this along with your proposed user so that we can understand the use. Without that new exports have no chance of going in anyway.

Quoted 7 lines
Quoted 2 lines

My driver is still under development. Do you recommend me to implement an example code for the new exports and re-submit the commit?

Hell no. The point is that we don't add new APIs unless we have actual users (not example code!). And as Alex mentioned the use case is rather questionable anyway, so without a user that actually shows a good use case which would remove those doubts it is a complete no-go.

I see. Thank you for your clarification. (I got confused with the use case.) I hope you can understand email communication results in some confusion from time to time, which is especially true for kernel newbie like me.