Introduce vdpa management tool
Subsystems (5)
cover Introduce vdpa management tool
This patchset covers user requirements for managing existing vdpa devices, using a tool and its internal design notes for kernel drivers.
Background and user requirements:
----------------------------------
(1) Currently VDPA device is created by driver when driver is loaded. However, user should have a choice when to create or not create a vdpa device for the underlying parent device.
For example, mlx5 PCI VF and subfunction device supports multiple classes of device such netdev, vdpa, rdma. Howevever it is not required to always created vdpa device for such device.
(2) In another use case, a device may support creating one or multiple vdpa device of same or different class such as net and block. Creating vdpa devices at driver load time further limits this use case.
(3) A user should be able to monitor and query vdpa queue level or device level statistics for a given vdpa device.
(4) A user should be able to query what class of vdpa devices are supported by its parent device.
(5) A user should be able to view supported features and negotiated features of the vdpa device.
(6) A user should be able to create a vdpa device in vendor agnostic manner using single tool.
Hence, it is required to have a tool through which user can create one or more vdpa devices from a parent device which addresses above user requirements.
Example devices:
----------------
+-----------+ +-----------+ +---------+ +--------+ +-----------+
|vdpa dev 0 | |vdpa dev 1 | |rdma dev | |netdev | |vdpa dev 3 |
|type=net | |type=block | |mlx5_0 | |ens3f0 | |type=net |
+----+------+ +-----+-----+ +----+----+ +-----+--+ +----+------+
| | | | |
| | | | |
+----+-----+ | +----+----+ | +----+----+
| mlx5 +--------+ |mlx5 +-------+ |mlx5 |
|pci vf 2 | |pci vf 4 | |pci sf 8 |
|03:00:2 | |03:00.4 | |mlx5_sf.8|
+----+-----+ +----+----+ +----+----+
| | |
| +----+-----+ |
+----------------------+mlx5 +----------------+
|pci pf 0 |
|03:00.0 |
+----------+vdpa tool:
----------
vdpa tool is a tool to create, delete vdpa devices from a parent device. It is a tool that enables user to query statistics, features and may be more attributes in future.
vdpa tool command draft:
------------------------
(a) List parent devices which supports creating vdpa devices. It also shows which class types supported by this parent device. In below command example two parent devices support vdpa device creation. First is PCI VF whose bdf is 03.00:2.
Second is PCI VF whose name is 03:00.4.
Third is PCI SF whose name is mlx5_core.sf.8
$ vdpa parentdev list
vdpasim
supported_classes
net
pci/0000:03.00:3
supported_classes
net block
pci/0000:03.00:4
supported_classes
net block
auxiliary/mlx5_core.sf.8
supported_classes
net(b) Now add a vdpa device of networking class and show the device.
$ vdpa dev add parentdev pci/0000:03.00:2 type net name foo0 $ vdpa dev show foo0 foo0: parentdev pci/0000:03.00:2 type network parentdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256
(c) Show features of a vdpa device
$ vdpa dev features show foo0 supported iommu platform version 1
(d) Dump vdpa device statistics
$ vdpa dev stats show foo0 kickdoorbells 10 wqes 100
(e) Now delete a vdpa device previously created.
$ vdpa dev del foo0
vdpa tool support in this patchset:
-----------------------------------
vdpa tool is created to create, delete and query vdpa devices. examples:
Show vdpa parent device that supports creating, deleting vdpa devices.
$ vdpa parentdev show
vdpasim:
supported_classes
net$ vdpa parentdev show -jp
{
"show": {
"vdpasim": {
"supported_classes": {
"net"
}
}
}Create a vdpa device of type networking named as "foo2" from the parent device vdpasim:
$ vdpa dev add parentdev vdpasim type net name foo2
Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network parentdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256
$ vdpa dev show foo2 -jp
{
"dev": {
"foo2": {
"type": "network",
"parentdev": "vdpasim",
"vendor_id": 0,
"max_vqs": 2,
"max_vq_size": 256
}
}
}Delete the vdpa device after its use:
$ vdpa dev del foo2
vdpa tool support by kernel:
----------------------------
vdpa tool user interface will be supported by existing vdpa kernel framework, i.e. drivers/vdpa/vdpa.c It services user command through a netlink interface.
Each parent device registers supported callback operations with vdpa subsystem through which vdpa device(s) can be managed.
FAQs:
-----
1. Where does userspace vdpa tool reside which users can use?
Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to create vdpa net devices.
2. Why not create and delete vdpa device using sysfs/configfs? Ans:
(a) A device creation may involve passing one or more attributes. Passing multiple attributes and returning error code and more verbose information for invalid attributes cannot be handled by sysfs/configfs.
(b) netlink framework is rich that enables user space and kernel driver to provide nested attributes.
(c) Exposing device specific file under sysfs without net namespace awareness exposes details to multiple containers. Instead exposing attributes via a netlink socket secures the communication channel with kernel.
(d) netlink socket interface enables to run syscaller kernel tests.
3. Why not use ioctl() interface?
Ans: ioctl() interface replicates the necessary plumbing which already exists through netlink socket.
4. What happens when one or more user created vdpa devices exist for a parent PCI VF or SF and such parent device is removed?
Ans: All user created vdpa devices are removed that belong to a parent.
[1] git://git.kernel.org/pub/scm/network/iproute2/iproute2-next.git
Next steps:
-----------
(a) Post this patchset and iproute2/vdpa inclusion, remaining two drivers will be coverted to support vdpa tool instead of creating unmanaged default device on driver load.
(b) More net specific parameters such as mac, mtu will be added.
(c) Features bits get and set interface will be added.
Parav Pandit (7):
vdpa: Add missing comment for virtqueue count
vdpa: Use simpler version of ida allocation
vdpa: Extend routine to accept vdpa device name
vdpa: Define vdpa parent device, ops and a netlink interface
vdpa: Enable a user to add and delete a vdpa device
vdpa: Enable user to query vdpa device info vdpa/vdpa_sim: Enable user to create vdpasim net devices
drivers/vdpa/Kconfig | 1 + drivers/vdpa/ifcvf/ifcvf_main.c | 2 +- drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +- drivers/vdpa/vdpa.c | 511 +++++++++++++++++++++++++++++- drivers/vdpa/vdpa_sim/vdpa_sim.c | 81 ++++- include/linux/vdpa.h | 46 ++- include/uapi/linux/vdpa.h | 41 +++ 7 files changed, 660 insertions(+), 24 deletions(-) create mode 100644 include/uapi/linux/vdpa.h
Signature · 1 line
2.26.2
PATCH 1/7 vdpa: Add missing comment for virtqueue count
Add missing comment for number of virtqueue.
--- include/linux/vdpa.h | 1 + 1 file changed, 1 insertion(+)
include/linux/vdpa.h +1 -0
@@ -42,6 +42,7 @@ struct vdpa_vq_state {* @config: the configuration ops for this device. * @index: device index * @features_valid: were features initialized? for legacy guests +* @nvqs: maximum number of supported virtqueues */ struct vdpa_device {struct device dev;
Signature · 1 line
2.26.2
PATCH 2/7 vdpa: Use simpler version of ida allocation
In reply to Parav Panditvdpa doesn't have any specific need to define start and end range of the device index.
Hence use the simper version of the ida allocator.
--- drivers/vdpa/vdpa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vdpa/vdpa.c +1 -1
@@ -89,7 +89,7 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent,if (!vdev) goto err; - err =ida_simple_get (&vdpa_index_ida,0 ,0 ,GFP_KERNEL ); + err =ida_alloc (&vdpa_index_ida,GFP_KERNEL );if (err <0 ) goto err_ida;
Signature · 1 line
2.26.2
PATCH 3/7 vdpa: Extend routine to accept vdpa device name
In reply to Parav PanditIn a subsequent patch, when user initiated command creates a vdpa device, the user chooses the name of the vdpa device.
To support it, extend the device allocation API to consider this name specified by the caller driver.
Split the device unregistration to device delete and device put so that device can be removed from the list after its deleted.
--- drivers/vdpa/ifcvf/ifcvf_main.c | 2 +- drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +- drivers/vdpa/vdpa.c | 36 +++++++++++++++++++++++++++---- drivers/vdpa/vdpa_sim/vdpa_sim.c | 2 +- include/linux/vdpa.h | 7 +++--- 5 files changed, 38 insertions(+), 11 deletions(-)
drivers/vdpa/ifcvf/ifcvf_main.c +1 -1
@@ -439,7 +439,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct pci_device_id *id) adapter = vdpa_alloc_device(struct ifcvf_adapter, vdpa, dev, &ifc_vdpa_ops, -IFCVF_MAX_QUEUE_PAIRS *2 ); +IFCVF_MAX_QUEUE_PAIRS *2 ,NULL );if (adapter ==NULL ) {IFCVF_ERR (pdev,"Failed to allocate vDPA structure" );return -ENOMEM ;
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -1958,7 +1958,7 @@ static int mlx5v_probe(struct auxiliary_device *adev, max_vqs =min_t (u32, max_vqs,MLX5_MAX_SUPPORTED_VQS ); ndev = vdpa_alloc_device(struct mlx5_vdpa_net, mvdev.vdev, mdev->device, &mlx5_vdpa_ops, -2 *mlx5_vdpa_max_qps (max_vqs)); +2 *mlx5_vdpa_max_qps (max_vqs),NULL );if (IS_ERR (ndev))return PTR_ERR (ndev);
drivers/vdpa/vdpa.c +32 -4
@@ -12,6 +12,8 @@#include <linux/slab.h> #include <linux/vdpa.h> +/* A global mutex that protects vdpa parent and device level operations. */ +static DEFINE_MUTEX (vdpa_dev_mutex);static DEFINE_IDA (vdpa_index_ida);static int vdpa_dev_probe (struct device *d)
@@ -63,6 +65,7 @@ static void vdpa_release_dev(struct device *d)* @config: the bus operations that is supported by this device * @nvqs: number of virtqueues supported by this device * @size: size of the parent structure that contains private data +* @name: name of the vdpa device; optional. * * Driver should use vdpa_alloc_device() wrapper macro instead of * using this directly.
@@ -72,8 +75,7 @@ static void vdpa_release_dev(struct device *d)*/ struct vdpa_device *__vdpa_alloc_device (struct device *parent,const struct vdpa_config_ops *config, -int nvqs, -size_t size) +int nvqs,size_t size,const char *name) {struct vdpa_device *vdev;int err = -EINVAL ;
@@ -101,7 +103,10 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent, vdev->features_valid = false; vdev->nvqs = nvqs; - err =dev_set_name (&vdev->dev,"vdpa%u" , vdev->index); +if (name) + err =dev_set_name (&vdev->dev,"%s" , name); +else + err =dev_set_name (&vdev->dev,"vdpa%u" , vdev->index);if (err) goto err_name;
@@ -118,6 +123,13 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent, }EXPORT_SYMBOL_GPL (__vdpa_alloc_device); +static int vdpa_name_match (struct device *dev,const void *data) +{ + struct vdpa_device *vdev = container_of(dev, struct vdpa_device, dev); + +return (strcmp (dev_name (&vdev->dev), data) ==0 ); +} +/** * vdpa_register_device - register a vDPA device * Callers must have a succeed call of vdpa_alloc_device() before.
@@ -127,7 +139,21 @@ EXPORT_SYMBOL_GPL(__vdpa_alloc_device);*/ int vdpa_register_device (struct vdpa_device *vdev) { -return device_add (&vdev->dev); +struct device *dev; +int err; + +mutex_lock (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL ,dev_name (&vdev->dev), vdpa_name_match); +if (dev) { +put_device (dev); + err = -EEXIST ; + goto name_err; + } + + err =device_add (&vdev->dev); +name_err: +mutex_unlock (&vdpa_dev_mutex); +return err; }EXPORT_SYMBOL_GPL (vdpa_register_device);
@@ -137,7 +163,9 @@ EXPORT_SYMBOL_GPL(vdpa_register_device);*/ void vdpa_unregister_device (struct vdpa_device *vdev) { +mutex_lock (&vdpa_dev_mutex);device_unregister (&vdev->dev); +mutex_unlock (&vdpa_dev_mutex); }EXPORT_SYMBOL_GPL (vdpa_unregister_device);
drivers/vdpa/vdpa_sim/vdpa_sim.c +1 -1
@@ -357,7 +357,7 @@ static struct vdpasim *vdpasim_create(void)else ops = &vdpasim_net_config_ops; - vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, VDPASIM_VQ_NUM); + vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, VDPASIM_VQ_NUM, NULL);if (!vdpasim) goto err_alloc;
include/linux/vdpa.h +3 -4
@@ -245,15 +245,14 @@ struct vdpa_config_ops {struct vdpa_device *__vdpa_alloc_device (struct device *parent,const struct vdpa_config_ops *config, -int nvqs, -size_t size); +int nvqs,size_t size,const char *name); -#define vdpa_alloc_device (dev_struct, member, parent, config, nvqs) \ +#define vdpa_alloc_device (dev_struct, member, parent, config, nvqs, name) \ container_of(__vdpa_alloc_device( \ parent, config, nvqs, \ sizeof(dev_struct) + \ BUILD_BUG_ON_ZERO(offsetof( \ - dev_struct, member))), \ + dev_struct, member)), name), \ dev_struct, member)int vdpa_register_device (struct vdpa_device *vdev);
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH 4/7 vdpa: Define vdpa parent device, ops and a netlink interface
In reply to Parav PanditTo add one or more VDPA devices, define a parent device which allows adding or removing vdpa device. A parent device defines set of callbacks to manage vdpa devices.
To begin with, it defines add and remove callbacks through which a user defined vdpa device can be added or removed.
A unique parent device is identified by its unique handle identified by parent device name and optionally the bus name.
Hence, introduce routine through which driver can register a parent device and its callback operations for adding and remove a vdpa device.
Introduce vdpa netlink socket family so that user can query parent device and its attributes.
Example of show vdpa parent device which allows creating vdpa device of networking class (device id = 0x1) of virtio specification 1.1 section 5.1.1.
$ vdpa parentdev show
vdpasim:
supported_classes:
netExample of showing vdpa parent device in JSON format.
$ vdpa parentdev show -jp
{
"show": {
"vdpasim": {
"supported_classes": [ "net" ]
}
}
}--- drivers/vdpa/Kconfig | 1 + drivers/vdpa/vdpa.c | 213 +++++++++++++++++++++++++++++++++++++- include/linux/vdpa.h | 32 ++++++ include/uapi/linux/vdpa.h | 32 ++++++ 4 files changed, 277 insertions(+), 1 deletion(-) create mode 100644 include/uapi/linux/vdpa.h
drivers/vdpa/Kconfig +1 -0
@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only menuconfig VDPA tristate "vDPA drivers" + depends on NET help Enable this module to support vDPA device that uses a datapath which complies with virtio specifications with
drivers/vdpa/vdpa.c +212 -1
236 lines not on this page · raw message
include/linux/vdpa.h +32 -0
@@ -35,6 +35,8 @@ struct vdpa_vq_state {u16 avail_index; }; +struct vdpa_parent_dev ; +/** * vDPA device - representation of a vDPA device * @dev: underlying device
@@ -335,4 +337,34 @@ static inline void vdpa_get_config(struct vdpa_device *vdev, unsigned offset, ops->get_config(vdev, offset, buf, len); } +/** +* vdpa_dev_ops - vdpa device ops +* @dev_add: Add a vdpa device using alloc and register +* @pdev: parent device to use for device addition +* @name: name of the new vdpa device +* @device_id: device id of the new vdpa device +* Driver need to add a new device using vdpa_register_device() after +* fully initializing the vdpa device. On successful addition driver +* must return a valid pointer of vdpa device or ERR_PTR for the error. +* @dev_del: Remove a vdpa device using unregister +* @pdev: parent device to use for device removal +* @dev: vdpa device to remove +* Driver need to remove the specified device by calling vdpa_unregister_device(). +*/ +struct vdpa_dev_ops { +struct vdpa_device * (*dev_add)(struct vdpa_parent_dev *pdev,const char *name, +u32 device_id); +void (*dev_del)(struct vdpa_parent_dev *pdev,struct vdpa_device *dev); +}; + +struct vdpa_parent_dev { +struct device *device; +const struct vdpa_dev_ops *ops; +const struct virtio_device_id *id_table;/* supported ids */ +struct list_head list; +}; + +int vdpa_parentdev_register (struct vdpa_parent_dev *pdev); +void vdpa_parentdev_unregister (struct vdpa_parent_dev *pdev); +#endif /* _LINUX_VDPA_H */
include/uapi/linux/vdpa.h new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +/* +* vdpa device management interface +* Copyright (c) 2020 Mellanox Technologies Ltd. All rights reserved. +*/ + +#ifndef _UAPI_LINUX_VDPA_H_ +#define _UAPI_LINUX_VDPA_H_ + +#define VDPA_GENL_NAME "vdpa" +#define VDPA_GENL_VERSION 0x1 +#define VDPA_GENL_MCGRP_CONFIG_NAME "config" + +enum vdpa_command { +VDPA_CMD_UNSPEC , +VDPA_CMD_PARENTDEV_NEW , +VDPA_CMD_PARENTDEV_GET ,/* can dump */ +}; + +enum vdpa_attr { +VDPA_ATTR_UNSPEC , + +/* bus name (optional) + dev name together make the parent device handle */ +VDPA_ATTR_PARENTDEV_BUS_NAME ,/* string */ +VDPA_ATTR_PARENTDEV_DEV_NAME ,/* string */ +VDPA_ATTR_PARENTDEV_SUPPORTED_CLASSES ,/* u64 */ + +/* new attributes must be added above here */ +VDPA_ATTR_MAX , +}; + +#endif
Signature · 1 line
2.26.2
PATCH 5/7 vdpa: Enable a user to add and delete a vdpa device
In reply to Parav PanditAdd the ability to add and delete a vdpa device.
Examples:
Create a vdpa device of type network named "foo2" from the parent device vdpasim:
$ vdpa dev add parentdev vdpasim type net name foo2
Delete the vdpa device after its use:
$ vdpa dev del foo2
--- drivers/vdpa/vdpa.c | 149 +++++++++++++++++++++++++++++++++++--- include/linux/vdpa.h | 6 ++ include/uapi/linux/vdpa.h | 5 ++ 3 files changed, 150 insertions(+), 10 deletions(-)
drivers/vdpa/vdpa.c +139 -10
@@ -136,6 +136,37 @@ static int vdpa_name_match(struct device *dev, const void *data)return (strcmp (dev_name (&vdev->dev), data) ==0 ); } +static int __vdpa_register_device (struct vdpa_device *vdev) +{ +struct device *dev; + +lockdep_assert_held (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL ,dev_name (&vdev->dev), vdpa_name_match); +if (dev) { +put_device (dev); +return -EEXIST ; + } +return device_add (&vdev->dev); +} + +/** +* _vdpa_register_device - register a vDPA device with vdpa lock held +* Caller must have a succeed call of vdpa_alloc_device() before. +* Caller must invoke this routine as part of parent device dev_add() callback +* after setting up valid parentdev for this vdpa device. +* @vdev: the vdpa device to be registered to vDPA bus +* +* Returns an error when fail to add device to vDPA bus +*/ +int _vdpa_register_device (struct vdpa_device *vdev) +{ +if (!vdev->pdev) +return -EINVAL ; + +return __vdpa_register_device (vdev); +} +EXPORT_SYMBOL_GPL (_vdpa_register_device); +/** * vdpa_register_device - register a vDPA device * Callers must have a succeed call of vdpa_alloc_device() before.
@@ -145,24 +176,28 @@ static int vdpa_name_match(struct device *dev, const void *data)*/ int vdpa_register_device (struct vdpa_device *vdev) { -struct device *dev;int err;mutex_lock (&vdpa_dev_mutex); - dev =bus_find_device (&vdpa_bus,NULL ,dev_name (&vdev->dev), vdpa_name_match); -if (dev) { -put_device (dev); - err = -EEXIST ; - goto name_err; - } - - err =device_add (&vdev->dev); -name_err: + err =__vdpa_register_device (vdev);mutex_unlock (&vdpa_dev_mutex);return err; }EXPORT_SYMBOL_GPL (vdpa_register_device); +/** +* _vdpa_unregister_device - unregister a vDPA device +* Caller must invoke this routine as part of parent device dev_del() callback. +* @vdev: the vdpa device to be unregisted from vDPA bus +*/ +void _vdpa_unregister_device (struct vdpa_device *vdev) +{ +lockdep_assert_held (&vdpa_dev_mutex); +WARN_ON (!vdev->pdev); +device_unregister (&vdev->dev); +} +EXPORT_SYMBOL_GPL (_vdpa_unregister_device); +/** * vdpa_unregister_device - unregister a vDPA device * @vdev: the vdpa device to be unregisted from vDPA bus
@@ -221,10 +256,25 @@ int vdpa_parentdev_register(struct vdpa_parent_dev *pdev) }EXPORT_SYMBOL_GPL (vdpa_parentdev_register); +static int vdpa_match_remove (struct device *dev,void *data) +{ + struct vdpa_device *vdev = container_of(dev, struct vdpa_device, dev); +struct vdpa_parent_dev *pdev = vdev->pdev; + +if (pdev == data) + pdev->ops->dev_del (pdev, vdev); +return 0 ; +} +void vdpa_parentdev_unregister (struct vdpa_parent_dev *pdev) {mutex_lock (&vdpa_dev_mutex); +list_del (&pdev->list); + +/* Filter out all the entries belong to this parent device and delete it. */ +bus_for_each_dev (&vdpa_bus,NULL , pdev, vdpa_match_remove); +mutex_unlock (&vdpa_dev_mutex); }EXPORT_SYMBOL_GPL (vdpa_parentdev_unregister);
@@ -368,9 +418,76 @@ vdpa_nl_cmd_parentdev_get_dumpit(struct sk_buff *msg, struct netlink_callback *creturn msg->len; } +static int vdpa_nl_cmd_dev_add_set_doit (struct sk_buff *skb,struct genl_info *info) +{ +struct vdpa_parent_dev *pdev; +struct vdpa_device *vdev; +const char *name; +u32 device_id; +int err =0 ; + +if (!info->attrs[VDPA_ATTR_DEV_NAME ] || !info->attrs[VDPA_ATTR_DEV_ID ]) +return -EINVAL ; + + name =nla_data (info->attrs[VDPA_ATTR_DEV_NAME ]); + device_id =nla_get_u32 (info->attrs[VDPA_ATTR_DEV_ID ]); + +mutex_lock (&vdpa_dev_mutex); + pdev =vdpa_parentdev_get_from_attr (info->attrs); +if (IS_ERR (pdev)) { +NL_SET_ERR_MSG_MOD (info->extack,"Fail to find the specified parent device" ); + err =PTR_ERR (pdev); + goto err; + } + + vdev = pdev->ops->dev_add (pdev, name, device_id); +if (IS_ERR (vdev)) + goto err; + +err: +mutex_unlock (&vdpa_dev_mutex); +return err; +} + +static int vdpa_nl_cmd_dev_del_set_doit (struct sk_buff *skb,struct genl_info *info) +{ +struct vdpa_parent_dev *pdev; +struct vdpa_device *vdev; +struct device *dev; +const char *name; +int err =0 ; + +if (!info->attrs[VDPA_ATTR_DEV_NAME ]) +return -EINVAL ; + name =nla_data (info->attrs[VDPA_ATTR_DEV_NAME ]); + +mutex_lock (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL , name, vdpa_name_match); +if (!dev) { +NL_SET_ERR_MSG_MOD (info->extack,"device not found" ); + err = -ENODEV ; + goto dev_err; + } + vdev = container_of(dev, struct vdpa_device, dev); +if (!vdev->pdev) { +NL_SET_ERR_MSG_MOD (info->extack,"Only user created device can be deleted by user" ); + err = -EINVAL ; + goto pdev_err; + } + pdev = vdev->pdev; + pdev->ops->dev_del (pdev, vdev); +pdev_err: +put_device (dev); +dev_err: +mutex_unlock (&vdpa_dev_mutex); +return err; +} +static const struct nla_policy vdpa_nl_policy[VDPA_ATTR_MAX ] = { [VDPA_ATTR_PARENTDEV_BUS_NAME ] = { .type =NLA_NUL_STRING }, [VDPA_ATTR_PARENTDEV_DEV_NAME ] = { .type =NLA_STRING }, + [VDPA_ATTR_DEV_NAME ] = { .type =NLA_STRING }, + [VDPA_ATTR_DEV_ID ] = { .type =NLA_U32 }, };static const struct genl_ops vdpa_nl_ops[] = {
@@ -380,6 +497,18 @@ static const struct genl_ops vdpa_nl_ops[] = { .doit = vdpa_nl_cmd_parentdev_get_doit, .dumpit = vdpa_nl_cmd_parentdev_get_dumpit, }, + { + .cmd = VDPA_CMD_DEV_NEW, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, + .doit = vdpa_nl_cmd_dev_add_set_doit, + .flags = GENL_ADMIN_PERM, + }, + { + .cmd = VDPA_CMD_DEV_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, + .doit = vdpa_nl_cmd_dev_del_set_doit, + .flags = GENL_ADMIN_PERM, + }, }; static struct genl_family vdpa_nl_family __ro_after_init = {
include/linux/vdpa.h +6 -0
@@ -45,6 +45,8 @@ struct vdpa_parent_dev;* @index: device index * @features_valid: were features initialized? for legacy guests * @nvqs: maximum number of supported virtqueues +* @pdev: parent device pointer; caller must setup when registering device as part +* of dev_add() parentdev ops callback before invoking _vdpa_register_device(). */ struct vdpa_device {struct device dev;
@@ -53,6 +55,7 @@ struct vdpa_device {unsigned int index;bool features_valid;int nvqs; +struct vdpa_parent_dev *pdev; };/**
@@ -260,6 +263,9 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent,int vdpa_register_device (struct vdpa_device *vdev);void vdpa_unregister_device (struct vdpa_device *vdev); +int _vdpa_register_device (struct vdpa_device *vdev); +void _vdpa_unregister_device (struct vdpa_device *vdev); +/** * vdpa_driver - operations for a vDPA driver * @driver: underlying device driver
include/uapi/linux/vdpa.h +5 -0
@@ -15,6 +15,8 @@ enum vdpa_command {VDPA_CMD_UNSPEC ,VDPA_CMD_PARENTDEV_NEW ,VDPA_CMD_PARENTDEV_GET ,/* can dump */ +VDPA_CMD_DEV_NEW , + VDPA_CMD_DEV_DEL, }; enum vdpa_attr {
@@ -25,6 +27,9 @@ enum vdpa_attr {VDPA_ATTR_PARENTDEV_DEV_NAME ,/* string */ VDPA_ATTR_PARENTDEV_SUPPORTED_CLASSES ,/* u64 */ +VDPA_ATTR_DEV_NAME ,/* string */ +VDPA_ATTR_DEV_ID ,/* u32 */ +/* new attributes must be added above here */ VDPA_ATTR_MAX, };
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH 6/7 vdpa: Enable user to query vdpa device info
In reply to Parav PanditEnable user to query vdpa device information.
$ vdpa dev add parentdev vdpasim type net name foo2
Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network parentdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256
$ vdpa dev show foo2 -jp
{
"dev": {
"foo2": {
"type": "network",
"parentdev": "vdpasim",
"vendor_id": 0,
"max_vqs": 2,
"max_vq_size": 256
}
}
}--- drivers/vdpa/vdpa.c | 131 ++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vdpa.h | 4 ++ 2 files changed, 135 insertions(+)
drivers/vdpa/vdpa.c +131 -0
@@ -483,6 +483,131 @@ static int vdpa_nl_cmd_dev_del_set_doit(struct sk_buff *skb, struct genl_info *ireturn err; } +static int +vdpa_dev_fill (struct vdpa_device *vdev,struct sk_buff *msg,u32 portid,u32 seq, +int flags,struct netlink_ext_ack *extack) +{ +u16 max_vq_size; +u32 device_id; +u32 vendor_id; +void *hdr; +int err; + + hdr =genlmsg_put (msg, portid, seq, &vdpa_nl_family, flags,VDPA_CMD_DEV_NEW ); +if (!hdr) +return -EMSGSIZE ; + + err =vdpa_nl_parentdev_handle_fill (msg, vdev->pdev); +if (err) + goto msg_err; + + device_id = vdev->config->get_device_id (vdev); + vendor_id = vdev->config->get_vendor_id (vdev); + max_vq_size = vdev->config->get_vq_num_max (vdev); + + err = -EMSGSIZE ; +if (nla_put_string (msg,VDPA_ATTR_DEV_NAME ,dev_name (&vdev->dev))) + goto msg_err; +if (nla_put_u32 (msg,VDPA_ATTR_DEV_ID , device_id)) + goto msg_err; +if (nla_put_u32 (msg,VDPA_ATTR_DEV_VENDOR_ID , vendor_id)) + goto msg_err; +if (nla_put_u32 (msg,VDPA_ATTR_DEV_MAX_VQS , vdev->nvqs)) + goto msg_err; +if (nla_put_u16 (msg,VDPA_ATTR_DEV_MAX_VQ_SIZE , max_vq_size)) + goto msg_err; + +genlmsg_end (msg, hdr); +return 0 ; + +msg_err: +genlmsg_cancel (msg, hdr); +return err; +} + +static int vdpa_nl_cmd_dev_get_doit (struct sk_buff *skb,struct genl_info *info) +{ +struct vdpa_device *vdev; +struct sk_buff *msg; +const char *devname; +struct device *dev; +int err; + +if (!info->attrs[VDPA_ATTR_DEV_NAME ]) +return -EINVAL ; + devname =nla_data (info->attrs[VDPA_ATTR_DEV_NAME ]); + msg =nlmsg_new (NLMSG_DEFAULT_SIZE ,GFP_KERNEL ); +if (!msg) +return -ENOMEM ; + +mutex_lock (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL , devname, vdpa_name_match); +if (!dev) { +mutex_unlock (&vdpa_dev_mutex); +NL_SET_ERR_MSG_MOD (info->extack,"device not found" ); +return -ENODEV ; + } + vdev = container_of(dev, struct vdpa_device, dev); +if (!vdev->pdev) { +mutex_unlock (&vdpa_dev_mutex); +put_device (dev); +return -EINVAL ; + } + err =vdpa_dev_fill (vdev, msg, info->snd_portid, info->snd_seq,0 , info->extack); +if (!err) + err =genlmsg_reply (msg, info); +put_device (dev); +mutex_unlock (&vdpa_dev_mutex); + +if (err) +nlmsg_free (msg); +return err; +} + +struct vdpa_dev_dump_info { +struct sk_buff *msg; +struct netlink_callback *cb; +int start_idx; +int idx; +}; + +static int vdpa_dev_dump (struct device *dev,void *data) +{ + struct vdpa_device *vdev = container_of(dev, struct vdpa_device, dev); +struct vdpa_dev_dump_info *info = data; +int err; + +if (!vdev->pdev) +return 0 ; +if (info->idx < info->start_idx) { + info->idx++; +return 0 ; + } + err =vdpa_dev_fill (vdev, info->msg,NETLINK_CB (info->cb->skb).portid, + info->cb->nlh->nlmsg_seq,NLM_F_MULTI , info->cb->extack); +if (err) +return err; + + info->idx++; +return 0 ; +} + +static int vdpa_nl_cmd_dev_get_dumpit (struct sk_buff *msg,struct netlink_callback *cb) +{ +struct vdpa_dev_dump_info info; + + info.msg = msg; + info.cb = cb; + info.start_idx = cb->args[0 ]; + info.idx =0 ; + +mutex_lock (&vdpa_dev_mutex); +bus_for_each_dev (&vdpa_bus,NULL , &info, vdpa_dev_dump); +mutex_unlock (&vdpa_dev_mutex); + cb->args[0 ] = info.idx; +return msg->len; +} +static const struct nla_policy vdpa_nl_policy[VDPA_ATTR_MAX ] = { [VDPA_ATTR_PARENTDEV_BUS_NAME ] = { .type =NLA_NUL_STRING }, [VDPA_ATTR_PARENTDEV_DEV_NAME ] = { .type =NLA_STRING },
@@ -509,6 +634,12 @@ static const struct genl_ops vdpa_nl_ops[] = { .doit = vdpa_nl_cmd_dev_del_set_doit, .flags = GENL_ADMIN_PERM, }, + { + .cmd = VDPA_CMD_DEV_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, + .doit = vdpa_nl_cmd_dev_get_doit, + .dumpit = vdpa_nl_cmd_dev_get_dumpit, + }, };static struct genl_family vdpa_nl_family__ro_after_init = {
include/uapi/linux/vdpa.h +4 -0
@@ -17,6 +17,7 @@ enum vdpa_command {VDPA_CMD_PARENTDEV_GET ,/* can dump */ VDPA_CMD_DEV_NEW ,VDPA_CMD_DEV_DEL , + VDPA_CMD_DEV_GET, /* can dump */ }; enum vdpa_attr {
@@ -29,6 +30,9 @@ enum vdpa_attr {VDPA_ATTR_DEV_NAME ,/* string */ VDPA_ATTR_DEV_ID ,/* u32 */ +VDPA_ATTR_DEV_VENDOR_ID ,/* u32 */ +VDPA_ATTR_DEV_MAX_VQS ,/* u32 */ +VDPA_ATTR_DEV_MAX_VQ_SIZE ,/* u16 */ /* new attributes must be added above here */ VDPA_ATTR_MAX,
Signature · 1 line
2.26.2
PATCH 7/7 vdpa/vdpa_sim: Enable user to create vdpasim net devices
In reply to Parav PanditEnable user to create vdpasim net simulate devices.
Show vdpa parent device that supports creating, deleting vdpa devices.
$ vdpa parentdev show
vdpasim:
supported_classes
net$ vdpa parentdev show -jp
{
"show": {
"vdpasim": {
"supported_classes": {
"net"
}
}
}Create a vdpa device of type networking named as "foo2" from the parent device vdpasim:
$ vdpa dev add parentdev vdpasim type net name foo2
Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network parentdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256
$ vdpa dev show foo2 -jp
{
"dev": {
"foo2": {
"type": "network",
"parentdev": "vdpasim",
"vendor_id": 0,
"max_vqs": 2,
"max_vq_size": 256
}
}
}Delete the vdpa device after its use:
$ vdpa dev del foo2
--- drivers/vdpa/vdpa_sim/vdpa_sim.c | 81 +++++++++++++++++++++++++++----- 1 file changed, 69 insertions(+), 12 deletions(-)
drivers/vdpa/vdpa_sim/vdpa_sim.c +69 -12
@@ -28,6 +28,7 @@#include <linux/vhost_iotlb.h> #include <uapi/linux/virtio_config.h> #include <uapi/linux/virtio_net.h> +#include <uapi/linux/vdpa.h> #define DRV_VERSION "0.1"#define DRV_AUTHOR "Jason Wang <[email protected]>"
@@ -42,6 +43,17 @@ static char *macaddr;module_param (macaddr, charp,0 );MODULE_PARM_DESC (macaddr,"Ethernet MAC address" ); +static struct vdpa_parent_dev parent_dev; + +static void vdpasim_parent_release (struct device *dev) +{ +} + +static struct device vdpasim_parent = { + .init_name ="vdpasim" , + .release = vdpasim_parent_release, +}; +struct vdpasim_virtqueue {struct vringh vring;struct vringh_kiov iov;
@@ -101,8 +113,6 @@ static inline __virtio16 cpu_to_vdpasim16(struct vdpasim *vdpasim, u16 val)return __cpu_to_virtio16 (vdpasim_is_little_endian (vdpasim), val); } -static struct vdpasim *vdpasim_dev; -static struct vdpasim *vdpa_to_sim (struct vdpa_device *vdpa) { return container_of(vdpa, struct vdpasim, vdpa);
@@ -345,7 +355,7 @@ static const struct dma_map_ops vdpasim_dma_ops = {static const struct vdpa_config_ops vdpasim_net_config_ops;static const struct vdpa_config_ops vdpasim_net_batch_config_ops; -static struct vdpasim *vdpasim_create (void ) +static struct vdpasim *vdpasim_create (const char *name) {const struct vdpa_config_ops *ops;struct vdpasim *vdpasim;
@@ -357,7 +367,7 @@ static struct vdpasim *vdpasim_create(void)else ops = &vdpasim_net_config_ops; - vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, VDPASIM_VQ_NUM, NULL); + vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, VDPASIM_VQ_NUM, name);if (!vdpasim) goto err_alloc;
@@ -393,7 +403,8 @@ static struct vdpasim *vdpasim_create(void)vringh_set_iotlb (&vdpasim->vqs[1 ].vring, vdpasim->iommu); vdpasim->vdpa.dma_dev = dev; - ret =vdpa_register_device (&vdpasim->vdpa); + vdpasim->vdpa.pdev = &parent_dev; + ret =_vdpa_register_device (&vdpasim->vdpa);if (ret) goto err_iommu;
@@ -714,21 +725,67 @@ static const struct vdpa_config_ops vdpasim_net_batch_config_ops = { .free = vdpasim_free, }; +static struct vdpa_device * +vdpa_dev_add (struct vdpa_parent_dev *pdev,const char *name,u32 device_id) +{ +struct vdpasim *simdev; + +if (device_id !=VIRTIO_ID_NET ) +return ERR_PTR (-EOPNOTSUPP ); + + simdev =vdpasim_create (name); +if (IS_ERR (simdev)) +return (struct vdpa_device *)simdev; + +return &simdev->vdpa; +} + +static void vdpa_dev_del (struct vdpa_parent_dev *pdev,struct vdpa_device *dev) +{ + struct vdpasim *simdev = container_of(dev, struct vdpasim, vdpa); + +_vdpa_unregister_device (&simdev->vdpa); +} + +static const struct vdpa_dev_ops vdpa_dev_parent_ops = { + .dev_add = vdpa_dev_add, + .dev_del = vdpa_dev_del +}; + +static struct virtio_device_id id_table[] = { + {VIRTIO_ID_NET ,VIRTIO_DEV_ANY_ID }, + {0 }, +}; + +static struct vdpa_parent_dev parent_dev = { + .device = &vdpasim_parent, + .id_table = id_table, + .ops = &vdpa_dev_parent_ops, +}; +static int __init vdpasim_dev_init (void ) { - vdpasim_dev =vdpasim_create (); +int ret; -if (!IS_ERR (vdpasim_dev)) -return 0 ; + ret =device_register (&vdpasim_parent); +if (ret) +return ret; + + ret =vdpa_parentdev_register (&parent_dev); +if (ret) + goto parent_err; -return PTR_ERR (vdpasim_dev); +return 0 ; + +parent_err: +device_unregister (&vdpasim_parent); +return ret; } static void __exit vdpasim_dev_exit(void) { -struct vdpa_device *vdpa = &vdpasim_dev->vdpa; - -vdpa_unregister_device (vdpa); +vdpa_parentdev_unregister (&parent_dev); +device_unregister (&vdpasim_parent); }module_init (vdpasim_dev_init)
Signature · 1 line
2.26.2
Great! A few questions and comments:
How are configuration parameters passed in during device creation (e.g. MAC address, number of queues)?
Can configuration parameters be changed at runtime (e.g. link up/down)?
Does the configuration parameter interface distinguish between standard and vendor-specific parameters? Are they namespaced to prevent naming collisions?
How are software-only parent drivers supported? It's kind of a shame to modprobe unconditionally if they won't be used. Does vdpatool have some way of requesting loading a parent driver? That way software drivers can be loaded on demand.
What is the benefit of making it part of iproute2? If there is not a significant advantage like sharing code, then I suggest using a separate repository and package so vdpatool can be installed separately (e.g. even on AF_VSOCK-only guests without Ethernet).
Stefan
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 8 lines
From: Stefan Hajnoczi <[email protected]>
Sent: Monday, November 16, 2020 3:11 PM
Great! A few questions and comments:How are configuration parameters passed in during device creation (e.g. MAC address, number of queues)?
During device creation time more parameters to be added.
Quoted 3 lines
Can configuration parameters be changed at runtime (e.g. link up/down)?
For representor eswitch based devices, it is usually controlled through it.
For others, I haven't thought about it. If the device supports it, I believe so.
If multiple vpda devices are created over single VF/PF/SF, virtualizing the link for up/down (not just changing the vdpa config bits) can be a challenge.
Quoted 3 lines
Does the configuration parameter interface distinguish between standard and vendor-specific parameters? Are they namespaced to prevent naming collisions?
Do you have an example of vendor specific parameters?
Since this tool exposes virtio compliant vdpa devices, I didn't consider any vendor specific params.
Quoted 5 lines
How are software-only parent drivers supported? It's kind of a shame to modprobe unconditionally if they won't be used. Does vdpatool have some way of requesting loading a parent driver? That way software drivers can be loaded on demand.
Well, since each parent or management device registers for it, and their type is same, there isn't a way right not to auto load the module. This will require user to learn what type of vendor device driver to be loaded, which kinds of defeats the purpose.
Quoted 5 lines
What is the benefit of making it part of iproute2? If there is not a significant advantage like sharing code, then I suggest using a separate repository and package so vdpatool can be installed separately (e.g. even on AF_VSOCK- only guests without Ethernet).
Given that vdpa tool intents to create network specific devices, iproute2 seems a better fit than a own repository. It mainly uses libmnl.
Quoted 2 lines
Stefan
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Jakub Kicinski
maintainerQuoted 10 lines · On Thu, 12 Nov 2020 08:39:58 +0200 Parav Pandit wrote:
FAQs:
-----
1. Where does userspace vdpa tool reside which users can use?
Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to create vdpa net devices.2. Why not create and delete vdpa device using sysfs/configfs? Ans:
3. Why not use
ioctl()interface?
Obviously I'm gonna ask you - why can't you use devlink?
Quoted 6 lines
Next steps:
-----------
(a) Post this patchset and iproute2/vdpa inclusion, remaining two drivers will be coverted to support vdpa tool instead of creating unmanaged default device on driver load.
(b) More net specific parameters such as mac, mtu will be added.
How does MAC and MTU belong in this new VDPA thing?
Quoted 12 lines
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, November 17, 2020 3:53 AMQuoted 6 lines · On Thu, 12 Nov 2020 08:39:58 +0200 Parav Pandit wrote:
FAQs:
-----
1. Where does userspace vdpa tool reside which users can use?
Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to create vdpa net devices.
Quoted 7 lines
Quoted 4 lines
2. Why not create and delete vdpa device using sysfs/configfs? Ans:
3. Why not use
ioctl()interface?Obviously I'm gonna ask you - why can't you use devlink?
This was considered.
However it seems that extending devlink for vdpa specific stats, devices, config sounds overloading devlink beyond its defined scope.
Quoted 8 lines
Quoted 6 lines
Next steps:
-----------
(a) Post this patchset and iproute2/vdpa inclusion, remaining two drivers will be coverted to support vdpa tool instead of creating unmanaged default device on driver load.
(b) More net specific parameters such as mac, mtu will be added.How does MAC and MTU belong in this new VDPA thing?
MAC only make sense when user wants to run VF/SF Netdev and vdpa together with different mac address. Otherwise existing devlink well defined API to have one MAC per function is fine.
Same for MTU, if queues of vdpa vs VF/SF Netdev queues wants have different MTU it make sense to add configure per vdpa device.
Michael S. Tsirkin
maintainerQuoted 15 lines · Parav Pandit
Quoted 15 lines
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, November 17, 2020 3:53 AMQuoted 8 lines · On Thu, 12 Nov 2020 08:39:58 +0200 Parav Pandit wrote:
FAQs:
-----
1. Where does userspace vdpa tool reside which users can use?
Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to create vdpa net devices.2. Why not create and delete vdpa device using sysfs/configfs? Ans:
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
3. Why not use
ioctl()interface?Obviously I'm gonna ask you - why can't you use devlink?
This was considered.
However it seems that extending devlink for vdpa specific stats, devices, config sounds overloading devlink beyond its defined scope.
kuba what's your thinking here? Should I merge this as is?
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
Next steps:
-----------
(a) Post this patchset and iproute2/vdpa inclusion, remaining two drivers will be coverted to support vdpa tool instead of creating unmanaged default device on driver load.
(b) More net specific parameters such as mac, mtu will be added.How does MAC and MTU belong in this new VDPA thing?
MAC only make sense when user wants to run VF/SF Netdev and vdpa together with different mac address. Otherwise existing devlink well defined API to have one MAC per function is fine.
Same for MTU, if queues of vdpa vs VF/SF Netdev queues wants have different MTU it make sense to add configure per vdpa device.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
A message that is not in the archive (sent off-list or not yet received): 20201216080610.08541f44@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com
Quoted 8 lines
From: Jakub Kicinski <[email protected]>
Sent: Wednesday, December 16, 2020 9:36 PMQuoted 4 lines · On Wed, 16 Dec 2020 04:13:51 -0500 Michael S. Tsirkin wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
3. Why not use
ioctl()interface?Obviously I'm gonna ask you - why can't you use devlink?
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
This was considered.
However it seems that extending devlink for vdpa specific stats, devices,config sounds overloading devlink beyond its defined scope.
Quoted 2 lines
kuba what's your thinking here? Should I merge this as is?
No objections from me if people familiar with VDPA like it.
I was too occupied with the recent work on subfunction series.
I wanted to change the "parentdev" to "mgmtdev" to make it little more clear for vdpa management tool to see vdpa mgmt device and operate on it. What do you think? Should I revise v2 or its late?
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 12 lines · Parav Pandit
Quoted 12 lines
From: Jakub Kicinski <[email protected]>
Sent: Wednesday, December 16, 2020 9:36 PMQuoted 6 lines · On Wed, 16 Dec 2020 04:13:51 -0500 Michael S. Tsirkin wrote:
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
3. Why not use
ioctl()interface?Obviously I'm gonna ask you - why can't you use devlink?
This was considered.
However it seems that extending devlink for vdpa specific stats, devices,config sounds overloading devlink beyond its defined scope.
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
kuba what's your thinking here? Should I merge this as is?
No objections from me if people familiar with VDPA like it.
I was too occupied with the recent work on subfunction series.
I wanted to change the "parentdev" to "mgmtdev" to make it little more clear for vdpa management tool to see vdpa mgmt device and operate on it. What do you think? Should I revise v2 or its late?
I need a rebase anyway, so sure.
Signature · 6 lines
MST _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 21 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, December 17, 2020 1:28 AMQuoted 10 lines · Parav Pandit
Quoted 10 lines
From: Jakub Kicinski <[email protected]>
Sent: Wednesday, December 16, 2020 9:36 PMQuoted 6 lines · On Wed, 16 Dec 2020 04:13:51 -0500 Michael S. Tsirkin wrote:
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
3. Why not use
ioctl()interface?Obviously I'm gonna ask you - why can't you use devlink?
This was considered.
However it seems that extending devlink for vdpa specific stats,devices,
Quoted 4 lines
Quoted 4 lines
config sounds overloading devlink beyond its defined scope.
Quoted 2 lines
kuba what's your thinking here? Should I merge this as is?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
No objections from me if people familiar with VDPA like it.
I was too occupied with the recent work on subfunction series. I wanted to change the "parentdev" to "mgmtdev" to make it little more
clear for vdpa management tool to see vdpa mgmt device and operate on it.
Quoted 1 line
What do you think? Should I revise v2 or its late?
I need a rebase anyway, so sure.
ok. Thanks.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 183 lines · Parav Pandit
This patchset covers user requirements for managing existing vdpa devices, using a tool and its internal design notes for kernel drivers.
Background and user requirements:
----------------------------------
(1) Currently VDPA device is created by driver when driver is loaded. However, user should have a choice when to create or not create a vdpa device for the underlying parent device.For example, mlx5 PCI VF and subfunction device supports multiple classes of device such netdev, vdpa, rdma. Howevever it is not required to always created vdpa device for such device.
(2) In another use case, a device may support creating one or multiple vdpa device of same or different class such as net and block. Creating vdpa devices at driver load time further limits this use case.
(3) A user should be able to monitor and query vdpa queue level or device level statistics for a given vdpa device.
(4) A user should be able to query what class of vdpa devices are supported by its parent device.
(5) A user should be able to view supported features and negotiated features of the vdpa device.
(6) A user should be able to create a vdpa device in vendor agnostic manner using single tool.
Hence, it is required to have a tool through which user can create one or more vdpa devices from a parent device which addresses above user requirements.
Example devices:
----------------+-----------+ +-----------+ +---------+ +--------+ +-----------+ |vdpa dev 0 | |vdpa dev 1 | |rdma dev | |netdev | |vdpa dev 3 | |type=net | |type=block | |mlx5_0 | |ens3f0 | |type=net | +----+------+ +-----+-----+ +----+----+ +-----+--+ +----+------+ | | | | | | | | | | +----+-----+ | +----+----+ | +----+----+ | mlx5 +--------+ |mlx5 +-------+ |mlx5 | |pci vf 2 | |pci vf 4 | |pci sf 8 | |03:00:2 | |03:00.4 | |mlx5_sf.8| +----+-----+ +----+----+ +----+----+ | | | | +----+-----+ | +----------------------+mlx5 +----------------+ |pci pf 0 | |03:00.0 | +----------+vdpa tool:
----------
vdpa tool is a tool to create, delete vdpa devices from a parent device. It is a tool that enables user to query statistics, features and may be more attributes in future.vdpa tool command draft:
------------------------
(a) List parent devices which supports creating vdpa devices. It also shows which class types supported by this parent device. In below command example two parent devices support vdpa device creation. First is PCI VF whose bdf is 03.00:2.
Second is PCI VF whose name is 03:00.4.
Third is PCI SF whose name is mlx5_core.sf.8$ vdpa parentdev list vdpasim supported_classes net pci/0000:03.00:3 supported_classes net block pci/0000:03.00:4 supported_classes net block auxiliary/mlx5_core.sf.8 supported_classes net(b) Now add a vdpa device of networking class and show the device.
$ vdpa dev add parentdev pci/0000:03.00:2 type net name foo0 $ vdpa dev show foo0 foo0: parentdev pci/0000:03.00:2 type network parentdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256(c) Show features of a vdpa device
$ vdpa dev features show foo0 supported iommu platform version 1(d) Dump vdpa device statistics
$ vdpa dev stats show foo0 kickdoorbells 10 wqes 100(e) Now delete a vdpa device previously created.
$ vdpa dev del foo0vdpa tool support in this patchset:
-----------------------------------
vdpa tool is created to create, delete and query vdpa devices. examples:
Show vdpa parent device that supports creating, deleting vdpa devices.$ vdpa parentdev show vdpasim: supported_classes net$ vdpa parentdev show -jp { "show": { "vdpasim": { "supported_classes": { "net" } } }Create a vdpa device of type networking named as "foo2" from the parent device vdpasim:
$ vdpa dev add parentdev vdpasim type net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network parentdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256$ vdpa dev show foo2 -jp { "dev": { "foo2": { "type": "network", "parentdev": "vdpasim", "vendor_id": 0, "max_vqs": 2, "max_vq_size": 256 } } }Delete the vdpa device after its use:
$ vdpa dev del foo2vdpa tool support by kernel:
----------------------------
vdpa tool user interface will be supported by existing vdpa kernel framework, i.e.drivers/vdpa/vdpa.cIt services user command through a netlink interface.Each parent device registers supported callback operations with vdpa subsystem through which vdpa device(s) can be managed.
FAQs:
-----
1. Where does userspace vdpa tool reside which users can use?
Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to create vdpa net devices.2. Why not create and delete vdpa device using sysfs/configfs? Ans:
(a) A device creation may involve passing one or more attributes. Passing multiple attributes and returning error code and more verbose information for invalid attributes cannot be handled by sysfs/configfs.(b) netlink framework is rich that enables user space and kernel driver to provide nested attributes.
(c) Exposing device specific file under sysfs without net namespace awareness exposes details to multiple containers. Instead exposing attributes via a netlink socket secures the communication channel with kernel.
(d) netlink socket interface enables to run syscaller kernel tests.
3. Why not use
ioctl()interface?
Ans:ioctl()interface replicates the necessary plumbing which already exists through netlink socket.4. What happens when one or more user created vdpa devices exist for a parent PCI VF or SF and such parent device is removed?
Ans: All user created vdpa devices are removed that belong to a parent.[1]
git://git.kernel.org/pub/scm/network/iproute2/iproute2-next.git
Quoted 7 lines
Next steps:
-----------
(a) Post this patchset and iproute2/vdpa inclusion, remaining two drivers will be coverted to support vdpa tool instead of creating unmanaged default device on driver load.
(b) More net specific parameters such as mac, mtu will be added.
(c) Features bits get and set interface will be added.
Adding Yong Ji for sharing some thoughts from the view of userspace vDPA device.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
A message that is not in the archive (sent off-list or not yet received): CACycT3sYScObb9nN3g7L3cesjE7sCZWxZ5_5R1usGU9ePZEeqA@mail.gmail.com
Quoted 237 lines · Yongji Xie
On Fri, Nov 27, 2020 at 11:53 AM Jason Wang <[email protected] <mailto:[email protected]>> wrote:
On 2020/11/12 下午2:39, Parav Pandit wrote:
> This patchset covers user requirements for managing existing vdpa devices,
> using a tool and its internal design notes for kernel drivers. >
> Background and user requirements:
> ----------------------------------
> (1) Currently VDPA device is created by driver when driver is loaded.
> However, user should have a choice when to create or not create a vdpa device
> for the underlying parent device.
>
> For example, mlx5 PCI VF and subfunction device supports multiple classes of
> device such netdev, vdpa, rdma. Howevever it is not required to always created
> vdpa device for such device.
>
> (2) In another use case, a device may support creating one or multiple vdpa
> device of same or different class such as net and block. > Creating vdpa devices at driver load time further limits this use case.
>
> (3) A user should be able to monitor and query vdpa queue level or device level
> statistics for a given vdpa device.
>
> (4) A user should be able to query what class of vdpa devices are supported
> by its parent device.
>
> (5) A user should be able to view supported features and negotiated features
> of the vdpa device.
>
> (6) A user should be able to create a vdpa device in vendor agnostic manner
> using single tool.
>
> Hence, it is required to have a tool through which user can create one or more
> vdpa devices from a parent device which addresses above user requirements.
>
> Example devices:
> ----------------
> +-----------+ +-----------+ +---------+ +--------+ +-----------+ > |vdpa dev 0 | |vdpa dev 1 | |rdma dev | |netdev | |vdpa dev 3 | > |type=net | |type=block | |mlx5_0 | |ens3f0 | |type=net | > +----+------+ +-----+-----+ +----+----+ +-----+--+ +----+------+ > | | | | | > | | | | | > +----+-----+ | +----+----+ | +----+----+ > | mlx5 +--------+ |mlx5 +-------+ |mlx5 | > |pci vf 2 | |pci vf 4 | |pci sf 8 |
> |03:00:2 | |03:00.4 | |mlx5_sf.8|
> +----+-----+ +----+----+ +----+----+
> | | |
> | +----+-----+ |
> +----------------------+mlx5 +----------------+ > |pci pf 0 |
> |03:00.0 |
> +----------+
>
> vdpa tool:
> ----------
> vdpa tool is a tool to create, delete vdpa devices from a parent device. It is a
> tool that enables user to query statistics, features and may be more attributes
> in future.
>
> vdpa tool command draft:
> ------------------------
> (a) List parent devices which supports creating vdpa devices. > It also shows which class types supported by this parent device. > In below command example two parent devices support vdpa device creation.
> First is PCI VF whose bdf is 03.00:2.
> Second is PCI VF whose name is 03:00.4.
> Third is PCI SF whose name is mlx5_core.sf.8
>
> $ vdpa parentdev list
> vdpasim
>supported_classes
> net
> pci/0000:03.00:3
>supported_classes
> net block
> pci/0000:03.00:4
>supported_classes
> net block
> auxiliary/mlx5_core.sf.8
>supported_classes
> net
>
> (b) Now add a vdpa device of networking class and show the device. > $ vdpa dev add parentdev pci/0000:03.00:2 type net name foo0 $ vdpa dev show foo0
> foo0: parentdev pci/0000:03.00:2 type network parentdev vdpasimvendor_id0max_vqs2max_vq_size256
>
> (c) Show features of a vdpa device
> $ vdpa dev features show foo0
> supported
> iommu platform
> version 1
>
> (d) Dump vdpa device statistics
> $ vdpa dev stats show foo0
> kickdoorbells 10
> wqes 100
>
> (e) Now delete a vdpa device previously created.
> $ vdpa dev del foo0
>
> vdpa tool support in this patchset:
> -----------------------------------
> vdpa tool is created to create, delete and query vdpa devices. > examples:
> Show vdpa parent device that supports creating, deleting vdpa devices.
>
> $ vdpa parentdev show
> vdpasim:
>supported_classes
> net
>
> $ vdpa parentdev show -jp
> {
> "show": {
> "vdpasim": {
> "supported_classes": {
> "net"
> }
> }
> }
>
> Create a vdpa device of type networking named as "foo2" from the parent device vdpasim:
>
> $ vdpa dev add parentdev vdpasim type net name foo2
>
> Show the newly created vdpa device by its name:
> $ vdpa dev show foo2
> foo2: type network parentdev vdpasimvendor_id0max_vqs2max_vq_size256
>
> $ vdpa dev show foo2 -jp
> {
> "dev": {
> "foo2": {
> "type": "network",
> "parentdev": "vdpasim",
> "vendor_id": 0,
> "max_vqs": 2,
> "max_vq_size": 256
> }
> }
> }
>
> Delete the vdpa device after its use:
> $ vdpa dev del foo2
>
> vdpa tool support by kernel:
> ----------------------------
> vdpa tool user interface will be supported by existing vdpa kernel framework,
> i.e.drivers/vdpa/vdpa.cIt services user command through a netlink interface.
>
> Each parent device registers supported callback operations with vdpa subsystem
> through which vdpa device(s) can be managed.
>
> FAQs:
> -----
> 1. Where does userspace vdpa tool reside which users can use? > Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to
> create vdpa net devices.
>
> 2. Why not create and delete vdpa device using sysfs/configfs? > Ans:
> (a) A device creation may involve passing one or more attributes. > Passing multiple attributes and returning error code and more verbose
> information for invalid attributes cannot be handled by sysfs/configfs.
>
> (b) netlink framework is rich that enables user space and kernel driver to
> provide nested attributes.
>
> (c) Exposing device specific file under sysfs without net namespace > awareness exposes details to multiple containers. Instead exposing > attributes via a netlink socket secures the communication channel with kernel.
>
> (d) netlink socket interface enables to run syscaller kernel tests. >
> 3. Why not useioctl()interface?
> Ans:ioctl()interface replicates the necessary plumbing which already
> exists through netlink socket.
>
> 4. What happens when one or more user created vdpa devices exist for a
> parent PCI VF or SF and such parent device is removed? > Ans: All user created vdpa devices are removed that belong to a parent.
>
> [1]
git://git.kernel.org/pub/scm/network/iproute2/iproute2-next.git
<http://git.kernel.org/pub/scm/network/iproute2/iproute2-next.git>
>
> Next steps:
> -----------
> (a) Post this patchset and iproute2/vdpa inclusion, remaining two drivers
> will be coverted to support vdpa tool instead of creating unmanaged default
> device on driver load.
> (b) More net specific parameters such as mac, mtu will be added. > (c) Features bits get and set interface will be added.Adding Yong Ji for sharing some thoughts from the view of userspace vDPA
device.
Quoted 8 lines
Thanks for adding me, Jason!
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But in vduse case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (callvdpa_register_device()).So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEandVDPA_CMD_DEV_DISABLEto make the config path more flexible.
Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of this.
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an VDUSE_GET_DEVICE_FD ioctl?
Thanks
Quoted 4 lines
Thanks,
Yongji
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 258 lines · Jason Wang
Quoted 250 lines · Yongji Xie
On Fri, Nov 27, 2020 at 11:53 AM Jason Wang <[email protected] <mailto:[email protected]>> wrote:
On 2020/11/12 下午2:39, Parav Pandit wrote:
> This patchset covers user requirements for managing existing vdpa devices,
> using a tool and its internal design notes for kernel drivers. >
> Background and user requirements:
> ----------------------------------
> (1) Currently VDPA device is created by driver when driver is loaded.
> However, user should have a choice when to create or not create a vdpa device
> for the underlying parent device.
>
> For example, mlx5 PCI VF and subfunction device supports multiple classes of
> device such netdev, vdpa, rdma. Howevever it is not required to always created
> vdpa device for such device.
>
> (2) In another use case, a device may support creating one or multiple vdpa
> device of same or different class such as net and block. > Creating vdpa devices at driver load time further limits this use case.
>
> (3) A user should be able to monitor and query vdpa queue level or device level
> statistics for a given vdpa device.
>
> (4) A user should be able to query what class of vdpa devices are supported
> by its parent device.
>
> (5) A user should be able to view supported features and negotiated features
> of the vdpa device.
>
> (6) A user should be able to create a vdpa device in vendor agnostic manner
> using single tool.
>
> Hence, it is required to have a tool through which user can create one or more
> vdpa devices from a parent device which addresses above user requirements.
>
> Example devices:
> ----------------
> +-----------+ +-----------+ +---------+ +--------+ +-----------+
> |vdpa dev 0 | |vdpa dev 1 | |rdma dev | |netdev | |vdpa dev 3 |
> |type=net | |type=block | |mlx5_0 | |ens3f0 | |type=net |
> +----+------+ +-----+-----+ +----+----+ +-----+--+ +----+------+
> | | | | |
> | | | | |
> +----+-----+ | +----+----+ | +----+----+
> | mlx5 +--------+ |mlx5 +-------+ |mlx5 |
> |pci vf 2 | |pci vf 4 | |pci sf 8 |
> |03:00:2 | |03:00.4 | |mlx5_sf.8|
> +----+-----+ +----+----+ +----+----+
> | | |
> | +----+-----+ |
> +----------------------+mlx5 +----------------+
> |pci pf 0 |
> |03:00.0 |
> +----------+
>
> vdpa tool:
> ----------
> vdpa tool is a tool to create, delete vdpa devices from a parent device. It is a
> tool that enables user to query statistics, features and may be more attributes
> in future.
>
> vdpa tool command draft:
> ------------------------
> (a) List parent devices which supports creating vdpa devices. > It also shows which class types supported by this parent device. > In below command example two parent devices support vdpa device creation.
> First is PCI VF whose bdf is 03.00:2.
> Second is PCI VF whose name is 03:00.4.
> Third is PCI SF whose name is mlx5_core.sf.8
>
> $ vdpa parentdev list
> vdpasim
>supported_classes
> net
> pci/0000:03.00:3
>supported_classes
> net block
> pci/0000:03.00:4
>supported_classes
> net block
> auxiliary/mlx5_core.sf.8
>supported_classes
> net
>
> (b) Now add a vdpa device of networking class and show the device. > $ vdpa dev add parentdev pci/0000:03.00:2 type net name foo0 $ vdpa dev show foo0
> foo0: parentdev pci/0000:03.00:2 type network parentdev vdpasimvendor_id0max_vqs2max_vq_size256
>
> (c) Show features of a vdpa device
> $ vdpa dev features show foo0
> supported
> iommu platform
> version 1
>
> (d) Dump vdpa device statistics
> $ vdpa dev stats show foo0
> kickdoorbells 10
> wqes 100
>
> (e) Now delete a vdpa device previously created.
> $ vdpa dev del foo0
>
> vdpa tool support in this patchset:
> -----------------------------------
> vdpa tool is created to create, delete and query vdpa devices. > examples:
> Show vdpa parent device that supports creating, deleting vdpa devices.
>
> $ vdpa parentdev show
> vdpasim:
>supported_classes
> net
>
> $ vdpa parentdev show -jp
> {
> "show": {
> "vdpasim": {
> "supported_classes": {
> "net"
> }
> }
> }
>
> Create a vdpa device of type networking named as "foo2" from the parent device vdpasim:
>
> $ vdpa dev add parentdev vdpasim type net name foo2
>
> Show the newly created vdpa device by its name:
> $ vdpa dev show foo2
> foo2: type network parentdev vdpasimvendor_id0max_vqs2max_vq_size256
>
> $ vdpa dev show foo2 -jp
> {
> "dev": {
> "foo2": {
> "type": "network",
> "parentdev": "vdpasim",
> "vendor_id": 0,
> "max_vqs": 2,
> "max_vq_size": 256
> }
> }
> }
>
> Delete the vdpa device after its use:
> $ vdpa dev del foo2
>
> vdpa tool support by kernel:
> ----------------------------
> vdpa tool user interface will be supported by existing vdpa kernel framework,
> i.e.drivers/vdpa/vdpa.cIt services user command through a netlink interface.
>
> Each parent device registers supported callback operations with vdpa subsystem
> through which vdpa device(s) can be managed.
>
> FAQs:
> -----
> 1. Where does userspace vdpa tool reside which users can use? > Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to
> create vdpa net devices.
>
> 2. Why not create and delete vdpa device using sysfs/configfs? > Ans:
> (a) A device creation may involve passing one or more attributes. > Passing multiple attributes and returning error code and more verbose
> information for invalid attributes cannot be handled by sysfs/configfs.
>
> (b) netlink framework is rich that enables user space and kernel driver to
> provide nested attributes.
>
> (c) Exposing device specific file under sysfs without net namespace > awareness exposes details to multiple containers. Instead exposing > attributes via a netlink socket secures the communication channel with kernel.
>
> (d) netlink socket interface enables to run syscaller kernel tests. >
> 3. Why not useioctl()interface?
> Ans:ioctl()interface replicates the necessary plumbing which already
> exists through netlink socket.
>
> 4. What happens when one or more user created vdpa devices exist for a
> parent PCI VF or SF and such parent device is removed? > Ans: All user created vdpa devices are removed that belong to a parent.
>
> [1]
git://git.kernel.org/pub/scm/network/iproute2/iproute2-next.git
<http://git.kernel.org/pub/scm/network/iproute2/iproute2-next.git>
>
> Next steps:
> -----------
> (a) Post this patchset and iproute2/vdpa inclusion, remaining two drivers
> will be coverted to support vdpa tool instead of creating unmanaged default
> device on driver load.
> (b) More net specific parameters such as mac, mtu will be added. > (c) Features bits get and set interface will be added.Adding Yong Ji for sharing some thoughts from the view of userspace vDPA
device.Thanks for adding me, Jason!
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But in vduse case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (callvdpa_register_device()).Quoted 4 lines
So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEandVDPA_CMD_DEV_DISABLEto make the config path more flexible.
Quoted 6 lines
Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of this.
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?
Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
Thanks,
Yongji
Quoted 19 lines · Yongji Xie
Quoted 19 lines
Quoted 12 lines
Thanks for adding me, Jason!
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But in vduse case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (callvdpa_register_device()).Quoted 4 lines
So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEandVDPA_CMD_DEV_DISABLEto make the config path more flexible.Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of this.
Quoted 7 lines
Quoted 3 lines
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't opened.
Thanks
Quoted 4 lines
Thanks,
Yongji
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 31 lines · Jason Wang
Quoted 19 lines · Yongji Xie
Quoted 19 lines
Quoted 12 lines
Thanks for adding me, Jason!
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But in vduse case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (callvdpa_register_device()).Quoted 4 lines
So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEandVDPA_CMD_DEV_DISABLEto make the config path more flexible.Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of this.
Quoted 7 lines
Quoted 3 lines
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
Quoted 4 lines
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't opened.
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine to me.
Thanks,
Yongji
Quoted 39 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMQuoted 10 lines · Jason Wang
Quoted 7 lines · Yongji Xie
Quoted 7 lines
Quoted 7 lines
Thanks for adding me, Jason!
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But in vduse case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (callvdpa_register_device()).So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEand
Quoted 14 lines
Quoted 12 lines
Quoted 8 lines
Quoted 2 lines
VDPA_CMD_DEV_DISABLEto make the config path more flexible.Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of this.
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
Quoted 8 lines
Quoted 4 lines
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't opened.
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine to me.
I probably do not well understand. I tried reading patch [1] and few things do not look correct as below. Creating the vdpa device on the bus device and destroying the device from the workqueue seems unnecessary and racy.
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command, instead of connecting two sides separately and ensuring race free access to it.
So VDUSE_DEV_START and VDUSE_DEV_STOP should possibly be avoided.
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2
When above command is executed it creates necessary vdpa device foo2 on the bus.
When user binds foo2 device with the vduse driver, in the probe(), it creates respective char device to access it from user space.
Depending on which driver foo2 device is bound it, it can be used, either via (a) existing vhost stack or (b) some vdpa Netdev driver? (not sure its current state), or (c) vduse user space.
This will have sane model to me without races unless I am missing something fundamental here. This way there are not two ways to create vdpa devices from user space. Consumers can be of different types (vhost, vduse etc) of the bus device as above mentioned.
[1] https://www.spinics.net/lists/linux-mm/msg231581.html
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 49 lines · Parav Pandit
Quoted 40 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMQuoted 10 lines · Jason Wang
Quoted 7 lines · Yongji Xie
Quoted 7 lines
Quoted 7 lines
Thanks for adding me, Jason!
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But in vduse case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (callvdpa_register_device()).So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEand
Quoted 14 lines
Quoted 12 lines
Quoted 8 lines
Quoted 2 lines
VDPA_CMD_DEV_DISABLEto make the config path more flexible.Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of this.
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
Quoted 8 lines
Quoted 4 lines
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't opened.
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine to me.
Quoted 8 lines
I probably do not well understand. I tried reading patch [1] and few things do not look correct as below. Creating the vdpa device on the bus device and destroying the device from the workqueue seems unnecessary and racy.
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command, instead of connecting two sides separately and ensuring race free access to it.So
VDUSE_DEV_STARTandVDUSE_DEV_STOPshould possibly be avoided.
Yes, we can avoid these two ioctls with the help of the management tool.
Quoted 4 lines
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2When above command is executed it creates necessary vdpa device foo2 on the bus.
When user binds foo2 device with the vduse driver, in theprobe(), it creates respective char device to access it from user space.
But vduse driver is not a vdpa bus driver. It works like vdpasim driver, but offloads the data plane and control plane to a user space process.
Thanks,
Yongji
Quoted 72 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 7:49 PMQuoted 7 lines · Parav Pandit
Quoted 7 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMOn Tue, Dec 1, 2020 at 2:25 PM Jason Wang <[email protected]>
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines · Yongji Xie
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Thanks for adding me, Jason!
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But invduse
Quoted 22 lines
Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (call
vdpa_register_device()).So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEand
Quoted 14 lines
Quoted 12 lines
Quoted 8 lines
Quoted 2 lines
VDPA_CMD_DEV_DISABLEto make the config path more flexible.Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of this.
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't
opened.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine to me.I probably do not well understand. I tried reading patch [1] and few things
do not look correct as below.
Quoted 1 line
Creating the vdpa device on the bus device and destroying the device from
the workqueue seems unnecessary and racy.
Quoted 3 lines
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command,instead of connecting two sides separately and ensuring race free access to it.
Quoted 3 lines
So
VDUSE_DEV_STARTandVDUSE_DEV_STOPshould possibly be avoided.
Quoted 9 lines
Yes, we can avoid these two ioctls with the help of the management tool.
Quoted 3 lines
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2When above command is executed it creates necessary vdpa device foo2
on the bus.
Quoted 1 line
When user binds foo2 device with the vduse driver, in the
probe(), itcreates respective char device to access it from user space.
I see. So vduse cannot work with any existing vdpa devices like ifc, mlx5 or netdevsim. It has its own implementation similar to fuse with its own backend of choice. More below.
Quoted 2 lines
But vduse driver is not a vdpa bus driver. It works like vdpasim driver, but offloads the data plane and control plane to a user space process.
In that case to draw parallel lines,
1. netdevsim:
(a) create resources in kernel sw
(b) datapath simulates in kernel
2. ifc + mlx5 vdpa dev:
(a) creates resource in hw
(b) data path is in hw
3. vduse:
(a) creates resources in userspace sw
(b) data path is in user space.
hence creates data path resources for user space.
So char device is created, removed as result of vdpa device creation.
For example,
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2
Above command will create char device for user space.
Similar command for ifc/mlx5 would have created similar channel for rest of the config commands in hw. vduse channel = char device, eventfd etc.
ifc/mlx5 hw channel = bar, irq, command interface etc
Netdev sim channel = sw direct calls
Does it make sense?
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 108 lines · Parav Pandit
Quoted 73 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 7:49 PMQuoted 7 lines · Parav Pandit
Quoted 7 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMOn Tue, Dec 1, 2020 at 2:25 PM Jason Wang <[email protected]>
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines · Yongji Xie
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Thanks for adding me, Jason!
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But invduse
Quoted 22 lines
Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (call
vdpa_register_device()).So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEand
Quoted 14 lines
Quoted 12 lines
Quoted 8 lines
Quoted 2 lines
VDPA_CMD_DEV_DISABLEto make the config path more flexible.Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of this.
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't
opened.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine to me.I probably do not well understand. I tried reading patch [1] and few things
do not look correct as below.
Quoted 1 line
Creating the vdpa device on the bus device and destroying the device from
the workqueue seems unnecessary and racy.
Quoted 3 lines
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command,instead of connecting two sides separately and ensuring race free access to it.
Quoted 3 lines
So
VDUSE_DEV_STARTandVDUSE_DEV_STOPshould possibly be avoided.Quoted 9 lines
Yes, we can avoid these two ioctls with the help of the management tool.
Quoted 3 lines
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2When above command is executed it creates necessary vdpa device foo2
on the bus.
Quoted 1 line
When user binds foo2 device with the vduse driver, in the
probe(), itcreates respective char device to access it from user space.
I see. So vduse cannot work with any existing vdpa devices like ifc, mlx5 or netdevsim. It has its own implementation similar to fuse with its own backend of choice. More below.
Quoted 2 lines
But vduse driver is not a vdpa bus driver. It works like vdpasim driver, but offloads the data plane and control plane to a user space process.
In that case to draw parallel lines,
1. netdevsim:
(a) create resources in kernel sw
(b) datapath simulates in kernel2. ifc + mlx5 vdpa dev:
(a) creates resource in hw
(b) data path is in hw3. vduse:
(a) creates resources in userspace sw
(b) data path is in user space.
hence creates data path resources for user space.
So char device is created, removed as result of vdpa device creation.For example,
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2
Quoted 8 lines
Above command will create char device for user space.
Similar command for ifc/mlx5 would have created similar channel for rest of the config commands in hw. vduse channel = char device, eventfd etc.
ifc/mlx5 hw channel = bar, irq, command interface etc
Netdev sim channel = sw direct callsDoes it make sense?
In my understanding, to make vdpa work, we need a backend (datapath resources) and a frontend (a vdpa device attached to a vdpa bus). In the above example, it looks like we use the command "vdpa dev add ..."
to create a backend, so do we need another command to create a
frontend?
Thanks,
Yongji
Quoted 133 lines
From: Yongji Xie <[email protected]>
Sent: Wednesday, December 2, 2020 9:00 AMQuoted 33 lines · Parav Pandit
Quoted 33 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 7:49 PMQuoted 7 lines · Parav Pandit
Quoted 7 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMOn Tue, Dec 1, 2020 at 2:25 PM Jason Wang <[email protected]>
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines · Yongji Xie
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Thanks for adding me, Jason!
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But invduse
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (call
vdpa_register_device()).Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEand
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
VDPA_CMD_DEV_DISABLEto make the config path more flexible.Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of
this.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 1 line
Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't
Quoted 6 lines
Quoted 6 lines
opened.
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine tome.
Quoted 21 lines
Quoted 21 lines
Quoted 3 lines
I probably do not well understand. I tried reading patch [1] and few things
do not look correct as below.
Quoted 2 lines
Creating the vdpa device on the bus device and destroying the device from
the workqueue seems unnecessary and racy.
Quoted 4 lines
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command,instead of connecting two sides separately and ensuring race free access to it.
Quoted 3 lines
So
VDUSE_DEV_STARTandVDUSE_DEV_STOPshould possibly be avoided.Yes, we can avoid these two ioctls with the help of the management tool.
Quoted 2 lines
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
When above command is executed it creates necessary vdpa device foo2
on the bus.
Quoted 2 lines
When user binds foo2 device with the vduse driver, in the
probe(), itcreates respective char device to access it from user space.
I see. So vduse cannot work with any existing vdpa devices like ifc, mlx5 or
netdevsim.
Quoted 22 lines
It has its own implementation similar to fuse with its own backend of choice. More below.
Quoted 2 lines
But vduse driver is not a vdpa bus driver. It works like vdpasim driver, but offloads the data plane and control plane to a user space process.
In that case to draw parallel lines,
1. netdevsim:
(a) create resources in kernel sw
(b) datapath simulates in kernel2. ifc + mlx5 vdpa dev:
(a) creates resource in hw
(b) data path is in hw3. vduse:
(a) creates resources in userspace sw
(b) data path is in user space.
hence creates data path resources for user space.
So char device is created, removed as result of vdpa device creation.Quoted 6 lines
For example,
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Above command will create char device for user space.
Similar command for ifc/mlx5 would have created similar channel for rest of
the config commands in hw.
Quoted 4 lines
vduse channel = char device, eventfd etc.
ifc/mlx5 hw channel = bar, irq, command interface etc Netdev sim channel = sw direct calls
Quoted 7 lines
Quoted 1 line
Does it make sense?
In my understanding, to make vdpa work, we need a backend (datapath resources) and a frontend (a vdpa device attached to a vdpa bus). In the above example, it looks like we use the command "vdpa dev add ..."
to create a backend, so do we need another command to create a frontend?
For block device there is certainly some backend to process the IOs.
Sometimes backend to be setup first, before its front end is exposed.
"vdpa dev add" is the front end command who connects to the backend (implicitly) for network device.
vhost->vdpa_block_device->backend_io_processor (usr,hw,kernel).
And it needs a way to connect to backend when explicitly specified during creation time. Something like,
$ vdpa dev add parentdev vdpa_vduse type block name foo3 handle <uuid> In above example some vendor device specific unique handle is passed based on backend setup in hardware/user space.
In below 3 examples, vdpa block simulator is connecting to backend block or file.
$ vdpa dev add parentdev vdpa_blocksim type block name foo4 blockdev /dev/zero
$ vdpa dev add parentdev vdpa_blocksim type block name foo5 blockdev /dev/sda2 size=100M offset=10M
$ vdpa dev add parentdev vdpa_block filebackend_sim type block name foo6 file /root/file_backend.txt
Or may be backend connects to the created vdpa device is bound to the driver.
Can vduse attach to the created vdpa block device through the char device and establish the channel to receive IOs, and to setup the block config space?
Quoted 2 lines
Thanks,
Yongji
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 138 lines · Parav Pandit
Quoted 119 lines
From: Yongji Xie <[email protected]>
Sent: Wednesday, December 2, 2020 9:00 AMQuoted 30 lines · Parav Pandit
Quoted 30 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 7:49 PMQuoted 6 lines · Parav Pandit
Quoted 6 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMOn Tue, Dec 1, 2020 at 2:25 PM Jason Wang <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines · Yongji Xie
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Thanks for adding me, Jason!
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But invduse
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (call
vdpa_register_device()).Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEand
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
VDPA_CMD_DEV_DISABLEto make the config path more flexible.Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of
this.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't
opened.
Quoted 3 lines
Quoted 3 lines
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine tome.
Quoted 17 lines
Quoted 17 lines
Quoted 2 lines
I probably do not well understand. I tried reading patch [1] and few things
do not look correct as below.
Quoted 2 lines
Creating the vdpa device on the bus device and destroying the device from
the workqueue seems unnecessary and racy.
Quoted 3 lines
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command,instead of connecting two sides separately and ensuring race free access to it.
Quoted 2 lines
So
VDUSE_DEV_STARTandVDUSE_DEV_STOPshould possibly be avoided.Yes, we can avoid these two ioctls with the help of the management tool.
Quoted 2 lines
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
When above command is executed it creates necessary vdpa device foo2
on the bus.
Quoted 2 lines
When user binds foo2 device with the vduse driver, in the
probe(), itcreates respective char device to access it from user space.
I see. So vduse cannot work with any existing vdpa devices like ifc, mlx5 or
netdevsim.
Quoted 21 lines
It has its own implementation similar to fuse with its own backend of choice. More below.
Quoted 2 lines
But vduse driver is not a vdpa bus driver. It works like vdpasim driver, but offloads the data plane and control plane to a user space process.
In that case to draw parallel lines,
1. netdevsim:
(a) create resources in kernel sw
(b) datapath simulates in kernel2. ifc + mlx5 vdpa dev:
(a) creates resource in hw
(b) data path is in hw3. vduse:
(a) creates resources in userspace sw
(b) data path is in user space.
hence creates data path resources for user space.
So char device is created, removed as result of vdpa device creation.Quoted 6 lines
For example,
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Above command will create char device for user space.
Similar command for ifc/mlx5 would have created similar channel for rest of
the config commands in hw.
Quoted 4 lines
vduse channel = char device, eventfd etc.
ifc/mlx5 hw channel = bar, irq, command interface etc Netdev sim channel = sw direct callsQuoted 6 lines
Quoted 1 line
Does it make sense?
In my understanding, to make vdpa work, we need a backend (datapath resources) and a frontend (a vdpa device attached to a vdpa bus). In the above example, it looks like we use the command "vdpa dev add ..."
to create a backend, so do we need another command to create a frontend?For block device there is certainly some backend to process the IOs.
Sometimes backend to be setup first, before its front end is exposed.
"vdpa dev add" is the front end command who connects to the backend (implicitly) for network device.
vhost->vdpa_block_device->backend_io_processor(usr,hw,kernel).And it needs a way to connect to backend when explicitly specified during creation time. Something like,
$ vdpa dev add parentdev vdpa_vduse type block name foo3 handle <uuid> In above example some vendor device specific unique handle is passed based on backend setup in hardware/user space.In below 3 examples, vdpa block simulator is connecting to backend block or file.
Quoted 8 lines
$ vdpa dev add parentdev vdpa_blocksim type block name foo4 blockdev /dev/zero$ vdpa dev add parentdev vdpa_blocksim type block name foo5 blockdev /dev/sda2 size=100M offset=10M$ vdpa dev add parentdev vdpa_block filebackend_sim type block name foo6 file /root/file_backend.txtOr may be backend connects to the created vdpa device is bound to the driver.
Can vduse attach to the created vdpa block device through the char device and establish the channel to receive IOs, and to setup the block config space?
I think it can work.
Another thing I wonder it that, do we consider more than one VDUSE parentdev(or management dev)? This allows us to have separated devices implemented via different processes.
If yes, VDUSE ioctl needs to be extended to register/unregister parentdev.
Thanks
Quoted 3 lines
Quoted 3 lines
Thanks,
Yongji
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 173 lines
From: Jason Wang <[email protected]>
Sent: Wednesday, December 2, 2020 11:21 AMQuoted 11 lines · Parav Pandit
Quoted 11 lines
From: Yongji Xie <[email protected]>
Sent: Wednesday, December 2, 2020 9:00 AMQuoted 6 lines · Parav Pandit
Quoted 6 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 7:49 PMOn Tue, Dec 1, 2020 at 7:32 PM Parav Pandit <[email protected]>
Quoted 6 lines · wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMOn Tue, Dec 1, 2020 at 2:25 PM Jason Wang
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 11 lines · wrote:
Quoted 11 lines
Quoted 11 lines · Yongji Xie
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Thanks for adding me, Jason!
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But invduse
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (call
vdpa_register_device()).Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VDPA_CMD_DEV_DISABLEto make the config path moreflexible.
Quoted 13 lines
Quoted 13 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of
this.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't
opened.
Quoted 2 lines
Quoted 2 lines
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, thenuse
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the name/vduse_id as a attribute to create vdpa device? It looks fine to
me.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
I probably do not well understand. I tried reading patch [1] and few things
do not look correct as below.
Quoted 2 lines
Creating the vdpa device on the bus device and destroying the device from
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
the workqueue seems unnecessary and racy.
Quoted 3 lines
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command,instead of connecting two sides separately and ensuring race free access to it.
Quoted 1 line
So
VDUSE_DEV_STARTandVDUSE_DEV_STOPshould possibly beavoided.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Yes, we can avoid these two ioctls with the help of the management
tool.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2When above command is executed it creates necessary vdpa device foo2
on the bus.
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
When user binds foo2 device with the vduse driver, in the
probe(), itcreates respective char device to access it from user space.
I see. So vduse cannot work with any existing vdpa devices like ifc, mlx5 or
netdevsim.
Quoted 1 line
It has its own implementation similar to fuse with its own backend of
choice.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
More below.
Quoted 2 lines
But vduse driver is not a vdpa bus driver. It works like vdpasim driver, but offloads the data plane and control plane to a user space
process.
Quoted 28 lines
Quoted 28 lines
Quoted 16 lines
In that case to draw parallel lines,
1. netdevsim:
(a) create resources in kernel sw
(b) datapath simulates in kernel2. ifc + mlx5 vdpa dev:
(a) creates resource in hw
(b) data path is in hw3. vduse:
(a) creates resources in userspace sw
(b) data path is in user space.
hence creates data path resources for user space.
So char device is created, removed as result of vdpa device creation.Quoted 7 lines
For example,
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Above command will create char device for user space.
Similar command for ifc/mlx5 would have created similar channel for rest of
the config commands in hw.
Quoted 4 lines
vduse channel = char device, eventfd etc.
ifc/mlx5 hw channel = bar, irq, command interface etc Netdev sim channel = sw direct callsQuoted 5 lines
Quoted 5 lines
Quoted 1 line
Does it make sense?
In my understanding, to make vdpa work, we need a backend (datapath resources) and a frontend (a vdpa device attached to a vdpa bus). In the above example, it looks like we use the command "vdpa dev add ..."
to create a backend, so do we need another command to create afrontend?
Quoted 4 lines
Quoted 1 line
For block device there is certainly some backend to process the IOs. Sometimes backend to be setup first, before its front end is exposed. "vdpa dev add" is the front end command who connects to the backend
(implicitly) for network device.
Quoted 4 lines
vhost->vdpa_block_device->backend_io_processor(usr,hw,kernel).And it needs a way to connect to backend when explicitly specified during
creation time.
Quoted 2 lines
Something like,
$ vdpa dev add parentdev vdpa_vduse type block name foo3 handle<uuid>
Quoted 1 line
In above example some vendor device specific unique handle is passed
based on backend setup in hardware/user space.
Quoted 2 lines
In below 3 examples, vdpa block simulator is connecting to backend block
or file.
Quoted 11 lines
$ vdpa dev add parentdev vdpa_blocksim type block name foo4 blockdev /dev/zero$ vdpa dev add parentdev vdpa_blocksim type block name foo5 blockdev /dev/sda2 size=100M offset=10M$ vdpa dev add parentdev vdpa_block filebackend_sim type block name foo6 file /root/file_backend.txtOr may be backend connects to the created vdpa device is bound to the
driver.
Quoted 1 line
Can vduse attach to the created vdpa block device through the char device
Quoted 8 lines
and establish the channel to receive IOs, and to setup the block config space?
I think it can work.
Another thing I wonder it that, do we consider more than one VDUSE parentdev(or management dev)? This allows us to have separated devices implemented via different processes.
Multiple parentdev should be possible per one driver. for example mlx5_vdpa.ko will create multiple parent dev, one for each PCI VFs, SFs. vdpa dev add can certainly use one parent/mgmt dev to create multiple vdpa devices.
Not sure why do we need to create multiple parent dev for that.
I guess there is just one parent/mgmt. dev for VDUSE. What will each mgmtdev do differently?
Demux of IOs, events will be per individual char dev level?
Quoted 9 lines
If yes, VDUSE ioctl needs to be extended to register/unregister parentdev.
Thanks
Quoted 3 lines
Quoted 3 lines
Thanks,
Yongji
Quoted 165 lines · Parav Pandit
Quoted 165 lines
From: Jason Wang <[email protected]>
Sent: Wednesday, December 2, 2020 11:21 AMQuoted 9 lines · Parav Pandit
Quoted 9 lines
From: Yongji Xie <[email protected]>
Sent: Wednesday, December 2, 2020 9:00 AMQuoted 5 lines · Parav Pandit
Quoted 5 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 7:49 PMOn Tue, Dec 1, 2020 at 7:32 PM Parav Pandit <[email protected]>
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMOn Tue, Dec 1, 2020 at 2:25 PM Jason Wang
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 10 lines · Yongji Xie
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Thanks for adding me, Jason!
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But invduse
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (call
vdpa_register_device()).Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VDPA_CMD_DEV_DISABLEto make the config path moreflexible.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of
this.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't
opened.
Quoted 2 lines
Quoted 2 lines
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, thenuse
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the name/vduse_id as a attribute to create vdpa device? It looks fine to
me.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
I probably do not well understand. I tried reading patch [1] and few things
do not look correct as below.
Quoted 2 lines
Creating the vdpa device on the bus device and destroying the device from
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
the workqueue seems unnecessary and racy.
Quoted 3 lines
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command,instead of connecting two sides separately and ensuring race free access to it.
Quoted 1 line
So
VDUSE_DEV_STARTandVDUSE_DEV_STOPshould possibly beavoided.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Yes, we can avoid these two ioctls with the help of the management
tool.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2When above command is executed it creates necessary vdpa device foo2
on the bus.
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
When user binds foo2 device with the vduse driver, in the
probe(), itcreates respective char device to access it from user space.
I see. So vduse cannot work with any existing vdpa devices like ifc, mlx5 or
netdevsim.
Quoted 1 line
It has its own implementation similar to fuse with its own backend of
choice.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
More below.
Quoted 2 lines
But vduse driver is not a vdpa bus driver. It works like vdpasim driver, but offloads the data plane and control plane to a user space
process.
Quoted 28 lines
Quoted 28 lines
Quoted 16 lines
In that case to draw parallel lines,
1. netdevsim:
(a) create resources in kernel sw
(b) datapath simulates in kernel2. ifc + mlx5 vdpa dev:
(a) creates resource in hw
(b) data path is in hw3. vduse:
(a) creates resources in userspace sw
(b) data path is in user space.
hence creates data path resources for user space.
So char device is created, removed as result of vdpa device creation.Quoted 7 lines
For example,
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Above command will create char device for user space.
Similar command for ifc/mlx5 would have created similar channel for rest of
the config commands in hw.
Quoted 4 lines
vduse channel = char device, eventfd etc.
ifc/mlx5 hw channel = bar, irq, command interface etc Netdev sim channel = sw direct callsQuoted 5 lines
Quoted 5 lines
Quoted 1 line
Does it make sense?
In my understanding, to make vdpa work, we need a backend (datapath resources) and a frontend (a vdpa device attached to a vdpa bus). In the above example, it looks like we use the command "vdpa dev add ..."
to create a backend, so do we need another command to create afrontend?
Quoted 3 lines
For block device there is certainly some backend to process the IOs. Sometimes backend to be setup first, before its front end is exposed. "vdpa dev add" is the front end command who connects to the backend
(implicitly) for network device.
Quoted 3 lines
vhost->vdpa_block_device->backend_io_processor(usr,hw,kernel).And it needs a way to connect to backend when explicitly specified during
creation time.
Quoted 2 lines
Something like,
$ vdpa dev add parentdev vdpa_vduse type block name foo3 handle<uuid>
Quoted 1 line
In above example some vendor device specific unique handle is passed
based on backend setup in hardware/user space.
Quoted 1 line
In below 3 examples, vdpa block simulator is connecting to backend block
or file.
Quoted 10 lines
$ vdpa dev add parentdev vdpa_blocksim type block name foo4 blockdev /dev/zero$ vdpa dev add parentdev vdpa_blocksim type block name foo5 blockdev /dev/sda2 size=100M offset=10M$ vdpa dev add parentdev vdpa_block filebackend_sim type block name foo6 file /root/file_backend.txtOr may be backend connects to the created vdpa device is bound to the
driver.
Quoted 1 line
Can vduse attach to the created vdpa block device through the char device
and establish the channel to receive IOs, and to setup the block config space?
I think it can work.
Quoted 8 lines
Quoted 3 lines
Another thing I wonder it that, do we consider more than one VDUSE parentdev(or management dev)? This allows us to have separated devices implemented via different processes.
Multiple parentdev should be possible per one driver. for example mlx5_vdpa.ko will create multiple parent dev, one for each PCI VFs, SFs. vdpa dev add can certainly use one parent/mgmt dev to create multiple vdpa devices.
Not sure why do we need to create multiple parent dev for that.
I guess there is just one parent/mgmt. dev for VDUSE. What will each mgmtdev do differently?
Demux of IOs, events will be per individual char dev level?
It could be something like how it works for different hardware vendors. E.g IFCVF and mlx5 will register different parentdevs. For userspace, we need to allow different software vendors to manage their instances individually.
Thanks
Quoted 8 lines
Quoted 8 lines
If yes, VDUSE ioctl needs to be extended to register/unregister parentdev.
Thanks
Quoted 2 lines
Quoted 2 lines
Thanks,
Yongji
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 151 lines · Jason Wang
Quoted 138 lines · Parav Pandit
Quoted 119 lines
From: Yongji Xie <[email protected]>
Sent: Wednesday, December 2, 2020 9:00 AMQuoted 30 lines · Parav Pandit
Quoted 30 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 7:49 PMQuoted 6 lines · Parav Pandit
Quoted 6 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMOn Tue, Dec 1, 2020 at 2:25 PM Jason Wang <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines · Yongji Xie
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Thanks for adding me, Jason!
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But invduse
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (call
vdpa_register_device()).Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEand
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
VDPA_CMD_DEV_DISABLEto make the config path more flexible.Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of
this.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't
opened.
Quoted 3 lines
Quoted 3 lines
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine tome.
Quoted 17 lines
Quoted 17 lines
Quoted 2 lines
I probably do not well understand. I tried reading patch [1] and few things
do not look correct as below.
Quoted 2 lines
Creating the vdpa device on the bus device and destroying the device from
the workqueue seems unnecessary and racy.
Quoted 3 lines
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command,instead of connecting two sides separately and ensuring race free access to it.
Quoted 2 lines
So
VDUSE_DEV_STARTandVDUSE_DEV_STOPshould possibly be avoided.Yes, we can avoid these two ioctls with the help of the management tool.
Quoted 2 lines
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
When above command is executed it creates necessary vdpa device foo2
on the bus.
Quoted 2 lines
When user binds foo2 device with the vduse driver, in the
probe(), itcreates respective char device to access it from user space.
I see. So vduse cannot work with any existing vdpa devices like ifc, mlx5 or
netdevsim.
Quoted 21 lines
It has its own implementation similar to fuse with its own backend of choice. More below.
Quoted 2 lines
But vduse driver is not a vdpa bus driver. It works like vdpasim driver, but offloads the data plane and control plane to a user space process.
In that case to draw parallel lines,
1. netdevsim:
(a) create resources in kernel sw
(b) datapath simulates in kernel2. ifc + mlx5 vdpa dev:
(a) creates resource in hw
(b) data path is in hw3. vduse:
(a) creates resources in userspace sw
(b) data path is in user space.
hence creates data path resources for user space.
So char device is created, removed as result of vdpa device creation.Quoted 6 lines
For example,
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Above command will create char device for user space.
Similar command for ifc/mlx5 would have created similar channel for rest of
the config commands in hw.
Quoted 4 lines
vduse channel = char device, eventfd etc.
ifc/mlx5 hw channel = bar, irq, command interface etc Netdev sim channel = sw direct callsQuoted 6 lines
Quoted 1 line
Does it make sense?
In my understanding, to make vdpa work, we need a backend (datapath resources) and a frontend (a vdpa device attached to a vdpa bus). In the above example, it looks like we use the command "vdpa dev add ..."
to create a backend, so do we need another command to create a frontend?For block device there is certainly some backend to process the IOs.
Sometimes backend to be setup first, before its front end is exposed.
"vdpa dev add" is the front end command who connects to the backend (implicitly) for network device.
vhost->vdpa_block_device->backend_io_processor(usr,hw,kernel).And it needs a way to connect to backend when explicitly specified during creation time. Something like,
$ vdpa dev add parentdev vdpa_vduse type block name foo3 handle <uuid> In above example some vendor device specific unique handle is passed based on backend setup in hardware/user space.In below 3 examples, vdpa block simulator is connecting to backend block or file.
Quoted 8 lines
$ vdpa dev add parentdev vdpa_blocksim type block name foo4 blockdev /dev/zero$ vdpa dev add parentdev vdpa_blocksim type block name foo5 blockdev /dev/sda2 size=100M offset=10M$ vdpa dev add parentdev vdpa_block filebackend_sim type block name foo6 file /root/file_backend.txtOr may be backend connects to the created vdpa device is bound to the driver.
Can vduse attach to the created vdpa block device through the char device and establish the channel to receive IOs, and to setup the block config space?
Quoted 8 lines
I think it can work.
Another thing I wonder it that, do we consider more than one VDUSE parentdev(or management dev)? This allows us to have separated devices implemented via different processes.
If yes, VDUSE ioctl needs to be extended to register/unregister parentdev.
Yes, we need to extend the ioctl to support that. Now we only have one parentdev represented by /dev/vduse.
Thanks,
Yongji
Quoted 136 lines · Parav Pandit
Quoted 136 lines
From: Yongji Xie <[email protected]>
Sent: Wednesday, December 2, 2020 9:00 AMQuoted 33 lines · Parav Pandit
Quoted 33 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 7:49 PMQuoted 7 lines · Parav Pandit
Quoted 7 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMOn Tue, Dec 1, 2020 at 2:25 PM Jason Wang <[email protected]>
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines · Yongji Xie
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Thanks for adding me, Jason!
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But invduse
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (call
vdpa_register_device()).Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEand
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
VDPA_CMD_DEV_DISABLEto make the config path more flexible.Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of
this.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 1 line
Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't
Quoted 6 lines
Quoted 6 lines
opened.
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine tome.
Quoted 21 lines
Quoted 21 lines
Quoted 3 lines
I probably do not well understand. I tried reading patch [1] and few things
do not look correct as below.
Quoted 2 lines
Creating the vdpa device on the bus device and destroying the device from
the workqueue seems unnecessary and racy.
Quoted 4 lines
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command,instead of connecting two sides separately and ensuring race free access to it.
Quoted 3 lines
So
VDUSE_DEV_STARTandVDUSE_DEV_STOPshould possibly be avoided.Yes, we can avoid these two ioctls with the help of the management tool.
Quoted 2 lines
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
When above command is executed it creates necessary vdpa device foo2
on the bus.
Quoted 2 lines
When user binds foo2 device with the vduse driver, in the
probe(), itcreates respective char device to access it from user space.
I see. So vduse cannot work with any existing vdpa devices like ifc, mlx5 or
netdevsim.
Quoted 22 lines
It has its own implementation similar to fuse with its own backend of choice. More below.
Quoted 2 lines
But vduse driver is not a vdpa bus driver. It works like vdpasim driver, but offloads the data plane and control plane to a user space process.
In that case to draw parallel lines,
1. netdevsim:
(a) create resources in kernel sw
(b) datapath simulates in kernel2. ifc + mlx5 vdpa dev:
(a) creates resource in hw
(b) data path is in hw3. vduse:
(a) creates resources in userspace sw
(b) data path is in user space.
hence creates data path resources for user space.
So char device is created, removed as result of vdpa device creation.Quoted 6 lines
For example,
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Above command will create char device for user space.
Similar command for ifc/mlx5 would have created similar channel for rest of
the config commands in hw.
Quoted 5 lines
vduse channel = char device, eventfd etc.
ifc/mlx5 hw channel = bar, irq, command interface etc Netdev sim channel = sw direct callsDoes it make sense?
Quoted 7 lines
Quoted 5 lines
In my understanding, to make vdpa work, we need a backend (datapath resources) and a frontend (a vdpa device attached to a vdpa bus). In the above example, it looks like we use the command "vdpa dev add ..."
to create a backend, so do we need another command to create a frontend?For block device there is certainly some backend to process the IOs. Sometimes backend to be setup first, before its front end is exposed.
Yes, the backend need to be setup firstly, this is vendor device specific, not vdpa specific.
Quoted 9 lines
"vdpa dev add" is the front end command who connects to the backend (implicitly) for network device.
vhost->vdpa_block_device->backend_io_processor(usr,hw,kernel).And it needs a way to connect to backend when explicitly specified during creation time. Something like,
$ vdpa dev add parentdev vdpa_vduse type block name foo3 handle <uuid> In above example some vendor device specific unique handle is passed based on backend setup in hardware/user space.
Yes, we can work like this. After we setup a backend through an anonymous inode(fd) from /dev/vduse, we can get a unique handle. Then use it to create a frontend which will connect to the specific backend.
Quoted 11 lines
In below 3 examples, vdpa block simulator is connecting to backend block or file.
$ vdpa dev add parentdev vdpa_blocksim type block name foo4 blockdev /dev/zero$ vdpa dev add parentdev vdpa_blocksim type block name foo5 blockdev /dev/sda2 size=100M offset=10M$ vdpa dev add parentdev vdpa_block filebackend_sim type block name foo6 file /root/file_backend.txtOr may be backend connects to the created vdpa device is bound to the driver.
Can vduse attach to the created vdpa block device through the char device and establish the channel to receive IOs, and to setup the block config space?
How to create the vdpa block device? If we use the command "vdpa dev add..", the command will hang there until a vduse process attaches to the vdpa block device.
Thanks,
Yongji
Quoted 179 lines
From: Yongji Xie <[email protected]>
Sent: Wednesday, December 2, 2020 2:52 PMQuoted 14 lines · Parav Pandit
Quoted 14 lines
From: Yongji Xie <[email protected]>
Sent: Wednesday, December 2, 2020 9:00 AMQuoted 7 lines · Parav Pandit
Quoted 7 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 7:49 PMOn Tue, Dec 1, 2020 at 7:32 PM Parav Pandit <[email protected]>
Quoted 28 lines · wrote:
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 8 lines
Quoted 8 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMOn Tue, Dec 1, 2020 at 2:25 PM Jason Wang
<[email protected]>Quoted 11 lines · wrote:
Quoted 11 lines
Quoted 11 lines · Yongji Xie
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Thanks for adding me, Jason!
Now I'm working on a v2 patchset for VDUSE (vDPA Device in
Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But invduse
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (call
vdpa_register_device()).Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
So I'd like to know whether it's possible (or have some
plans) to add two new netlink msgs something like:VDPA_CMD_DEV_ENABLEand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VDPA_CMD_DEV_DISABLEto make the config path moreflexible.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of
this.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate
step.
Quoted 24 lines
Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The fd should be binded to a name or something else which need to be configured before.
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't
Quoted 7 lines
Quoted 7 lines
opened.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine tome.
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
I probably do not well understand. I tried reading patch [1] and few things
do not look correct as below.
Quoted 2 lines
Creating the vdpa device on the bus device and destroying the device from
the workqueue seems unnecessary and racy.
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command,instead of connecting two sides separately and ensuring race free access to it.
Quoted 2 lines
So
VDUSE_DEV_STARTandVDUSE_DEV_STOPshould possibly beavoided.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Yes, we can avoid these two ioctls with the help of the management
tool.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2When above command is executed it creates necessary vdpa device
foo2on the bus.
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
When user binds foo2 device with the vduse driver, in the
probe(), itcreates respective char device to access it from user space.
I see. So vduse cannot work with any existing vdpa devices like ifc, mlx5 or
netdevsim.
Quoted 1 line
It has its own implementation similar to fuse with its own backend of
choice.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
More below.
Quoted 2 lines
But vduse driver is not a vdpa bus driver. It works like vdpasim driver, but offloads the data plane and control plane to a user space
process.
Quoted 29 lines
Quoted 29 lines
Quoted 17 lines
In that case to draw parallel lines,
1. netdevsim:
(a) create resources in kernel sw
(b) datapath simulates in kernel2. ifc + mlx5 vdpa dev:
(a) creates resource in hw
(b) data path is in hw3. vduse:
(a) creates resources in userspace sw
(b) data path is in user space.
hence creates data path resources for user space.
So char device is created, removed as result of vdpa device creation.Quoted 7 lines
For example,
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Above command will create char device for user space.
Similar command for ifc/mlx5 would have created similar channel for rest of
the config commands in hw.
Quoted 4 lines
vduse channel = char device, eventfd etc.
ifc/mlx5 hw channel = bar, irq, command interface etc Netdev sim channel = sw direct callsQuoted 6 lines
Quoted 6 lines
Quoted 1 line
Does it make sense?
In my understanding, to make vdpa work, we need a backend (datapath resources) and a frontend (a vdpa device attached to a vdpa bus). In the above example, it looks like we use the command "vdpa dev add ..."
to create a backend, so do we need another command to create afrontend?
Quoted 3 lines
Quoted 1 line
For block device there is certainly some backend to process the IOs. Sometimes backend to be setup first, before its front end is exposed.
Yes, the backend need to be setup firstly, this is vendor device specific, not vdpa specific.
Quoted 1 line
"vdpa dev add" is the front end command who connects to the backend
(implicitly) for network device.
Quoted 4 lines
vhost->vdpa_block_device->backend_io_processor(usr,hw,kernel).And it needs a way to connect to backend when explicitly specified during
creation time.
Quoted 2 lines
Something like,
$ vdpa dev add parentdev vdpa_vduse type block name foo3 handle
Quoted 8 lines
<uuid>
Quoted 1 line
In above example some vendor device specific unique handle is passed
based on backend setup in hardware/user space.
Quoted 1 line
Yes, we can work like this. After we setup a backend through an anonymous inode(fd) from
/dev/vduse, we can get a unique handle. Then use it to create a frontend which will connect to the specific backend.
I do not fully understand the inode. But I assume this is some unique handle say uuid or something that both sides backend and vdpa device understand. It cannot be some kernel internal handle expose to user space.
Quoted 14 lines
Quoted 2 lines
In below 3 examples, vdpa block simulator is connecting to backend block
or file.
Quoted 11 lines
$ vdpa dev add parentdev vdpa_blocksim type block name foo4 blockdev /dev/zero$ vdpa dev add parentdev vdpa_blocksim type block name foo5 blockdev /dev/sda2 size=100M offset=10M$ vdpa dev add parentdev vdpa_block filebackend_sim type block name foo6 file /root/file_backend.txtOr may be backend connects to the created vdpa device is bound to the
Quoted 8 lines
driver.
Quoted 1 line
Can vduse attach to the created vdpa block device through the char device
and establish the channel to receive IOs, and to setup the block config space?
Quoted 1 line
How to create the vdpa block device? If we use the command "vdpa dev add..", the command will hang there until a vduse process attaches to the vdpa block device.
I was suggesting that vdpa device is created, but it doesn’t have backend attached to it.
It is attached to the backend when ioctl() side does enough setup. This state is handled internally the vduse driver.
But the above method of preparing backend looks more sane.
Regardless of which method is preferred, vduse driver must need a state to detach the vdpa bus device queues etc from the user space. This is needed because user space process can terminate anytime resulting in detaching dpa bus device in_use by the vhost side.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 185 lines · Parav Pandit
Quoted 185 lines
From: Yongji Xie <[email protected]>
Sent: Wednesday, December 2, 2020 2:52 PMQuoted 14 lines · Parav Pandit
Quoted 14 lines
From: Yongji Xie <[email protected]>
Sent: Wednesday, December 2, 2020 9:00 AMQuoted 7 lines · Parav Pandit
Quoted 7 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 7:49 PMOn Tue, Dec 1, 2020 at 7:32 PM Parav Pandit <[email protected]>
Quoted 28 lines · wrote:
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 8 lines
Quoted 8 lines
From: Yongji Xie <[email protected]>
Sent: Tuesday, December 1, 2020 3:26 PMOn Tue, Dec 1, 2020 at 2:25 PM Jason Wang
<[email protected]>Quoted 11 lines · wrote:
Quoted 11 lines
Quoted 11 lines · Yongji Xie
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Thanks for adding me, Jason!
Now I'm working on a v2 patchset for VDUSE (vDPA Device in
Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But invduse
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (call
vdpa_register_device()).Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
So I'd like to know whether it's possible (or have some
plans) to add two new netlink msgs something like:VDPA_CMD_DEV_ENABLEand
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
VDPA_CMD_DEV_DISABLEto make the config path moreflexible.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of
this.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate
step.
Quoted 24 lines
Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
The fd should be binded to a name or something else which need to be configured before.
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't
Quoted 7 lines
Quoted 7 lines
opened.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine tome.
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
I probably do not well understand. I tried reading patch [1] and few things
do not look correct as below.
Quoted 2 lines
Creating the vdpa device on the bus device and destroying the device from
the workqueue seems unnecessary and racy.
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
It seems vduse driver needs
This is something should be done as part of the vdpa dev add command,instead of connecting two sides separately and ensuring race free access to it.
Quoted 2 lines
So
VDUSE_DEV_STARTandVDUSE_DEV_STOPshould possibly beavoided.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Yes, we can avoid these two ioctls with the help of the management
tool.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2When above command is executed it creates necessary vdpa device
foo2on the bus.
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
When user binds foo2 device with the vduse driver, in the
probe(), itcreates respective char device to access it from user space.
I see. So vduse cannot work with any existing vdpa devices like ifc, mlx5 or
netdevsim.
Quoted 1 line
It has its own implementation similar to fuse with its own backend of
choice.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
More below.
Quoted 2 lines
But vduse driver is not a vdpa bus driver. It works like vdpasim driver, but offloads the data plane and control plane to a user space
process.
Quoted 29 lines
Quoted 29 lines
Quoted 17 lines
In that case to draw parallel lines,
1. netdevsim:
(a) create resources in kernel sw
(b) datapath simulates in kernel2. ifc + mlx5 vdpa dev:
(a) creates resource in hw
(b) data path is in hw3. vduse:
(a) creates resources in userspace sw
(b) data path is in user space.
hence creates data path resources for user space.
So char device is created, removed as result of vdpa device creation.Quoted 7 lines
For example,
$ vdpa dev add parentdev vduse_mgmtdev type net name foo2Above command will create char device for user space.
Similar command for ifc/mlx5 would have created similar channel for rest of
the config commands in hw.
Quoted 4 lines
vduse channel = char device, eventfd etc.
ifc/mlx5 hw channel = bar, irq, command interface etc Netdev sim channel = sw direct callsQuoted 6 lines
Quoted 6 lines
Quoted 1 line
Does it make sense?
In my understanding, to make vdpa work, we need a backend (datapath resources) and a frontend (a vdpa device attached to a vdpa bus). In the above example, it looks like we use the command "vdpa dev add ..."
to create a backend, so do we need another command to create afrontend?
Quoted 3 lines
Quoted 1 line
For block device there is certainly some backend to process the IOs. Sometimes backend to be setup first, before its front end is exposed.
Yes, the backend need to be setup firstly, this is vendor device specific, not vdpa specific.
Quoted 1 line
"vdpa dev add" is the front end command who connects to the backend
(implicitly) for network device.
Quoted 4 lines
vhost->vdpa_block_device->backend_io_processor(usr,hw,kernel).And it needs a way to connect to backend when explicitly specified during
creation time.
Quoted 2 lines
Something like,
$ vdpa dev add parentdev vdpa_vduse type block name foo3 handle<uuid>
Quoted 1 line
In above example some vendor device specific unique handle is passed
based on backend setup in hardware/user space.
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Yes, we can work like this. After we setup a backend through an anonymous inode(fd) from
/dev/vduse, we can get a unique handle. Then use it to create a frontend which will connect to the specific backend.I do not fully understand the inode. But I assume this is some unique handle say uuid or something that both sides backend and vdpa device understand. It cannot be some kernel internal handle expose to user space.
Yes, the unique handle should be a user-defined stuff.
Quoted 22 lines
Quoted 14 lines
Quoted 2 lines
In below 3 examples, vdpa block simulator is connecting to backend block
or file.
Quoted 11 lines
$ vdpa dev add parentdev vdpa_blocksim type block name foo4 blockdev /dev/zero$ vdpa dev add parentdev vdpa_blocksim type block name foo5 blockdev /dev/sda2 size=100M offset=10M$ vdpa dev add parentdev vdpa_block filebackend_sim type block name foo6 file /root/file_backend.txtOr may be backend connects to the created vdpa device is bound to the
Quoted 8 lines
driver.
Quoted 1 line
Can vduse attach to the created vdpa block device through the char device
and establish the channel to receive IOs, and to setup the block config space?
Quoted 1 line
How to create the vdpa block device? If we use the command "vdpa dev add..", the command will hang there until a vduse process attaches to the vdpa block device.
Quoted 7 lines
I was suggesting that vdpa device is created, but it doesn’t have backend attached to it.
It is attached to the backend whenioctl()side does enough setup. This state is handled internally the vduse driver.But the above method of preparing backend looks more sane.
Regardless of which method is preferred, vduse driver must need a state to detach the vdpa bus device queues etc from the user space. This is needed because user space process can terminate anytime resulting in detaching dpa bus device
in_useby the vhost side.
I think the vdpa device should only be detached by the command "vdpa dev del...". The vduse driver can support reconnecting when user space process is terminated.
Thanks,
Yongji
Quoted 30 lines · Yongji Xie
Quoted 29 lines · Jason Wang
Quoted 26 lines · Yongji Xie
Quoted 16 lines
Quoted 12 lines
Thanks for adding me, Jason!
Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) [1]. This tool is very useful for the vduse device. So I'm considering integrating this into my v2 patchset. But there is one problem:
In this tool, vdpa device config action and enable action are combined into one netlink msg:
VDPA_CMD_DEV_NEW. But in vduse case, it needs to be splitted because a chardev should be created and opened by a userspace process before we enable the vdpa device (callvdpa_register_device()).Quoted 4 lines
So I'd like to know whether it's possible (or have some plans) to add two new netlink msgs something like:
VDPA_CMD_DEV_ENABLEandVDPA_CMD_DEV_DISABLEto make the config path more flexible.Quoted 6 lines
Actually, we've discussed such intermediate step in some early discussion. It looks to me VDUSE could be one of the users of this.
Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE then fetching it via an
VDUSE_GET_DEVICE_FDioctl?Yes, we can. Actually the current implementation in VDUSE is like this. But seems like this is still a intermediate step. The fd should be binded to a name or something else which need to be configured before.
Quoted 8 lines
Quoted 4 lines
The name could be specified via the netlink. It looks to me the real issue is that until the device is connected with a userspace, it can't be used. So we also need to fail the enabling if it doesn't opened.
Yes, that's true. So you mean we can firstly try to fetch the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD, then use the name/vduse_id as a attribute to create vdpa device? It looks fine to me.
Yes, something like this. The anonymous fd will be created during dev_add() and the fd will be carried in the msg to userspace.
Thanks
Quoted 4 lines
Thanks,
Yongji
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 3 lines · Jason Wang
1. Where does userspace vdpa tool reside which users can use?
Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to create vdpa net devices.
iproute2 package is fine with us, but there are some expectations:
syntax, command options and documentation need to be consistent with other iproute2 commands (this thread suggests it will be but just being clear), and it needs to re-use code as much as possible (e.g., json functions). If there is overlap with other tools (devlink, dcb, etc), you should refactor into common code used by all. Petr Machata has done this quite a bit for dcb and is a good example to follow.
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Hi David,
Quoted 8 lines
From: David Ahern <[email protected]>
Sent: Wednesday, December 9, 2020 4:17 AMQuoted 3 lines · Jason Wang
1. Where does userspace vdpa tool reside which users can use?
Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to create vdpa net devices.
Quoted 7 lines
iproute2 package is fine with us, but there are some expectations:
syntax, command options and documentation need to be consistent with other iproute2 commands (this thread suggests it will be but just being clear), and it needs to re-use code as much as possible (e.g., json functions). If there is overlap with other tools (devlink, dcb, etc), you should refactor into common code used by all. Petr Machata has done this quite a bit for dcb and is a good example to follow.
Sorry for my late reply. I missed your message until yesterday.
Thanks for the ack and inputs.
Yes, I migrated the iproute2/vdpa to use now uses the common code introduced by dcb tool. Waiting for kernel side to finish.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 2 lines · Parav Pandit
This patchset covers user requirements for managing existing vdpa devices, using a tool and its internal design notes for kernel drivers.
I applied bugfix patches 1 and 2.
Others conflict with vdpa sim block support, pls rebase.
Quoted 197 lines
Background and user requirements:
----------------------------------
(1) Currently VDPA device is created by driver when driver is loaded. However, user should have a choice when to create or not create a vdpa device for the underlying parent device.For example, mlx5 PCI VF and subfunction device supports multiple classes of device such netdev, vdpa, rdma. Howevever it is not required to always created vdpa device for such device.
(2) In another use case, a device may support creating one or multiple vdpa device of same or different class such as net and block. Creating vdpa devices at driver load time further limits this use case.
(3) A user should be able to monitor and query vdpa queue level or device level statistics for a given vdpa device.
(4) A user should be able to query what class of vdpa devices are supported by its parent device.
(5) A user should be able to view supported features and negotiated features of the vdpa device.
(6) A user should be able to create a vdpa device in vendor agnostic manner using single tool.
Hence, it is required to have a tool through which user can create one or more vdpa devices from a parent device which addresses above user requirements.
Example devices:
----------------+-----------+ +-----------+ +---------+ +--------+ +-----------+ |vdpa dev 0 | |vdpa dev 1 | |rdma dev | |netdev | |vdpa dev 3 | |type=net | |type=block | |mlx5_0 | |ens3f0 | |type=net | +----+------+ +-----+-----+ +----+----+ +-----+--+ +----+------+ | | | | | | | | | | +----+-----+ | +----+----+ | +----+----+ | mlx5 +--------+ |mlx5 +-------+ |mlx5 | |pci vf 2 | |pci vf 4 | |pci sf 8 | |03:00:2 | |03:00.4 | |mlx5_sf.8| +----+-----+ +----+----+ +----+----+ | | | | +----+-----+ | +----------------------+mlx5 +----------------+ |pci pf 0 | |03:00.0 | +----------+vdpa tool:
----------
vdpa tool is a tool to create, delete vdpa devices from a parent device. It is a tool that enables user to query statistics, features and may be more attributes in future.vdpa tool command draft:
------------------------
(a) List parent devices which supports creating vdpa devices. It also shows which class types supported by this parent device. In below command example two parent devices support vdpa device creation. First is PCI VF whose bdf is 03.00:2.
Second is PCI VF whose name is 03:00.4.
Third is PCI SF whose name is mlx5_core.sf.8$ vdpa parentdev list vdpasim supported_classes net pci/0000:03.00:3 supported_classes net block pci/0000:03.00:4 supported_classes net block auxiliary/mlx5_core.sf.8 supported_classes net(b) Now add a vdpa device of networking class and show the device.
$ vdpa dev add parentdev pci/0000:03.00:2 type net name foo0 $ vdpa dev show foo0 foo0: parentdev pci/0000:03.00:2 type network parentdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256(c) Show features of a vdpa device
$ vdpa dev features show foo0 supported iommu platform version 1(d) Dump vdpa device statistics
$ vdpa dev stats show foo0 kickdoorbells 10 wqes 100(e) Now delete a vdpa device previously created.
$ vdpa dev del foo0vdpa tool support in this patchset:
-----------------------------------
vdpa tool is created to create, delete and query vdpa devices. examples:
Show vdpa parent device that supports creating, deleting vdpa devices.$ vdpa parentdev show vdpasim: supported_classes net$ vdpa parentdev show -jp { "show": { "vdpasim": { "supported_classes": { "net" } } }Create a vdpa device of type networking named as "foo2" from the parent device vdpasim:
$ vdpa dev add parentdev vdpasim type net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network parentdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256$ vdpa dev show foo2 -jp { "dev": { "foo2": { "type": "network", "parentdev": "vdpasim", "vendor_id": 0, "max_vqs": 2, "max_vq_size": 256 } } }Delete the vdpa device after its use:
$ vdpa dev del foo2vdpa tool support by kernel:
----------------------------
vdpa tool user interface will be supported by existing vdpa kernel framework, i.e.drivers/vdpa/vdpa.cIt services user command through a netlink interface.Each parent device registers supported callback operations with vdpa subsystem through which vdpa device(s) can be managed.
FAQs:
-----
1. Where does userspace vdpa tool reside which users can use?
Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to create vdpa net devices.2. Why not create and delete vdpa device using sysfs/configfs? Ans:
(a) A device creation may involve passing one or more attributes. Passing multiple attributes and returning error code and more verbose information for invalid attributes cannot be handled by sysfs/configfs.(b) netlink framework is rich that enables user space and kernel driver to provide nested attributes.
(c) Exposing device specific file under sysfs without net namespace awareness exposes details to multiple containers. Instead exposing attributes via a netlink socket secures the communication channel with kernel.
(d) netlink socket interface enables to run syscaller kernel tests.
3. Why not use
ioctl()interface?
Ans:ioctl()interface replicates the necessary plumbing which already exists through netlink socket.4. What happens when one or more user created vdpa devices exist for a parent PCI VF or SF and such parent device is removed?
Ans: All user created vdpa devices are removed that belong to a parent.[1]
git://git.kernel.org/pub/scm/network/iproute2/iproute2-next.gitNext steps:
-----------
(a) Post this patchset and iproute2/vdpa inclusion, remaining two drivers will be coverted to support vdpa tool instead of creating unmanaged default device on driver load.
(b) More net specific parameters such as mac, mtu will be added.
(c) Features bits get and set interface will be added.Parav Pandit (7):
vdpa: Add missing comment for virtqueue count
vdpa: Use simpler version of ida allocation
vdpa: Extend routine to accept vdpa device name
vdpa: Define vdpa parent device, ops and a netlink interface
vdpa: Enable a user to add and delete a vdpa device
vdpa: Enable user to query vdpa device info vdpa/vdpa_sim: Enable user to create vdpasim net devices
Quoted 12 lines
drivers/vdpa/Kconfig | 1 + drivers/vdpa/ifcvf/ifcvf_main.c | 2 +- drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +- drivers/vdpa/vdpa.c | 511 +++++++++++++++++++++++++++++- drivers/vdpa/vdpa_sim/vdpa_sim.c | 81 ++++- include/linux/vdpa.h | 46 ++- include/uapi/linux/vdpa.h | 41 +++ 7 files changed, 660 insertions(+), 24 deletions(-) create mode 100644 include/uapi/linux/vdpa.hSignature · 1 line
2.26.2
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
v2 cover Introduce vdpa management tool
In reply to Parav PanditThis patchset covers user requirements for managing existing vdpa devices, using a tool and its internal design notes for kernel drivers.
Background and user requirements:
----------------------------------
(1) Currently VDPA device is created by driver when driver is loaded. However, user should have a choice when to create or not create a vdpa device for the underlying management device.
For example, mlx5 PCI VF and subfunction device supports multiple classes of device such netdev, vdpa, rdma. Howevever it is not required to always created vdpa device for such device.
(2) In another use case, a device may support creating one or multiple vdpa device of same or different class such as net and block. Creating vdpa devices at driver load time further limits this use case.
(3) A user should be able to monitor and query vdpa queue level or device level statistics for a given vdpa device.
(4) A user should be able to query what class of vdpa devices are supported by its management device.
(5) A user should be able to view supported features and negotiated features of the vdpa device.
(6) A user should be able to create a vdpa device in vendor agnostic manner using single tool.
Hence, it is required to have a tool through which user can create one or more vdpa devices from a management device which addresses above user requirements.
Example devices:
----------------
+-----------+ +-----------+ +---------+ +--------+ +-----------+
|vdpa dev 0 | |vdpa dev 1 | |rdma dev | |netdev | |vdpa dev 3 |
|type=net | |type=net | |mlx5_0 | |ens3f0 | |type=net |
+----+------+ +-----+-----+ +----+----+ +-----+--+ +----+------+
| | | | |
| | | | |
+----+-----+ | +----+----+ | +----+----+
| mlx5 +--------+ |mlx5 +-------+ |mlx5 |
|pci vf 2 | |pci vf 4 | |pci sf 8 |
|03:00:2 | |03:00.4 | |mlx5_sf.8|
+----+-----+ +----+----+ +----+----+
| | |
| +----+-----+ |
+----------------------+mlx5 +----------------+
|pci pf 0 |
|03:00.0 |
+----------+vdpa tool:
----------
vdpa tool is a tool to create, delete vdpa devices from a management device. It is a tool that enables user to query statistics, features and may be more attributes in future.
vdpa tool command draft:
------------------------
(a) List management devices which support creating vdpa devices. It also shows which class types supported by this management device. In below command example four management devices support vdpa device creation. First is PCI VF whose bdf is 03.00:2.
Second is PCI VF whose name is 03:00.4.
Third is PCI SF whose name is mlx5_core.sf.8
$ vdpa mgmtdev list
vdpasim_net
supported_classes
net
pci/0000:03.00:0
supported_classes
net
pci/0000:03.00:4
supported_classes
net
auxiliary/mlx5_core.sf.8
supported_classes
net(b) Now add a vdpa device of networking class and show the device.
$ vdpa dev add mgmtdev pci/0000:03.00:0 name foo0
$ vdpa dev show foo0 foo0: mgmtdev pci/0000:03.00:2 type network vendor_id 0 max_vqs 2 max_vq_size 256
(c) Show features of a vdpa device
$ vdpa dev features show foo0 supported iommu platform version 1
(d) Dump vdpa device statistics
$ vdpa dev stats show foo0 kickdoorbells 10 wqes 100
(e) Now delete a vdpa device previously created.
$ vdpa dev del foo0
vdpa tool support in this patchset:
-----------------------------------
vdpa tool is created to create, delete and query vdpa devices. examples:
Show vdpa management device that supports creating, deleting vdpa devices.
$ vdpa mgmtdev show
vdpasim_net:
supported_classes
net$ vdpa mgmtdev show -jp
{
"show": {
"vdpasim_net": {
"supported_classes": {
"net"
}
}
}Create a vdpa device of type networking named as "foo2" from the management device vdpasim:
$ vdpa dev add mgmtdev vdpasim type net name foo2
Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256
$ vdpa dev show foo2 -jp
{
"dev": {
"foo2": {
"type": "network",
"mgmtdev": "vdpasim_net",
"vendor_id": 0,
"max_vqs": 2,
"max_vq_size": 256
}
}
}Delete the vdpa device after its use:
$ vdpa dev del foo2
vdpa tool support by kernel:
----------------------------
vdpa tool user interface will be supported by existing vdpa kernel framework, i.e. drivers/vdpa/vdpa.c It services user command through a netlink interface.
Each management device registers supported callback operations with vdpa subsystem through which vdpa device(s) can be managed.
Patch summary:
--------------
Patch-1 Makes mac address array static
Patch-2 adds module parameter to avoid breaking backward compatibility
for default device
Patch-3 Extends API to accept vdpa device name
Patch-4 Defines management device interface
Patch-5 Extends netlink interface to add, delete vdpa devices Patch-6 Extends netlink interface to query vdpa device attributes Patch-7 Extends vdpa_sim_net driver to add/delete simulated vdpa devices
FAQs:
-----
1. Where does userspace vdpa tool reside which users can use?
Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to handler vdpa network devices.
2. Why not create and delete vdpa device using sysfs/configfs? Ans:
(a) A device creation may involve passing one or more attributes. Passing multiple attributes and returning error code and more verbose information for invalid attributes cannot be handled by sysfs/configfs.
(b) netlink framework is rich that enables user space and kernel driver to provide nested attributes.
(c) Exposing device specific file under sysfs without net namespace awareness exposes details to multiple containers. Instead exposing attributes via a netlink socket secures the communication channel with kernel.
(d) netlink socket interface enables to run syscaller kernel tests.
3. Why not use ioctl() interface?
Ans: ioctl() interface replicates the necessary plumbing which already exists through netlink socket.
4. What happens when one or more user created vdpa devices exist for a management PCI VF or SF and such management device is removed?
Ans: All user created vdpa devices are removed that belong to a management device.
[1] git://git.kernel.org/pub/scm/network/iproute2/iproute2-next.git
Next steps:
-----------
(a) Post this patchset and iproute2/vdpa inclusion, remaining two drivers will be coverted to support vdpa tool instead of creating unmanaged default device on driver load.
(b) More net specific parameters such as mac, mtu will be added.
(c) Features bits get and set interface will be added.
Parav Pandit (7):
vdpa_sim_net: Make mac address array static
vdpa_sim_net: Add module param to disable default vdpa net device
vdpa: Extend routine to accept vdpa device name
vdpa: Define vdpa mgmt device, ops and a netlink interface
vdpa: Enable a user to add and delete a vdpa device
vdpa: Enable user to query vdpa device info
vdpa_sim_net: Add support for user supported devices
drivers/vdpa/Kconfig | 1 + drivers/vdpa/ifcvf/ifcvf_main.c | 2 +- drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +- drivers/vdpa/vdpa.c | 503 ++++++++++++++++++++++++++- drivers/vdpa/vdpa_sim/vdpa_sim.c | 3 +- drivers/vdpa/vdpa_sim/vdpa_sim.h | 2 + drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 135 ++++++- include/linux/vdpa.h | 44 ++- include/uapi/linux/vdpa.h | 40 +++ 9 files changed, 707 insertions(+), 25 deletions(-) create mode 100644 include/uapi/linux/vdpa.h
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v2 1/7 vdpa_sim_net: Make mac address array static
MAC address array is used only in vdpa_sim_net.c.
Hence, keep it static.
---
Changelog:v1->v2:
- new patch
--- drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +1 -1
@@ -33,7 +33,7 @@ static char *macaddr;module_param (macaddr, charp,0 );MODULE_PARM_DESC (macaddr,"Ethernet MAC address" ); -u8 macaddr_buf[ETH_ALEN ]; +static u8 macaddr_buf[ETH_ALEN ];static struct vdpasim *vdpasim_net_dev;
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH linux-next v2 1/7] vdpa_sim_net: Make mac address array static
Quoted 17 lines · Parav Pandit
MAC address array is used only in
vdpa_sim_net.c.
Hence, keep it static.Signed-off-by Parav Pandit <[email protected]>---
Changelog:v1->v2:
- new patch--- drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/vdpa/vdpa_sim/vdpa_sim_net.c +0
-0
drivers/vdpa/vdpa_sim/vdpa_sim_net.c static char *macaddr; · line 39
Quoted 8 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +1
-1@@ -33,7 +33,7 @@ static char *macaddr;module_param (macaddr, charp,0 );MODULE_PARM_DESC (macaddr,"Ethernet MAC address" ); -u8 macaddr_buf[ETH_ALEN ]; +static u8 macaddr_buf[ETH_ALEN ];static struct vdpasim *vdpasim_net_dev;
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v2 2/7 vdpa_sim_net: Add module param to disable default vdpa net device
In reply to Parav PanditTo support creating multiple vdpa devices and to allow user to manage them, add a knob to disable a default vdpa net device.
---
Changelog:v1->v2:
- new patch
--- drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 41 ++++++++++++++++++++-------- 1 file changed, 29 insertions(+), 12 deletions(-)
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +29 -12
@@ -33,6 +33,10 @@ static char *macaddr;module_param (macaddr, charp,0 );MODULE_PARM_DESC (macaddr,"Ethernet MAC address" ); +static bool default_device = true; +module_param (default_device, bool,0 ); +MODULE_PARM_DESC (default_device,"Support single default VDPA device" ); +static u8 macaddr_buf[ETH_ALEN ];static struct vdpasim *vdpasim_net_dev;
@@ -120,21 +124,11 @@ static void vdpasim_net_get_config(struct vdpasim *vdpasim, void *config)memcpy (net_config->mac, macaddr_buf,ETH_ALEN ); } -static int __init vdpasim_net_init (void ) +static int vdpasim_net_default_dev_register (void ) {struct vdpasim_dev_attr dev_attr = {};int ret; -if (macaddr) { -mac_pton (macaddr, macaddr_buf); -if (!is_valid_ether_addr (macaddr_buf)) { - ret = -EADDRNOTAVAIL ; - goto out; - } - }else { -eth_random_addr (macaddr_buf); - } - dev_attr.id =VIRTIO_ID_NET ; dev_attr.supported_features =VDPASIM_NET_FEATURES ; dev_attr.nvqs =VDPASIM_NET_VQ_NUM ;
@@ -161,13 +155,36 @@ static int __init vdpasim_net_init(void)return ret; } -static void __exit vdpasim_net_exit(void) +static void vdpasim_net_default_dev_unregister (void ) {struct vdpa_device *vdpa = &vdpasim_net_dev->vdpa;vdpa_unregister_device (vdpa); } +static int __init vdpasim_net_init (void ) +{ +int ret =0 ; + +if (macaddr) { +mac_pton (macaddr, macaddr_buf); +if (!is_valid_ether_addr (macaddr_buf)) +return -EADDRNOTAVAIL ; + }else { +eth_random_addr (macaddr_buf); + } + +if (default_device) + ret =vdpasim_net_default_dev_register (); +return ret; +} + +static void __exit vdpasim_net_exit(void) +{ +if (default_device) +vdpasim_net_default_dev_unregister (); +} +module_init (vdpasim_net_init);module_exit (vdpasim_net_exit);
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v2 3/7 vdpa: Extend routine to accept vdpa device name
In reply to Parav PanditIn a subsequent patch, when user initiated command creates a vdpa device, the user chooses the name of the vdpa device.
To support it, extend the device allocation API to consider this name specified by the caller driver.
---
Changelog:v1->v2:
- rebased
--- drivers/vdpa/ifcvf/ifcvf_main.c | 2 +- drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +- drivers/vdpa/vdpa.c | 36 +++++++++++++++++++++++++++---- drivers/vdpa/vdpa_sim/vdpa_sim.c | 2 +- include/linux/vdpa.h | 7 +++--- 5 files changed, 38 insertions(+), 11 deletions(-)
drivers/vdpa/ifcvf/ifcvf_main.c +1 -1
@@ -432,7 +432,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct pci_device_id *id) adapter = vdpa_alloc_device(struct ifcvf_adapter, vdpa, dev, &ifc_vdpa_ops, -IFCVF_MAX_QUEUE_PAIRS *2 ); +IFCVF_MAX_QUEUE_PAIRS *2 ,NULL );if (adapter ==NULL ) {IFCVF_ERR (pdev,"Failed to allocate vDPA structure" );return -ENOMEM ;
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -1946,7 +1946,7 @@ void *mlx5_vdpa_add_dev(struct mlx5_core_dev *mdev) max_vqs =min_t (u32, max_vqs,MLX5_MAX_SUPPORTED_VQS ); ndev = vdpa_alloc_device(struct mlx5_vdpa_net, mvdev.vdev, mdev->device, &mlx5_vdpa_ops, -2 *mlx5_vdpa_max_qps (max_vqs)); +2 *mlx5_vdpa_max_qps (max_vqs),NULL );if (IS_ERR (ndev))return ndev;
drivers/vdpa/vdpa.c +32 -4
@@ -12,6 +12,8 @@#include <linux/slab.h> #include <linux/vdpa.h> +/* A global mutex that protects vdpa management device and device level operations. */ +static DEFINE_MUTEX (vdpa_dev_mutex);static DEFINE_IDA (vdpa_index_ida);static int vdpa_dev_probe (struct device *d)
@@ -63,6 +65,7 @@ static void vdpa_release_dev(struct device *d)* @config: the bus operations that is supported by this device * @nvqs: number of virtqueues supported by this device * @size: size of the parent structure that contains private data +* @name: name of the vdpa device; optional. * * Driver should use vdpa_alloc_device() wrapper macro instead of * using this directly.
@@ -72,8 +75,7 @@ static void vdpa_release_dev(struct device *d)*/ struct vdpa_device *__vdpa_alloc_device (struct device *parent,const struct vdpa_config_ops *config, -int nvqs, -size_t size) +int nvqs,size_t size,const char *name) {struct vdpa_device *vdev;int err = -EINVAL ;
@@ -101,7 +103,10 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent, vdev->features_valid = false; vdev->nvqs = nvqs; - err =dev_set_name (&vdev->dev,"vdpa%u" , vdev->index); +if (name) + err =dev_set_name (&vdev->dev,"%s" , name); +else + err =dev_set_name (&vdev->dev,"vdpa%u" , vdev->index);if (err) goto err_name;
@@ -118,6 +123,13 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent, }EXPORT_SYMBOL_GPL (__vdpa_alloc_device); +static int vdpa_name_match (struct device *dev,const void *data) +{ + struct vdpa_device *vdev = container_of(dev, struct vdpa_device, dev); + +return (strcmp (dev_name (&vdev->dev), data) ==0 ); +} +/** * vdpa_register_device - register a vDPA device * Callers must have a succeed call of vdpa_alloc_device() before.
@@ -127,7 +139,21 @@ EXPORT_SYMBOL_GPL(__vdpa_alloc_device);*/ int vdpa_register_device (struct vdpa_device *vdev) { -return device_add (&vdev->dev); +struct device *dev; +int err; + +mutex_lock (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL ,dev_name (&vdev->dev), vdpa_name_match); +if (dev) { +put_device (dev); + err = -EEXIST ; + goto name_err; + } + + err =device_add (&vdev->dev); +name_err: +mutex_unlock (&vdpa_dev_mutex); +return err; }EXPORT_SYMBOL_GPL (vdpa_register_device);
@@ -137,7 +163,9 @@ EXPORT_SYMBOL_GPL(vdpa_register_device);*/ void vdpa_unregister_device (struct vdpa_device *vdev) { +mutex_lock (&vdpa_dev_mutex);device_unregister (&vdev->dev); +mutex_unlock (&vdpa_dev_mutex); }EXPORT_SYMBOL_GPL (vdpa_unregister_device);
drivers/vdpa/vdpa_sim/vdpa_sim.c +1 -1
@@ -235,7 +235,7 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr) ops = &vdpasim_config_ops; vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, - dev_attr->nvqs); + dev_attr->nvqs,NULL );if (!vdpasim) goto err_alloc;
include/linux/vdpa.h +3 -4
@@ -245,15 +245,14 @@ struct vdpa_config_ops {struct vdpa_device *__vdpa_alloc_device (struct device *parent,const struct vdpa_config_ops *config, -int nvqs, -size_t size); +int nvqs,size_t size,const char *name); -#define vdpa_alloc_device (dev_struct, member, parent, config, nvqs) \ +#define vdpa_alloc_device (dev_struct, member, parent, config, nvqs, name) \ container_of(__vdpa_alloc_device( \ parent, config, nvqs, \ sizeof(dev_struct) + \ BUILD_BUG_ON_ZERO(offsetof( \ - dev_struct, member))), \ + dev_struct, member)), name), \ dev_struct, member)int vdpa_register_device (struct vdpa_device *vdev);
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v2 4/7 vdpa: Define vdpa mgmt device, ops and a netlink interface
In reply to Parav PanditTo add one or more VDPA devices, define a management device which allows adding or removing vdpa device. A management device defines set of callbacks to manage vdpa devices.
To begin with, it defines add and remove callbacks through which a user defined vdpa device can be added or removed.
A unique management device is identified by its unique handle identified by management device name and optionally the bus name.
Hence, introduce routine through which driver can register a management device and its callback operations for adding and remove a vdpa device.
Introduce vdpa netlink socket family so that user can query management device and its attributes.
Example of show vdpa management device which allows creating vdpa device of networking class (device id = 0x1) of virtio specification 1.1 section 5.1.1.
$ vdpa mgmtdev show
vdpasim_net:
supported_classes:
netExample of showing vdpa management device in JSON format.
$ vdpa mgmtdev show -jp
{
"show": {
"vdpasim_net": {
"supported_classes": [ "net" ]
}
}
}---
Changelog:v1->v2:
- rebased
- updated commit log example for management device name from "vdpasim" to "vdpasim_net"
- removed device_id as net and block management devices are separated
- dev_add() return type is changed from struct vdpa_device to int
--- drivers/vdpa/Kconfig | 1 + drivers/vdpa/vdpa.c | 213 +++++++++++++++++++++++++++++++++++++- include/linux/vdpa.h | 31 ++++++ include/uapi/linux/vdpa.h | 31 ++++++ 4 files changed, 275 insertions(+), 1 deletion(-) create mode 100644 include/uapi/linux/vdpa.h
drivers/vdpa/Kconfig +1 -0
@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only menuconfig VDPA tristate "vDPA drivers" + depends on NET help Enable this module to support vDPA device that uses a datapath which complies with virtio specifications with
drivers/vdpa/vdpa.c +212 -1
236 lines not on this page · raw message
include/linux/vdpa.h +31 -0
@@ -35,6 +35,8 @@ struct vdpa_vq_state {u16 avail_index; }; +struct vdpa_mgmt_dev ; +/** * vDPA device - representation of a vDPA device * @dev: underlying device
@@ -335,4 +337,33 @@ static inline void vdpa_get_config(struct vdpa_device *vdev, unsigned offset, ops->get_config(vdev, offset, buf, len); } +/** +* vdpa_mgmtdev_ops - vdpa device ops +* @dev_add: Add a vdpa device using alloc and register +* @mdev: parent device to use for device addition +* @name: name of the new vdpa device +* Driver need to add a new device using _vdpa_register_device() +* after fully initializing the vdpa device. Driver must return 0 +* on success or appropriate error code. +* @dev_del: Remove a vdpa device using unregister +* @mdev: parent device to use for device removal +* @dev: vdpa device to remove +* Driver need to remove the specified device by calling +* _vdpa_unregister_device(). +*/ +struct vdpa_mgmtdev_ops { +int (*dev_add)(struct vdpa_mgmt_dev *mdev,const char *name); +void (*dev_del)(struct vdpa_mgmt_dev *mdev,struct vdpa_device *dev); +}; + +struct vdpa_mgmt_dev { +struct device *device; +const struct vdpa_mgmtdev_ops *ops; +const struct virtio_device_id *id_table;/* supported ids */ +struct list_head list; +}; + +int vdpa_mgmtdev_register (struct vdpa_mgmt_dev *mdev); +void vdpa_mgmtdev_unregister (struct vdpa_mgmt_dev *mdev); +#endif /* _LINUX_VDPA_H */
include/uapi/linux/vdpa.h new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +/* +* vdpa device management interface +* Copyright (c) 2020 Mellanox Technologies Ltd. All rights reserved. +*/ + +#ifndef _UAPI_LINUX_VDPA_H_ +#define _UAPI_LINUX_VDPA_H_ + +#define VDPA_GENL_NAME "vdpa" +#define VDPA_GENL_VERSION 0x1 + +enum vdpa_command { +VDPA_CMD_UNSPEC , +VDPA_CMD_MGMTDEV_NEW , +VDPA_CMD_MGMTDEV_GET ,/* can dump */ +}; + +enum vdpa_attr { +VDPA_ATTR_UNSPEC , + +/* bus name (optional) + dev name together make the parent device handle */ +VDPA_ATTR_MGMTDEV_BUS_NAME ,/* string */ +VDPA_ATTR_MGMTDEV_DEV_NAME ,/* string */ +VDPA_ATTR_MGMTDEV_SUPPORTED_CLASSES ,/* u64 */ + +/* new attributes must be added above here */ +VDPA_ATTR_MAX , +}; + +#endif
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH linux-next v2 4/7] vdpa: Define vdpa mgmt device, ops and a netlink interface
Quoted 40 lines · Parav Pandit
To add one or more VDPA devices, define a management device which allows adding or removing vdpa device. A management device defines set of callbacks to manage vdpa devices.
To begin with, it defines add and remove callbacks through which a user defined vdpa device can be added or removed.
A unique management device is identified by its unique handle identified by management device name and optionally the bus name.
Hence, introduce routine through which driver can register a management device and its callback operations for adding and remove a vdpa device.
Introduce vdpa netlink socket family so that user can query management device and its attributes.
Example of show vdpa management device which allows creating vdpa device of networking class (device id =
0x1) of virtio specification 1.1 section 5.1.1.$ vdpa mgmtdev show vdpasim_net: supported_classes: netExample of showing vdpa management device in JSON format.
$ vdpa mgmtdev show -jp { "show": { "vdpasim_net": { "supported_classes": [ "net" ] } } }Signed-off-by Parav Pandit<[email protected]>Reviewed-by Eli Cohen<[email protected]>Reviewed-by Jason Wang<[email protected]>
Quoted 7 lines
---
Changelog:v1->v2:
- rebased
- updated commit log example for management device name from "vdpasim" to "vdpasim_net"
- removeddevice_idas net and block management devices are separated
So I wonder whether there could be a type of management devices that can deal with multiple types of virtio devices. If yes, we probably need to add device id back.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v2 4/7] vdpa: Define vdpa mgmt device, ops and a netlink interface
Quoted 56 lines
From: Jason Wang <[email protected]>
Sent: Monday, January 4, 2021 12:33 PMQuoted 40 lines · Parav Pandit
To add one or more VDPA devices, define a management device which allows adding or removing vdpa device. A management device defines set of callbacks to manage vdpa devices.
To begin with, it defines add and remove callbacks through which a user defined vdpa device can be added or removed.
A unique management device is identified by its unique handle identified by management device name and optionally the bus name.
Hence, introduce routine through which driver can register a management device and its callback operations for adding and remove a vdpa device.
Introduce vdpa netlink socket family so that user can query management device and its attributes.
Example of show vdpa management device which allows creating vdpa device of networking class (device id =
0x1) of virtio specification 1.1 section 5.1.1.$ vdpa mgmtdev show vdpasim_net: supported_classes: netExample of showing vdpa management device in JSON format.
$ vdpa mgmtdev show -jp { "show": { "vdpasim_net": { "supported_classes": [ "net" ] } } }Signed-off-by Parav Pandit<[email protected]>Reviewed-by Eli Cohen<[email protected]>Reviewed-by Jason Wang<[email protected]>Quoted 8 lines
---
Changelog:v1->v2:
- rebased
- updated commit log example for management device name from "vdpasim" to "vdpasim_net"
- removeddevice_idas net and block management devices are
separated
Quoted 3 lines
So I wonder whether there could be a type of management devices that can deal with multiple types of virtio devices. If yes, we probably need to add device id back.
At this point mlx5 plan to support only net.
It is useful to see what type of vdpa device is supported by a management device.
In future if a mgmt dev supports multiple types, user needs to choose desired type. I guess we can differ this optional type to future, when such mgmt. device will/may be available.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH linux-next v2 4/7] vdpa: Define vdpa mgmt device, ops and a netlink interface
Quoted 54 lines · Parav Pandit
Quoted 54 lines
From: Jason Wang <[email protected]>
Sent: Monday, January 4, 2021 12:33 PMQuoted 48 lines · Parav Pandit
To add one or more VDPA devices, define a management device which allows adding or removing vdpa device. A management device defines set of callbacks to manage vdpa devices.
To begin with, it defines add and remove callbacks through which a user defined vdpa device can be added or removed.
A unique management device is identified by its unique handle identified by management device name and optionally the bus name.
Hence, introduce routine through which driver can register a management device and its callback operations for adding and remove a vdpa device.
Introduce vdpa netlink socket family so that user can query management device and its attributes.
Example of show vdpa management device which allows creating vdpa device of networking class (device id =
0x1) of virtio specification 1.1 section 5.1.1.$ vdpa mgmtdev show vdpasim_net: supported_classes: netExample of showing vdpa management device in JSON format.
$ vdpa mgmtdev show -jp { "show": { "vdpasim_net": { "supported_classes": [ "net" ] } } }Signed-off-by Parav Pandit<[email protected]>Reviewed-by Eli Cohen<[email protected]>Reviewed-by Jason Wang<[email protected]>---
Changelog:v1->v2:
- rebased
- updated commit log example for management device name from "vdpasim" to "vdpasim_net"
- removeddevice_idas net and block management devices are
separated
Quoted 8 lines
Quoted 3 lines
So I wonder whether there could be a type of management devices that can deal with multiple types of virtio devices. If yes, we probably need to add device id back.
At this point mlx5 plan to support only net.
It is useful to see what type of vdpa device is supported by a management device.In future if a mgmt dev supports multiple types, user needs to choose desired type. I guess we can differ this optional type to future, when such mgmt. device will/may be available.
I worry if we remove device_id, it may gives a hint that multiple mgmt devices needs to be registered if it supports multiple types.
So if possible I would like to keep the device_id here.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v2 4/7] vdpa: Define vdpa mgmt device, ops and a netlink interface
Quoted 67 lines
From: Jason Wang <[email protected]>
Sent: Tuesday, January 5, 2021 9:40 AMQuoted 54 lines · Parav Pandit
Quoted 54 lines
From: Jason Wang <[email protected]>
Sent: Monday, January 4, 2021 12:33 PMQuoted 48 lines · Parav Pandit
To add one or more VDPA devices, define a management device which allows adding or removing vdpa device. A management device defines set of callbacks to manage vdpa devices.
To begin with, it defines add and remove callbacks through which a user defined vdpa device can be added or removed.
A unique management device is identified by its unique handle identified by management device name and optionally the bus name.
Hence, introduce routine through which driver can register a management device and its callback operations for adding and remove a vdpa device.
Introduce vdpa netlink socket family so that user can query management device and its attributes.
Example of show vdpa management device which allows creating vdpa device of networking class (device id =
0x1) of virtio specification 1.1 section 5.1.1.$ vdpa mgmtdev show vdpasim_net: supported_classes: netExample of showing vdpa management device in JSON format.
$ vdpa mgmtdev show -jp { "show": { "vdpasim_net": { "supported_classes": [ "net" ] } } }Signed-off-by Parav Pandit<[email protected]>Reviewed-by Eli Cohen<[email protected]>Reviewed-by Jason Wang<[email protected]>---
Changelog:v1->v2:
- rebased
- updated commit log example for management device name from "vdpasim" to "vdpasim_net"
- removeddevice_idas net and block management devices are
separatedQuoted 5 lines
Quoted 3 lines
So I wonder whether there could be a type of management devices that can deal with multiple types of virtio devices. If yes, we probably need to add device id back.
At this point mlx5 plan to support only net.
It is useful to see what type of vdpa device is supported by a managementdevice.
Quoted 1 line
Quoted 9 lines
Quoted 1 line
In future if a mgmt dev supports multiple types, user needs to choose
desired type.
Quoted 1 line
I guess we can differ this optional type to future, when such mgmt. device
will/may be available.
I worry if we remove
device_id, it may gives a hint that multiple mgmt devices needs to be registered if it supports multiple types.
No it shouldn't. because we do expose multiple supported types in mgmtdev attributes.
Quoted 2 lines
So if possible I would like to keep the
device_idhere.
Its possible to keep it. But with current drivers, mainly mlx5 and vdpa_sim, it is redundant. Not sure of the ifc's plan.
We have been splitting modules to handle net and block differently in mlx5 as well as vdpa_sim. So it looks to me that both may be separate management drivers (and management devices). Such as vdpasim_net and vdpasim_block.
mlx5 doesn't have plan for block yet.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH linux-next v2 4/7] vdpa: Define vdpa mgmt device, ops and a netlink interface
Quoted 65 lines · Parav Pandit
Quoted 65 lines
From: Jason Wang <[email protected]>
Sent: Tuesday, January 5, 2021 9:40 AMQuoted 52 lines · Parav Pandit
Quoted 52 lines
From: Jason Wang <[email protected]>
Sent: Monday, January 4, 2021 12:33 PMQuoted 48 lines · Parav Pandit
To add one or more VDPA devices, define a management device which allows adding or removing vdpa device. A management device defines set of callbacks to manage vdpa devices.
To begin with, it defines add and remove callbacks through which a user defined vdpa device can be added or removed.
A unique management device is identified by its unique handle identified by management device name and optionally the bus name.
Hence, introduce routine through which driver can register a management device and its callback operations for adding and remove a vdpa device.
Introduce vdpa netlink socket family so that user can query management device and its attributes.
Example of show vdpa management device which allows creating vdpa device of networking class (device id =
0x1) of virtio specification 1.1 section 5.1.1.$ vdpa mgmtdev show vdpasim_net: supported_classes: netExample of showing vdpa management device in JSON format.
$ vdpa mgmtdev show -jp { "show": { "vdpasim_net": { "supported_classes": [ "net" ] } } }Signed-off-by Parav Pandit<[email protected]>Reviewed-by Eli Cohen<[email protected]>Reviewed-by Jason Wang<[email protected]>---
Changelog:v1->v2:
- rebased
- updated commit log example for management device name from "vdpasim" to "vdpasim_net"
- removeddevice_idas net and block management devices are
separatedQuoted 5 lines
Quoted 3 lines
So I wonder whether there could be a type of management devices that can deal with multiple types of virtio devices. If yes, we probably need to add device id back.
At this point mlx5 plan to support only net.
It is useful to see what type of vdpa device is supported by a managementdevice.
Quoted 1 line
In future if a mgmt dev supports multiple types, user needs to choose
desired type.
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
I guess we can differ this optional type to future, when such mgmt. device
will/may be available.
I worry if we remove
device_id, it may gives a hint that multiple mgmt devices needs to be registered if it supports multiple types.No it shouldn't. because we do expose multiple supported types in mgmtdev attributes.
Right.
Quoted 9 lines
Quoted 3 lines
So if possible I would like to keep the
device_idhere.Its possible to keep it. But with current drivers, mainly mlx5 and
vdpa_sim, it is redundant. Not sure of the ifc's plan.
We have been splitting modules to handle net and block differently in mlx5 as well asvdpa_sim. So it looks to me that both may be separate management drivers (and management devices). Such asvdpasim_netandvdpasim_block.
mlx5 doesn't have plan for block yet.
Ok. Then it's fine.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v2 5/7 vdpa: Enable a user to add and delete a vdpa device
In reply to Parav PanditAdd the ability to add and delete a vdpa device.
Examples:
Create a vdpa device of type network named "foo2" from the management device vdpasim:
$ vdpa dev add mgmtdev vdpasim_net name foo2
Delete the vdpa device after its use:
$ vdpa dev del foo2
---
Changelog:v1->v2:
- using int return type for dev_add callback
- removed device_id (type) as current drivers only supports single type
--- drivers/vdpa/vdpa.c | 143 +++++++++++++++++++++++++++++++++++--- include/linux/vdpa.h | 6 ++ include/uapi/linux/vdpa.h | 4 ++ 3 files changed, 143 insertions(+), 10 deletions(-)
drivers/vdpa/vdpa.c +133 -10
@@ -136,6 +136,37 @@ static int vdpa_name_match(struct device *dev, const void *data)return (strcmp (dev_name (&vdev->dev), data) ==0 ); } +static int __vdpa_register_device (struct vdpa_device *vdev) +{ +struct device *dev; + +lockdep_assert_held (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL ,dev_name (&vdev->dev), vdpa_name_match); +if (dev) { +put_device (dev); +return -EEXIST ; + } +return device_add (&vdev->dev); +} + +/** +* _vdpa_register_device - register a vDPA device with vdpa lock held +* Caller must have a succeed call of vdpa_alloc_device() before. +* Caller must invoke this routine in the management device dev_add() +* callback after setting up valid mgmtdev for this vdpa device. +* @vdev: the vdpa device to be registered to vDPA bus +* +* Returns an error when fail to add device to vDPA bus +*/ +int _vdpa_register_device (struct vdpa_device *vdev) +{ +if (!vdev->mdev) +return -EINVAL ; + +return __vdpa_register_device (vdev); +} +EXPORT_SYMBOL_GPL (_vdpa_register_device); +/** * vdpa_register_device - register a vDPA device * Callers must have a succeed call of vdpa_alloc_device() before.
@@ -145,24 +176,29 @@ static int vdpa_name_match(struct device *dev, const void *data)*/ int vdpa_register_device (struct vdpa_device *vdev) { -struct device *dev;int err;mutex_lock (&vdpa_dev_mutex); - dev =bus_find_device (&vdpa_bus,NULL ,dev_name (&vdev->dev), vdpa_name_match); -if (dev) { -put_device (dev); - err = -EEXIST ; - goto name_err; - } - - err =device_add (&vdev->dev); -name_err: + err =__vdpa_register_device (vdev);mutex_unlock (&vdpa_dev_mutex);return err; }EXPORT_SYMBOL_GPL (vdpa_register_device); +/** +* _vdpa_unregister_device - unregister a vDPA device +* Caller must invoke this routine as part of management device dev_del() +* callback. +* @vdev: the vdpa device to be unregisted from vDPA bus +*/ +void _vdpa_unregister_device (struct vdpa_device *vdev) +{ +lockdep_assert_held (&vdpa_dev_mutex); +WARN_ON (!vdev->mdev); +device_unregister (&vdev->dev); +} +EXPORT_SYMBOL_GPL (_vdpa_unregister_device); +/** * vdpa_unregister_device - unregister a vDPA device * @vdev: the vdpa device to be unregisted from vDPA bus
@@ -221,10 +257,25 @@ int vdpa_mgmtdev_register(struct vdpa_mgmt_dev *mdev) }EXPORT_SYMBOL_GPL (vdpa_mgmtdev_register); +static int vdpa_match_remove (struct device *dev,void *data) +{ + struct vdpa_device *vdev = container_of(dev, struct vdpa_device, dev); +struct vdpa_mgmt_dev *mdev = vdev->mdev; + +if (mdev == data) + mdev->ops->dev_del (mdev, vdev); +return 0 ; +} +void vdpa_mgmtdev_unregister (struct vdpa_mgmt_dev *mdev) {mutex_lock (&vdpa_dev_mutex); +list_del (&mdev->list); + +/* Filter out all the entries belong to this management device and delete it. */ +bus_for_each_dev (&vdpa_bus,NULL , mdev, vdpa_match_remove); +mutex_unlock (&vdpa_dev_mutex); }EXPORT_SYMBOL_GPL (vdpa_mgmtdev_unregister);
@@ -368,9 +419,69 @@ vdpa_nl_cmd_mgmtdev_get_dumpit(struct sk_buff *msg, struct netlink_callback *cb)return msg->len; } +static int vdpa_nl_cmd_dev_add_set_doit (struct sk_buff *skb,struct genl_info *info) +{ +struct vdpa_mgmt_dev *mdev; +const char *name; +int err =0 ; + +if (!info->attrs[VDPA_ATTR_DEV_NAME ]) +return -EINVAL ; + + name =nla_data (info->attrs[VDPA_ATTR_DEV_NAME ]); + +mutex_lock (&vdpa_dev_mutex); + mdev =vdpa_mgmtdev_get_from_attr (info->attrs); +if (IS_ERR (mdev)) { +NL_SET_ERR_MSG_MOD (info->extack,"Fail to find the specified management device" ); + err =PTR_ERR (mdev); + goto err; + } + + err = mdev->ops->dev_add (mdev, name); +err: +mutex_unlock (&vdpa_dev_mutex); +return err; +} + +static int vdpa_nl_cmd_dev_del_set_doit (struct sk_buff *skb,struct genl_info *info) +{ +struct vdpa_mgmt_dev *mdev; +struct vdpa_device *vdev; +struct device *dev; +const char *name; +int err =0 ; + +if (!info->attrs[VDPA_ATTR_DEV_NAME ]) +return -EINVAL ; + name =nla_data (info->attrs[VDPA_ATTR_DEV_NAME ]); + +mutex_lock (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL , name, vdpa_name_match); +if (!dev) { +NL_SET_ERR_MSG_MOD (info->extack,"device not found" ); + err = -ENODEV ; + goto dev_err; + } + vdev = container_of(dev, struct vdpa_device, dev); +if (!vdev->mdev) { +NL_SET_ERR_MSG_MOD (info->extack,"Only user created device can be deleted by user" ); + err = -EINVAL ; + goto mdev_err; + } + mdev = vdev->mdev; + mdev->ops->dev_del (mdev, vdev); +mdev_err: +put_device (dev); +dev_err: +mutex_unlock (&vdpa_dev_mutex); +return err; +} +static const struct nla_policy vdpa_nl_policy[VDPA_ATTR_MAX ] = { [VDPA_ATTR_MGMTDEV_BUS_NAME ] = { .type =NLA_NUL_STRING }, [VDPA_ATTR_MGMTDEV_DEV_NAME ] = { .type =NLA_STRING }, + [VDPA_ATTR_DEV_NAME ] = { .type =NLA_STRING }, };static const struct genl_ops vdpa_nl_ops[] = {
@@ -380,6 +491,18 @@ static const struct genl_ops vdpa_nl_ops[] = { .doit = vdpa_nl_cmd_mgmtdev_get_doit, .dumpit = vdpa_nl_cmd_mgmtdev_get_dumpit, }, + { + .cmd = VDPA_CMD_DEV_NEW, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, + .doit = vdpa_nl_cmd_dev_add_set_doit, + .flags = GENL_ADMIN_PERM, + }, + { + .cmd = VDPA_CMD_DEV_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, + .doit = vdpa_nl_cmd_dev_del_set_doit, + .flags = GENL_ADMIN_PERM, + }, }; static struct genl_family vdpa_nl_family __ro_after_init = {
include/linux/vdpa.h +6 -0
@@ -45,6 +45,8 @@ struct vdpa_mgmt_dev;* @index: device index * @features_valid: were features initialized? for legacy guests * @nvqs: maximum number of supported virtqueues +* @mdev: management device pointer; caller must setup when registering device as part +* of dev_add() mgmtdev ops callback before invoking _vdpa_register_device(). */ struct vdpa_device {struct device dev;
@@ -53,6 +55,7 @@ struct vdpa_device {unsigned int index;bool features_valid;int nvqs; +struct vdpa_mgmt_dev *mdev; };/**
@@ -260,6 +263,9 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent,int vdpa_register_device (struct vdpa_device *vdev);void vdpa_unregister_device (struct vdpa_device *vdev); +int _vdpa_register_device (struct vdpa_device *vdev); +void _vdpa_unregister_device (struct vdpa_device *vdev); +/** * vdpa_driver - operations for a vDPA driver * @driver: underlying device driver
include/uapi/linux/vdpa.h +4 -0
@@ -14,6 +14,8 @@ enum vdpa_command {VDPA_CMD_UNSPEC ,VDPA_CMD_MGMTDEV_NEW ,VDPA_CMD_MGMTDEV_GET ,/* can dump */ +VDPA_CMD_DEV_NEW , + VDPA_CMD_DEV_DEL, }; enum vdpa_attr {
@@ -24,6 +26,8 @@ enum vdpa_attr {VDPA_ATTR_MGMTDEV_DEV_NAME ,/* string */ VDPA_ATTR_MGMTDEV_SUPPORTED_CLASSES ,/* u64 */ +VDPA_ATTR_DEV_NAME ,/* string */ +/* new attributes must be added above here */ VDPA_ATTR_MAX, };
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v2 6/7 vdpa: Enable user to query vdpa device info
In reply to Parav PanditEnable user to query vdpa device information.
$ vdpa dev add mgmtdev vdpasim_net name foo2
Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256
$ vdpa dev show foo2 -jp
{
"dev": {
"foo2": {
"type": "network",
"mgmtdev": "vdpasim_net",
"vendor_id": 0,
"max_vqs": 2,
"max_vq_size": 256
}
}
}--- drivers/vdpa/vdpa.c | 131 ++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vdpa.h | 5 ++ 2 files changed, 136 insertions(+)
drivers/vdpa/vdpa.c +131 -0
@@ -478,6 +478,131 @@ static int vdpa_nl_cmd_dev_del_set_doit(struct sk_buff *skb, struct genl_info *ireturn err; } +static int +vdpa_dev_fill (struct vdpa_device *vdev,struct sk_buff *msg,u32 portid,u32 seq, +int flags,struct netlink_ext_ack *extack) +{ +u16 max_vq_size; +u32 device_id; +u32 vendor_id; +void *hdr; +int err; + + hdr =genlmsg_put (msg, portid, seq, &vdpa_nl_family, flags,VDPA_CMD_DEV_NEW ); +if (!hdr) +return -EMSGSIZE ; + + err =vdpa_nl_mgmtdev_handle_fill (msg, vdev->mdev); +if (err) + goto msg_err; + + device_id = vdev->config->get_device_id (vdev); + vendor_id = vdev->config->get_vendor_id (vdev); + max_vq_size = vdev->config->get_vq_num_max (vdev); + + err = -EMSGSIZE ; +if (nla_put_string (msg,VDPA_ATTR_DEV_NAME ,dev_name (&vdev->dev))) + goto msg_err; +if (nla_put_u32 (msg,VDPA_ATTR_DEV_ID , device_id)) + goto msg_err; +if (nla_put_u32 (msg,VDPA_ATTR_DEV_VENDOR_ID , vendor_id)) + goto msg_err; +if (nla_put_u32 (msg,VDPA_ATTR_DEV_MAX_VQS , vdev->nvqs)) + goto msg_err; +if (nla_put_u16 (msg,VDPA_ATTR_DEV_MAX_VQ_SIZE , max_vq_size)) + goto msg_err; + +genlmsg_end (msg, hdr); +return 0 ; + +msg_err: +genlmsg_cancel (msg, hdr); +return err; +} + +static int vdpa_nl_cmd_dev_get_doit (struct sk_buff *skb,struct genl_info *info) +{ +struct vdpa_device *vdev; +struct sk_buff *msg; +const char *devname; +struct device *dev; +int err; + +if (!info->attrs[VDPA_ATTR_DEV_NAME ]) +return -EINVAL ; + devname =nla_data (info->attrs[VDPA_ATTR_DEV_NAME ]); + msg =nlmsg_new (NLMSG_DEFAULT_SIZE ,GFP_KERNEL ); +if (!msg) +return -ENOMEM ; + +mutex_lock (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL , devname, vdpa_name_match); +if (!dev) { +mutex_unlock (&vdpa_dev_mutex); +NL_SET_ERR_MSG_MOD (info->extack,"device not found" ); +return -ENODEV ; + } + vdev = container_of(dev, struct vdpa_device, dev); +if (!vdev->mdev) { +mutex_unlock (&vdpa_dev_mutex); +put_device (dev); +return -EINVAL ; + } + err =vdpa_dev_fill (vdev, msg, info->snd_portid, info->snd_seq,0 , info->extack); +if (!err) + err =genlmsg_reply (msg, info); +put_device (dev); +mutex_unlock (&vdpa_dev_mutex); + +if (err) +nlmsg_free (msg); +return err; +} + +struct vdpa_dev_dump_info { +struct sk_buff *msg; +struct netlink_callback *cb; +int start_idx; +int idx; +}; + +static int vdpa_dev_dump (struct device *dev,void *data) +{ + struct vdpa_device *vdev = container_of(dev, struct vdpa_device, dev); +struct vdpa_dev_dump_info *info = data; +int err; + +if (!vdev->mdev) +return 0 ; +if (info->idx < info->start_idx) { + info->idx++; +return 0 ; + } + err =vdpa_dev_fill (vdev, info->msg,NETLINK_CB (info->cb->skb).portid, + info->cb->nlh->nlmsg_seq,NLM_F_MULTI , info->cb->extack); +if (err) +return err; + + info->idx++; +return 0 ; +} + +static int vdpa_nl_cmd_dev_get_dumpit (struct sk_buff *msg,struct netlink_callback *cb) +{ +struct vdpa_dev_dump_info info; + + info.msg = msg; + info.cb = cb; + info.start_idx = cb->args[0 ]; + info.idx =0 ; + +mutex_lock (&vdpa_dev_mutex); +bus_for_each_dev (&vdpa_bus,NULL , &info, vdpa_dev_dump); +mutex_unlock (&vdpa_dev_mutex); + cb->args[0 ] = info.idx; +return msg->len; +} +static const struct nla_policy vdpa_nl_policy[VDPA_ATTR_MAX ] = { [VDPA_ATTR_MGMTDEV_BUS_NAME ] = { .type =NLA_NUL_STRING }, [VDPA_ATTR_MGMTDEV_DEV_NAME ] = { .type =NLA_STRING },
@@ -503,6 +628,12 @@ static const struct genl_ops vdpa_nl_ops[] = { .doit = vdpa_nl_cmd_dev_del_set_doit, .flags = GENL_ADMIN_PERM, }, + { + .cmd = VDPA_CMD_DEV_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, + .doit = vdpa_nl_cmd_dev_get_doit, + .dumpit = vdpa_nl_cmd_dev_get_dumpit, + }, };static struct genl_family vdpa_nl_family__ro_after_init = {
include/uapi/linux/vdpa.h +5 -0
@@ -16,6 +16,7 @@ enum vdpa_command {VDPA_CMD_MGMTDEV_GET ,/* can dump */ VDPA_CMD_DEV_NEW ,VDPA_CMD_DEV_DEL , + VDPA_CMD_DEV_GET, /* can dump */ }; enum vdpa_attr {
@@ -27,6 +28,10 @@ enum vdpa_attr {VDPA_ATTR_MGMTDEV_SUPPORTED_CLASSES ,/* u64 */ VDPA_ATTR_DEV_NAME ,/* string */ +VDPA_ATTR_DEV_ID ,/* u32 */ +VDPA_ATTR_DEV_VENDOR_ID ,/* u32 */ +VDPA_ATTR_DEV_MAX_VQS ,/* u32 */ +VDPA_ATTR_DEV_MAX_VQ_SIZE ,/* u16 */ /* new attributes must be added above here */ VDPA_ATTR_MAX,
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v2 7/7 vdpa_sim_net: Add support for user supported devices
In reply to Parav PanditEnable user to create vdpasim net simulate devices.
Show vdpa management device that supports creating, deleting vdpa devices.
$ vdpa mgmtdev show
vdpasim_net:
supported_classes
net$ vdpa mgmtdev show -jp
{
"show": {
"vdpasim_net": {
"supported_classes": {
"net"
}
}
}Create a vdpa device of type networking named as "foo2" from the management device vdpasim:
$ vdpa dev add mgmtdev vdpasim_net name foo2
Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256
$ vdpa dev show foo2 -jp
{
"dev": {
"foo2": {
"type": "network",
"mgmtdev": "vdpasim_net",
"vendor_id": 0,
"max_vqs": 2,
"max_vq_size": 256
}
}
}Delete the vdpa device after its use:
$ vdpa dev del foo2
---
Changelog:v1->v2:
- rebased
--- drivers/vdpa/vdpa_sim/vdpa_sim.c | 3 +- drivers/vdpa/vdpa_sim/vdpa_sim.h | 2 + drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 92 ++++++++++++++++++++++++++++ 3 files changed, 96 insertions(+), 1 deletion(-)
drivers/vdpa/vdpa_sim/vdpa_sim.c +2 -1
@@ -235,7 +235,7 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr) ops = &vdpasim_config_ops; vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, - dev_attr->nvqs,NULL ); + dev_attr->nvqs, dev_attr->name);if (!vdpasim) goto err_alloc;
@@ -249,6 +249,7 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr)if (dma_set_mask_and_coherent (dev,DMA_BIT_MASK (64 ))) goto err_iommu;set_dma_ops (dev, &vdpasim_dma_ops); + vdpasim->vdpa.mdev = dev_attr->mgmt_dev; vdpasim->config =kzalloc (dev_attr->config_size,GFP_KERNEL );if (!vdpasim->config)
drivers/vdpa/vdpa_sim/vdpa_sim.h +2 -0
@@ -33,6 +33,8 @@ struct vdpasim_virtqueue { };struct vdpasim_dev_attr { +struct vdpa_mgmt_dev *mgmt_dev; +const char *name;u64 supported_features;size_t config_size;size_t buffer_size;
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +92 -0
110 lines not on this page · raw message
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH linux-next v2 7/7] vdpa_sim_net: Add support for user supported devices
Quoted 13 lines · Parav Pandit
static int __init vdpasim_net_init(void) { int ret = 0;drivers/vdpa/vdpa_sim/vdpa_sim_net.c +2
-0@@ -176,6 +264,8 @@ static int __init vdpasim_net_init(void)if (default_device) ret =vdpasim_net_default_dev_register (); +else + ret =vdpasim_net_mgmtdev_init ();return ret; }
drivers/vdpa/vdpa_sim/vdpa_sim_net.c static void __exit vdpasim_net_exit(void)
Quoted 9 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +2
-1@@ -183,6 +273,8 @@ static void __exit vdpasim_net_exit(void) {if (default_device)vdpasim_net_default_dev_unregister (); +else +vdpasim_net_mgmtdev_cleanup (); }module_init (vdpasim_net_init); -- 2.26.2
I wonder what's the value of keeping the default device that is out of the control of management API.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v2 7/7] vdpa_sim_net: Add support for user supported devices
Quoted 26 lines
From: Jason Wang <[email protected]>
Sent: Monday, January 4, 2021 12:35 PMQuoted 20 lines · Parav Pandit
static int __init vdpasim_net_init(void) { int ret = 0;drivers/vdpa/vdpa_sim/vdpa_sim_net.c +4
-0@@ -176,6 +264,8 @@ static int __init vdpasim_net_init(void)if (default_device) ret =vdpasim_net_default_dev_register (); +else + ret =vdpasim_net_mgmtdev_init ();return ret; }@@ -183,6 +273,8 @@ static void __exit vdpasim_net_exit(void) {if (default_device)vdpasim_net_default_dev_unregister (); +else +vdpasim_net_mgmtdev_cleanup (); }
Quoted 6 lines
Review static void __exit vdpasim_net_exit(void)
Quoted 2 lines
(quoted diff) +0
-1module_init(vdpasim_net_init); -- 2.26.2I wonder what's the value of keeping the default device that is out of the control of management API.
I think we can remove it like how I did in the v1 version. And actual vendor drivers like mlx5_vdpa will likely should do only user created devices. I added only for backward compatibility purpose, but we can remove the default simulated vdpa net device.
What do you recommend?
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH linux-next v2 7/7] vdpa_sim_net: Add support for user supported devices
Quoted 25 lines · Parav Pandit
Quoted 25 lines
From: Jason Wang <[email protected]>
Sent: Monday, January 4, 2021 12:35 PMQuoted 20 lines · Parav Pandit
static int __init vdpasim_net_init(void) { int ret = 0;drivers/vdpa/vdpa_sim/vdpa_sim_net.c +4
-0@@ -176,6 +264,8 @@ static int __init vdpasim_net_init(void)if (default_device) ret =vdpasim_net_default_dev_register (); +else + ret =vdpasim_net_mgmtdev_init ();return ret; }@@ -183,6 +273,8 @@ static void __exit vdpasim_net_exit(void) {if (default_device)vdpasim_net_default_dev_unregister (); +else +vdpasim_net_mgmtdev_cleanup (); }
Quoted 8 lines
Quoted 5 lines
Review static void __exit vdpasim_net_exit(void)
Quoted 2 lines
(quoted diff) +0
-1module_init(vdpasim_net_init); -- 2.26.2I wonder what's the value of keeping the default device that is out of the control of management API.
I think we can remove it like how I did in the v1 version. And actual vendor drivers like
mlx5_vdpawill likely should do only user created devices. I added only for backward compatibility purpose, but we can remove the default simulated vdpa net device.
What do you recommend?
I think we'd better mandate this management API. This can avoid vendor specific configuration that may complex management layer.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v2 7/7] vdpa_sim_net: Add support for user supported devices
Quoted 38 lines
From: Jason Wang <[email protected]>
Sent: Tuesday, January 5, 2021 9:36 AMQuoted 25 lines · Parav Pandit
Quoted 25 lines
From: Jason Wang <[email protected]>
Sent: Monday, January 4, 2021 12:35 PMQuoted 20 lines · Parav Pandit
static int __init vdpasim_net_init(void) { int ret = 0;drivers/vdpa/vdpa_sim/vdpa_sim_net.c +4
-0@@ -176,6 +264,8 @@ static int __init vdpasim_net_init(void)if (default_device) ret =vdpasim_net_default_dev_register (); +else + ret =vdpasim_net_mgmtdev_init ();return ret; }@@ -183,6 +273,8 @@ static void __exit vdpasim_net_exit(void) {if (default_device)vdpasim_net_default_dev_unregister (); +else +vdpasim_net_mgmtdev_cleanup (); }Quoted 6 lines
Quoted 5 lines
Review static void __exit vdpasim_net_exit(void)
Quoted 2 lines
(quoted diff) +0
-1module_init(vdpasim_net_init); -- 2.26.2I wonder what's the value of keeping the default device that is out of the control of management API.
I think we can remove it like how I did in the v1 version. And actual vendor
Quoted 9 lines
drivers like
mlx5_vdpawill likely should do only user created devices.Quoted 1 line
I added only for backward compatibility purpose, but we can remove the
default simulated vdpa net device.
Quoted 1 line
What do you recommend?
I think we'd better mandate this management API. This can avoid vendor specific configuration that may complex management layer.
Sounds good.
I will drop the patch that allows vdpasim_net default device via module parameter. Will post v3 with that removal.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
v3 cover Introduce vdpa management tool
In reply to Parav PanditThis patchset covers user requirements for managing existing vdpa devices, using a tool and its internal design notes for kernel drivers.
Background and user requirements:
----------------------------------
(1) Currently VDPA device is created by driver when driver is loaded. However, user should have a choice when to create or not create a vdpa device for the underlying management device.
For example, mlx5 PCI VF and subfunction device supports multiple classes of device such netdev, vdpa, rdma. Howevever it is not required to always created vdpa device for such device.
(2) In another use case, a device may support creating one or multiple vdpa device of same or different class such as net and block. Creating vdpa devices at driver load time further limits this use case.
(3) A user should be able to monitor and query vdpa queue level or device level statistics for a given vdpa device.
(4) A user should be able to query what class of vdpa devices are supported by its management device.
(5) A user should be able to view supported features and negotiated features of the vdpa device.
(6) A user should be able to create a vdpa device in vendor agnostic manner using single tool.
Hence, it is required to have a tool through which user can create one or more vdpa devices from a management device which addresses above user requirements.
Example devices:
----------------
+-----------+ +-----------+ +---------+ +--------+ +-----------+
|vdpa dev 0 | |vdpa dev 1 | |rdma dev | |netdev | |vdpa dev 3 |
|type=net | |type=net | |mlx5_0 | |ens3f0 | |type=net |
+----+------+ +-----+-----+ +----+----+ +-----+--+ +----+------+
| | | | |
| | | | |
+----+-----+ | +----+----+ | +----+----+
| mlx5 +--------+ |mlx5 +-------+ |mlx5 |
|pci vf 2 | |pci vf 4 | |pci sf 8 |
|03:00:2 | |03:00.4 | |mlx5_sf.8|
+----+-----+ +----+----+ +----+----+
| | |
| +----+-----+ |
+----------------------+mlx5 +----------------+
|pci pf 0 |
|03:00.0 |
+----------+vdpa tool:
----------
vdpa tool is a tool to create, delete vdpa devices from a management device. It is a tool that enables user to query statistics, features and may be more attributes in future.
vdpa tool command draft:
------------------------
(a) List management devices which support creating vdpa devices. It also shows which class types supported by this management device. In below command example four management devices support vdpa device creation.
First is simulated vdpasim_net management device.
Second is PCI VF whose bdf is 03.00:2.
Third is PCI VF whose name is 03:00.4.
Forth is PCI SF whose name is mlx5_core.sf.8
$ vdpa mgmtdev list
vdpasim_net
supported_classes
net
pci/0000:03.00:0
supported_classes
net
pci/0000:03.00:4
supported_classes
net
auxiliary/mlx5_core.sf.8
supported_classes
net(b) Now add a vdpa device of networking class and show the device.
$ vdpa dev add mgmtdev pci/0000:03.00:0 name foo0
$ vdpa dev show foo0 foo0: mgmtdev pci/0000:03.00:2 type network vendor_id 0 max_vqs 2 max_vq_size 256
(c) Show features of a vdpa device
$ vdpa dev features show foo0 supported iommu platform version 1
(d) Dump vdpa device statistics
$ vdpa dev stats show foo0 kickdoorbells 10 wqes 100
(e) Now delete a vdpa device previously created.
$ vdpa dev del foo0
vdpa tool support in this patchset:
-----------------------------------
vdpa tool is created to create, delete and query vdpa devices. examples:
Show vdpa management device that supports creating, deleting vdpa devices.
$ vdpa mgmtdev show
vdpasim_net:
supported_classes
net$ vdpa mgmtdev show -jp
{
"show": {
"vdpasim_net": {
"supported_classes": {
"net"
}
}
}Create a vdpa device of type networking named as "foo2" from the management device vdpasim_net:
$ vdpa dev add mgmtdev vdpasim_net name foo2
Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256
$ vdpa dev show foo2 -jp
{
"dev": {
"foo2": {
"type": "network",
"mgmtdev": "vdpasim_net",
"vendor_id": 0,
"max_vqs": 2,
"max_vq_size": 256
}
}
}Delete the vdpa device after its use:
$ vdpa dev del foo2
vdpa tool support by kernel:
----------------------------
vdpa tool user interface is supported by existing vdpa kernel framework, i.e. drivers/vdpa/vdpa.c It services user command through a netlink interface.
Each management device registers supported callback operations with vdpa subsystem through which vdpa device(s) can be managed.
Patch summary:
--------------
Patch-1 Makes mac address array static
Patch-2 Extends API to accept vdpa device name
Patch-3 Defines management device interface
Patch-4 Extends netlink interface to add, delete vdpa devices Patch-5 Extends netlink interface to query vdpa device attributes Patch-6 Extends vdpa_sim_net driver to add/delete simulated vdpa devices
Changelog:
----------v2->v3:
- removed default device module param patch
- removed code branches due to removal of default device module param patch
- removed two merged patches from v1
- added patch to make mac address staticv1->v2:
- rebased
- moved code from vdpasim to vdpa_sim_net module as code is split between two modules
- removed device_id field during device create as its not used currently
- updated examples in commit log for management device name and device_id removal
- changed parentdev to mgmtdev as tool reflects management functionality
FAQs:
-----
1. Where does userspace vdpa tool reside which users can use?
Ans: vdpa tool can possibly reside in iproute2 [1] as it enables user to handler vdpa network devices.
2. Why not create and delete vdpa device using sysfs/configfs? Ans:
(a) A device creation may involve passing one or more attributes. Passing multiple attributes and returning error code and more verbose information for invalid attributes cannot be handled by sysfs/configfs.
(b) netlink framework is rich that enables user space and kernel driver to provide nested attributes.
(c) Exposing device specific file under sysfs without net namespace awareness exposes details to multiple containers. Instead exposing attributes via a netlink socket secures the communication channel with kernel.
(d) netlink socket interface enables to run syscaller kernel tests.
3. Why not use ioctl() interface?
Ans: ioctl() interface replicates the necessary plumbing which already exists through netlink socket.
4. What happens when one or more user created vdpa devices exist for a management PCI VF or SF and such management device is removed?
Ans: All user created vdpa devices are removed that belong to a management device.
[1] git://git.kernel.org/pub/scm/network/iproute2/iproute2-next.git
Next steps:
-----------
(a) Post this patchset and iproute2/vdpa inclusion, remaining two drivers will be coverted to support vdpa tool instead of creating unmanaged default device on driver load.
(b) More net specific parameters such as mac, mtu will be added.
(c) Features bits get and set interface will be added.
Parav Pandit (6):
vdpa_sim_net: Make mac address array static
vdpa: Extend routine to accept vdpa device name
vdpa: Define vdpa mgmt device, ops and a netlink interface
vdpa: Enable a user to add and delete a vdpa device
vdpa: Enable user to query vdpa device info
vdpa_sim_net: Add support for user supported devices
drivers/vdpa/Kconfig | 1 + drivers/vdpa/ifcvf/ifcvf_main.c | 2 +- drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +- drivers/vdpa/vdpa.c | 503 ++++++++++++++++++++++++++- drivers/vdpa/vdpa_sim/vdpa_sim.c | 3 +- drivers/vdpa/vdpa_sim/vdpa_sim.h | 2 + drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 98 ++++-- include/linux/vdpa.h | 44 ++- include/uapi/linux/vdpa.h | 40 +++ 9 files changed, 657 insertions(+), 38 deletions(-) create mode 100644 include/uapi/linux/vdpa.h
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v3 1/6 vdpa_sim_net: Make mac address array static
MAC address array is used only in vdpa_sim_net.c.
Hence, keep it static.
---
Changelog:v1->v2:
- new patch
--- drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +1 -1
@@ -33,7 +33,7 @@ static char *macaddr;module_param (macaddr, charp,0 );MODULE_PARM_DESC (macaddr,"Ethernet MAC address" ); -u8 macaddr_buf[ETH_ALEN ]; +static u8 macaddr_buf[ETH_ALEN ];static struct vdpasim *vdpasim_net_dev;
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH linux-next v3 1/6] vdpa_sim_net: Make mac address array static
Quoted 5 lines · Parav Pandit
MAC address array is used only in
vdpa_sim_net.c. Hence, keep it static.Signed-off-by Parav Pandit <[email protected]>Acked-by Jason Wang <[email protected]>
Quoted 7 lines
---
Changelog:v1->v2:
- new patch
---drivers/vdpa/vdpa_sim/vdpa_sim_net.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 22 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +1
-1@@ -33,7 +33,7 @@ static char *macaddr;module_param (macaddr, charp,0 );MODULE_PARM_DESC (macaddr,"Ethernet MAC address" ); -u8 macaddr_buf[ETH_ALEN ]; +static u8 macaddr_buf[ETH_ALEN ];static struct vdpasim *vdpasim_net_dev;Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v3 2/6 vdpa: Extend routine to accept vdpa device name
In reply to Parav PanditIn a subsequent patch, when user initiated command creates a vdpa device, the user chooses the name of the vdpa device.
To support it, extend the device allocation API to consider this name specified by the caller driver.
---
Changelog:v1->v2:
- rebased
--- drivers/vdpa/ifcvf/ifcvf_main.c | 2 +- drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +- drivers/vdpa/vdpa.c | 36 +++++++++++++++++++++++++++---- drivers/vdpa/vdpa_sim/vdpa_sim.c | 2 +- include/linux/vdpa.h | 7 +++--- 5 files changed, 38 insertions(+), 11 deletions(-)
drivers/vdpa/ifcvf/ifcvf_main.c +1 -1
@@ -432,7 +432,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct pci_device_id *id) adapter = vdpa_alloc_device(struct ifcvf_adapter, vdpa, dev, &ifc_vdpa_ops, -IFCVF_MAX_QUEUE_PAIRS *2 ); +IFCVF_MAX_QUEUE_PAIRS *2 ,NULL );if (adapter ==NULL ) {IFCVF_ERR (pdev,"Failed to allocate vDPA structure" );return -ENOMEM ;
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -1946,7 +1946,7 @@ void *mlx5_vdpa_add_dev(struct mlx5_core_dev *mdev) max_vqs =min_t (u32, max_vqs,MLX5_MAX_SUPPORTED_VQS ); ndev = vdpa_alloc_device(struct mlx5_vdpa_net, mvdev.vdev, mdev->device, &mlx5_vdpa_ops, -2 *mlx5_vdpa_max_qps (max_vqs)); +2 *mlx5_vdpa_max_qps (max_vqs),NULL );if (IS_ERR (ndev))return ndev;
drivers/vdpa/vdpa.c +32 -4
@@ -12,6 +12,8 @@#include <linux/slab.h> #include <linux/vdpa.h> +/* A global mutex that protects vdpa management device and device level operations. */ +static DEFINE_MUTEX (vdpa_dev_mutex);static DEFINE_IDA (vdpa_index_ida);static int vdpa_dev_probe (struct device *d)
@@ -63,6 +65,7 @@ static void vdpa_release_dev(struct device *d)* @config: the bus operations that is supported by this device * @nvqs: number of virtqueues supported by this device * @size: size of the parent structure that contains private data +* @name: name of the vdpa device; optional. * * Driver should use vdpa_alloc_device() wrapper macro instead of * using this directly.
@@ -72,8 +75,7 @@ static void vdpa_release_dev(struct device *d)*/ struct vdpa_device *__vdpa_alloc_device (struct device *parent,const struct vdpa_config_ops *config, -int nvqs, -size_t size) +int nvqs,size_t size,const char *name) {struct vdpa_device *vdev;int err = -EINVAL ;
@@ -101,7 +103,10 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent, vdev->features_valid = false; vdev->nvqs = nvqs; - err =dev_set_name (&vdev->dev,"vdpa%u" , vdev->index); +if (name) + err =dev_set_name (&vdev->dev,"%s" , name); +else + err =dev_set_name (&vdev->dev,"vdpa%u" , vdev->index);if (err) goto err_name;
@@ -118,6 +123,13 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent, }EXPORT_SYMBOL_GPL (__vdpa_alloc_device); +static int vdpa_name_match (struct device *dev,const void *data) +{ + struct vdpa_device *vdev = container_of(dev, struct vdpa_device, dev); + +return (strcmp (dev_name (&vdev->dev), data) ==0 ); +} +/** * vdpa_register_device - register a vDPA device * Callers must have a succeed call of vdpa_alloc_device() before.
@@ -127,7 +139,21 @@ EXPORT_SYMBOL_GPL(__vdpa_alloc_device);*/ int vdpa_register_device (struct vdpa_device *vdev) { -return device_add (&vdev->dev); +struct device *dev; +int err; + +mutex_lock (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL ,dev_name (&vdev->dev), vdpa_name_match); +if (dev) { +put_device (dev); + err = -EEXIST ; + goto name_err; + } + + err =device_add (&vdev->dev); +name_err: +mutex_unlock (&vdpa_dev_mutex); +return err; }EXPORT_SYMBOL_GPL (vdpa_register_device);
@@ -137,7 +163,9 @@ EXPORT_SYMBOL_GPL(vdpa_register_device);*/ void vdpa_unregister_device (struct vdpa_device *vdev) { +mutex_lock (&vdpa_dev_mutex);device_unregister (&vdev->dev); +mutex_unlock (&vdpa_dev_mutex); }EXPORT_SYMBOL_GPL (vdpa_unregister_device);
drivers/vdpa/vdpa_sim/vdpa_sim.c +1 -1
@@ -235,7 +235,7 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr) ops = &vdpasim_config_ops; vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, - dev_attr->nvqs); + dev_attr->nvqs,NULL );if (!vdpasim) goto err_alloc;
include/linux/vdpa.h +3 -4
@@ -245,15 +245,14 @@ struct vdpa_config_ops {struct vdpa_device *__vdpa_alloc_device (struct device *parent,const struct vdpa_config_ops *config, -int nvqs, -size_t size); +int nvqs,size_t size,const char *name); -#define vdpa_alloc_device (dev_struct, member, parent, config, nvqs) \ +#define vdpa_alloc_device (dev_struct, member, parent, config, nvqs, name) \ container_of(__vdpa_alloc_device( \ parent, config, nvqs, \ sizeof(dev_struct) + \ BUILD_BUG_ON_ZERO(offsetof( \ - dev_struct, member))), \ + dev_struct, member)), name), \ dev_struct, member)int vdpa_register_device (struct vdpa_device *vdev);
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v3 3/6 vdpa: Define vdpa mgmt device, ops and a netlink interface
In reply to Parav PanditTo add one or more VDPA devices, define a management device which allows adding or removing vdpa device. A management device defines set of callbacks to manage vdpa devices.
To begin with, it defines add and remove callbacks through which a user defined vdpa device can be added or removed.
A unique management device is identified by its unique handle identified by management device name and optionally the bus name.
Hence, introduce routine through which driver can register a management device and its callback operations for adding and remove a vdpa device.
Introduce vdpa netlink socket family so that user can query management device and its attributes.
Example of show vdpa management device which allows creating vdpa device of networking class (device id = 0x1) of virtio specification 1.1 section 5.1.1.
$ vdpa mgmtdev show
vdpasim_net:
supported_classes:
netExample of showing vdpa management device in JSON format.
$ vdpa mgmtdev show -jp
{
"show": {
"vdpasim_net": {
"supported_classes": [ "net" ]
}
}
}---
Changelog:v1->v2:
- rebased
- updated commit log example for management device name from "vdpasim" to "vdpasim_net"
- removed device_id as net and block management devices are separated
- dev_add() return type is changed from struct vdpa_device to int
--- drivers/vdpa/Kconfig | 1 + drivers/vdpa/vdpa.c | 213 +++++++++++++++++++++++++++++++++++++- include/linux/vdpa.h | 31 ++++++ include/uapi/linux/vdpa.h | 31 ++++++ 4 files changed, 275 insertions(+), 1 deletion(-) create mode 100644 include/uapi/linux/vdpa.h
drivers/vdpa/Kconfig +1 -0
@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only menuconfig VDPA tristate "vDPA drivers" + depends on NET help Enable this module to support vDPA device that uses a datapath which complies with virtio specifications with
drivers/vdpa/vdpa.c +212 -1
236 lines not on this page · raw message
include/linux/vdpa.h +31 -0
@@ -35,6 +35,8 @@ struct vdpa_vq_state {u16 avail_index; }; +struct vdpa_mgmt_dev ; +/** * vDPA device - representation of a vDPA device * @dev: underlying device
@@ -335,4 +337,33 @@ static inline void vdpa_get_config(struct vdpa_device *vdev, unsigned offset, ops->get_config(vdev, offset, buf, len); } +/** +* vdpa_mgmtdev_ops - vdpa device ops +* @dev_add: Add a vdpa device using alloc and register +* @mdev: parent device to use for device addition +* @name: name of the new vdpa device +* Driver need to add a new device using _vdpa_register_device() +* after fully initializing the vdpa device. Driver must return 0 +* on success or appropriate error code. +* @dev_del: Remove a vdpa device using unregister +* @mdev: parent device to use for device removal +* @dev: vdpa device to remove +* Driver need to remove the specified device by calling +* _vdpa_unregister_device(). +*/ +struct vdpa_mgmtdev_ops { +int (*dev_add)(struct vdpa_mgmt_dev *mdev,const char *name); +void (*dev_del)(struct vdpa_mgmt_dev *mdev,struct vdpa_device *dev); +}; + +struct vdpa_mgmt_dev { +struct device *device; +const struct vdpa_mgmtdev_ops *ops; +const struct virtio_device_id *id_table;/* supported ids */ +struct list_head list; +}; + +int vdpa_mgmtdev_register (struct vdpa_mgmt_dev *mdev); +void vdpa_mgmtdev_unregister (struct vdpa_mgmt_dev *mdev); +#endif /* _LINUX_VDPA_H */
include/uapi/linux/vdpa.h new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +/* +* vdpa device management interface +* Copyright (c) 2020 Mellanox Technologies Ltd. All rights reserved. +*/ + +#ifndef _UAPI_LINUX_VDPA_H_ +#define _UAPI_LINUX_VDPA_H_ + +#define VDPA_GENL_NAME "vdpa" +#define VDPA_GENL_VERSION 0x1 + +enum vdpa_command { +VDPA_CMD_UNSPEC , +VDPA_CMD_MGMTDEV_NEW , +VDPA_CMD_MGMTDEV_GET ,/* can dump */ +}; + +enum vdpa_attr { +VDPA_ATTR_UNSPEC , + +/* bus name (optional) + dev name together make the parent device handle */ +VDPA_ATTR_MGMTDEV_BUS_NAME ,/* string */ +VDPA_ATTR_MGMTDEV_DEV_NAME ,/* string */ +VDPA_ATTR_MGMTDEV_SUPPORTED_CLASSES ,/* u64 */ + +/* new attributes must be added above here */ +VDPA_ATTR_MAX , +}; + +#endif
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v3 4/6 vdpa: Enable a user to add and delete a vdpa device
In reply to Parav PanditAdd the ability to add and delete a vdpa device.
Examples:
Create a vdpa device of type network named "foo2" from the management device vdpasim:
$ vdpa dev add mgmtdev vdpasim_net name foo2
Delete the vdpa device after its use:
$ vdpa dev del foo2
---
Changelog:v1->v2:
- using int return type for dev_add callback
- removed device_id (type) as current drivers only supports single type
--- drivers/vdpa/vdpa.c | 143 +++++++++++++++++++++++++++++++++++--- include/linux/vdpa.h | 6 ++ include/uapi/linux/vdpa.h | 4 ++ 3 files changed, 143 insertions(+), 10 deletions(-)
drivers/vdpa/vdpa.c +133 -10
@@ -136,6 +136,37 @@ static int vdpa_name_match(struct device *dev, const void *data)return (strcmp (dev_name (&vdev->dev), data) ==0 ); } +static int __vdpa_register_device (struct vdpa_device *vdev) +{ +struct device *dev; + +lockdep_assert_held (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL ,dev_name (&vdev->dev), vdpa_name_match); +if (dev) { +put_device (dev); +return -EEXIST ; + } +return device_add (&vdev->dev); +} + +/** +* _vdpa_register_device - register a vDPA device with vdpa lock held +* Caller must have a succeed call of vdpa_alloc_device() before. +* Caller must invoke this routine in the management device dev_add() +* callback after setting up valid mgmtdev for this vdpa device. +* @vdev: the vdpa device to be registered to vDPA bus +* +* Returns an error when fail to add device to vDPA bus +*/ +int _vdpa_register_device (struct vdpa_device *vdev) +{ +if (!vdev->mdev) +return -EINVAL ; + +return __vdpa_register_device (vdev); +} +EXPORT_SYMBOL_GPL (_vdpa_register_device); +/** * vdpa_register_device - register a vDPA device * Callers must have a succeed call of vdpa_alloc_device() before.
@@ -145,24 +176,29 @@ static int vdpa_name_match(struct device *dev, const void *data)*/ int vdpa_register_device (struct vdpa_device *vdev) { -struct device *dev;int err;mutex_lock (&vdpa_dev_mutex); - dev =bus_find_device (&vdpa_bus,NULL ,dev_name (&vdev->dev), vdpa_name_match); -if (dev) { -put_device (dev); - err = -EEXIST ; - goto name_err; - } - - err =device_add (&vdev->dev); -name_err: + err =__vdpa_register_device (vdev);mutex_unlock (&vdpa_dev_mutex);return err; }EXPORT_SYMBOL_GPL (vdpa_register_device); +/** +* _vdpa_unregister_device - unregister a vDPA device +* Caller must invoke this routine as part of management device dev_del() +* callback. +* @vdev: the vdpa device to be unregisted from vDPA bus +*/ +void _vdpa_unregister_device (struct vdpa_device *vdev) +{ +lockdep_assert_held (&vdpa_dev_mutex); +WARN_ON (!vdev->mdev); +device_unregister (&vdev->dev); +} +EXPORT_SYMBOL_GPL (_vdpa_unregister_device); +/** * vdpa_unregister_device - unregister a vDPA device * @vdev: the vdpa device to be unregisted from vDPA bus
@@ -221,10 +257,25 @@ int vdpa_mgmtdev_register(struct vdpa_mgmt_dev *mdev) }EXPORT_SYMBOL_GPL (vdpa_mgmtdev_register); +static int vdpa_match_remove (struct device *dev,void *data) +{ + struct vdpa_device *vdev = container_of(dev, struct vdpa_device, dev); +struct vdpa_mgmt_dev *mdev = vdev->mdev; + +if (mdev == data) + mdev->ops->dev_del (mdev, vdev); +return 0 ; +} +void vdpa_mgmtdev_unregister (struct vdpa_mgmt_dev *mdev) {mutex_lock (&vdpa_dev_mutex); +list_del (&mdev->list); + +/* Filter out all the entries belong to this management device and delete it. */ +bus_for_each_dev (&vdpa_bus,NULL , mdev, vdpa_match_remove); +mutex_unlock (&vdpa_dev_mutex); }EXPORT_SYMBOL_GPL (vdpa_mgmtdev_unregister);
@@ -368,9 +419,69 @@ vdpa_nl_cmd_mgmtdev_get_dumpit(struct sk_buff *msg, struct netlink_callback *cb)return msg->len; } +static int vdpa_nl_cmd_dev_add_set_doit (struct sk_buff *skb,struct genl_info *info) +{ +struct vdpa_mgmt_dev *mdev; +const char *name; +int err =0 ; + +if (!info->attrs[VDPA_ATTR_DEV_NAME ]) +return -EINVAL ; + + name =nla_data (info->attrs[VDPA_ATTR_DEV_NAME ]); + +mutex_lock (&vdpa_dev_mutex); + mdev =vdpa_mgmtdev_get_from_attr (info->attrs); +if (IS_ERR (mdev)) { +NL_SET_ERR_MSG_MOD (info->extack,"Fail to find the specified management device" ); + err =PTR_ERR (mdev); + goto err; + } + + err = mdev->ops->dev_add (mdev, name); +err: +mutex_unlock (&vdpa_dev_mutex); +return err; +} + +static int vdpa_nl_cmd_dev_del_set_doit (struct sk_buff *skb,struct genl_info *info) +{ +struct vdpa_mgmt_dev *mdev; +struct vdpa_device *vdev; +struct device *dev; +const char *name; +int err =0 ; + +if (!info->attrs[VDPA_ATTR_DEV_NAME ]) +return -EINVAL ; + name =nla_data (info->attrs[VDPA_ATTR_DEV_NAME ]); + +mutex_lock (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL , name, vdpa_name_match); +if (!dev) { +NL_SET_ERR_MSG_MOD (info->extack,"device not found" ); + err = -ENODEV ; + goto dev_err; + } + vdev = container_of(dev, struct vdpa_device, dev); +if (!vdev->mdev) { +NL_SET_ERR_MSG_MOD (info->extack,"Only user created device can be deleted by user" ); + err = -EINVAL ; + goto mdev_err; + } + mdev = vdev->mdev; + mdev->ops->dev_del (mdev, vdev); +mdev_err: +put_device (dev); +dev_err: +mutex_unlock (&vdpa_dev_mutex); +return err; +} +static const struct nla_policy vdpa_nl_policy[VDPA_ATTR_MAX ] = { [VDPA_ATTR_MGMTDEV_BUS_NAME ] = { .type =NLA_NUL_STRING }, [VDPA_ATTR_MGMTDEV_DEV_NAME ] = { .type =NLA_STRING }, + [VDPA_ATTR_DEV_NAME ] = { .type =NLA_STRING }, };static const struct genl_ops vdpa_nl_ops[] = {
@@ -380,6 +491,18 @@ static const struct genl_ops vdpa_nl_ops[] = { .doit = vdpa_nl_cmd_mgmtdev_get_doit, .dumpit = vdpa_nl_cmd_mgmtdev_get_dumpit, }, + { + .cmd = VDPA_CMD_DEV_NEW, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, + .doit = vdpa_nl_cmd_dev_add_set_doit, + .flags = GENL_ADMIN_PERM, + }, + { + .cmd = VDPA_CMD_DEV_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, + .doit = vdpa_nl_cmd_dev_del_set_doit, + .flags = GENL_ADMIN_PERM, + }, }; static struct genl_family vdpa_nl_family __ro_after_init = {
include/linux/vdpa.h +6 -0
@@ -45,6 +45,8 @@ struct vdpa_mgmt_dev;* @index: device index * @features_valid: were features initialized? for legacy guests * @nvqs: maximum number of supported virtqueues +* @mdev: management device pointer; caller must setup when registering device as part +* of dev_add() mgmtdev ops callback before invoking _vdpa_register_device(). */ struct vdpa_device {struct device dev;
@@ -53,6 +55,7 @@ struct vdpa_device {unsigned int index;bool features_valid;int nvqs; +struct vdpa_mgmt_dev *mdev; };/**
@@ -260,6 +263,9 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent,int vdpa_register_device (struct vdpa_device *vdev);void vdpa_unregister_device (struct vdpa_device *vdev); +int _vdpa_register_device (struct vdpa_device *vdev); +void _vdpa_unregister_device (struct vdpa_device *vdev); +/** * vdpa_driver - operations for a vDPA driver * @driver: underlying device driver
include/uapi/linux/vdpa.h +4 -0
@@ -14,6 +14,8 @@ enum vdpa_command {VDPA_CMD_UNSPEC ,VDPA_CMD_MGMTDEV_NEW ,VDPA_CMD_MGMTDEV_GET ,/* can dump */ +VDPA_CMD_DEV_NEW , + VDPA_CMD_DEV_DEL, }; enum vdpa_attr {
@@ -24,6 +26,8 @@ enum vdpa_attr {VDPA_ATTR_MGMTDEV_DEV_NAME ,/* string */ VDPA_ATTR_MGMTDEV_SUPPORTED_CLASSES ,/* u64 */ +VDPA_ATTR_DEV_NAME ,/* string */ +/* new attributes must be added above here */ VDPA_ATTR_MAX, };
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v3 5/6 vdpa: Enable user to query vdpa device info
In reply to Parav PanditEnable user to query vdpa device information.
$ vdpa dev add mgmtdev vdpasim_net name foo2
Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256
$ vdpa dev show foo2 -jp
{
"dev": {
"foo2": {
"type": "network",
"mgmtdev": "vdpasim_net",
"vendor_id": 0,
"max_vqs": 2,
"max_vq_size": 256
}
}
}--- drivers/vdpa/vdpa.c | 131 ++++++++++++++++++++++++++++++++++++++ include/uapi/linux/vdpa.h | 5 ++ 2 files changed, 136 insertions(+)
drivers/vdpa/vdpa.c +131 -0
@@ -478,6 +478,131 @@ static int vdpa_nl_cmd_dev_del_set_doit(struct sk_buff *skb, struct genl_info *ireturn err; } +static int +vdpa_dev_fill (struct vdpa_device *vdev,struct sk_buff *msg,u32 portid,u32 seq, +int flags,struct netlink_ext_ack *extack) +{ +u16 max_vq_size; +u32 device_id; +u32 vendor_id; +void *hdr; +int err; + + hdr =genlmsg_put (msg, portid, seq, &vdpa_nl_family, flags,VDPA_CMD_DEV_NEW ); +if (!hdr) +return -EMSGSIZE ; + + err =vdpa_nl_mgmtdev_handle_fill (msg, vdev->mdev); +if (err) + goto msg_err; + + device_id = vdev->config->get_device_id (vdev); + vendor_id = vdev->config->get_vendor_id (vdev); + max_vq_size = vdev->config->get_vq_num_max (vdev); + + err = -EMSGSIZE ; +if (nla_put_string (msg,VDPA_ATTR_DEV_NAME ,dev_name (&vdev->dev))) + goto msg_err; +if (nla_put_u32 (msg,VDPA_ATTR_DEV_ID , device_id)) + goto msg_err; +if (nla_put_u32 (msg,VDPA_ATTR_DEV_VENDOR_ID , vendor_id)) + goto msg_err; +if (nla_put_u32 (msg,VDPA_ATTR_DEV_MAX_VQS , vdev->nvqs)) + goto msg_err; +if (nla_put_u16 (msg,VDPA_ATTR_DEV_MAX_VQ_SIZE , max_vq_size)) + goto msg_err; + +genlmsg_end (msg, hdr); +return 0 ; + +msg_err: +genlmsg_cancel (msg, hdr); +return err; +} + +static int vdpa_nl_cmd_dev_get_doit (struct sk_buff *skb,struct genl_info *info) +{ +struct vdpa_device *vdev; +struct sk_buff *msg; +const char *devname; +struct device *dev; +int err; + +if (!info->attrs[VDPA_ATTR_DEV_NAME ]) +return -EINVAL ; + devname =nla_data (info->attrs[VDPA_ATTR_DEV_NAME ]); + msg =nlmsg_new (NLMSG_DEFAULT_SIZE ,GFP_KERNEL ); +if (!msg) +return -ENOMEM ; + +mutex_lock (&vdpa_dev_mutex); + dev =bus_find_device (&vdpa_bus,NULL , devname, vdpa_name_match); +if (!dev) { +mutex_unlock (&vdpa_dev_mutex); +NL_SET_ERR_MSG_MOD (info->extack,"device not found" ); +return -ENODEV ; + } + vdev = container_of(dev, struct vdpa_device, dev); +if (!vdev->mdev) { +mutex_unlock (&vdpa_dev_mutex); +put_device (dev); +return -EINVAL ; + } + err =vdpa_dev_fill (vdev, msg, info->snd_portid, info->snd_seq,0 , info->extack); +if (!err) + err =genlmsg_reply (msg, info); +put_device (dev); +mutex_unlock (&vdpa_dev_mutex); + +if (err) +nlmsg_free (msg); +return err; +} + +struct vdpa_dev_dump_info { +struct sk_buff *msg; +struct netlink_callback *cb; +int start_idx; +int idx; +}; + +static int vdpa_dev_dump (struct device *dev,void *data) +{ + struct vdpa_device *vdev = container_of(dev, struct vdpa_device, dev); +struct vdpa_dev_dump_info *info = data; +int err; + +if (!vdev->mdev) +return 0 ; +if (info->idx < info->start_idx) { + info->idx++; +return 0 ; + } + err =vdpa_dev_fill (vdev, info->msg,NETLINK_CB (info->cb->skb).portid, + info->cb->nlh->nlmsg_seq,NLM_F_MULTI , info->cb->extack); +if (err) +return err; + + info->idx++; +return 0 ; +} + +static int vdpa_nl_cmd_dev_get_dumpit (struct sk_buff *msg,struct netlink_callback *cb) +{ +struct vdpa_dev_dump_info info; + + info.msg = msg; + info.cb = cb; + info.start_idx = cb->args[0 ]; + info.idx =0 ; + +mutex_lock (&vdpa_dev_mutex); +bus_for_each_dev (&vdpa_bus,NULL , &info, vdpa_dev_dump); +mutex_unlock (&vdpa_dev_mutex); + cb->args[0 ] = info.idx; +return msg->len; +} +static const struct nla_policy vdpa_nl_policy[VDPA_ATTR_MAX ] = { [VDPA_ATTR_MGMTDEV_BUS_NAME ] = { .type =NLA_NUL_STRING }, [VDPA_ATTR_MGMTDEV_DEV_NAME ] = { .type =NLA_STRING },
@@ -503,6 +628,12 @@ static const struct genl_ops vdpa_nl_ops[] = { .doit = vdpa_nl_cmd_dev_del_set_doit, .flags = GENL_ADMIN_PERM, }, + { + .cmd = VDPA_CMD_DEV_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, + .doit = vdpa_nl_cmd_dev_get_doit, + .dumpit = vdpa_nl_cmd_dev_get_dumpit, + }, };static struct genl_family vdpa_nl_family__ro_after_init = {
include/uapi/linux/vdpa.h +5 -0
@@ -16,6 +16,7 @@ enum vdpa_command {VDPA_CMD_MGMTDEV_GET ,/* can dump */ VDPA_CMD_DEV_NEW ,VDPA_CMD_DEV_DEL , + VDPA_CMD_DEV_GET, /* can dump */ }; enum vdpa_attr {
@@ -27,6 +28,10 @@ enum vdpa_attr {VDPA_ATTR_MGMTDEV_SUPPORTED_CLASSES ,/* u64 */ VDPA_ATTR_DEV_NAME ,/* string */ +VDPA_ATTR_DEV_ID ,/* u32 */ +VDPA_ATTR_DEV_VENDOR_ID ,/* u32 */ +VDPA_ATTR_DEV_MAX_VQS ,/* u32 */ +VDPA_ATTR_DEV_MAX_VQ_SIZE ,/* u16 */ /* new attributes must be added above here */ VDPA_ATTR_MAX,
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v3 6/6 vdpa_sim_net: Add support for user supported devices
In reply to Parav PanditEnable user to create vdpasim net simulate devices.
Show vdpa management device that supports creating, deleting vdpa devices.
$ vdpa mgmtdev show
vdpasim_net:
supported_classes
net$ vdpa mgmtdev show -jp
{
"show": {
"vdpasim_net": {
"supported_classes": {
"net"
}
}
}Create a vdpa device of type networking named as "foo2" from the management device vdpasim:
$ vdpa dev add mgmtdev vdpasim_net name foo2
Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256
$ vdpa dev show foo2 -jp
{
"dev": {
"foo2": {
"type": "network",
"mgmtdev": "vdpasim_net",
"vendor_id": 0,
"max_vqs": 2,
"max_vq_size": 256
}
}
}Delete the vdpa device after its use:
$ vdpa dev del foo2
---
Changelog:v2->v3:
- removed code branches due to default device removal patchv1->v2:
- rebased
--- drivers/vdpa/vdpa_sim/vdpa_sim.c | 3 +- drivers/vdpa/vdpa_sim/vdpa_sim.h | 2 + drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 96 ++++++++++++++++++++-------- 3 files changed, 75 insertions(+), 26 deletions(-)
drivers/vdpa/vdpa_sim/vdpa_sim.c +2 -1
@@ -235,7 +235,7 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr) ops = &vdpasim_config_ops; vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, - dev_attr->nvqs,NULL ); + dev_attr->nvqs, dev_attr->name);if (!vdpasim) goto err_alloc;
@@ -249,6 +249,7 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr)if (dma_set_mask_and_coherent (dev,DMA_BIT_MASK (64 ))) goto err_iommu;set_dma_ops (dev, &vdpasim_dma_ops); + vdpasim->vdpa.mdev = dev_attr->mgmt_dev; vdpasim->config =kzalloc (dev_attr->config_size,GFP_KERNEL );if (!vdpasim->config)
drivers/vdpa/vdpa_sim/vdpa_sim.h +2 -0
@@ -33,6 +33,8 @@ struct vdpasim_virtqueue { };struct vdpasim_dev_attr { +struct vdpa_mgmt_dev *mgmt_dev; +const char *name;u64 supported_features;size_t config_size;size_t buffer_size;
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +71 -25
129 lines not on this page · raw message
Signature · 6 lines
2.26.2 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 28 lines · Parav Pandit
Enable user to create vdpasim net simulate devices.
Show vdpa management device that supports creating, deleting vdpa devices.
$ vdpa mgmtdev show vdpasim_net: supported_classes net$ vdpa mgmtdev show -jp { "show": { "vdpasim_net": { "supported_classes": { "net" } } }Create a vdpa device of type networking named as "foo2" from the management device vdpasim:
$ vdpa dev add mgmtdev vdpasim_net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256
Quoted 12 lines
$ vdpa dev show foo2 -jp
{
"dev": {
"foo2": {
"type": "network",
"mgmtdev": "vdpasim_net",
"vendor_id": 0,
"max_vqs": 2,
"max_vq_size": 256
}
}
}I'd like an example of how do device specific
(e.g. net specific) interfaces tie in to this.
Quoted 148 lines
Delete the vdpa device after its use:
$ vdpa dev del foo2Signed-off-by Parav Pandit <[email protected]>Reviewed-by Eli Cohen <[email protected]>Acked-by Jason Wang <[email protected]>---
Changelog:v2->v3:
- removed code branches due to default device removal patchv1->v2:
- rebased--- drivers/vdpa/vdpa_sim/vdpa_sim.c | 3 +- drivers/vdpa/vdpa_sim/vdpa_sim.h | 2 + drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 96 ++++++++++++++++++++-------- 3 files changed, 75 insertions(+), 26 deletions(-)drivers/vdpa/vdpa_sim/vdpa_sim.c +2
-1@@ -235,7 +235,7 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr) ops = &vdpasim_config_ops; vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, - dev_attr->nvqs,NULL ); + dev_attr->nvqs, dev_attr->name);if (!vdpasim) goto err_alloc;@@ -249,6 +249,7 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr)if (dma_set_mask_and_coherent (dev,DMA_BIT_MASK (64 ))) goto err_iommu;set_dma_ops (dev, &vdpasim_dma_ops); + vdpasim->vdpa.mdev = dev_attr->mgmt_dev; vdpasim->config =kzalloc (dev_attr->config_size,GFP_KERNEL );if (!vdpasim->config)drivers/vdpa/vdpa_sim/vdpa_sim.h +2
-0@@ -33,6 +33,8 @@ struct vdpasim_virtqueue { };struct vdpasim_dev_attr { +struct vdpa_mgmt_dev *mgmt_dev; +const char *name;u64 supported_features;size_t config_size;size_t buffer_size;drivers/vdpa/vdpa_sim/vdpa_sim_net.c +53
-17@@ -35,8 +35,6 @@ MODULE_PARM_DESC(macaddr, "Ethernet MAC address");static u8 macaddr_buf[ETH_ALEN ]; -static struct vdpasim *vdpasim_net_dev; -static void vdpasim_net_work (struct work_struct *work) { struct vdpasim *vdpasim = container_of(work, struct vdpasim, work);@@ -120,21 +118,23 @@ static void vdpasim_net_get_config(struct vdpasim *vdpasim, void *config)memcpy (net_config->mac, macaddr_buf,ETH_ALEN ); } -static int __init vdpasim_net_init (void ) +static void vdpasim_net_mgmtdev_release (struct device *dev) +{ +} + +static struct device vdpasim_net_mgmtdev = { + .init_name ="vdpasim_net" , + .release = vdpasim_net_mgmtdev_release, +}; + +static int vdpasim_net_dev_add (struct vdpa_mgmt_dev *mdev,const char *name) {struct vdpasim_dev_attr dev_attr = {}; +struct vdpasim *simdev;int ret; -if (macaddr) { -mac_pton (macaddr, macaddr_buf); -if (!is_valid_ether_addr (macaddr_buf)) { - ret = -EADDRNOTAVAIL ; - goto out; - } - }else { -eth_random_addr (macaddr_buf); - } - + dev_attr.mgmt_dev = mdev; + dev_attr.name = name; dev_attr.id =VIRTIO_ID_NET ; dev_attr.supported_features =VDPASIM_NET_FEATURES ; dev_attr.nvqs =VDPASIM_NET_VQ_NUM ;@@ -143,29 +143,75 @@ static int __init vdpasim_net_init(void) dev_attr.work_fn = vdpasim_net_work; dev_attr.buffer_size =PAGE_SIZE ; - vdpasim_net_dev =vdpasim_create (&dev_attr); -if (IS_ERR (vdpasim_net_dev)) { - ret =PTR_ERR (vdpasim_net_dev); - goto out; + simdev =vdpasim_create (&dev_attr); +if (IS_ERR (simdev)) +return PTR_ERR (simdev); + + ret =_vdpa_register_device (&simdev->vdpa); +if (ret) + goto reg_err; + +return 0 ; + +reg_err: +put_device (&simdev->vdpa.dev); +return ret; +} + +static void vdpasim_net_dev_del (struct vdpa_mgmt_dev *mdev, +struct vdpa_device *dev) +{ + struct vdpasim *simdev = container_of(dev, struct vdpasim, vdpa); + +_vdpa_unregister_device (&simdev->vdpa); +} + +static const struct vdpa_mgmtdev_ops vdpasim_net_mgmtdev_ops = { + .dev_add = vdpasim_net_dev_add, + .dev_del = vdpasim_net_dev_del +}; + +static struct virtio_device_id id_table[] = { + {VIRTIO_ID_NET ,VIRTIO_DEV_ANY_ID }, + {0 }, +}; + +static struct vdpa_mgmt_dev mgmt_dev = { + .device = &vdpasim_net_mgmtdev, + .id_table = id_table, + .ops = &vdpasim_net_mgmtdev_ops, +}; + +static int __init vdpasim_net_init (void )
drivers/vdpa/vdpa_sim/vdpa_sim_net.c static int __init vdpasim_net_init(void) · line 195
Quoted 10 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +9
-0+{ +int ret; + +if (macaddr) { +mac_pton (macaddr, macaddr_buf); +if (!is_valid_ether_addr (macaddr_buf)) +return -EADDRNOTAVAIL ; + }else { +eth_random_addr (macaddr_buf); }
Hmm so all devices start out with the same MAC
until changed? And how is the change effected?
Quoted 31 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +9
-8- ret =vdpa_register_device (&vdpasim_net_dev->vdpa); + ret =device_register (&vdpasim_net_mgmtdev);if (ret) - goto put_dev; +return ret; + ret =vdpa_mgmtdev_register (&mgmt_dev); +if (ret) + goto parent_err;return 0 ; -put_dev: -put_device (&vdpasim_net_dev->vdpa.dev); -out: +parent_err: +device_unregister (&vdpasim_net_mgmtdev);return ret; } static void __exit vdpasim_net_exit(void) { -struct vdpa_device *vdpa = &vdpasim_net_dev->vdpa; - -vdpa_unregister_device (vdpa); +vdpa_mgmtdev_unregister (&mgmt_dev); +device_unregister (&vdpasim_net_mgmtdev); }module_init (vdpasim_net_init);Signature · 1 line
2.26.2
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 31 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:19 PMQuoted 11 lines · Parav Pandit
Enable user to create vdpasim net simulate devices.
$ vdpa dev add mgmtdev vdpasim_net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256Quoted 12 lines
$ vdpa dev show foo2 -jp { "dev": { "foo2": { "type": "network", "mgmtdev": "vdpasim_net", "vendor_id": 0, "max_vqs": 2, "max_vq_size": 256 } } }
Quoted 2 lines
I'd like an example of how do device specific (e.g. net specific) interfaces tie in to this.
Not sure I follow your question.
Do you mean how to set mac address or mtu of this vdpa device of type net?
If so, dev add command will be extended shortly in subsequent series to set this net specific attributes. (I did mention in the next steps in cover letter).
Quoted 5 lines
Quoted 5 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +5
-0+static int __init vdpasim_net_init (void ) { +int ret; + +if (macaddr) { +mac_pton (macaddr, macaddr_buf);
Quoted 8 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c static int __init vdpasim_net_init(void) · line 195
Quoted 5 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +4
-0+if (!is_valid_ether_addr (macaddr_buf)) +return -EADDRNOTAVAIL ; + }else { +eth_random_addr (macaddr_buf); }Hmm so all devices start out with the same MAC until changed? And how is the change effected?
Post this patchset and post we have iproute2 vdpa in the tree, will add the mac address as the input attribute during "vdpa dev add" command. So that each different vdpa device can have user specified (different) mac address.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 51 lines · Parav Pandit
Quoted 31 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:19 PMQuoted 11 lines · Parav Pandit
Enable user to create vdpasim net simulate devices.
$ vdpa dev add mgmtdev vdpasim_net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256Quoted 12 lines
$ vdpa dev show foo2 -jp { "dev": { "foo2": { "type": "network", "mgmtdev": "vdpasim_net", "vendor_id": 0, "max_vqs": 2, "max_vq_size": 256 } } }Quoted 2 lines
I'd like an example of how do device specific (e.g. net specific) interfaces tie in to this.
Not sure I follow your question.
Do you mean how to set mac address or mtu of this vdpa device of type net?
If so, dev add command will be extended shortly in subsequent series to set this net specific attributes. (I did mention in the next steps in cover letter).Quoted 5 lines
Quoted 5 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +5
-0+static int __init vdpasim_net_init (void ) { +int ret; + +if (macaddr) { +mac_pton (macaddr, macaddr_buf);Quoted 8 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c static int __init vdpasim_net_init(void) · line 195
Quoted 5 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +4
-0+if (!is_valid_ether_addr (macaddr_buf)) +return -EADDRNOTAVAIL ; + }else { +eth_random_addr (macaddr_buf); }Hmm so all devices start out with the same MAC until changed? And how is the change effected?
Quoted 2 lines
Post this patchset and post we have iproute2 vdpa in the tree, will add the mac address as the input attribute during "vdpa dev add" command. So that each different vdpa device can have user specified (different) mac address.
For now maybe just avoid VIRTIO_NET_F_MAC then for new devices then?
Signature · 6 lines
MST _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 59 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:45 PMQuoted 31 lines · Parav Pandit
Quoted 31 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:19 PMQuoted 23 lines · Parav Pandit
Enable user to create vdpasim net simulate devices.
$ vdpa dev add mgmtdev vdpasim_net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256$ vdpa dev show foo2 -jp { "dev": { "foo2": { "type": "network", "mgmtdev": "vdpasim_net", "vendor_id": 0, "max_vqs": 2, "max_vq_size": 256 } } }Quoted 4 lines
Quoted 2 lines
I'd like an example of how do device specific (e.g. net specific) interfaces tie in to this.
Not sure I follow your question.
Do you mean how to set mac address or mtu of this vdpa device of typenet?
Quoted 1 line
If so, dev add command will be extended shortly in subsequent series to
set this net specific attributes.
Quoted 15 lines
(I did mention in the next steps in cover letter).
Quoted 13 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c static int __init vdpasim_net_init(void) · line 195
Quoted 10 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +9
-0+static int __init vdpasim_net_init (void ) { +int ret; + +if (macaddr) { +mac_pton (macaddr, macaddr_buf); +if (!is_valid_ether_addr (macaddr_buf)) +return -EADDRNOTAVAIL ; + }else { +eth_random_addr (macaddr_buf); }Hmm so all devices start out with the same MAC until changed? And how is the change effected?
Quoted 6 lines
Quoted 1 line
Post this patchset and post we have iproute2 vdpa in the tree, will add the
mac address as the input attribute during "vdpa dev add" command.
Quoted 1 line
So that each different vdpa device can have user specified (different) mac
address.
For now maybe just avoid
VIRTIO_NET_F_MACthen for new devices then?
That would require book keeping existing net vdpa_sim devices created to avoid setting VIRTIO_NET_F_MAC. Such book keeping code will be short lived anyway.
Not sure if its worth it.
Until now only one device was created. So not sure two vdpa devices with same mac address will be a real issue.
When we add mac address attribute in add command, at that point also remove the module parameter macaddr.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 64 lines · Parav Pandit
Quoted 64 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:45 PMQuoted 31 lines · Parav Pandit
Quoted 31 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:19 PMQuoted 23 lines · Parav Pandit
Enable user to create vdpasim net simulate devices.
$ vdpa dev add mgmtdev vdpasim_net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256$ vdpa dev show foo2 -jp { "dev": { "foo2": { "type": "network", "mgmtdev": "vdpasim_net", "vendor_id": 0, "max_vqs": 2, "max_vq_size": 256 } } }Quoted 4 lines
Quoted 2 lines
I'd like an example of how do device specific (e.g. net specific) interfaces tie in to this.
Not sure I follow your question.
Do you mean how to set mac address or mtu of this vdpa device of typenet?
Quoted 1 line
If so, dev add command will be extended shortly in subsequent series to
set this net specific attributes.
Quoted 15 lines
(I did mention in the next steps in cover letter).
Quoted 5 lines
Quoted 5 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +5
-0+static int __init vdpasim_net_init (void ) { +int ret; + +if (macaddr) { +mac_pton (macaddr, macaddr_buf);Quoted 8 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c static int __init vdpasim_net_init(void) · line 195
Quoted 5 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +4
-0+if (!is_valid_ether_addr (macaddr_buf)) +return -EADDRNOTAVAIL ; + }else { +eth_random_addr (macaddr_buf); }Hmm so all devices start out with the same MAC until changed? And how is the change effected?
Quoted 1 line
Post this patchset and post we have iproute2 vdpa in the tree, will add the
mac address as the input attribute during "vdpa dev add" command.
Quoted 1 line
So that each different vdpa device can have user specified (different) mac
address.
Quoted 8 lines
Quoted 1 line
For now maybe just avoid
VIRTIO_NET_F_MACthen for new devices then?That would require book keeping existing net
vdpa_simdevices created to avoid settingVIRTIO_NET_F_MAC. Such book keeping code will be short lived anyway.
Not sure if its worth it.
Until now only one device was created. So not sure two vdpa devices with same mac address will be a real issue.When we add mac address attribute in add command, at that point also remove the module parameter macaddr.
Will that be mandatory? I'm not to happy with a UAPI we intend to break straight away ...
Signature · 6 lines
MST _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 82 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 6:53 PMQuoted 61 lines · Parav Pandit
Quoted 61 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:45 PMQuoted 31 lines · Parav Pandit
Quoted 31 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:19 PMQuoted 23 lines · Parav Pandit
Enable user to create vdpasim net simulate devices.
$ vdpa dev add mgmtdev vdpasim_net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256$ vdpa dev show foo2 -jp { "dev": { "foo2": { "type": "network", "mgmtdev": "vdpasim_net", "vendor_id": 0, "max_vqs": 2, "max_vq_size": 256 } } }Quoted 5 lines
Quoted 2 lines
I'd like an example of how do device specific (e.g. net specific) interfaces tie in to this.
Not sure I follow your question.
Do you mean how to set mac address or mtu of this vdpa device of typenet?
Quoted 2 lines
If so, dev add command will be extended shortly in subsequent series to
set this net specific attributes.
Quoted 15 lines
(I did mention in the next steps in cover letter).
Quoted 13 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c static int __init vdpasim_net_init(void) · line 195
Quoted 10 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +9
-0+static int __init vdpasim_net_init (void ) { +int ret; + +if (macaddr) { +mac_pton (macaddr, macaddr_buf); +if (!is_valid_ether_addr (macaddr_buf)) +return -EADDRNOTAVAIL ; + }else { +eth_random_addr (macaddr_buf); }Hmm so all devices start out with the same MAC until changed? And how is the change effected?
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Post this patchset and post we have iproute2 vdpa in the tree, will add the
mac address as the input attribute during "vdpa dev add" command.
Quoted 2 lines
So that each different vdpa device can have user specified (different) mac
address.
For now maybe just avoid
VIRTIO_NET_F_MACthen for new devicesthen?
Quoted 2 lines
That would require book keeping existing net
vdpa_simdevices created toavoid setting
VIRTIO_NET_F_MAC.Quoted 3 lines
Such book keeping code will be short lived anyway.
Not sure if its worth it.
Until now only one device was created. So not sure two vdpa devices with
Quoted 7 lines
same mac address will be a real issue.
Quoted 2 lines
When we add mac address attribute in add command, at that point also
remove the module parameter macaddr.
Will that be mandatory? I'm not to happy with a UAPI we intend to break straight away ...
No. Specifying mac address shouldn't be mandatory. UAPI wont' be broken.
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Hi Michael,
Quoted 9 lines
From: Virtualization <[email protected]> On Behalf Of Parav Pandit
Quoted 6 lines
Quoted 2 lines
When we add mac address attribute in add command, at that point also
remove the module parameter macaddr.
Will that be mandatory? I'm not to happy with a UAPI we intend to break straight away ...
No. Specifying mac address shouldn't be mandatory. UAPI wont' be broken.
Shall we please proceed with this patchset?
I would like to complete the iproute2 part and converting remaining two drivers to follow mgmt tool subsequent to this series.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
In reply to Parav PanditQuoted 75 lines · Parav Pandit
Quoted 75 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 6:53 PMQuoted 56 lines · Parav Pandit
Quoted 56 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:45 PMQuoted 28 lines · Parav Pandit
Quoted 28 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:19 PMQuoted 22 lines · Parav Pandit
Enable user to create vdpasim net simulate devices.
$ vdpa dev add mgmtdev vdpasim_net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256$ vdpa dev show foo2 -jp { "dev": { "foo2": { "type": "network", "mgmtdev": "vdpasim_net", "vendor_id": 0, "max_vqs": 2, "max_vq_size": 256 } } }Quoted 5 lines
Quoted 2 lines
I'd like an example of how do device specific (e.g. net specific) interfaces tie in to this.
Not sure I follow your question.
Do you mean how to set mac address or mtu of this vdpa device of typenet?
Quoted 2 lines
If so, dev add command will be extended shortly in subsequent series to
set this net specific attributes.
Quoted 14 lines
(I did mention in the next steps in cover letter).
Quoted 12 lines
Review
Quoted 10 lines
(quoted diff) +9
-0+static int __init vdpasim_net_init(void) { + int ret; + + if (macaddr) { + mac_pton(macaddr, macaddr_buf); + if (!is_valid_ether_addr(macaddr_buf)) + return -EADDRNOTAVAIL; + } else { + eth_random_addr(macaddr_buf); }Hmm so all devices start out with the same MAC until changed? And how is the change effected?
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Post this patchset and post we have iproute2 vdpa in the tree, will add the
mac address as the input attribute during "vdpa dev add" command.
Quoted 2 lines
So that each different vdpa device can have user specified (different) mac
address.
For now maybe just avoid
VIRTIO_NET_F_MACthen for new devicesthen?
Quoted 1 line
That would require book keeping existing net
vdpa_simdevices created toavoid setting
VIRTIO_NET_F_MAC.Quoted 3 lines
Such book keeping code will be short lived anyway.
Not sure if its worth it.
Until now only one device was created. So not sure two vdpa devices with
Quoted 7 lines
Quoted 6 lines
same mac address will be a real issue.
Quoted 1 line
When we add mac address attribute in add command, at that point also
remove the module parameter macaddr.
Will that be mandatory? I'm not to happy with a UAPI we intend to break straight away ...
No. Specifying mac address shouldn't be mandatory. UAPI wont' be broken.
If it's not mandatory. Does it mean the vDPA parent need to use its own logic to generate a validate mac? I'm not sure this is what management (libvirt want).
Thanks
Quoted 1 line
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 89 lines
From: Jason Wang <[email protected]>
Sent: Thursday, January 14, 2021 9:48 AMQuoted 83 lines · Parav Pandit
Quoted 83 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 6:53 PMQuoted 56 lines · Parav Pandit
Quoted 56 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:45 PMQuoted 28 lines · Parav Pandit
Quoted 28 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:19 PMQuoted 22 lines · Parav Pandit
Enable user to create vdpasim net simulate devices.
$ vdpa dev add mgmtdev vdpasim_net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256$ vdpa dev show foo2 -jp { "dev": { "foo2": { "type": "network", "mgmtdev": "vdpasim_net", "vendor_id": 0, "max_vqs": 2, "max_vq_size": 256 } } }Quoted 5 lines
Quoted 2 lines
I'd like an example of how do device specific (e.g. net specific) interfaces tie in to this.
Not sure I follow your question.
Do you mean how to set mac address or mtu of this vdpa device of typenet?
Quoted 2 lines
If so, dev add command will be extended shortly in subsequent series to
set this net specific attributes.
Quoted 14 lines
(I did mention in the next steps in cover letter).
Quoted 12 lines
Review
Quoted 10 lines
(quoted diff) +9
-0+static int __init vdpasim_net_init(void) { + int ret; + + if (macaddr) { + mac_pton(macaddr, macaddr_buf); + if (!is_valid_ether_addr(macaddr_buf)) + return -EADDRNOTAVAIL; + } else { + eth_random_addr(macaddr_buf); }Hmm so all devices start out with the same MAC until changed? And how is the change effected?
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Post this patchset and post we have iproute2 vdpa in the tree, will add the
mac address as the input attribute during "vdpa dev add" command.
Quoted 2 lines
So that each different vdpa device can have user specified (different) mac
address.
For now maybe just avoid
VIRTIO_NET_F_MACthen for new devicesthen?
Quoted 2 lines
That would require book keeping existing net
vdpa_simdevices created toavoid setting
VIRTIO_NET_F_MAC.Quoted 4 lines
Such book keeping code will be short lived anyway.
Not sure if its worth it.
Until now only one device was created. So not sure two vdpa devices withsame mac address will be a real issue.
Quoted 1 line
When we add mac address attribute in add command, at that point also
remove the module parameter macaddr.
Will that be mandatory? I'm not to happy with a UAPI we intend to break straight away ...
Quoted 8 lines
Quoted 1 line
No. Specifying mac address shouldn't be mandatory. UAPI wont' be
broken.
If it's not mandatory. Does it mean the vDPA parent need to use its own logic to generate a validate mac? I'm not sure this is what management (libvirt want).
There are few use cases that I see with PFs, VFs and SFs supporting vdpa devices.
1. User wants to use the VF only for vdpa purpose. Here user got the VF which was pre-setup by the sysadmin. In this case whatever MAC assigned to the VF can be used by its vdpa device.
Here, user doesn't need to pass the mac address during vdpa device creation time.
This is done as the same MAC has been setup in the ACL rules on the switch side.
Non VDPA users of a VF typically use the VF this way for Netdev and rdma functionality. They might continue same way for vdpa application as well.
Here VF mac is either set using
(a) devlink port function set hw_addr command or using
(b) ip link set vf mac
So vdpa tool didn't pass the mac. (optional).
Though VIRTIO_NET_F_MAC is still valid.
2. User may want to create one or more vdpa device out of the mgmt. device.
Here user wants to more/full control of all features, overriding what sysadmin has setup as MAC of the VF/SF. In this case user will specify the MAC via mgmt tool.
(a) This is also used by those vdpa devices which doesn't have eswitch offloads.
(b) This will work with eswitch offloads as well who does source learning.
(c) User chose to use the vdpa device of a VF while VF Netdev and rdma device are used by hypervisor for something else as well. VIRTIO_NET_F_MAC remains valid in all 2.{a,b,c}.
3. A vendor mgmt. device always expects it user to provide mac for its vdpa devices.
So when it is not provided, it can fail with error message string in extack or clear the VIRTIO_NET_F_MAC and let it work using virtio spec's 5.1.5 point 5 to proceed.
As common denominator of all above cases, if QEMU or user pass the MAC during creation, it will almost always work. Advance user and QEMU with switchdev mode support who has done 1.a/1.b, will omit it.
I do not know how deep integration of QEMU exist with the switchdev mode support.
With that mac, mtu as optional input fields provide the necessary flexibility for different stacks to take appropriate shape as they desire.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 114 lines · Parav Pandit
Quoted 84 lines
From: Jason Wang <[email protected]>
Sent: Thursday, January 14, 2021 9:48 AMQuoted 79 lines · Parav Pandit
Quoted 79 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 6:53 PMQuoted 53 lines · Parav Pandit
Quoted 53 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:45 PMQuoted 26 lines · Parav Pandit
Quoted 26 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:19 PMQuoted 22 lines · Parav Pandit
Enable user to create vdpasim net simulate devices.
$ vdpa dev add mgmtdev vdpasim_net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0 max_vqs 2 max_vq_size 256$ vdpa dev show foo2 -jp { "dev": { "foo2": { "type": "network", "mgmtdev": "vdpasim_net", "vendor_id": 0, "max_vqs": 2, "max_vq_size": 256 } } }Quoted 5 lines
Quoted 2 lines
I'd like an example of how do device specific (e.g. net specific) interfaces tie in to this.
Not sure I follow your question.
Do you mean how to set mac address or mtu of this vdpa device of typenet?
Quoted 2 lines
If so, dev add command will be extended shortly in subsequent series to
set this net specific attributes.
Quoted 14 lines
(I did mention in the next steps in cover letter).
Quoted 12 lines
Review
Quoted 10 lines
(quoted diff) +9
-0+static int __init vdpasim_net_init(void) { + int ret; + + if (macaddr) { + mac_pton(macaddr, macaddr_buf); + if (!is_valid_ether_addr(macaddr_buf)) + return -EADDRNOTAVAIL; + } else { + eth_random_addr(macaddr_buf); }Hmm so all devices start out with the same MAC until changed? And how is the change effected?
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Post this patchset and post we have iproute2 vdpa in the tree, will add the
mac address as the input attribute during "vdpa dev add" command.
Quoted 2 lines
So that each different vdpa device can have user specified (different) mac
address.
For now maybe just avoid
VIRTIO_NET_F_MACthen for new devicesthen?
Quoted 2 lines
That would require book keeping existing net
vdpa_simdevices created toavoid setting
VIRTIO_NET_F_MAC.Quoted 4 lines
Such book keeping code will be short lived anyway.
Not sure if its worth it.
Until now only one device was created. So not sure two vdpa devices withsame mac address will be a real issue.
Quoted 1 line
When we add mac address attribute in add command, at that point also
remove the module parameter macaddr.
Will that be mandatory? I'm not to happy with a UAPI we intend to break straight away ...
Quoted 8 lines
Quoted 1 line
No. Specifying mac address shouldn't be mandatory. UAPI wont' be
broken.
If it's not mandatory. Does it mean the vDPA parent need to use its own logic to generate a validate mac? I'm not sure this is what management (libvirt want).
There are few use cases that I see with PFs, VFs and SFs supporting vdpa devices.
1. User wants to use the VF only for vdpa purpose. Here user got the VF which was pre-setup by the sysadmin. In this case whatever MAC assigned to the VF can be used by its vdpa device.
Here, user doesn't need to pass the mac address during vdpa device creation time.
This is done as the same MAC has been setup in the ACL rules on the switch side.
Non VDPA users of a VF typically use the VF this way for Netdev and rdma functionality. They might continue same way for vdpa application as well.
Here VF mac is either set using
(a) devlink port function sethw_addrcommand or using
(b) ip link set vf mac
So vdpa tool didn't pass the mac. (optional).
ThoughVIRTIO_NET_F_MACis still valid.2. User may want to create one or more vdpa device out of the mgmt. device.
Here user wants to more/full control of all features, overriding what sysadmin has setup as MAC of the VF/SF. In this case user will specify the MAC via mgmt tool.
(a) This is also used by those vdpa devices which doesn't have eswitch offloads.
(b) This will work with eswitch offloads as well who does source learning.
(c) User chose to use the vdpa device of a VF while VF Netdev and rdma device are used by hypervisor for something else as well.VIRTIO_NET_F_MACremains valid in all 2.{a,b,c}.
Quoted 8 lines
3. A vendor mgmt. device always expects it user to provide mac for its vdpa devices.
So when it is not provided, it can fail with error message string in extack or clear theVIRTIO_NET_F_MACand let it work using virtio spec's 5.1.5 point 5 to proceed.As common denominator of all above cases, if QEMU or user pass the MAC during creation, it will almost always work. Advance user and QEMU with switchdev mode support who has done 1.a/1.b, will omit it.
I do not know how deep integration of QEMU exist with the switchdev mode support.With that mac, mtu as optional input fields provide the necessary flexibility for different stacks to take appropriate shape as they desire.
Thanks for the clarification. I think we'd better document the above in the patch that introduces the mac setting from management API.
Quoted 1 line
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 144 lines
From: Jason Wang <[email protected]>
Sent: Friday, January 15, 2021 11:09 AMQuoted 17 lines · Parav Pandit
Quoted 17 lines
From: Jason Wang <[email protected]>
Sent: Thursday, January 14, 2021 9:48 AMQuoted 12 lines · Parav Pandit
Quoted 12 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 6:53 PMQuoted 8 lines · Parav Pandit
Quoted 8 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:45 PMQuoted 4 lines · Parav Pandit
Quoted 4 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 5, 2021 5:19 PMOn Tue, Jan 05, 2021 at 12:32:03PM +0200, Parav Pandit wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Enable user to create vdpasim net simulate devices.
$ vdpa dev add mgmtdev vdpasim_net name foo2Show the newly created vdpa device by its name:
$ vdpa dev show foo2 foo2: type network mgmtdev vdpasim_net vendor_id 0
max_vqs2Quoted 61 lines
Quoted 61 lines
Quoted 61 lines
Quoted 53 lines
Quoted 37 lines
Quoted 37 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
max_vq_size256$ vdpa dev show foo2 -jp { "dev": { "foo2": { "type": "network", "mgmtdev": "vdpasim_net", "vendor_id": 0, "max_vqs": 2, "max_vq_size": 256 } } }Quoted 5 lines
Quoted 2 lines
I'd like an example of how do device specific (e.g. net specific) interfaces tie in to this.
Not sure I follow your question.
Do you mean how to set mac address or mtu of this vdpa device of typenet?
Quoted 2 lines
If so, dev add command will be extended shortly in subsequent series to
set this net specific attributes.
Quoted 14 lines
(I did mention in the next steps in cover letter).
Quoted 12 lines
Review
Quoted 10 lines
(quoted diff) +9
-0+static int __init vdpasim_net_init(void) { + int ret; + + if (macaddr) { + mac_pton(macaddr, macaddr_buf); + if (!is_valid_ether_addr(macaddr_buf)) + return -EADDRNOTAVAIL; + } else { + eth_random_addr(macaddr_buf); }Hmm so all devices start out with the same MAC until changed? And how is the change effected?
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Post this patchset and post we have iproute2 vdpa in the tree, will add the
mac address as the input attribute during "vdpa dev add" command.
Quoted 2 lines
So that each different vdpa device can have user specified (different) mac
address.
For now maybe just avoid
VIRTIO_NET_F_MACthen for new devicesthen?
Quoted 2 lines
That would require book keeping existing net
vdpa_simdevices created toavoid setting
VIRTIO_NET_F_MAC.Quoted 4 lines
Such book keeping code will be short lived anyway. Not sure if its worth it.
Until now only one device was created. So not sure two vdpa devices withQuoted 7 lines
same mac address will be a real issue.
Quoted 2 lines
When we add mac address attribute in add command, at that point also
remove the module parameter macaddr.
Will that be mandatory? I'm not to happy with a UAPI we intend to break straight away ...
No. Specifying mac address shouldn't be mandatory. UAPI wont' be
Quoted 8 lines
Quoted 7 lines
broken.
If it's not mandatory. Does it mean the vDPA parent need to use its own logic to generate a validate mac? I'm not sure this is what management (libvirt want).
There are few use cases that I see with PFs, VFs and SFs supporting vdpa
devices.
Quoted 2 lines
1. User wants to use the VF only for vdpa purpose. Here user got the VF
which was pre-setup by the sysadmin.
Quoted 1 line
In this case whatever MAC assigned to the VF can be used by its vdpa
device.
Quoted 1 line
Here, user doesn't need to pass the mac address during vdpa device
creation time.
Quoted 1 line
This is done as the same MAC has been setup in the ACL rules on the switch
side.
Quoted 1 line
Non VDPA users of a VF typically use the VF this way for Netdev and rdma
functionality.
Quoted 8 lines
They might continue same way for vdpa application as well. Here VF mac is either set using
(a) devlink port function sethw_addrcommand or using
(b) ip link set vf mac
So vdpa tool didn't pass the mac. (optional). ThoughVIRTIO_NET_F_MACis still valid.2. User may want to create one or more vdpa device out of the mgmt.
device.
Quoted 1 line
Here user wants to more/full control of all features, overriding what
sysadmin has setup as MAC of the VF/SF.
Quoted 2 lines
In this case user will specify the MAC via mgmt tool.
(a) This is also used by those vdpa devices which doesn't have eswitchoffloads.
Quoted 2 lines
(b) This will work with eswitch offloads as well who does source learning.
(c) User chose to use the vdpa device of a VF while VF Netdev and rdmadevice are used by hypervisor for something else as well.
Quoted 3 lines
VIRTIO_NET_F_MACremains valid in all 2.{a,b,c}.3. A vendor mgmt. device always expects it user to provide mac for its
vdpa devices.
Quoted 1 line
So when it is not provided, it can fail with error message string in extack or
clear the
VIRTIO_NET_F_MACand let it work using virtio spec's 5.1.5 point 5 to proceed.Quoted 2 lines
As common denominator of all above cases, if QEMU or user pass the MAC
during creation, it will almost always work.
Quoted 1 line
Advance user and QEMU with switchdev mode support who has done
1.a/1.b, will omit it.
Quoted 1 line
I do not know how deep integration of QEMU exist with the switchdev
Quoted 8 lines
mode support.
Quoted 2 lines
With that mac, mtu as optional input fields provide the necessary flexibility
for different stacks to take appropriate shape as they desire.
Thanks for the clarification. I think we'd better document the above in the patch that introduces the mac setting from management API.
Yes. Will do.
Thanks.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 8 lines · Parav Pandit
Quoted 6 lines
Quoted 1 line
With that mac, mtu as optional input fields provide the necessary flexibility
for different stacks to take appropriate shape as they desire.
Thanks for the clarification. I think we'd better document the above in the patch that introduces the mac setting from management API.
Yes. Will do.
Thanks.
Adding Sean.
Regarding to mac address setting. Do we plan to allow to modify mac address after the creation? It looks like Openstack wants this.
Sean may share more information on this.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 17 lines
From: Jason Wang <[email protected]>
Sent: Tuesday, January 19, 2021 4:39 PM
To: Parav Pandit <[email protected]>; Michael S. Tsirkin <[email protected]>
Cc: [email protected]; Eli Cohen <[email protected]>; [email protected]; Sean Mooney <[email protected]>
Subject: Re: [PATCH linux-next v3 6/6]vdpa_sim_net: Add support for user supported devicesQuoted 7 lines · Parav Pandit
Quoted 7 lines
Quoted 2 lines
With that mac, mtu as optional input fields provide the necessary flexibility
for different stacks to take appropriate shape as they desire.
Thanks for the clarification. I think we'd better document the above in the patch that introduces the mac setting from management API.
Quoted 9 lines
Quoted 2 lines
Yes. Will do.
Thanks.Adding Sean.
Regarding to mac address setting. Do we plan to allow to modify mac address after the creation? It looks like Openstack wants this.
Mac address is exposed in the features so yes, it should be possible to modify it as part of features modify command. (in future). User needs to make sure that device is not attached to vhost or higher layer stack when device configuration layout is modified.
Quoted 3 lines
Sean may share more information on this.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 31 lines
From: Virtualization <[email protected]> On Behalf Of Parav Pandit
Quoted 11 lines
From: Jason Wang <[email protected]>
Sent: Tuesday, January 19, 2021 4:39 PM
To: Parav Pandit <[email protected]>; Michael S. Tsirkin <[email protected]>
Cc: [email protected]; Eli Cohen <[email protected]>; [email protected]; Sean Mooney <[email protected]>
Subject: Re: [PATCH linux-next v3 6/6]vdpa_sim_net: Add support for user supported devicesQuoted 8 lines
Quoted 7 lines · Parav Pandit
Quoted 7 lines
Quoted 2 lines
With that mac, mtu as optional input fields provide the necessary flexibility
for different stacks to take appropriate shape as they desire.
Thanks for the clarification. I think we'd better document the above in the patch that introduces the mac setting from management
API.
Quoted 6 lines
Quoted 2 lines
Yes. Will do.
Thanks.Adding Sean.
Quoted 8 lines
Quoted 3 lines
Regarding to mac address setting. Do we plan to allow to modify mac address after the creation? It looks like Openstack wants this.
Mac address is exposed in the features so yes, it should be possible to modify it as part of features modify command. (in future).
User needs to make sure that device is not attached to vhost or higher layer stack when device configuration layout is modified.
Just curious, why Openstack cannot set the mac address at device creation time? Is vdpa device created by non Openstack software?
Does it always want to set the mac address of vdpa device?
Quoted 1 line
Quoted 1 line
Sean may share more information on this.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
In reply to Jason WangHi Michael, Jason,
Quoted 6 lines
From: Jason Wang <[email protected]>
Sent: Friday, January 15, 2021 11:09 AMThanks for the clarification. I think we'd better document the above in the patch that introduces the mac setting from management API.
Can we proceed with this patchset?
We like to progress next to iproute2/vdpa, mac and other drivers post this series in this kernel version.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices
Quoted 11 lines · Parav Pandit
Hi Michael, Jason,
Quoted 6 lines
From: Jason Wang <[email protected]>
Sent: Friday, January 15, 2021 11:09 AMThanks for the clarification. I think we'd better document the above in the patch that introduces the mac setting from management API.
Can we proceed with this patchset?
We like to progress next to iproute2/vdpa, mac and other drivers post this series in this kernel version.
Let me put this in next so it can get some testing there for a week or so.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization