Introduce a vfio driver over virtio devices
Subsystems (5)
Yishai Hadas
maintainerv7 cover Introduce a vfio driver over virtio devices
This series introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
Background, from the virtio spec [1].
--------------------------------------------------------------------
In some systems, there is a need to support a virtio legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.
For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.
--------------------------------------------------------------------
The first 6 patches are in the virtio area and handle the below:
- Introduce the admin virtqueue infrastcture.
- Expose the layout of the commands that should be used for supporting the legacy access.
- Expose APIs to enable upper layers as of vfio, net, etc to execute admin commands.
The above follows the virtio spec that was lastly accepted in that area [1].
The last 3 patches are in the vfio area and handle the below:
- Expose some APIs from vfio/pci to be used by the vfio/virtio driver.
- Introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
The series was tested successfully over virtio-net VFs in the host, while running in the guest both modern and legacy drivers.
[1]
https://github.com/oasis-tcs/virtio-spec/commit/03c2d32e5093ca9f2a17797242fbef88efe94b8c
Changes from V6: https://lore.kernel.org/kvm/[email protected]/
Vfio:
- Put the pm_runtime stuff into translate_io_bar_to_mem_bar() and
organize the callers to be more success oriented, as suggested by Jason.
- Add missing 'ops' (i.e. 'detach_ioas' and 'device_feature'), as mentioned by Jason.
- Clean virtiovf_bar0_exists() to cast to bool automatically, as
suggested by Jason.
- Add Reviewed-by: Jason Gunthorpe <[email protected]>.
Changes from V5: https://lore.kernel.org/kvm/[email protected]/
Vfio:
- Rename vfio_pci_iowrite64 to vfio_pci_core_iowrite64 as was mentioned
by Alex.
Changes from V4: https://lore.kernel.org/all/[email protected]/T/
Virtio:
- Drop the unused macro 'VIRTIO_ADMIN_MAX_CMD_OPCODE' as was asked by Michael.
- Add Acked-by: Michael S. Tsirkin <[email protected]>
Vfio:
- Export vfio_pci_core_setup_barmap() in place and rename vfio_pci_iowrite/read<xxx> to have the 'core' prefix as part of the functions names, as was discussed with Alex.
- Improve packing of struct virtiovf_pci_core_device, as was suggested
by Alex.
- Upon reset, set 'pci_cmd' back to zero, in addition, if
the user didn't set the 'PCI_COMMAND_IO' bit, return -EIO upon any read/write towards the IO bar, as was suggested by Alex.
- Enforce by BUILD_BUG_ON that 'bar0_virtual_buf_size' is power of 2 as part of virtiovf_pci_init_device() and clean the 'sizing calculation' code accordingly, as was suggested by Alex.
Changes from V3: https://www.spinics.net/lists/kvm/msg333008.html
Virtio:
- Rebase on top of 6.7 rc3.
Vfio:
- Fix a typo, drop 'acc' from 'virtiovf_acc_vfio_pci_tran_ops'.
Changes from V2: https://lore.kernel.org/all/[email protected]/T/
Virtio:
- Rebase on top of 6.7 rc1.
- Add a mutex to serialize admin commands execution and virtqueue
deletion, as was suggested by Michael.
- Remove the 'ref_count' usage which is not needed any more.
- Reduce the depth of the admin vq to match a single command at a given time.
- Add a supported check upon command execution and move to use a single flow of virtqueue_exec_admin_cmd().
- Improve the description of the exported commands to better match the specification and the expected usage as was asked by Michael.
Vfio:
- Upon calling to virtio_pci_admin_legacy/common_device_io_read/write() supply the 'offset' within the relevant configuration area, following the virtio exported APIs.
Changes from V1: https://lore.kernel.org/all/[email protected]/T/
Virtio:
- Drop its first patch, it was accepted upstream already.
- Add a new patch (#6) which initializes the supported admin commands
upon admin queue activation as was suggested by Michael.
- Split the legacy_io_read/write commands per common/device
configuration as was asked by Michael.
- Don't expose any more the list query/used APIs outside of virtio.
- Instead, expose an API to check whether the legacy io functionality is
supported as was suggested by Michael.
- Fix some Krobot's note by adding the missing include file.
Vfio:
- Refer specifically to virtio-net as part of the driver/module description as Alex asked.
- Change to check MSIX enablement based on the irq type of the given vfio core device. In addition, drop its capable checking from the probe flow as was asked by Alex.
- Adapt to use the new virtio exposed APIs and clean some code accordingly.
- Adapt to some cleaner style code in some places (if/else) as was suggested by Alex.
- Fix the range_intersect_range() function and adapt its usage as was pointed by Alex.
- Make struct virtiovf_pci_core_device better packed.
- Overwrite the subsystem vendor ID to be 0x1af4 as was discussed in the ML.
- Add support for the 'bar sizing negotiation' as was asked by Alex.
- Drop the 'acc' from the 'ops' as Alex asked.
Changes from V0: https://www.spinics.net/lists/linux-virtualization/msg63802.html
Virtio:
- Fix the common config map size issue that was reported by Michael Tsirkin.
- Do not use vp_dev->vqs[] array upon vp_del_vqs() as was asked by Michael, instead skip the AQ specifically.
- Move admin vq implementation into virtio_pci_modern.c as was asked by Michael.
- Rename structure virtio_avq to virtio_pci_admin_vq and some extra corresponding renames.
- Remove exported symbols virtio_pci_vf_get_pf_dev(), virtio_admin_cmd_exec() as now callers are local to the module.
- Handle inflight commands as part of the device reset flow.
- Introduce APIs per admin command in virtio-pci as was asked by Michael.
Vfio:
- Change to use EXPORT_SYMBOL_GPL instead of EXPORT_SYMBOL for vfio_pci_core_setup_barmap() and vfio_pci_iowrite#xxx() as pointed by Alex.
- Drop the intermediate patch which prepares the commands and calls the generic virtio admin command API (i.e. virtio_admin_cmd_exec()).
- Instead, call directly to the new APIs per admin command that are exported from Virtio - based on Michael's request.
- Enable only virtio-net as part of the pci_device_id table to enforce upon binding only what is supported as suggested by Alex.
- Add support for byte-wise access (read/write) over the device config region as was asked by Alex.
- Consider whether MSIX is practically enabled/disabled to choose the right opcode upon issuing read/write admin command, as mentioned by Michael.
- Move to use VIRTIO_PCI_CONFIG_OFF instead of adding some new defines as was suggested by Michael.
- Set the '.close_device' op to vfio_pci_core_close_device() as was pointed by Alex.
- Adapt to Vfio multi-line comment style in a few places.
- Add [email protected] in the MAINTAINERS file to be CCed for the new driver as was suggested by Jason.
Yishai
Feng Liu (4):
virtio: Define feature bit for administration virtqueue
virtio-pci: Introduce admin virtqueue
virtio-pci: Introduce admin command sending function
virtio-pci: Introduce admin commands
Yishai Hadas (5):
virtio-pci: Initialize the supported admin commands
virtio-pci: Introduce APIs to execute legacy IO admin commands vfio/pci: Expose vfio_pci_core_setup_barmap() vfio/pci: Expose vfio_pci_core_iowrite/read##size() vfio/virtio: Introduce a vfio driver over virtio devices
MAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/vfio_pci_rdwr.c | 57 +-- drivers/vfio/pci/virtio/Kconfig | 16 + drivers/vfio/pci/virtio/Makefile | 4 + drivers/vfio/pci/virtio/main.c | 567 +++++++++++++++++++++++++ drivers/virtio/virtio.c | 37 +- drivers/virtio/virtio_pci_common.c | 14 + drivers/virtio/virtio_pci_common.h | 21 +- drivers/virtio/virtio_pci_modern.c | 503 +++++++++++++++++++++- drivers/virtio/virtio_pci_modern_dev.c | 24 +- include/linux/vfio_pci_core.h | 20 + include/linux/virtio.h | 8 + include/linux/virtio_config.h | 4 + include/linux/virtio_pci_admin.h | 21 + include/linux/virtio_pci_modern.h | 2 + include/uapi/linux/virtio_config.h | 8 +- include/uapi/linux/virtio_pci.h | 68 +++ 19 files changed, 1349 insertions(+), 36 deletions(-) create mode 100644 drivers/vfio/pci/virtio/Kconfig create mode 100644 drivers/vfio/pci/virtio/Makefile create mode 100644 drivers/vfio/pci/virtio/main.c create mode 100644 include/linux/virtio_pci_admin.h
Signature · 1 line
2.27.0
Yishai Hadas
maintainerPATCH v7 1/9 virtio: Define feature bit for administration virtqueue
From: Feng Liu <[email protected]>
Introduce VIRTIO_F_ADMIN_VQ which is used for administration virtqueue support.
--- include/uapi/linux/virtio_config.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
include/uapi/linux/virtio_config.h +7 -1
@@ -52,7 +52,7 @@* rest are per-device feature bits. */ #define VIRTIO_TRANSPORT_F_START 28 -#define VIRTIO_TRANSPORT_F_END 41 +#define VIRTIO_TRANSPORT_F_END 42#ifndef VIRTIO_CONFIG_NO_LEGACY /* Do we get callbacks when the ring is completely used, even if we've
@@ -114,4 +114,10 @@* This feature indicates that the driver can reset a queue individually. */ #define VIRTIO_F_RING_RESET 40 + +/* +* This feature indicates that the device support administration virtqueues. +*/ +#define VIRTIO_F_ADMIN_VQ 41 +#endif /* _UAPI_LINUX_VIRTIO_CONFIG_H */
Signature · 1 line
2.27.0
Yishai Hadas
maintainerPATCH v7 2/9 virtio-pci: Introduce admin virtqueue
In reply to Yishai HadasFrom: Feng Liu <[email protected]>
Introduce support for the admin virtqueue. By negotiating VIRTIO_F_ADMIN_VQ feature, driver detects capability and creates one administration virtqueue. Administration virtqueue implementation in virtio pci generic layer, enables multiple types of upper layer drivers such as vfio, net, blk to utilize it.
--- drivers/virtio/virtio.c | 37 +++++++++++-- drivers/virtio/virtio_pci_common.c | 3 ++ drivers/virtio/virtio_pci_common.h | 15 +++++- drivers/virtio/virtio_pci_modern.c | 75 +++++++++++++++++++++++++- drivers/virtio/virtio_pci_modern_dev.c | 24 ++++++++- include/linux/virtio_config.h | 4 ++ include/linux/virtio_pci_modern.h | 2 + include/uapi/linux/virtio_pci.h | 5 ++ 8 files changed, 157 insertions(+), 8 deletions(-)
drivers/virtio/virtio.c +33 -4
@@ -302,9 +302,15 @@ static int virtio_dev_probe(struct device *_d)if (err) goto err; +if (dev->config->create_avq) { + err = dev->config->create_avq (dev); +if (err) + goto err; + } + err = drv->probe (dev);if (err) - goto err; + goto err_probe;/* If probe didn't do it, mark device DRIVER_OK ourselves. */ if (!(dev->config->get_status (dev) &VIRTIO_CONFIG_S_DRIVER_OK ))
@@ -316,6 +322,10 @@ static int virtio_dev_probe(struct device *_d)virtio_config_enable (dev);return 0 ; + +err_probe: +if (dev->config->destroy_avq) + dev->config->destroy_avq (dev); err:virtio_add_status (dev,VIRTIO_CONFIG_S_FAILED );return err;
@@ -331,6 +341,9 @@ static void virtio_dev_remove(struct device *_d) drv->remove (dev); +if (dev->config->destroy_avq) + dev->config->destroy_avq (dev); +/* Driver should have reset device. */ WARN_ON_ONCE (dev->config->get_status (dev));
@@ -489,13 +502,20 @@ EXPORT_SYMBOL_GPL(unregister_virtio_device);int virtio_device_freeze (struct virtio_device *dev) {struct virtio_driver *drv =drv_to_virtio (dev->dev.driver); +int ret;virtio_config_disable (dev); dev->failed = dev->config->get_status (dev) &VIRTIO_CONFIG_S_FAILED ; -if (drv && drv->freeze) -return drv->freeze (dev); +if (drv && drv->freeze) { + ret = drv->freeze (dev); +if (ret) +return ret; + } + +if (dev->config->destroy_avq) + dev->config->destroy_avq (dev);return 0 ; }
@@ -532,10 +552,16 @@ int virtio_device_restore(struct virtio_device *dev)if (ret) goto err; +if (dev->config->create_avq) { + ret = dev->config->create_avq (dev); +if (ret) + goto err; + } +if (drv->restore) { ret = drv->restore (dev);if (ret) - goto err; + goto err_restore; }/* If restore didn't do it, mark device DRIVER_OK ourselves. */
@@ -546,6 +572,9 @@ int virtio_device_restore(struct virtio_device *dev)return 0 ; +err_restore: +if (dev->config->destroy_avq) + dev->config->destroy_avq (dev); err:virtio_add_status (dev,VIRTIO_CONFIG_S_FAILED );return ret;
drivers/virtio/virtio_pci_common.c +3 -0
@@ -236,6 +236,9 @@ void vp_del_vqs(struct virtio_device *vdev)int i;list_for_each_entry_safe (vq, n, &vdev->vqs, list) { +if (vp_dev->is_avq (vdev, vq->index)) +continue ; +if (vp_dev->per_vq_vectors) {int v = vp_dev->vqs[vq->index]->msix_vector;
drivers/virtio/virtio_pci_common.h +14 -1
@@ -41,6 +41,14 @@ struct virtio_pci_vq_info {unsigned int msix_vector; }; +struct virtio_pci_admin_vq { +/* Virtqueue info associated with this admin queue. */ +struct virtio_pci_vq_info info; +/* Name of the admin queue: avq.$vq_index. */ +char name[10 ]; +u16 vq_index; +}; +/* Our device structure */ struct virtio_pci_device {struct virtio_device vdev;
@@ -58,9 +66,13 @@ struct virtio_pci_device {spinlock_t lock;struct list_head virtqueues; -/* array of all queues for house-keeping */ +/* Array of all virtqueues reported in the +* PCI common config num_queues field +*/ struct virtio_pci_vq_info **vqs; +struct virtio_pci_admin_vq admin_vq; +/* MSI-X support */ int msix_enabled;int intx_enabled;
@@ -86,6 +98,7 @@ struct virtio_pci_device {void (*del_vq)(struct virtio_pci_vq_info *info);u16 (*config_vector)(struct virtio_pci_device *vp_dev,u16 vector); +bool (*is_avq)(struct virtio_device *vdev,unsigned int index); };/* Constants for MSI-X */
drivers/virtio/virtio_pci_modern.c +73 -2
150 lines not on this page · raw message
drivers/virtio/virtio_pci_modern_dev.c +23 -1
@@ -207,6 +207,10 @@ static inline void check_offsets(void) offsetof(struct virtio_pci_modern_common_cfg, queue_notify_data));BUILD_BUG_ON (VIRTIO_PCI_COMMON_Q_RESET != offsetof(struct virtio_pci_modern_common_cfg , queue_reset)); +BUILD_BUG_ON (VIRTIO_PCI_COMMON_ADM_Q_IDX != + offsetof(struct virtio_pci_modern_common_cfg , admin_queue_index)); +BUILD_BUG_ON (VIRTIO_PCI_COMMON_ADM_Q_NUM != + offsetof(struct virtio_pci_modern_common_cfg , admin_queue_num)); }/*
@@ -296,7 +300,7 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev) mdev->common =vp_modern_map_capability (mdev, common,sizeof (struct virtio_pci_common_cfg ),4 ,0 , offsetofend(struct virtio_pci_modern_common_cfg, - queue_reset), + admin_queue_num), &mdev->common_len,NULL );if (!mdev->common) goto err_map_common;
@@ -719,6 +723,24 @@ void __iomem *vp_modern_map_vq_notify(struct virtio_pci_modern_device *mdev, }EXPORT_SYMBOL_GPL (vp_modern_map_vq_notify); +u16 vp_modern_avq_num (struct virtio_pci_modern_device *mdev) +{ +struct virtio_pci_modern_common_cfg __iomem *cfg; + + cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; +return vp_ioread16 (&cfg->admin_queue_num); +} +EXPORT_SYMBOL_GPL (vp_modern_avq_num); + +u16 vp_modern_avq_index (struct virtio_pci_modern_device *mdev) +{ +struct virtio_pci_modern_common_cfg __iomem *cfg; + + cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; +return vp_ioread16 (&cfg->admin_queue_index); +} +EXPORT_SYMBOL_GPL (vp_modern_avq_index); +MODULE_VERSION ("0.1" );MODULE_DESCRIPTION ("Modern Virtio PCI Device" );MODULE_AUTHOR ("Jason Wang <[email protected]>" );
include/linux/virtio_config.h +4 -0
@@ -93,6 +93,8 @@ typedef void vq_callback_t(struct virtqueue *);* Returns 0 on success or error status * If disable_vq_and_reset is set, then enable_vq_after_reset must also be * set. +* @create_avq: create admin virtqueue resource. +* @destroy_avq: destroy admin virtqueue resource. */ struct virtio_config_ops { void (*get)(struct virtio_device *vdev, unsigned offset,
@@ -120,6 +122,8 @@ struct virtio_config_ops { struct virtio_shm_region *region, u8 id);int (*disable_vq_and_reset)(struct virtqueue *vq);int (*enable_vq_after_reset)(struct virtqueue *vq); +int (*create_avq)(struct virtio_device *vdev); +void (*destroy_avq)(struct virtio_device *vdev); };/* If driver didn't advertise the feature, it will never appear. */
include/linux/virtio_pci_modern.h +2 -0
@@ -125,4 +125,6 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev);void vp_modern_remove (struct virtio_pci_modern_device *mdev);int vp_modern_get_queue_reset (struct virtio_pci_modern_device *mdev,u16 index);void vp_modern_set_queue_reset (struct virtio_pci_modern_device *mdev,u16 index); +u16 vp_modern_avq_num (struct virtio_pci_modern_device *mdev); +u16 vp_modern_avq_index (struct virtio_pci_modern_device *mdev);#endif
include/uapi/linux/virtio_pci.h +5 -0
@@ -175,6 +175,9 @@ struct virtio_pci_modern_common_cfg {__le16 queue_notify_data;/* read-write */ __le16 queue_reset;/* read-write */ + +__le16 admin_queue_index;/* read-only */ +__le16 admin_queue_num;/* read-only */ };/* Fields in VIRTIO_PCI_CAP_PCI_CFG: */
@@ -215,6 +218,8 @@ struct virtio_pci_cfg_cap {#define VIRTIO_PCI_COMMON_Q_USEDHI 52#define VIRTIO_PCI_COMMON_Q_NDATA 56#define VIRTIO_PCI_COMMON_Q_RESET 58 +#define VIRTIO_PCI_COMMON_ADM_Q_IDX 60 +#define VIRTIO_PCI_COMMON_ADM_Q_NUM 62#endif /* VIRTIO_PCI_NO_MODERN */
Signature · 1 line
2.27.0
Yishai Hadas
maintainerPATCH v7 3/9 virtio-pci: Introduce admin command sending function
In reply to Yishai HadasFrom: Feng Liu <[email protected]>
Add support for sending admin command through admin virtqueue interface. Abort any inflight admin commands once device reset completes. Activate admin queue when device becomes ready; deactivate on device reset.
To comply to the below specification statement [1], the admin virtqueue is activated for upper layer users only after setting DRIVER_OK status.
[1] The driver MUST NOT send any buffer available notifications to the device before setting DRIVER_OK.
--- drivers/virtio/virtio_pci_common.h | 3 + drivers/virtio/virtio_pci_modern.c | 143 ++++++++++++++++++++++++++++- include/linux/virtio.h | 8 ++ include/uapi/linux/virtio_pci.h | 22 +++++ 4 files changed, 174 insertions(+), 2 deletions(-)
drivers/virtio/virtio_pci_common.h +3 -0
@@ -29,6 +29,7 @@#include <linux/virtio_pci_modern.h> #include <linux/highmem.h> #include <linux/spinlock.h> +#include <linux/mutex.h> struct virtio_pci_vq_info {/* the actual virtqueue */
@@ -44,6 +45,8 @@ struct virtio_pci_vq_info {struct virtio_pci_admin_vq {/* Virtqueue info associated with this admin queue. */ struct virtio_pci_vq_info info; +/* serializing admin commands execution and virtqueue deletion */ +struct mutex cmd_lock;/* Name of the admin queue: avq.$vq_index. */ char name[10 ];u16 vq_index;
drivers/virtio/virtio_pci_modern.c +141 -2
186 lines not on this page · raw message
include/linux/virtio.h +8 -0
@@ -103,6 +103,14 @@ int virtqueue_resize(struct virtqueue *vq, u32 num,int virtqueue_reset (struct virtqueue *vq,void (*recycle)(struct virtqueue *vq,void *buf)); +struct virtio_admin_cmd { +__le16 opcode; +__le16 group_type; +__le64 group_member_id; +struct scatterlist *data_sg; +struct scatterlist *result_sg; +}; +/** * struct virtio_device - representation of a device using virtio * @index: unique position on the virtio bus
include/uapi/linux/virtio_pci.h +22 -0
@@ -223,4 +223,26 @@ struct virtio_pci_cfg_cap {#endif /* VIRTIO_PCI_NO_MODERN */ +/* Admin command status. */ +#define VIRTIO_ADMIN_STATUS_OK 0 + +struct __packed virtio_admin_cmd_hdr { +__le16 opcode; +/* +* 1 - SR-IOV +* 2-65535 - reserved +*/ +__le16 group_type; +/* Unused, reserved for future extensions. */ +__u8 reserved1[12 ]; +__le64 group_member_id; +}; + +struct __packed virtio_admin_cmd_status { +__le16 status; +__le16 status_qualifier; +/* Unused, reserved for future extensions. */ +__u8 reserved2[4 ]; +}; +#endif
Signature · 1 line
2.27.0
Yishai Hadas
maintainerPATCH v7 4/9 virtio-pci: Introduce admin commands
In reply to Yishai HadasFrom: Feng Liu <[email protected]>
Introduces admin commands, as follow:
The "list query" command can be used by the driver to query the set of admin commands supported by the virtio device. The "list use" command is used to inform the virtio device which admin commands the driver will use.
The "legacy common cfg rd/wr" commands are used to read from/write into the legacy common configuration structure.
The "legacy dev cfg rd/wr" commands are used to read from/write into the legacy device configuration structure.
The "notify info" command is used to query the notification region information.
--- include/uapi/linux/virtio_pci.h | 41 +++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
include/uapi/linux/virtio_pci.h +41 -0
@@ -226,6 +226,20 @@ struct virtio_pci_cfg_cap {/* Admin command status. */ #define VIRTIO_ADMIN_STATUS_OK 0 +/* Admin command opcode. */ +#define VIRTIO_ADMIN_CMD_LIST_QUERY 0x0 +#define VIRTIO_ADMIN_CMD_LIST_USE 0x1 + +/* Admin command group type. */ +#define VIRTIO_ADMIN_GROUP_TYPE_SRIOV 0x1 + +/* Transitional device admin command. */ +#define VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE 0x2 +#define VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ 0x3 +#define VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE 0x4 +#define VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ 0x5 +#define VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO 0x6 +struct __packed virtio_admin_cmd_hdr {__le16 opcode;/*
@@ -245,4 +259,31 @@ struct __packed virtio_admin_cmd_status { __u8 reserved2[4]; }; +struct __packed virtio_admin_cmd_legacy_wr_data { +__u8 offset;/* Starting offset of the register(s) to write. */ +__u8 reserved[7 ]; +__u8 registers[]; +}; + +struct __packed virtio_admin_cmd_legacy_rd_data { +__u8 offset;/* Starting offset of the register(s) to read. */ +}; + +#define VIRTIO_ADMIN_CMD_NOTIFY_INFO_FLAGS_END 0 +#define VIRTIO_ADMIN_CMD_NOTIFY_INFO_FLAGS_OWNER_DEV 0x1 +#define VIRTIO_ADMIN_CMD_NOTIFY_INFO_FLAGS_OWNER_MEM 0x2 + +#define VIRTIO_ADMIN_CMD_MAX_NOTIFY_INFO 4 + +struct __packed virtio_admin_cmd_notify_info_data { +__u8 flags;/* 0 = end of list, 1 = owner device, 2 = member device */ +__u8 bar;/* BAR of the member or the owner device */ +__u8 padding[6 ]; +__le64 offset;/* Offset within bar. */ +}; + +struct virtio_admin_cmd_notify_info_result { +struct virtio_admin_cmd_notify_info_data entries[VIRTIO_ADMIN_CMD_MAX_NOTIFY_INFO ]; +}; +#endif
Signature · 1 line
2.27.0
Yishai Hadas
maintainerPATCH v7 5/9 virtio-pci: Initialize the supported admin commands
In reply to Yishai HadasInitialize the supported admin commands upon activating the admin queue.
The supported commands are saved as part of the admin queue context.
Next patches in this series will expose APIs to use them.
--- drivers/virtio/virtio_pci_common.h | 1 + drivers/virtio/virtio_pci_modern.c | 48 ++++++++++++++++++++++++++++-- 2 files changed, 47 insertions(+), 2 deletions(-)
drivers/virtio/virtio_pci_common.h +1 -0
@@ -47,6 +47,7 @@ struct virtio_pci_admin_vq {struct virtio_pci_vq_info info;/* serializing admin commands execution and virtqueue deletion */ struct mutex cmd_lock; +u64 supported_cmds;/* Name of the admin queue: avq.$vq_index. */ char name[10 ];u16 vq_index;
drivers/virtio/virtio_pci_modern.c +46 -2
78 lines not on this page · raw message
Signature · 1 line
2.27.0
Yishai Hadas
maintainerPATCH v7 6/9 virtio-pci: Introduce APIs to execute legacy IO admin commands
In reply to Yishai HadasIntroduce APIs to execute legacy IO admin commands.
It includes: io_legacy_read/write for both common and the device configuration, io_legacy_notify_info.
In addition, exposing an API to check whether the legacy IO commands are supported. (i.e. virtio_pci_admin_has_legacy_io()).
Those APIs will be used by the next patches from this series.
--- drivers/virtio/virtio_pci_common.c | 11 ++ drivers/virtio/virtio_pci_common.h | 2 + drivers/virtio/virtio_pci_modern.c | 245 +++++++++++++++++++++++++++++ include/linux/virtio_pci_admin.h | 21 +++ 4 files changed, 279 insertions(+) create mode 100644 include/linux/virtio_pci_admin.h
drivers/virtio/virtio_pci_common.c +11 -0
@@ -645,6 +645,17 @@ static struct pci_driver virtio_pci_driver = { .sriov_configure = virtio_pci_sriov_configure, }; +struct virtio_device *virtio_pci_vf_get_pf_dev (struct pci_dev *pdev) +{ +struct virtio_pci_device *pf_vp_dev; + + pf_vp_dev =pci_iov_get_pf_drvdata (pdev, &virtio_pci_driver); +if (IS_ERR (pf_vp_dev)) +return NULL ; + +return &pf_vp_dev->vdev; +} +module_pci_driver (virtio_pci_driver);MODULE_AUTHOR ("Anthony Liguori <[email protected]>" );
drivers/virtio/virtio_pci_common.h +2 -0
@@ -156,4 +156,6 @@ static inline void virtio_pci_legacy_remove(struct virtio_pci_device *vp_dev)int virtio_pci_modern_probe (struct virtio_pci_device *);void virtio_pci_modern_remove (struct virtio_pci_device *); +struct virtio_device *virtio_pci_vf_get_pf_dev (struct pci_dev *pdev); +#endif
drivers/virtio/virtio_pci_modern.c +245 -0
257 lines not on this page · raw message
include/linux/virtio_pci_admin.h new file +21 -0
@@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_VIRTIO_PCI_ADMIN_H +#define _LINUX_VIRTIO_PCI_ADMIN_H + +#include <linux/types.h> +#include <linux/pci.h> + +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev); +int virtio_pci_admin_legacy_common_io_write (struct pci_dev *pdev,u8 offset, +u8 size,u8 *buf); +int virtio_pci_admin_legacy_common_io_read (struct pci_dev *pdev,u8 offset, +u8 size,u8 *buf); +int virtio_pci_admin_legacy_device_io_write (struct pci_dev *pdev,u8 offset, +u8 size,u8 *buf); +int virtio_pci_admin_legacy_device_io_read (struct pci_dev *pdev,u8 offset, +u8 size,u8 *buf); +int virtio_pci_admin_legacy_io_notify_info (struct pci_dev *pdev, +u8 req_bar_flags,u8 *bar, +u64 *bar_offset); + +#endif /* _LINUX_VIRTIO_PCI_ADMIN_H */
Signature · 1 line
2.27.0
Yishai Hadas
maintainerPATCH v7 7/9 vfio/pci: Expose vfio_pci_core_setup_barmap()
In reply to Yishai HadasExpose vfio_pci_core_setup_barmap() to be used by drivers.
This will let drivers to mmap a BAR and re-use it from both vfio and the driver when it's applicable.
This API will be used in the next patches by the vfio/virtio coming driver.
--- drivers/vfio/pci/vfio_pci_rdwr.c | 7 ++++--- include/linux/vfio_pci_core.h | 1 + 2 files changed, 5 insertions(+), 3 deletions(-)
drivers/vfio/pci/vfio_pci_rdwr.c +4 -3
@@ -200,7 +200,7 @@ static ssize_t do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,return done; } -static int vfio_pci_setup_barmap (struct vfio_pci_core_device *vdev,int bar) +int vfio_pci_core_setup_barmap (struct vfio_pci_core_device *vdev,int bar) {struct pci_dev *pdev = vdev->pdev;int ret;
@@ -223,6 +223,7 @@ static int vfio_pci_setup_barmap(struct vfio_pci_core_device *vdev, int bar)return 0 ; } +EXPORT_SYMBOL_GPL (vfio_pci_core_setup_barmap);ssize_t vfio_pci_bar_rw (struct vfio_pci_core_device *vdev,char __user *buf,size_t count,loff_t *ppos,bool iswrite)
@@ -262,7 +263,7 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf, } x_end = end; }else { -int ret =vfio_pci_setup_barmap (vdev, bar); +int ret =vfio_pci_core_setup_barmap (vdev, bar);if (ret) { done = ret; goto out;
@@ -438,7 +439,7 @@ int vfio_pci_ioeventfd(struct vfio_pci_core_device *vdev, loff_t offset,return -EINVAL ;#endif - ret =vfio_pci_setup_barmap (vdev, bar); + ret =vfio_pci_core_setup_barmap (vdev, bar);if (ret)return ret;
include/linux/vfio_pci_core.h +1 -0
@@ -127,6 +127,7 @@ int vfio_pci_core_match(struct vfio_device *core_vdev, char *buf);int vfio_pci_core_enable (struct vfio_pci_core_device *vdev);void vfio_pci_core_disable (struct vfio_pci_core_device *vdev);void vfio_pci_core_finish_enable (struct vfio_pci_core_device *vdev); +int vfio_pci_core_setup_barmap (struct vfio_pci_core_device *vdev,int bar);pci_ers_result_t vfio_pci_core_aer_err_detected (struct pci_dev *pdev,pci_channel_state_t state);
Signature · 1 line
2.27.0
RE: [PATCH V7 vfio 7/9] vfio/pci: Expose vfio_pci_core_setup_barmap()
Quoted 5 lines
From: Yishai Hadas <[email protected]>
Sent: Thursday, December 7, 2023 6:28 PMExpose
vfio_pci_core_setup_barmap()to be used by drivers.
Quoted 8 lines
This will let drivers to mmap a BAR and re-use it from both vfio and the driver when it's applicable.
This API will be used in the next patches by the vfio/virtio coming driver.
Reviewed-by Jason Gunthorpe <[email protected]>Signed-off-by Yishai Hadas <[email protected]>
Yishai Hadas
maintainerPATCH v7 8/9 vfio/pci: Expose vfio_pci_core_iowrite/read##size()
In reply to Yishai HadasExpose vfio_pci_core_iowrite/read##size() to let it be used by drivers.
This functionality is needed to enable direct access to some physical BAR of the device with the proper locks/checks in place.
The next patches from this series will use this functionality on a data path flow when a direct access to the BAR is needed.
--- drivers/vfio/pci/vfio_pci_rdwr.c | 50 +++++++++++++++++--------------- include/linux/vfio_pci_core.h | 19 ++++++++++++ 2 files changed, 45 insertions(+), 24 deletions(-)
drivers/vfio/pci/vfio_pci_rdwr.c +26 -24
@@ -38,7 +38,7 @@#define vfio_iowrite8 iowrite8#define VFIO_IOWRITE (size) \ -static int vfio_pci_iowrite##size(struct vfio_pci_core_device *vdev, \ +int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev, \ bool test_mem, u##size val, void __iomem *io) \ { \ if (test_mem) { \
@@ -55,7 +55,8 @@ static int vfio_pci_iowrite##size(struct vfio_pci_core_device *vdev, \up_read (&vdev->memory_lock); \ \return 0 ; \ -} +} \ +EXPORT_SYMBOL_GPL(vfio_pci_core_iowrite##size); VFIO_IOWRITE(8)VFIO_IOWRITE (16 )
@@ -65,7 +66,7 @@ VFIO_IOWRITE(64)#endif #define VFIO_IOREAD (size) \ -static int vfio_pci_ioread##size(struct vfio_pci_core_device *vdev, \ +int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev, \ bool test_mem, u##size *val, void __iomem *io) \ { \ if (test_mem) { \
@@ -82,7 +83,8 @@ static int vfio_pci_ioread##size(struct vfio_pci_core_device *vdev, \up_read (&vdev->memory_lock); \ \return 0 ; \ -} +} \ +EXPORT_SYMBOL_GPL(vfio_pci_core_ioread##size); VFIO_IOREAD(8)VFIO_IOREAD (16 )
@@ -119,13 +121,13 @@ static ssize_t do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,if (copy_from_user (&val, buf,4 ))return -EFAULT ; - ret =vfio_pci_iowrite32 (vdev, test_mem, - val, io + off); + ret =vfio_pci_core_iowrite32 (vdev, test_mem, + val, io + off);if (ret)return ret; }else { - ret =vfio_pci_ioread32 (vdev, test_mem, - &val, io + off); + ret =vfio_pci_core_ioread32 (vdev, test_mem, + &val, io + off);if (ret)return ret;
@@ -141,13 +143,13 @@ static ssize_t do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,if (copy_from_user (&val, buf,2 ))return -EFAULT ; - ret =vfio_pci_iowrite16 (vdev, test_mem, - val, io + off); + ret =vfio_pci_core_iowrite16 (vdev, test_mem, + val, io + off);if (ret)return ret; }else { - ret =vfio_pci_ioread16 (vdev, test_mem, - &val, io + off); + ret =vfio_pci_core_ioread16 (vdev, test_mem, + &val, io + off);if (ret)return ret;
@@ -163,13 +165,13 @@ static ssize_t do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,if (copy_from_user (&val, buf,1 ))return -EFAULT ; - ret =vfio_pci_iowrite8 (vdev, test_mem, - val, io + off); + ret =vfio_pci_core_iowrite8 (vdev, test_mem, + val, io + off);if (ret)return ret; }else { - ret =vfio_pci_ioread8 (vdev, test_mem, - &val, io + off); + ret =vfio_pci_core_ioread8 (vdev, test_mem, + &val, io + off);if (ret)return ret;
@@ -364,21 +366,21 @@ static void vfio_pci_ioeventfd_do_write(struct vfio_pci_ioeventfd *ioeventfd, {switch (ioeventfd->count) {case 1 : -vfio_pci_iowrite8 (ioeventfd->vdev, test_mem, - ioeventfd->data, ioeventfd->addr); +vfio_pci_core_iowrite8 (ioeventfd->vdev, test_mem, + ioeventfd->data, ioeventfd->addr);break ;case 2 : -vfio_pci_iowrite16 (ioeventfd->vdev, test_mem, - ioeventfd->data, ioeventfd->addr); +vfio_pci_core_iowrite16 (ioeventfd->vdev, test_mem, + ioeventfd->data, ioeventfd->addr);break ;case 4 : -vfio_pci_iowrite32 (ioeventfd->vdev, test_mem, - ioeventfd->data, ioeventfd->addr); +vfio_pci_core_iowrite32 (ioeventfd->vdev, test_mem, + ioeventfd->data, ioeventfd->addr);break ;#ifdef iowrite64case 8 : -vfio_pci_iowrite64 (ioeventfd->vdev, test_mem, - ioeventfd->data, ioeventfd->addr); +vfio_pci_core_iowrite64 (ioeventfd->vdev, test_mem, + ioeventfd->data, ioeventfd->addr);break ;#endif }
include/linux/vfio_pci_core.h +19 -0
@@ -131,4 +131,23 @@ int vfio_pci_core_setup_barmap(struct vfio_pci_core_device *vdev, int bar);pci_ers_result_t vfio_pci_core_aer_err_detected (struct pci_dev *pdev,pci_channel_state_t state); +#define VFIO_IOWRITE_DECLATION (size) \ +int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev, \ + bool test_mem, u##size val, void __iomem *io); + +VFIO_IOWRITE_DECLATION(8) +VFIO_IOWRITE_DECLATION(16) +VFIO_IOWRITE_DECLATION(32) +#ifdef iowrite64 +VFIO_IOWRITE_DECLATION(64) +#endif + +#define VFIO_IOREAD_DECLATION (size) \ +int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev, \ + bool test_mem, u##size *val, void __iomem *io); + +VFIO_IOREAD_DECLATION(8) +VFIO_IOREAD_DECLATION(16) +VFIO_IOREAD_DECLATION(32) +#endif /* VFIO_PCI_CORE_H */
Signature · 1 line
2.27.0
RE: [PATCH V7 vfio 8/9] vfio/pci: Expose vfio_pci_core_iowrite/read##size()
Quoted 5 lines
From: Yishai Hadas <[email protected]>
Sent: Thursday, December 7, 2023 6:28 PMExpose vfio_pci_core_iowrite/read##size() to let it be used by drivers.
Quoted 8 lines
This functionality is needed to enable direct access to some physical BAR of the device with the proper locks/checks in place.
The next patches from this series will use this functionality on a data path flow when a direct access to the BAR is needed.
Reviewed-by Jason Gunthorpe <[email protected]>Signed-off-by Yishai Hadas <[email protected]>
Yishai Hadas
maintainerPATCH v7 9/9 vfio/virtio: Introduce a vfio driver over virtio devices
In reply to Yishai HadasIntroduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
Background, from the virtio spec [1].
--------------------------------------------------------------------
In some systems, there is a need to support a virtio legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.
For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.
--------------------------------------------------------------------
Specifically, this driver adds support for a virtio-net VF to be exposed as a transitional device to a guest driver and allows the legacy IO BAR functionality on top.
This allows a VM which uses a legacy virtio-net driver in the guest to work transparently over a VF which its driver in the host is that new driver.
The driver can be extended easily to support some other types of virtio devices (e.g virtio-blk), by adding in a few places the specific type properties as was done for virtio-net.
For now, only the virtio-net use case was tested and as such we introduce the support only for such a device.
Practically,
Upon probing a VF for a virtio-net device, in case its PF supports legacy access over the virtio admin commands and the VF doesn't have BAR 0, we set some specific 'vfio_device_ops' to be able to simulate in SW a transitional device with I/O BAR in BAR 0.
The existence of the simulated I/O bar is reported later on by overwriting the VFIO_DEVICE_GET_REGION_INFO command and the device exposes itself as a transitional device by overwriting some properties upon reading its config space.
Once we report the existence of I/O BAR as BAR 0 a legacy driver in the guest may use it via read/write calls according to the virtio specification.
Any read/write towards the control parts of the BAR will be captured by the new driver and will be translated into admin commands towards the device.
Any data path read/write access (i.e. virtio driver notifications) will be forwarded to the physical BAR which its properties were supplied by the admin command VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO upon the probing/init flow.
With that code in place a legacy driver in the guest has the look and feel as if having a transitional device with legacy support for both its control and data path flows.
[1]
https://github.com/oasis-tcs/virtio-spec/commit/03c2d32e5093ca9f2a17797242fbef88efe94b8c
--- MAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/virtio/Kconfig | 16 + drivers/vfio/pci/virtio/Makefile | 4 + drivers/vfio/pci/virtio/main.c | 567 +++++++++++++++++++++++++++++++ 6 files changed, 598 insertions(+) create mode 100644 drivers/vfio/pci/virtio/Kconfig create mode 100644 drivers/vfio/pci/virtio/Makefile create mode 100644 drivers/vfio/pci/virtio/main.c
MAINTAINERS +7 -0
@@ -22872,6 +22872,13 @@ L: [email protected] S: Maintained F: drivers/vfio/pci/mlx5/ +VFIO VIRTIO PCI DRIVER +M: Yishai Hadas <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +F: drivers/vfio/pci/virtio + VFIO PCI DEVICE SPECIFIC DRIVERS R: Jason Gunthorpe <[email protected]> R: Yishai Hadas <[email protected]>
drivers/vfio/pci/Kconfig +2 -0
@@ -65,4 +65,6 @@ source "drivers/vfio/pci/hisilicon/Kconfig" source "drivers/vfio/pci/pds/Kconfig" +source "drivers/vfio/pci/virtio/Kconfig" + endmenu
drivers/vfio/pci/Makefile +2 -0
@@ -13,3 +13,5 @@ obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ obj-$(CONFIG_PDS_VFIO_PCI) += pds/ + +obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/
drivers/vfio/pci/virtio/Kconfig new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices" + depends on VIRTIO_PCI + select VFIO_PCI_CORE + help + This provides support for exposing VIRTIO NET VF devices which support + legacy IO access, using the VFIO framework that can work with a legacy + virtio driver in the guest. + Based on PCIe spec, VFs do not support I/O Space; thus, VF BARs shall + not indicate I/O Space. + As of that this driver emulated I/O BAR in software to let a VF be + seen as a transitional device in the guest and let it work with + a legacy driver. + + If you don't know what to do here, say N.
drivers/vfio/pci/virtio/Makefile new file +4 -0
@@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio-vfio-pci.o +virtio-vfio-pci-y := main.o +
drivers/vfio/pci/virtio/main.c new file +567 -0
567 lines not on this page · raw message
Signature · 1 line
2.27.0
Tian, Kevin
reviewerRE: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 11 lines
From: Yishai Hadas <[email protected]>
Sent: Thursday, December 7, 2023 6:28 PMAny read/write towards the control parts of the BAR will be captured by the new driver and will be translated into admin commands towards the device.
Any data path read/write access (i.e. virtio driver notifications) will be forwarded to the physical BAR which its properties were supplied by the admin command
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFOupon the probing/init flow.
this is still captured by the new driver. Just the difference between using admin cmds vs. directly accessing bar when emulating the access.
drivers/vfio/pci/virtio/Kconfig line 11
Quoted 12 lines
drivers/vfio/pci/virtio/Kconfig new file +6
-0+config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices" + depends on VIRTIO_PCI + select VFIO_PCI_CORE + help + This provides support for exposing VIRTIO NET VF devices whichsupport
+ legacy IO access, using the VFIO framework that can work with a
legacydrivers/vfio/pci/virtio/Kconfig new file +3
-0+ virtio driver in the guest. + Based on PCIe spec, VFs do not support I/O Space; thus, VF BARs shall + not indicate I/O Space.
"thus, ..." duplicates with the former part.
Quoted 1 line
+ As of that this driver emulated I/O BAR in software to let a VF be
s/emulated/emulates/
drivers/vfio/pci/virtio/Kconfig line 14
Quoted 2 lines
drivers/vfio/pci/virtio/Kconfig new file +2
-0+ seen as a transitional device in the guest and let it work with + a legacy driver.
VFIO is not specific to the guest. a native application including a legacy virtio driver could also benefit. let's not write it in a way specific to virt.
drivers/vfio/pci/virtio/main.c line 81
Quoted 5 lines
drivers/vfio/pci/virtio/main.c new file +5
-0+ +static int +translate_io_bar_to_mem_bar (struct virtiovf_pci_core_device *virtvdev, +loff_t pos,char __user *buf, +size_t count,bool read)
this name only talks about the behavior for VIRTIO_PCI_QUEUE_NOTIFY.
for legacy admin cmd it's unclear whether it's actually conveyed to a mem bar.
is it clearer to call it virtiovf_pci_bar0_rw()?
Quoted 40 lines
drivers/vfio/pci/virtio/main.c new file +22
-0+ +static ssize_t virtiovf_pci_read_config (struct vfio_device *core_vdev, +char __user *buf,size_t count, +loff_t *ppos) +{ +struct virtiovf_pci_core_device *virtvdev =container_of ( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +loff_t pos = *ppos &VFIO_PCI_OFFSET_MASK ; +size_t register_offset; +loff_t copy_offset; +size_t copy_count; +__le32 val32; +__le16 val16; +u8 val8; +int ret; + + ret =vfio_pci_core_read (core_vdev, buf, count, ppos); +if (ret <0 ) +return ret; + + if (range_intersect_range(pos, count, PCI_DEVICE_ID, sizeof(val16), + ©_offset, ©_count,
®ister_offset)) {(quoted diff) +2
-0+ val16 = cpu_to_le16(VIRTIO_TRANS_ID_NET); + if (copy_to_user(buf + copy_offset, (void *)&val16 +
register_offset,copy_count))drivers/vfio/pci/virtio/main.c new file +6
-0+return -EFAULT ; + } + +if ((le16_to_cpu (virtvdev->pci_cmd) &PCI_COMMAND_IO ) && + range_intersect_range(pos, count, PCI_COMMAND, sizeof(val16), + ©_offset, ©_count,
®ister_offset)) {(quoted diff) +1
-0+ if (copy_from_user((void *)&val16 + register_offset, buf +
copy_offset,drivers/vfio/pci/virtio/main.c new file +4
-0+ copy_count)) +return -EFAULT ; + val16 |=cpu_to_le16 (PCI_COMMAND_IO ); + if (copy_to_user(buf + copy_offset, (void *)&val16 +
register_offset,
drivers/vfio/pci/virtio/main.c line 198
Quoted 3 lines
drivers/vfio/pci/virtio/main.c new file +3
-0+ copy_count)) +return -EFAULT ; + }
the write handler calls vfio_pci_core_write() for PCI_COMMAND so the core vconfig should have the latest copy of the IO bit value which is copied to the user buffer by vfio_pci_core_read(). then not necessary to update it again.
btw the approach in this patch sounds a bit hackish - it modifies the result before/after vfio pci core emulation instead of directly injecting its specific emulation logic in vfio vconfig. It's probably being that vfio vconfig currently has a global permission/handler scheme for all pci devices. Extending it to support per-device tweak might need lots of change.
So I'm not advocating that big change at this point, especially when only this driver imposes such requirement now. But in the future when more drivers e.g. Ankit's nvgrace-gpu want to do similar tweak we may consider such possibility.
Quoted 13 lines
(quoted diff) +2
-0+ + if (range_intersect_range(pos, count, PCI_BASE_ADDRESS_0,sizeof(val32),
(quoted diff) +1
-0+ ©_offset, ©_count,
®ister_offset)) {(quoted diff) +2
-0+ u32 bar_mask = ~(virtvdev->bar0_virtual_buf_size - 1); + u32 pci_base_addr_0 = le32_to_cpu(virtvdev-Quoted 1 line
pci_base_addr_0);(quoted diff) +2
-0+ + val32 = cpu_to_le32((pci_base_addr_0 & bar_mask) |
PCI_BASE_ADDRESS_SPACE_IO);(quoted diff) +1
-0+ if (copy_to_user(buf + copy_offset, (void *)&val32 +
register_offset,copy_count))
Review
Quoted 2 lines
(quoted diff) +2
-0+ return -EFAULT; + }
Do we care about the initial value of bar0? this patch leaves it as 0, unlike other real bars initialized with the hw value. In reality this may not be a problem as software usually writes all 1's to detect the size as the first step.
raise it just in case others may see a potential issue.
Quoted 36 lines
drivers/vfio/pci/virtio/main.c new file +3
-0+ +static ssize_t +virtiovf_pci_core_write(struct vfio_device *core_vdev, const char __user*buf,
drivers/vfio/pci/virtio/main.c new file +16
-0+ size_t count, loff_t *ppos) +{ + struct virtiovf_pci_core_device *virtvdev = container_of( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +unsigned int index =VFIO_PCI_OFFSET_TO_INDEX (*ppos); +loff_t pos = *ppos &VFIO_PCI_OFFSET_MASK ; + +if (!count) +return 0 ; + +if (index ==VFIO_PCI_CONFIG_REGION_INDEX ) { +size_t register_offset; +loff_t copy_offset; +size_t copy_count; + + if (range_intersect_range(pos, count, PCI_COMMAND,
sizeof(virtvdev->pci_cmd),drivers/vfio/pci/virtio/main.c new file +3
-0+ ©_offset, ©_count, + ®ister_offset)) { + if (copy_from_user((void *)&virtvdev->pci_cmd +
register_offset,drivers/vfio/pci/virtio/main.c new file +10
-0+ buf + copy_offset, + copy_count)) +return -EFAULT ; + } + +if (range_intersect_range (pos, count,PCI_BASE_ADDRESS_0 , +sizeof (virtvdev->pci_base_addr_0), + ©_offset, ©_count, + ®ister_offset)) { + if (copy_from_user((void *)&virtvdev-Quoted 1 line
pci_base_addr_0+register_offset,
drivers/vfio/pci/virtio/main.c line 296
Quoted 5 lines
drivers/vfio/pci/virtio/main.c new file +5
-0+ buf + copy_offset, + copy_count)) +return -EFAULT ; + } + }
wrap above into virtiovf_pci_write_config() to be symmetric with the read path.
Quoted 19 lines
drivers/vfio/pci/virtio/main.c new file +19
-0+static int virtiovf_pci_init_device (struct vfio_device *core_vdev) +{ +struct virtiovf_pci_core_device *virtvdev =container_of ( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +struct pci_dev *pdev; +int ret; + + ret =vfio_pci_core_init_dev (core_vdev); +if (ret) +return ret; + + pdev = virtvdev->core_device.pdev; + ret =virtiovf_read_notify_info (virtvdev); +if (ret) +return ret; + +/* Being ready with a buffer that supports MSIX */ + virtvdev->bar0_virtual_buf_size =VIRTIO_PCI_CONFIG_OFF (true) + + virtiovf_get_device_config_size(pdev-
Quoted 1 line
Quoted 1 line
device);
which code is relevant to MSIX?
drivers/vfio/pci/virtio/main.c line 477
Quoted 6 lines
drivers/vfio/pci/virtio/main.c new file +6
-0+ +static const struct vfio_device_ops virtiovf_vfio_pci_ops = { + .name ="virtio-vfio-pci" , + .init = vfio_pci_core_init_dev, + .release = vfio_pci_core_release_dev, + .open_device = virtiovf_pci_open_device,
could be vfio_pci_core_open_device(). Given virtiovf specific init func is not called virtiovf_pci_open_device() is essentially same as the core func.
drivers/vfio/pci/virtio/main.c line 508
Quoted 11 lines
drivers/vfio/pci/virtio/main.c new file +11
-0+ +static int virtiovf_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *id) +{ +const struct vfio_device_ops *ops = &virtiovf_vfio_pci_ops; +struct virtiovf_pci_core_device *virtvdev; +int ret; + +if (pdev->is_virtfn &&virtio_pci_admin_has_legacy_io (pdev) && + !virtiovf_bar0_exists (pdev)) + ops = &virtiovf_vfio_pci_tran_ops;
I have a confusion here.
why do we want to allow this driver binding to non-matching VF or even PF?
if that is the intention then the naming/description should be adjusted to not specific to vf throughout this patch.
e.g. don't use "virtiovf_" prefix...
the config option is generic:
+config VIRTIO_VFIO_PCI
+ tristate "VFIO support for VIRTIO NET PCI devices"
but the description is specific to vf:
+ This provides support for exposing VIRTIO NET VF devices which support + legacy IO access, using the VFIO framework that can work with a legacy + virtio driver in the guest.
then the module description is generic again:
+MODULE_DESCRIPTION(
+ "VIRTIO VFIO PCI - User Level meta-driver for VIRTIO NET devices");
Yishai Hadas
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 7 lines · Kevin
Quoted 7 lines
From: Yishai Hadas <[email protected]>
Sent: Thursday, December 7, 2023 6:28 PMAny read/write towards the control parts of the BAR will be captured by the new driver and will be translated into admin commands towards the device.
Quoted 7 lines
Quoted 4 lines
Any data path read/write access (i.e. virtio driver notifications) will be forwarded to the physical BAR which its properties were supplied by the admin command
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFOupon the probing/init flow.this is still captured by the new driver. Just the difference between using admin cmds vs. directly accessing bar when emulating the access.
OK, I can rephrase the above to clarify that.
Quoted 14 lines
drivers/vfio/pci/virtio/Kconfig line 11
Quoted 13 lines
drivers/vfio/pci/virtio/Kconfig new file +6
-0+config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices" + depends on VIRTIO_PCI + select VFIO_PCI_CORE + help + This provides support for exposing VIRTIO NET VF devices whichsupport
+ legacy IO access, using the VFIO framework that can work with a
legacydrivers/vfio/pci/virtio/Kconfig new file +3
-0+ virtio driver in the guest. + Based on PCIe spec, VFs do not support I/O Space; thus, VF BARs shall + not indicate I/O Space.
Quoted 5 lines
"thus, ..." duplicates with the former part.
Quoted 1 line
+ As of that this driver emulated I/O BAR in software to let a VF be
s/emulated/emulates/
OK
Quoted 7 lines
drivers/vfio/pci/virtio/Kconfig line 14
Quoted 3 lines
drivers/vfio/pci/virtio/Kconfig new file +2
-0+ seen as a transitional device in the guest and let it work with + a legacy driver.VFIO is not specific to the guest. a native application including a legacy virtio driver could also benefit. let's not write it in a way specific to virt.
OK, we can rephrase to the below.
" .. to let a VF be seen as a transitional device by its users and .."
Quoted 8 lines
drivers/vfio/pci/virtio/main.c line 81
Quoted 5 lines
drivers/vfio/pci/virtio/main.c new file +5
-0+ +static int +translate_io_bar_to_mem_bar (struct virtiovf_pci_core_device *virtvdev, +loff_t pos,char __user *buf, +size_t count,bool read)this name only talks about the behavior for
VIRTIO_PCI_QUEUE_NOTIFY.
Quoted 4 lines
for legacy admin cmd it's unclear whether it's actually conveyed to a mem bar.
is it clearer to call it
virtiovf_pci_bar0_rw()?
I'm fine with your rename suggestion, will be part of V8.
Quoted 41 lines
Quoted 41 lines
drivers/vfio/pci/virtio/main.c new file +22
-0+ +static ssize_t virtiovf_pci_read_config (struct vfio_device *core_vdev, +char __user *buf,size_t count, +loff_t *ppos) +{ +struct virtiovf_pci_core_device *virtvdev =container_of ( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +loff_t pos = *ppos &VFIO_PCI_OFFSET_MASK ; +size_t register_offset; +loff_t copy_offset; +size_t copy_count; +__le32 val32; +__le16 val16; +u8 val8; +int ret; + + ret =vfio_pci_core_read (core_vdev, buf, count, ppos); +if (ret <0 ) +return ret; + + if (range_intersect_range(pos, count, PCI_DEVICE_ID, sizeof(val16), + ©_offset, ©_count,
®ister_offset)) {(quoted diff) +2
-0+ val16 = cpu_to_le16(VIRTIO_TRANS_ID_NET); + if (copy_to_user(buf + copy_offset, (void *)&val16 +
register_offset,copy_count))drivers/vfio/pci/virtio/main.c new file +6
-0+return -EFAULT ; + } + +if ((le16_to_cpu (virtvdev->pci_cmd) &PCI_COMMAND_IO ) && + range_intersect_range(pos, count, PCI_COMMAND, sizeof(val16), + ©_offset, ©_count,
®ister_offset)) {(quoted diff) +1
-0+ if (copy_from_user((void *)&val16 + register_offset, buf +
copy_offset,drivers/vfio/pci/virtio/main.c new file +4
-0+ copy_count)) +return -EFAULT ; + val16 |=cpu_to_le16 (PCI_COMMAND_IO ); + if (copy_to_user(buf + copy_offset, (void *)&val16 +
register_offset,
Quoted 8 lines
drivers/vfio/pci/virtio/main.c line 198
Quoted 3 lines
drivers/vfio/pci/virtio/main.c new file +3
-0+ copy_count)) +return -EFAULT ; + }the write handler calls
vfio_pci_core_write()forPCI_COMMANDso the core vconfig should have the latest copy of the IO bit value which is copied to the user buffer byvfio_pci_core_read(). then not necessary to update it again.
You assume the the 'vconfig' mechanism/flow is always applicable for that specific field, this should be double-checked.
However, as for now the driver doesn't rely / use the vconfig for other fields as it doesn't match and need a big refactor, I prefer to not rely on it at all and have it here.
Quoted 7 lines
btw the approach in this patch sounds a bit hackish - it modifies the result before/after vfio pci core emulation instead of directly injecting its specific emulation logic in vfio vconfig. It's probably being that vfio vconfig currently has a global permission/handler scheme for all pci devices. Extending it to support per-device tweak might need lots of change.
Right, the vconfig is not ready for that and might require lots of change, for now all is done by the driver layer.
Quoted 5 lines
So I'm not advocating that big change at this point, especially when only this driver imposes such requirement now. But in the future when more drivers e.g. Ankit's nvgrace-gpu want to do similar tweak we may consider such possibility.
This can be some orthogonal future refactoring once we'll have it.
Quoted 23 lines
Quoted 14 lines
(quoted diff) +2
-0+ + if (range_intersect_range(pos, count, PCI_BASE_ADDRESS_0,sizeof(val32),
(quoted diff) +1
-0+ ©_offset, ©_count,
®ister_offset)) {(quoted diff) +2
-0+ u32 bar_mask = ~(virtvdev->bar0_virtual_buf_size - 1); + u32 pci_base_addr_0 = le32_to_cpu(virtvdev-Quoted 1 line
pci_base_addr_0);(quoted diff) +2
-0+ + val32 = cpu_to_le32((pci_base_addr_0 & bar_mask) |
PCI_BASE_ADDRESS_SPACE_IO);(quoted diff) +1
-0+ if (copy_to_user(buf + copy_offset, (void *)&val32 +
register_offset,copy_count))Review
Quoted 2 lines
(quoted diff) +2
-0+ return -EFAULT; + }Do we care about the initial value of bar0? this patch leaves it as 0, unlike other real bars initialized with the hw value. In reality this may not be a problem as software usually writes all 1's to detect the size as the first step.
Quoted 1 line
raise it just in case others may see a potential issue.
We don't see here an issue as you mentioned above.
Quoted 37 lines
Quoted 37 lines
drivers/vfio/pci/virtio/main.c new file +3
-0+ +static ssize_t +virtiovf_pci_core_write(struct vfio_device *core_vdev, const char __user*buf,
drivers/vfio/pci/virtio/main.c new file +16
-0+ size_t count, loff_t *ppos) +{ + struct virtiovf_pci_core_device *virtvdev = container_of( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +unsigned int index =VFIO_PCI_OFFSET_TO_INDEX (*ppos); +loff_t pos = *ppos &VFIO_PCI_OFFSET_MASK ; + +if (!count) +return 0 ; + +if (index ==VFIO_PCI_CONFIG_REGION_INDEX ) { +size_t register_offset; +loff_t copy_offset; +size_t copy_count; + + if (range_intersect_range(pos, count, PCI_COMMAND,
sizeof(virtvdev->pci_cmd),drivers/vfio/pci/virtio/main.c new file +3
-0+ ©_offset, ©_count, + ®ister_offset)) { + if (copy_from_user((void *)&virtvdev->pci_cmd +
register_offset,drivers/vfio/pci/virtio/main.c new file +10
-0+ buf + copy_offset, + copy_count)) +return -EFAULT ; + } + +if (range_intersect_range (pos, count,PCI_BASE_ADDRESS_0 , +sizeof (virtvdev->pci_base_addr_0), + ©_offset, ©_count, + ®ister_offset)) { + if (copy_from_user((void *)&virtvdev-Quoted 1 line
pci_base_addr_0+register_offset,
Quoted 8 lines
drivers/vfio/pci/virtio/main.c line 296
Quoted 5 lines
drivers/vfio/pci/virtio/main.c new file +5
-0+ buf + copy_offset, + copy_count)) +return -EFAULT ; + } + }wrap above into
virtiovf_pci_write_config()to be symmetric with the read path.
Upon the read path, we do the full flow and return to the user. Here we just save some data and continue to call the core, so I'm not sure that this worth a dedicated function.
However, this can be done, do you still suggest it for V8 ?
Quoted 20 lines
Quoted 20 lines
drivers/vfio/pci/virtio/main.c new file +19
-0+static int virtiovf_pci_init_device (struct vfio_device *core_vdev) +{ +struct virtiovf_pci_core_device *virtvdev =container_of ( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +struct pci_dev *pdev; +int ret; + + ret =vfio_pci_core_init_dev (core_vdev); +if (ret) +return ret; + + pdev = virtvdev->core_device.pdev; + ret =virtiovf_read_notify_info (virtvdev); +if (ret) +return ret; + +/* Being ready with a buffer that supports MSIX */ + virtvdev->bar0_virtual_buf_size =VIRTIO_PCI_CONFIG_OFF (true) + + virtiovf_get_device_config_size(pdev-
Quoted 3 lines
Quoted 1 line
Quoted 1 line
device);
which code is relevant to MSIX?
The buffer size must include the MSIX part to match the virtio-net specification.
As part of virtiovf_issue_legacy_rw_cmd() we may use the full buffer upon read/write.
Quoted 12 lines
drivers/vfio/pci/virtio/main.c line 477
Quoted 8 lines
drivers/vfio/pci/virtio/main.c new file +6
-0+ +static const struct vfio_device_ops virtiovf_vfio_pci_ops = { + .name ="virtio-vfio-pci" , + .init = vfio_pci_core_init_dev, + .release = vfio_pci_core_release_dev, + .open_device = virtiovf_pci_open_device,could be
vfio_pci_core_open_device(). Given virtiovf specific init func is not calledvirtiovf_pci_open_device()is essentially same as the core func.
We don't have today vfio_pci_core_open_device() as an exported function.
The virtiovf_pci_open_device() matches both cases so I don't see a real reason to export it now.
By the way, it follows other drivers in vfio, see for example here [1].
Quoted 15 lines
drivers/vfio/pci/virtio/main.c line 508
Quoted 12 lines
drivers/vfio/pci/virtio/main.c new file +11
-0+ +static int virtiovf_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *id) +{ +const struct vfio_device_ops *ops = &virtiovf_vfio_pci_ops; +struct virtiovf_pci_core_device *virtvdev; +int ret; + +if (pdev->is_virtfn &&virtio_pci_admin_has_legacy_io (pdev) && + !virtiovf_bar0_exists (pdev)) + ops = &virtiovf_vfio_pci_tran_ops;I have a confusion here.
Quoted 2 lines
why do we want to allow this driver binding to non-matching VF or even PF?
The intention is to allow the binding of any virtio-net device (i.e. PF, VF which is not transitional capable) to have a single driver over VFIO for all virtio-net devices.
This enables any user space application to bind and use any virtio-net device without the need to care.
In case the device is not transitional capable, it will simply use the generic vfio functionality.
Quoted 5 lines
if that is the intention then the naming/description should be adjusted to not specific to vf throughout this patch.
e.g. don't use "
virtiovf_" prefix...
The main functionality is to supply the transitional device to user space for the VF, hence the prefix and the description for that driver refers to VF.
Let's stay with that.
Quoted 15 lines
the config option is generic:
drivers/vfio/pci/virtio/Kconfig new file +2
-0+config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices"but the description is specific to vf:
drivers/vfio/pci/virtio/Kconfig new file +3
-0+ This provides support for exposing VIRTIO NET VF devices which support + legacy IO access, using the VFIO framework that can work with a legacy + virtio driver in the guest.then the module description is generic again:
drivers/vfio/pci/virtio/main.c line 567
Quoted 2 lines
drivers/vfio/pci/virtio/main.c new file +2
-0+MODULE_DESCRIPTION ( +"VIRTIO VFIO PCI - User Level meta-driver for VIRTIO NET devices" );
Yes, as the binding allows that, it looks fine to me.
Thanks,
Yishai
Re: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 23 lines · On Wed, 13 Dec 2023 14:25:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 14 lines · Kevin
drivers/vfio/pci/virtio/main.c line 508
Quoted 11 lines
drivers/vfio/pci/virtio/main.c new file +11
-0+ +static int virtiovf_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *id) +{ +const struct vfio_device_ops *ops = &virtiovf_vfio_pci_ops; +struct virtiovf_pci_core_device *virtvdev; +int ret; + +if (pdev->is_virtfn &&virtio_pci_admin_has_legacy_io (pdev) && + !virtiovf_bar0_exists (pdev)) + ops = &virtiovf_vfio_pci_tran_ops;I have a confusion here.
Quoted 2 lines
why do we want to allow this driver binding to non-matching VF or even PF?
The intention is to allow the binding of any virtio-net device (i.e. PF, VF which is not transitional capable) to have a single driver over VFIO for all virtio-net devices.
Quoted 5 lines
This enables any user space application to bind and use any virtio-net device without the need to care.
In case the device is not transitional capable, it will simply use the generic vfio functionality.
The algorithm we've suggested for finding the most appropriate variant driver for the device doesn't include a step of moving on to another driver if the binding fails. We lose determinism at that point. Therefore this driver needs to handle all devices matching the id table. The fact that virtio dictates various config space fields limits our ability to refine the match from the id table. Thanks,
Alex
Tian, Kevin
reviewerRE: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 35 lines
From: Alex Williamson <[email protected]>
Sent: Thursday, December 14, 2023 4:24 AMQuoted 23 lines · On Wed, 13 Dec 2023 14:25:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 14 lines · Kevin
drivers/vfio/pci/virtio/main.c line 508
Quoted 11 lines
drivers/vfio/pci/virtio/main.c new file +11
-0+ +static int virtiovf_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *id) +{ +const struct vfio_device_ops *ops = &virtiovf_vfio_pci_ops; +struct virtiovf_pci_core_device *virtvdev; +int ret; + +if (pdev->is_virtfn &&virtio_pci_admin_has_legacy_io (pdev) && + !virtiovf_bar0_exists (pdev)) + ops = &virtiovf_vfio_pci_tran_ops;I have a confusion here.
Quoted 2 lines
why do we want to allow this driver binding to non-matching VF or even PF?
The intention is to allow the binding of any virtio-net device (i.e. PF, VF which is not transitional capable) to have a single driver over VFIO for all virtio-net devices.
Quoted 5 lines
This enables any user space application to bind and use any virtio-net device without the need to care.
In case the device is not transitional capable, it will simply use the generic vfio functionality.
Quoted 7 lines
The algorithm we've suggested for finding the most appropriate variant driver for the device doesn't include a step of moving on to another driver if the binding fails. We lose determinism at that point. Therefore this driver needs to handle all devices matching the id table. The fact that virtio dictates various config space fields limits our ability to refine the match from the id table. Thanks,
OK, that makes sense.
Tian, Kevin
reviewerRE: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
In reply to Yishai HadasQuoted 56 lines
From: Yishai Hadas <[email protected]>
Sent: Wednesday, December 13, 2023 8:25 PMQuoted 43 lines · Kevin
Quoted 43 lines
From: Yishai Hadas <[email protected]>
Sent: Thursday, December 7, 2023 6:28 PMdrivers/vfio/pci/virtio/main.c new file +22
-0+ +static ssize_t virtiovf_pci_read_config (struct vfio_device *core_vdev, +char __user *buf,size_t count, +loff_t *ppos) +{ +struct virtiovf_pci_core_device *virtvdev =container_of ( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +loff_t pos = *ppos &VFIO_PCI_OFFSET_MASK ; +size_t register_offset; +loff_t copy_offset; +size_t copy_count; +__le32 val32; +__le16 val16; +u8 val8; +int ret; + + ret =vfio_pci_core_read (core_vdev, buf, count, ppos); +if (ret <0 ) +return ret; + + if (range_intersect_range(pos, count, PCI_DEVICE_ID, sizeof(val16), + ©_offset, ©_count,
®ister_offset)) {(quoted diff) +2
-0+ val16 = cpu_to_le16(VIRTIO_TRANS_ID_NET); + if (copy_to_user(buf + copy_offset, (void *)&val16 +
register_offset,copy_count))drivers/vfio/pci/virtio/main.c new file +6
-0+return -EFAULT ; + } + +if ((le16_to_cpu (virtvdev->pci_cmd) &PCI_COMMAND_IO ) && + range_intersect_range(pos, count, PCI_COMMAND, sizeof(val16), + ©_offset, ©_count,
®ister_offset)) {(quoted diff) +1
-0+ if (copy_from_user((void *)&val16 + register_offset, buf +
copy_offset,drivers/vfio/pci/virtio/main.c new file +4
-0+ copy_count)) +return -EFAULT ; + val16 |=cpu_to_le16 (PCI_COMMAND_IO ); + if (copy_to_user(buf + copy_offset, (void *)&val16 +
register_offset,Quoted 8 lines
drivers/vfio/pci/virtio/main.c line 198
Quoted 3 lines
drivers/vfio/pci/virtio/main.c new file +3
-0+ copy_count)) +return -EFAULT ; + }the write handler calls
vfio_pci_core_write()forPCI_COMMANDso the core vconfig should have the latest copy of the IO bit value which is copied to the user buffer byvfio_pci_core_read(). then not necessary to update it again.
Quoted 5 lines
You assume the the 'vconfig' mechanism/flow is always applicable for that specific field, this should be double-checked.
However, as for now the driver doesn't rely / use the vconfig for other fields as it doesn't match and need a big refactor, I prefer to not rely on it at all and have it here.
iiuc this driver does relies on vconfig for other fields. It first calls vfio_pci_core_read() and then modifies selected fields which needs special tweak in this driver.
btw what virtio-net specific tweak is applied to PCI_COMMAND? You basically cache the cmd value and then set PCI_COMMAND_IO if it's set in the cached value. But this is already covered by pci vconfig. If anything is broken there then we already have a big trouble.
The trick for bar0 makes sense as it doesn't exist physically then the vconfig mechanism may not give the expected result. But I didn't see the same rationale for PCI_COMMAND.
Quoted 45 lines
Quoted 36 lines
Quoted 36 lines
drivers/vfio/pci/virtio/main.c new file +3
-0+ +static ssize_t +virtiovf_pci_core_write(struct vfio_device *core_vdev, const char __user*buf,
drivers/vfio/pci/virtio/main.c new file +16
-0+ size_t count, loff_t *ppos) +{ + struct virtiovf_pci_core_device *virtvdev = container_of( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +unsigned int index =VFIO_PCI_OFFSET_TO_INDEX (*ppos); +loff_t pos = *ppos &VFIO_PCI_OFFSET_MASK ; + +if (!count) +return 0 ; + +if (index ==VFIO_PCI_CONFIG_REGION_INDEX ) { +size_t register_offset; +loff_t copy_offset; +size_t copy_count; + + if (range_intersect_range(pos, count, PCI_COMMAND,
sizeof(virtvdev->pci_cmd),drivers/vfio/pci/virtio/main.c new file +3
-0+ ©_offset, ©_count, + ®ister_offset)) { + if (copy_from_user((void *)&virtvdev->pci_cmd +
register_offset,drivers/vfio/pci/virtio/main.c new file +10
-0+ buf + copy_offset, + copy_count)) +return -EFAULT ; + } + +if (range_intersect_range (pos, count,PCI_BASE_ADDRESS_0 , +sizeof (virtvdev->pci_base_addr_0), + ©_offset, ©_count, + ®ister_offset)) { + if (copy_from_user((void *)&virtvdev-Quoted 1 line
pci_base_addr_0+register_offset,Quoted 8 lines
drivers/vfio/pci/virtio/main.c line 296
Quoted 5 lines
drivers/vfio/pci/virtio/main.c new file +5
-0+ buf + copy_offset, + copy_count)) +return -EFAULT ; + } + }wrap above into
virtiovf_pci_write_config()to be symmetric with the read path.
Quoted 3 lines
Upon the read path, we do the full flow and return to the user. Here we just save some data and continue to call the core, so I'm not sure that this worth a dedicated function.
I don't see a real difference.
for the read path you first call vfio_pci_core_read() then modifies some fields.
for the write path you save some data then call vfio_pci_core_write().
Quoted 2 lines
However, this can be done, do you still suggest it for V8 ?
yes
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
drivers/vfio/pci/virtio/main.c new file +19
-0+static int virtiovf_pci_init_device (struct vfio_device *core_vdev) +{ +struct virtiovf_pci_core_device *virtvdev =container_of ( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +struct pci_dev *pdev; +int ret; + + ret =vfio_pci_core_init_dev (core_vdev); +if (ret) +return ret; + + pdev = virtvdev->core_device.pdev; + ret =virtiovf_read_notify_info (virtvdev); +if (ret) +return ret; + +/* Being ready with a buffer that supports MSIX */ + virtvdev->bar0_virtual_buf_size =VIRTIO_PCI_CONFIG_OFF (true) + + virtiovf_get_device_config_size(pdev-Quoted 1 line
device);
Quoted 7 lines
Quoted 1 line
which code is relevant to MSIX?
The buffer size must include the MSIX part to match the virtio-net specification.
As part of
virtiovf_issue_legacy_rw_cmd()we may use the full buffer upon read/write.
at least mention that MSIX is in the device config region otherwise it's not helpful to people w/o virtio background.
Quoted 13 lines
Quoted 10 lines
drivers/vfio/pci/virtio/main.c line 477
Quoted 6 lines
drivers/vfio/pci/virtio/main.c new file +6
-0+ +static const struct vfio_device_ops virtiovf_vfio_pci_ops = { + .name ="virtio-vfio-pci" , + .init = vfio_pci_core_init_dev, + .release = vfio_pci_core_release_dev, + .open_device = virtiovf_pci_open_device,could be
vfio_pci_core_open_device(). Given virtiovf specific init func is not calledvirtiovf_pci_open_device()is essentially same as the core func.We don't have today
vfio_pci_core_open_device()as an exported function.
Quoted 8 lines
The
virtiovf_pci_open_device()matches both cases so I don't see a real reason to export it now.By the way, it follows other drivers in vfio, see for example here [1].
[1]
https://elixir.bootlin.com/linux/v6.7-
rc5/source/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c#L1383
ah, yes.
Quoted 21 lines
Quoted 14 lines
drivers/vfio/pci/virtio/main.c line 508
Quoted 11 lines
drivers/vfio/pci/virtio/main.c new file +11
-0+ +static int virtiovf_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *id) +{ +const struct vfio_device_ops *ops = &virtiovf_vfio_pci_ops; +struct virtiovf_pci_core_device *virtvdev; +int ret; + +if (pdev->is_virtfn &&virtio_pci_admin_has_legacy_io (pdev) && + !virtiovf_bar0_exists (pdev)) + ops = &virtiovf_vfio_pci_tran_ops;I have a confusion here.
Quoted 2 lines
why do we want to allow this driver binding to non-matching VF or even PF?
The intention is to allow the binding of any virtio-net device (i.e. PF, VF which is not transitional capable) to have a single driver over VFIO for all virtio-net devices.
Quoted 5 lines
This enables any user space application to bind and use any virtio-net device without the need to care.
In case the device is not transitional capable, it will simply use the generic vfio functionality.
Is it useful to print a message here?
Quoted 5 lines
Quoted 5 lines
if that is the intention then the naming/description should be adjusted to not specific to vf throughout this patch.
Quoted 7 lines
Quoted 1 line
e.g. don't use "
virtiovf_" prefix...The main functionality is to supply the transitional device to user space for the VF, hence the prefix and the description for that driver refers to VF.
Let's stay with that.
ok
Quoted 15 lines
Quoted 10 lines
the config option is generic:
drivers/vfio/pci/virtio/Kconfig new file +2
-0+config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices"but the description is specific to vf:
+ This provides support for exposing VIRTIO NET VF devices which
support
Quoted 1 line
+ legacy IO access, using the VFIO framework that can work with a
legacy
Quoted 2 lines
+ virtio driver in the guest.
Quoted 8 lines
drivers/vfio/pci/virtio/main.c line 567
Quoted 5 lines
then the module description is generic again:
drivers/vfio/pci/virtio/main.c new file +2
-0+MODULE_DESCRIPTION ( +"VIRTIO VFIO PCI - User Level meta-driver for VIRTIO NET devices" );Yes, as the binding allows that, it looks fine to me.
what about revising the kconfig message to make it clear that it's for all virtio-net device with special trick to make VF as a transitional device?
Yishai Hadas
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 62 lines · Kevin
Quoted 56 lines
From: Yishai Hadas <[email protected]>
Sent: Wednesday, December 13, 2023 8:25 PMQuoted 43 lines · Kevin
Quoted 43 lines
From: Yishai Hadas <[email protected]>
Sent: Thursday, December 7, 2023 6:28 PMdrivers/vfio/pci/virtio/main.c new file +22
-0+ +static ssize_t virtiovf_pci_read_config (struct vfio_device *core_vdev, +char __user *buf,size_t count, +loff_t *ppos) +{ +struct virtiovf_pci_core_device *virtvdev =container_of ( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +loff_t pos = *ppos &VFIO_PCI_OFFSET_MASK ; +size_t register_offset; +loff_t copy_offset; +size_t copy_count; +__le32 val32; +__le16 val16; +u8 val8; +int ret; + + ret =vfio_pci_core_read (core_vdev, buf, count, ppos); +if (ret <0 ) +return ret; + + if (range_intersect_range(pos, count, PCI_DEVICE_ID, sizeof(val16), + ©_offset, ©_count,
®ister_offset)) {(quoted diff) +2
-0+ val16 = cpu_to_le16(VIRTIO_TRANS_ID_NET); + if (copy_to_user(buf + copy_offset, (void *)&val16 +
register_offset,copy_count))drivers/vfio/pci/virtio/main.c new file +6
-0+return -EFAULT ; + } + +if ((le16_to_cpu (virtvdev->pci_cmd) &PCI_COMMAND_IO ) && + range_intersect_range(pos, count, PCI_COMMAND, sizeof(val16), + ©_offset, ©_count,
®ister_offset)) {(quoted diff) +1
-0+ if (copy_from_user((void *)&val16 + register_offset, buf +
copy_offset,drivers/vfio/pci/virtio/main.c new file +4
-0+ copy_count)) +return -EFAULT ; + val16 |=cpu_to_le16 (PCI_COMMAND_IO ); + if (copy_to_user(buf + copy_offset, (void *)&val16 +
register_offset,Quoted 8 lines
drivers/vfio/pci/virtio/main.c line 198
Quoted 3 lines
drivers/vfio/pci/virtio/main.c new file +3
-0+ copy_count)) +return -EFAULT ; + }the write handler calls
vfio_pci_core_write()forPCI_COMMANDso the core vconfig should have the latest copy of the IO bit value which is copied to the user buffer byvfio_pci_core_read(). then not necessary to update it again.Quoted 5 lines
You assume the the 'vconfig' mechanism/flow is always applicable for that specific field, this should be double-checked.
However, as for now the driver doesn't rely / use the vconfig for other fields as it doesn't match and need a big refactor, I prefer to not rely on it at all and have it here.
Quoted 3 lines
iiuc this driver does relies on vconfig for other fields. It first calls
vfio_pci_core_read()and then modifies selected fields which needs special tweak in this driver.
No, there is no dependency at all on vconfig for other fields in the driver.
vfio_pci_core_read() for most of its fields including the PCI_COMMAND goes directly over the PCI API/flow to the device and doesn't use the vconfig.
So, we must save/restore the PCI_COMMAND on the driver context to have it properly reported/emulated the PCI_COMMAND_IO bit.
Quoted 6 lines
btw what virtio-net specific tweak is applied to
PCI_COMMAND? You basically cache the cmd value and then setPCI_COMMAND_IOif it's set in the cached value. But this is already covered by pci vconfig. If anything is broken there then we already have a big trouble.
As I wrote above, this field (i.e.PCI_COMMAND) is not managed at all by vconfig, so we need to emulate it in the driver.
Quoted 59 lines
The trick for bar0 makes sense as it doesn't exist physically then the vconfig mechanism may not give the expected result. But I didn't see the same rationale for
PCI_COMMAND.Quoted 45 lines
Quoted 36 lines
Quoted 36 lines
drivers/vfio/pci/virtio/main.c new file +3
-0+ +static ssize_t +virtiovf_pci_core_write(struct vfio_device *core_vdev, const char __user*buf,
drivers/vfio/pci/virtio/main.c new file +16
-0+ size_t count, loff_t *ppos) +{ + struct virtiovf_pci_core_device *virtvdev = container_of( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +unsigned int index =VFIO_PCI_OFFSET_TO_INDEX (*ppos); +loff_t pos = *ppos &VFIO_PCI_OFFSET_MASK ; + +if (!count) +return 0 ; + +if (index ==VFIO_PCI_CONFIG_REGION_INDEX ) { +size_t register_offset; +loff_t copy_offset; +size_t copy_count; + + if (range_intersect_range(pos, count, PCI_COMMAND,
sizeof(virtvdev->pci_cmd),drivers/vfio/pci/virtio/main.c new file +3
-0+ ©_offset, ©_count, + ®ister_offset)) { + if (copy_from_user((void *)&virtvdev->pci_cmd +
register_offset,drivers/vfio/pci/virtio/main.c new file +10
-0+ buf + copy_offset, + copy_count)) +return -EFAULT ; + } + +if (range_intersect_range (pos, count,PCI_BASE_ADDRESS_0 , +sizeof (virtvdev->pci_base_addr_0), + ©_offset, ©_count, + ®ister_offset)) { + if (copy_from_user((void *)&virtvdev-Quoted 1 line
pci_base_addr_0+register_offset,Quoted 8 lines
drivers/vfio/pci/virtio/main.c line 296
Quoted 5 lines
drivers/vfio/pci/virtio/main.c new file +5
-0+ buf + copy_offset, + copy_count)) +return -EFAULT ; + } + }wrap above into
virtiovf_pci_write_config()to be symmetric with the read path.Quoted 3 lines
Upon the read path, we do the full flow and return to the user. Here we just save some data and continue to call the core, so I'm not sure that this worth a dedicated function.
I don't see a real difference.
for the read path you first call
vfio_pci_core_read()then modifies some fields.
Quoted 6 lines
for the write path you save some data then call
vfio_pci_core_write().Quoted 2 lines
However, this can be done, do you still suggest it for V8 ?
yes
OK, will add as part of V8.
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Quoted 20 lines
drivers/vfio/pci/virtio/main.c new file +19
-0+static int virtiovf_pci_init_device (struct vfio_device *core_vdev) +{ +struct virtiovf_pci_core_device *virtvdev =container_of ( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +struct pci_dev *pdev; +int ret; + + ret =vfio_pci_core_init_dev (core_vdev); +if (ret) +return ret; + + pdev = virtvdev->core_device.pdev; + ret =virtiovf_read_notify_info (virtvdev); +if (ret) +return ret; + +/* Being ready with a buffer that supports MSIX */ + virtvdev->bar0_virtual_buf_size =VIRTIO_PCI_CONFIG_OFF (true) + + virtiovf_get_device_config_size(pdev-Quoted 1 line
Quoted 1 line
device);
which code is relevant to MSIX?
Quoted 8 lines
Quoted 5 lines
The buffer size must include the MSIX part to match the virtio-net specification.
As part of
virtiovf_issue_legacy_rw_cmd()we may use the full buffer upon read/write.at least mention that MSIX is in the device config region otherwise it's not helpful to people w/o virtio background.
MSIX is not in the device configuration region, it follows the common one.
In any case, it looks like this comment doesn't give any real value, I'll simply drop it in V8.
Quoted 46 lines
Quoted 14 lines
Quoted 11 lines
drivers/vfio/pci/virtio/main.c line 477
Quoted 7 lines
drivers/vfio/pci/virtio/main.c new file +6
-0+ +static const struct vfio_device_ops virtiovf_vfio_pci_ops = { + .name ="virtio-vfio-pci" , + .init = vfio_pci_core_init_dev, + .release = vfio_pci_core_release_dev, + .open_device = virtiovf_pci_open_device,could be
vfio_pci_core_open_device(). Given virtiovf specific init func is not calledvirtiovf_pci_open_device()is essentially same as the core func.We don't have today
vfio_pci_core_open_device()as an exported function.Quoted 8 lines
The
virtiovf_pci_open_device()matches both cases so I don't see a real reason to export it now.By the way, it follows other drivers in vfio, see for example here [1].
[1]
https://elixir.bootlin.com/linux/v6.7-
rc5/source/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c#L1383ah, yes.
Quoted 21 lines
Quoted 14 lines
drivers/vfio/pci/virtio/main.c line 508
Quoted 11 lines
drivers/vfio/pci/virtio/main.c new file +11
-0+ +static int virtiovf_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *id) +{ +const struct vfio_device_ops *ops = &virtiovf_vfio_pci_ops; +struct virtiovf_pci_core_device *virtvdev; +int ret; + +if (pdev->is_virtfn &&virtio_pci_admin_has_legacy_io (pdev) && + !virtiovf_bar0_exists (pdev)) + ops = &virtiovf_vfio_pci_tran_ops;I have a confusion here.
Quoted 2 lines
why do we want to allow this driver binding to non-matching VF or even PF?
The intention is to allow the binding of any virtio-net device (i.e. PF, VF which is not transitional capable) to have a single driver over VFIO for all virtio-net devices.
Quoted 7 lines
Quoted 5 lines
This enables any user space application to bind and use any virtio-net device without the need to care.
In case the device is not transitional capable, it will simply use the generic vfio functionality.
Is it useful to print a message here?
I don't think so, we usually don't print such messages.
User space will work based on caps to know what is supported, as of other cases.
Quoted 40 lines
Quoted 6 lines
Quoted 6 lines
if that is the intention then the naming/description should be adjusted to not specific to vf throughout this patch.
Quoted 7 lines
Quoted 1 line
e.g. don't use "
virtiovf_" prefix...The main functionality is to supply the transitional device to user space for the VF, hence the prefix and the description for that driver refers to VF.
Let's stay with that.
ok
Quoted 15 lines
Quoted 10 lines
the config option is generic:
drivers/vfio/pci/virtio/Kconfig new file +2
-0+config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices"but the description is specific to vf:
+ This provides support for exposing VIRTIO NET VF devices which
support
Quoted 1 line
+ legacy IO access, using the VFIO framework that can work with a
legacy
Quoted 2 lines
+ virtio driver in the guest.
Quoted 8 lines
drivers/vfio/pci/virtio/main.c line 567
Quoted 5 lines
then the module description is generic again:
drivers/vfio/pci/virtio/main.c new file +2
-0+MODULE_DESCRIPTION ( +"VIRTIO VFIO PCI - User Level meta-driver for VIRTIO NET devices" );Yes, as the binding allows that, it looks fine to me.
Quoted 3 lines
what about revising the kconfig message to make it clear that it's for all virtio-net device with special trick to make VF as a transitional device?
Kconfig doesn't need to get into the details of how things were done. We already mention that it emulates I/O BAR, this seems to be good enough.
However, I'll improve it as you already suggested in the previous mail.
Yishai
Tian, Kevin
reviewerRE: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 70 lines
From: Yishai Hadas <[email protected]>
Sent: Thursday, December 14, 2023 4:58 PMQuoted 62 lines · Kevin
Quoted 56 lines
From: Yishai Hadas <[email protected]>
Sent: Wednesday, December 13, 2023 8:25 PMQuoted 43 lines · Kevin
Quoted 43 lines
From: Yishai Hadas <[email protected]>
Sent: Thursday, December 7, 2023 6:28 PMdrivers/vfio/pci/virtio/main.c new file +22
-0+ +static ssize_t virtiovf_pci_read_config (struct vfio_device *core_vdev, +char __user *buf,size_t count, +loff_t *ppos) +{ +struct virtiovf_pci_core_device *virtvdev =container_of ( + core_vdev, struct virtiovf_pci_core_device, core_device.vdev); +loff_t pos = *ppos &VFIO_PCI_OFFSET_MASK ; +size_t register_offset; +loff_t copy_offset; +size_t copy_count; +__le32 val32; +__le16 val16; +u8 val8; +int ret; + + ret =vfio_pci_core_read (core_vdev, buf, count, ppos); +if (ret <0 ) +return ret; + + if (range_intersect_range(pos, count, PCI_DEVICE_ID, sizeof(val16), + ©_offset, ©_count,
®ister_offset)) {(quoted diff) +2
-0+ val16 = cpu_to_le16(VIRTIO_TRANS_ID_NET); + if (copy_to_user(buf + copy_offset, (void *)&val16 +
register_offset,copy_count))drivers/vfio/pci/virtio/main.c new file +6
-0+return -EFAULT ; + } + +if ((le16_to_cpu (virtvdev->pci_cmd) &PCI_COMMAND_IO ) && + range_intersect_range(pos, count, PCI_COMMAND, sizeof(val16), + ©_offset, ©_count,
®ister_offset)) {(quoted diff) +1
-0+ if (copy_from_user((void *)&val16 + register_offset, buf +
copy_offset,drivers/vfio/pci/virtio/main.c new file +4
-0+ copy_count)) +return -EFAULT ; + val16 |=cpu_to_le16 (PCI_COMMAND_IO ); + if (copy_to_user(buf + copy_offset, (void *)&val16 +
register_offset,Quoted 8 lines
drivers/vfio/pci/virtio/main.c line 198
Quoted 3 lines
drivers/vfio/pci/virtio/main.c new file +3
-0+ copy_count)) +return -EFAULT ; + }the write handler calls
vfio_pci_core_write()forPCI_COMMANDso the core vconfig should have the latest copy of the IO bit value which is copied to the user buffer byvfio_pci_core_read(). then not necessary to update it again.Quoted 5 lines
You assume the the 'vconfig' mechanism/flow is always applicable for that specific field, this should be double-checked.
However, as for now the driver doesn't rely / use the vconfig for other fields as it doesn't match and need a big refactor, I prefer to not rely on it at all and have it here.Quoted 3 lines
iiuc this driver does relies on vconfig for other fields. It first calls
vfio_pci_core_read()and then modifies selected fields which needs special tweak in this driver.
Quoted 8 lines
No, there is no dependency at all on vconfig for other fields in the driver.
vfio_pci_core_read()for most of its fields including thePCI_COMMANDgoes directly over the PCI API/flow to the device and doesn't use the vconfig.So, we must save/restore the
PCI_COMMANDon the driver context to have it properly reported/emulated thePCI_COMMAND_IObit.
you are right. sorry that I ignored this fact.
Michael S. Tsirkin
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
In reply to Yishai HadasQuoted 132 lines · Yishai Hadas
Introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
Background, from the virtio spec [1].
--------------------------------------------------------------------
In some systems, there is a need to support a virtio legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.
--------------------------------------------------------------------Specifically, this driver adds support for a virtio-net VF to be exposed as a transitional device to a guest driver and allows the legacy IO BAR functionality on top.
This allows a VM which uses a legacy virtio-net driver in the guest to work transparently over a VF which its driver in the host is that new driver.
The driver can be extended easily to support some other types of virtio devices (e.g virtio-blk), by adding in a few places the specific type properties as was done for virtio-net.
For now, only the virtio-net use case was tested and as such we introduce the support only for such a device.
Practically,
Upon probing a VF for a virtio-net device, in case its PF supports legacy access over the virtio admin commands and the VF doesn't have BAR 0, we set some specific 'vfio_device_ops' to be able to simulate in SW a transitional device with I/O BAR in BAR 0.The existence of the simulated I/O bar is reported later on by overwriting the
VFIO_DEVICE_GET_REGION_INFOcommand and the device exposes itself as a transitional device by overwriting some properties upon reading its config space.Once we report the existence of I/O BAR as BAR 0 a legacy driver in the guest may use it via read/write calls according to the virtio specification.
Any read/write towards the control parts of the BAR will be captured by the new driver and will be translated into admin commands towards the device.
Any data path read/write access (i.e. virtio driver notifications) will be forwarded to the physical BAR which its properties were supplied by the admin command
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFOupon the probing/init flow.With that code in place a legacy driver in the guest has the look and feel as if having a transitional device with legacy support for both its control and data path flows.
[1]
https://github.com/oasis-tcs/virtio-spec/commit/03c2d32e5093ca9f2a17797242fbef88efe94b8cReviewed-by Jason Gunthorpe <[email protected]>Signed-off-by Yishai Hadas <[email protected]>--- MAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/virtio/Kconfig | 16 + drivers/vfio/pci/virtio/Makefile | 4 + drivers/vfio/pci/virtio/main.c | 567 +++++++++++++++++++++++++++++++ 6 files changed, 598 insertions(+) create mode 100644 drivers/vfio/pci/virtio/Kconfig create mode 100644 drivers/vfio/pci/virtio/Makefile create mode 100644 drivers/vfio/pci/virtio/main.cMAINTAINERS +7
-0@@ -22872,6 +22872,13 @@ L: [email protected] S: Maintained F: drivers/vfio/pci/mlx5/ +VFIO VIRTIO PCI DRIVER +M: Yishai Hadas <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +F: drivers/vfio/pci/virtio + VFIO PCI DEVICE SPECIFIC DRIVERS R: Jason Gunthorpe <[email protected]> R: Yishai Hadas <[email protected]>drivers/vfio/pci/Kconfig +2
-0@@ -65,4 +65,6 @@ source "drivers/vfio/pci/hisilicon/Kconfig" source "drivers/vfio/pci/pds/Kconfig" +source "drivers/vfio/pci/virtio/Kconfig" + endmenudrivers/vfio/pci/Makefile +2
-0@@ -13,3 +13,5 @@ obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ obj-$(CONFIG_PDS_VFIO_PCI) += pds/ + +obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/drivers/vfio/pci/virtio/Kconfig new file +6
-0@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices" + depends on VIRTIO_PCI + select VFIO_PCI_CORE + help
drivers/vfio/pci/virtio/Kconfig line 16
Quoted 10 lines
drivers/vfio/pci/virtio/Kconfig new file +10
-0+ This provides support for exposing VIRTIO NET VF devices which support + legacy IO access, using the VFIO framework that can work with a legacy + virtio driver in the guest. + Based on PCIe spec, VFs do not support I/O Space; thus, VF BARs shall + not indicate I/O Space. + As of that this driver emulated I/O BAR in software to let a VF be + seen as a transitional device in the guest and let it work with + a legacy driver. + + If you don't know what to do here, say N.
BTW shouldn't this driver be limited to X86? Things like lack of memory barriers will make legacy virtio racy on e.g. ARM will they not? And endian-ness will be broken on PPC ...
Signature · 1 line
MST
Yishai Hadas
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 139 lines · Michael S. Tsirkin
Quoted 138 lines · Yishai Hadas
Introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
Background, from the virtio spec [1].
--------------------------------------------------------------------
In some systems, there is a need to support a virtio legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.
--------------------------------------------------------------------Specifically, this driver adds support for a virtio-net VF to be exposed as a transitional device to a guest driver and allows the legacy IO BAR functionality on top.
This allows a VM which uses a legacy virtio-net driver in the guest to work transparently over a VF which its driver in the host is that new driver.
The driver can be extended easily to support some other types of virtio devices (e.g virtio-blk), by adding in a few places the specific type properties as was done for virtio-net.
For now, only the virtio-net use case was tested and as such we introduce the support only for such a device.
Practically,
Upon probing a VF for a virtio-net device, in case its PF supports legacy access over the virtio admin commands and the VF doesn't have BAR 0, we set some specific 'vfio_device_ops' to be able to simulate in SW a transitional device with I/O BAR in BAR 0.The existence of the simulated I/O bar is reported later on by overwriting the
VFIO_DEVICE_GET_REGION_INFOcommand and the device exposes itself as a transitional device by overwriting some properties upon reading its config space.Once we report the existence of I/O BAR as BAR 0 a legacy driver in the guest may use it via read/write calls according to the virtio specification.
Any read/write towards the control parts of the BAR will be captured by the new driver and will be translated into admin commands towards the device.
Any data path read/write access (i.e. virtio driver notifications) will be forwarded to the physical BAR which its properties were supplied by the admin command
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFOupon the probing/init flow.With that code in place a legacy driver in the guest has the look and feel as if having a transitional device with legacy support for both its control and data path flows.
[1]
https://github.com/oasis-tcs/virtio-spec/commit/03c2d32e5093ca9f2a17797242fbef88efe94b8cReviewed-by Jason Gunthorpe <[email protected]>Signed-off-by Yishai Hadas <[email protected]>--- MAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/virtio/Kconfig | 16 + drivers/vfio/pci/virtio/Makefile | 4 + drivers/vfio/pci/virtio/main.c | 567 +++++++++++++++++++++++++++++++ 6 files changed, 598 insertions(+)create mode 100644
drivers/vfio/pci/virtio/Kconfigcreate mode 100644drivers/vfio/pci/virtio/Makefilecreate mode 100644drivers/vfio/pci/virtio/main.cMAINTAINERS +7
-0@@ -22872,6 +22872,13 @@ L: [email protected] S: Maintained F: drivers/vfio/pci/mlx5/ +VFIO VIRTIO PCI DRIVER +M: Yishai Hadas <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +F: drivers/vfio/pci/virtio + VFIO PCI DEVICE SPECIFIC DRIVERS R: Jason Gunthorpe <[email protected]> R: Yishai Hadas <[email protected]>drivers/vfio/pci/Kconfig +2
-0@@ -65,4 +65,6 @@ source "drivers/vfio/pci/hisilicon/Kconfig" source "drivers/vfio/pci/pds/Kconfig" +source "drivers/vfio/pci/virtio/Kconfig" + endmenudrivers/vfio/pci/Makefile +2
-0@@ -13,3 +13,5 @@ obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ obj-$(CONFIG_PDS_VFIO_PCI) += pds/ + +obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/drivers/vfio/pci/virtio/Kconfig new file +12
-0@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices" + depends on VIRTIO_PCI + select VFIO_PCI_CORE + help + This provides support for exposing VIRTIO NET VF devices which support + legacy IO access, using the VFIO framework that can work with a legacy + virtio driver in the guest. + Based on PCIe spec, VFs do not support I/O Space; thus, VF BARs shall + not indicate I/O Space. + As of that this driver emulated I/O BAR in software to let a VF be
Quoted 9 lines
drivers/vfio/pci/virtio/Kconfig line 16
Quoted 4 lines
drivers/vfio/pci/virtio/Kconfig new file +4
-0+ seen as a transitional device in the guest and let it work with + a legacy driver. + + If you don't know what to do here, say N.BTW shouldn't this driver be limited to X86? Things like lack of memory barriers will make legacy virtio racy on e.g. ARM will they not? And endian-ness will be broken on PPC ...
OK, if so, we can come with the below extra code.
Makes sense ?
I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3 -0
@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;
Yishai
Michael S. Tsirkin
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 167 lines · Yishai Hadas
Quoted 135 lines · Michael S. Tsirkin
Quoted 134 lines · Yishai Hadas
Introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
Background, from the virtio spec [1].
--------------------------------------------------------------------
In some systems, there is a need to support a virtio legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.
--------------------------------------------------------------------Specifically, this driver adds support for a virtio-net VF to be exposed as a transitional device to a guest driver and allows the legacy IO BAR functionality on top.
This allows a VM which uses a legacy virtio-net driver in the guest to work transparently over a VF which its driver in the host is that new driver.
The driver can be extended easily to support some other types of virtio devices (e.g virtio-blk), by adding in a few places the specific type properties as was done for virtio-net.
For now, only the virtio-net use case was tested and as such we introduce the support only for such a device.
Practically,
Upon probing a VF for a virtio-net device, in case its PF supports legacy access over the virtio admin commands and the VF doesn't have BAR 0, we set some specific 'vfio_device_ops' to be able to simulate in SW a transitional device with I/O BAR in BAR 0.The existence of the simulated I/O bar is reported later on by overwriting the
VFIO_DEVICE_GET_REGION_INFOcommand and the device exposes itself as a transitional device by overwriting some properties upon reading its config space.Once we report the existence of I/O BAR as BAR 0 a legacy driver in the guest may use it via read/write calls according to the virtio specification.
Any read/write towards the control parts of the BAR will be captured by the new driver and will be translated into admin commands towards the device.
Any data path read/write access (i.e. virtio driver notifications) will be forwarded to the physical BAR which its properties were supplied by the admin command
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFOupon the probing/init flow.With that code in place a legacy driver in the guest has the look and feel as if having a transitional device with legacy support for both its control and data path flows.
[1]
https://github.com/oasis-tcs/virtio-spec/commit/03c2d32e5093ca9f2a17797242fbef88efe94b8cReviewed-by Jason Gunthorpe <[email protected]>Signed-off-by Yishai Hadas <[email protected]>--- MAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/virtio/Kconfig | 16 + drivers/vfio/pci/virtio/Makefile | 4 + drivers/vfio/pci/virtio/main.c | 567 +++++++++++++++++++++++++++++++ 6 files changed, 598 insertions(+)create mode 100644
drivers/vfio/pci/virtio/Kconfigcreate mode 100644drivers/vfio/pci/virtio/Makefilecreate mode 100644drivers/vfio/pci/virtio/main.cMAINTAINERS +7
-0@@ -22872,6 +22872,13 @@ L: [email protected] S: Maintained F: drivers/vfio/pci/mlx5/ +VFIO VIRTIO PCI DRIVER +M: Yishai Hadas <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +F: drivers/vfio/pci/virtio + VFIO PCI DEVICE SPECIFIC DRIVERS R: Jason Gunthorpe <[email protected]> R: Yishai Hadas <[email protected]>drivers/vfio/pci/Kconfig +2
-0@@ -65,4 +65,6 @@ source "drivers/vfio/pci/hisilicon/Kconfig" source "drivers/vfio/pci/pds/Kconfig" +source "drivers/vfio/pci/virtio/Kconfig" + endmenudrivers/vfio/pci/Makefile +2
-0@@ -13,3 +13,5 @@ obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ obj-$(CONFIG_PDS_VFIO_PCI) += pds/ + +obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/drivers/vfio/pci/virtio/Kconfig new file +12
-0@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices" + depends on VIRTIO_PCI + select VFIO_PCI_CORE + help + This provides support for exposing VIRTIO NET VF devices which support + legacy IO access, using the VFIO framework that can work with a legacy + virtio driver in the guest. + Based on PCIe spec, VFs do not support I/O Space; thus, VF BARs shall + not indicate I/O Space. + As of that this driver emulated I/O BAR in software to let a VF beQuoted 9 lines
drivers/vfio/pci/virtio/Kconfig line 16
Quoted 4 lines
drivers/vfio/pci/virtio/Kconfig new file +4
-0+ seen as a transitional device in the guest and let it work with + a legacy driver. + + If you don't know what to do here, say N.BTW shouldn't this driver be limited to X86? Things like lack of memory barriers will make legacy virtio racy on e.g. ARM will they not? And endian-ness will be broken on PPC ...
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
Signature · 1 line
MST
Yishai Hadas
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 168 lines · Michael S. Tsirkin
Quoted 167 lines · Yishai Hadas
Quoted 135 lines · Michael S. Tsirkin
Quoted 134 lines · Yishai Hadas
Introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
Background, from the virtio spec [1].
--------------------------------------------------------------------
In some systems, there is a need to support a virtio legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.
--------------------------------------------------------------------Specifically, this driver adds support for a virtio-net VF to be exposed as a transitional device to a guest driver and allows the legacy IO BAR functionality on top.
This allows a VM which uses a legacy virtio-net driver in the guest to work transparently over a VF which its driver in the host is that new driver.
The driver can be extended easily to support some other types of virtio devices (e.g virtio-blk), by adding in a few places the specific type properties as was done for virtio-net.
For now, only the virtio-net use case was tested and as such we introduce the support only for such a device.
Practically,
Upon probing a VF for a virtio-net device, in case its PF supports legacy access over the virtio admin commands and the VF doesn't have BAR 0, we set some specific 'vfio_device_ops' to be able to simulate in SW a transitional device with I/O BAR in BAR 0.The existence of the simulated I/O bar is reported later on by overwriting the
VFIO_DEVICE_GET_REGION_INFOcommand and the device exposes itself as a transitional device by overwriting some properties upon reading its config space.Once we report the existence of I/O BAR as BAR 0 a legacy driver in the guest may use it via read/write calls according to the virtio specification.
Any read/write towards the control parts of the BAR will be captured by the new driver and will be translated into admin commands towards the device.
Any data path read/write access (i.e. virtio driver notifications) will be forwarded to the physical BAR which its properties were supplied by the admin command
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFOupon the probing/init flow.With that code in place a legacy driver in the guest has the look and feel as if having a transitional device with legacy support for both its control and data path flows.
[1]
https://github.com/oasis-tcs/virtio-spec/commit/03c2d32e5093ca9f2a17797242fbef88efe94b8cReviewed-by Jason Gunthorpe <[email protected]>Signed-off-by Yishai Hadas <[email protected]>--- MAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/virtio/Kconfig | 16 + drivers/vfio/pci/virtio/Makefile | 4 + drivers/vfio/pci/virtio/main.c | 567 +++++++++++++++++++++++++++++++ 6 files changed, 598 insertions(+)create mode 100644
drivers/vfio/pci/virtio/Kconfigcreate mode 100644drivers/vfio/pci/virtio/Makefilecreate mode 100644drivers/vfio/pci/virtio/main.cMAINTAINERS +7
-0@@ -22872,6 +22872,13 @@ L: [email protected] S: Maintained F: drivers/vfio/pci/mlx5/ +VFIO VIRTIO PCI DRIVER +M: Yishai Hadas <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +F: drivers/vfio/pci/virtio + VFIO PCI DEVICE SPECIFIC DRIVERS R: Jason Gunthorpe <[email protected]> R: Yishai Hadas <[email protected]>drivers/vfio/pci/Kconfig +2
-0@@ -65,4 +65,6 @@ source "drivers/vfio/pci/hisilicon/Kconfig" source "drivers/vfio/pci/pds/Kconfig" +source "drivers/vfio/pci/virtio/Kconfig" + endmenudrivers/vfio/pci/Makefile +2
-0@@ -13,3 +13,5 @@ obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ obj-$(CONFIG_PDS_VFIO_PCI) += pds/ + +obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/drivers/vfio/pci/virtio/Kconfig new file +12
-0@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices" + depends on VIRTIO_PCI + select VFIO_PCI_CORE + help + This provides support for exposing VIRTIO NET VF devices which support + legacy IO access, using the VFIO framework that can work with a legacy + virtio driver in the guest. + Based on PCIe spec, VFs do not support I/O Space; thus, VF BARs shall + not indicate I/O Space. + As of that this driver emulated I/O BAR in software to let a VF beQuoted 9 lines
drivers/vfio/pci/virtio/Kconfig line 16
Quoted 4 lines
drivers/vfio/pci/virtio/Kconfig new file +4
-0+ seen as a transitional device in the guest and let it work with + a legacy driver. + + If you don't know what to do here, say N.BTW shouldn't this driver be limited to X86? Things like lack of memory barriers will make legacy virtio racy on e.g. ARM will they not? And endian-ness will be broken on PPC ...
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;
Quoted 5 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef CONFIG_X86 return false).
However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index 37a0035f8381..53e29824d404 100644
drivers/virtio/virtio_pci_modern.c +7 -0
@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO * commands are supported
@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);
Yishai
Re: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 197 lines · On Thu, 14 Dec 2023 11:37:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 168 lines · Michael S. Tsirkin
Quoted 167 lines · Yishai Hadas
Quoted 135 lines · Michael S. Tsirkin
Quoted 134 lines · Yishai Hadas
Introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
Background, from the virtio spec [1].
--------------------------------------------------------------------
In some systems, there is a need to support a virtio legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.
--------------------------------------------------------------------Specifically, this driver adds support for a virtio-net VF to be exposed as a transitional device to a guest driver and allows the legacy IO BAR functionality on top.
This allows a VM which uses a legacy virtio-net driver in the guest to work transparently over a VF which its driver in the host is that new driver.
The driver can be extended easily to support some other types of virtio devices (e.g virtio-blk), by adding in a few places the specific type properties as was done for virtio-net.
For now, only the virtio-net use case was tested and as such we introduce the support only for such a device.
Practically,
Upon probing a VF for a virtio-net device, in case its PF supports legacy access over the virtio admin commands and the VF doesn't have BAR 0, we set some specific 'vfio_device_ops' to be able to simulate in SW a transitional device with I/O BAR in BAR 0.The existence of the simulated I/O bar is reported later on by overwriting the
VFIO_DEVICE_GET_REGION_INFOcommand and the device exposes itself as a transitional device by overwriting some properties upon reading its config space.Once we report the existence of I/O BAR as BAR 0 a legacy driver in the guest may use it via read/write calls according to the virtio specification.
Any read/write towards the control parts of the BAR will be captured by the new driver and will be translated into admin commands towards the device.
Any data path read/write access (i.e. virtio driver notifications) will be forwarded to the physical BAR which its properties were supplied by the admin command
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFOupon the probing/init flow.With that code in place a legacy driver in the guest has the look and feel as if having a transitional device with legacy support for both its control and data path flows.
[1]
https://github.com/oasis-tcs/virtio-spec/commit/03c2d32e5093ca9f2a17797242fbef88efe94b8cReviewed-by Jason Gunthorpe <[email protected]>Signed-off-by Yishai Hadas <[email protected]>--- MAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/virtio/Kconfig | 16 + drivers/vfio/pci/virtio/Makefile | 4 + drivers/vfio/pci/virtio/main.c | 567 +++++++++++++++++++++++++++++++ 6 files changed, 598 insertions(+)create mode 100644
drivers/vfio/pci/virtio/Kconfigcreate mode 100644drivers/vfio/pci/virtio/Makefilecreate mode 100644drivers/vfio/pci/virtio/main.cMAINTAINERS +7
-0@@ -22872,6 +22872,13 @@ L: [email protected] S: Maintained F: drivers/vfio/pci/mlx5/ +VFIO VIRTIO PCI DRIVER +M: Yishai Hadas <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +F: drivers/vfio/pci/virtio + VFIO PCI DEVICE SPECIFIC DRIVERS R: Jason Gunthorpe <[email protected]> R: Yishai Hadas <[email protected]>drivers/vfio/pci/Kconfig +2
-0@@ -65,4 +65,6 @@ source "drivers/vfio/pci/hisilicon/Kconfig" source "drivers/vfio/pci/pds/Kconfig" +source "drivers/vfio/pci/virtio/Kconfig" + endmenudrivers/vfio/pci/Makefile +2
-0@@ -13,3 +13,5 @@ obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ obj-$(CONFIG_PDS_VFIO_PCI) += pds/ + +obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/drivers/vfio/pci/virtio/Kconfig new file +12
-0@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices" + depends on VIRTIO_PCI + select VFIO_PCI_CORE + help + This provides support for exposing VIRTIO NET VF devices which support + legacy IO access, using the VFIO framework that can work with a legacy + virtio driver in the guest. + Based on PCIe spec, VFs do not support I/O Space; thus, VF BARs shall + not indicate I/O Space. + As of that this driver emulated I/O BAR in software to let a VF beQuoted 9 lines
drivers/vfio/pci/virtio/Kconfig line 16
Quoted 4 lines
drivers/vfio/pci/virtio/Kconfig new file +4
-0+ seen as a transitional device in the guest and let it work with + a legacy driver. + + If you don't know what to do here, say N.BTW shouldn't this driver be limited to X86? Things like lack of memory barriers will make legacy virtio racy on e.g. ARM will they not? And endian-ness will be broken on PPC ...
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;Quoted 5 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef
CONFIG_X86return false).However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index
37a0035f8381..53e29824d404100644drivers/virtio/virtio_pci_modern.c +1
-0@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO* commands are supported
drivers/virtio/virtio_pci_modern.c +0
-0@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)
drivers/virtio/virtio_pci_modern.c bool virtio_pci_admin_has_legacy_io(struct pci_dev · line 818
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +6
-0return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);
Doesn't this also raise the question of the purpose of virtio-vfio-pci on non-x86? Without any other features it offers nothing over vfio-pci and we should likely adjust the Kconfig for x86 or COMPILE_TEST. Thanks,
Alex
Michael S. Tsirkin
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 216 lines · Alex Williamson
Quoted 197 lines · On Thu, 14 Dec 2023 11:37:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 168 lines · Michael S. Tsirkin
Quoted 167 lines · Yishai Hadas
Quoted 135 lines · Michael S. Tsirkin
Quoted 134 lines · Yishai Hadas
Introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
Background, from the virtio spec [1].
--------------------------------------------------------------------
In some systems, there is a need to support a virtio legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.
--------------------------------------------------------------------Specifically, this driver adds support for a virtio-net VF to be exposed as a transitional device to a guest driver and allows the legacy IO BAR functionality on top.
This allows a VM which uses a legacy virtio-net driver in the guest to work transparently over a VF which its driver in the host is that new driver.
The driver can be extended easily to support some other types of virtio devices (e.g virtio-blk), by adding in a few places the specific type properties as was done for virtio-net.
For now, only the virtio-net use case was tested and as such we introduce the support only for such a device.
Practically,
Upon probing a VF for a virtio-net device, in case its PF supports legacy access over the virtio admin commands and the VF doesn't have BAR 0, we set some specific 'vfio_device_ops' to be able to simulate in SW a transitional device with I/O BAR in BAR 0.The existence of the simulated I/O bar is reported later on by overwriting the
VFIO_DEVICE_GET_REGION_INFOcommand and the device exposes itself as a transitional device by overwriting some properties upon reading its config space.Once we report the existence of I/O BAR as BAR 0 a legacy driver in the guest may use it via read/write calls according to the virtio specification.
Any read/write towards the control parts of the BAR will be captured by the new driver and will be translated into admin commands towards the device.
Any data path read/write access (i.e. virtio driver notifications) will be forwarded to the physical BAR which its properties were supplied by the admin command
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFOupon the probing/init flow.With that code in place a legacy driver in the guest has the look and feel as if having a transitional device with legacy support for both its control and data path flows.
[1]
https://github.com/oasis-tcs/virtio-spec/commit/03c2d32e5093ca9f2a17797242fbef88efe94b8cReviewed-by Jason Gunthorpe <[email protected]>Signed-off-by Yishai Hadas <[email protected]>--- MAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/virtio/Kconfig | 16 + drivers/vfio/pci/virtio/Makefile | 4 + drivers/vfio/pci/virtio/main.c | 567 +++++++++++++++++++++++++++++++ 6 files changed, 598 insertions(+)create mode 100644
drivers/vfio/pci/virtio/Kconfigcreate mode 100644drivers/vfio/pci/virtio/Makefilecreate mode 100644drivers/vfio/pci/virtio/main.cMAINTAINERS +7
-0@@ -22872,6 +22872,13 @@ L: [email protected] S: Maintained F: drivers/vfio/pci/mlx5/ +VFIO VIRTIO PCI DRIVER +M: Yishai Hadas <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +F: drivers/vfio/pci/virtio + VFIO PCI DEVICE SPECIFIC DRIVERS R: Jason Gunthorpe <[email protected]> R: Yishai Hadas <[email protected]>drivers/vfio/pci/Kconfig +2
-0@@ -65,4 +65,6 @@ source "drivers/vfio/pci/hisilicon/Kconfig" source "drivers/vfio/pci/pds/Kconfig" +source "drivers/vfio/pci/virtio/Kconfig" + endmenudrivers/vfio/pci/Makefile +2
-0@@ -13,3 +13,5 @@ obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ obj-$(CONFIG_PDS_VFIO_PCI) += pds/ + +obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/drivers/vfio/pci/virtio/Kconfig new file +12
-0@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices" + depends on VIRTIO_PCI + select VFIO_PCI_CORE + help + This provides support for exposing VIRTIO NET VF devices which support + legacy IO access, using the VFIO framework that can work with a legacy + virtio driver in the guest. + Based on PCIe spec, VFs do not support I/O Space; thus, VF BARs shall + not indicate I/O Space. + As of that this driver emulated I/O BAR in software to let a VF beQuoted 9 lines
drivers/vfio/pci/virtio/Kconfig line 16
Quoted 4 lines
drivers/vfio/pci/virtio/Kconfig new file +4
-0+ seen as a transitional device in the guest and let it work with + a legacy driver. + + If you don't know what to do here, say N.BTW shouldn't this driver be limited to X86? Things like lack of memory barriers will make legacy virtio racy on e.g. ARM will they not? And endian-ness will be broken on PPC ...
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;Quoted 5 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef
CONFIG_X86return false).However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index
37a0035f8381..53e29824d404100644drivers/virtio/virtio_pci_modern.c +1
-0@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO* commands are supported
drivers/virtio/virtio_pci_modern.c +0
-0@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)drivers/virtio/virtio_pci_modern.c bool virtio_pci_admin_has_legacy_io(struct pci_dev · line 818
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +6
-0return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);Doesn't this also raise the question of the purpose of virtio-vfio-pci on non-x86? Without any other features it offers nothing over vfio-pci and we should likely adjust the
Kconfigfor x86 orCOMPILE_TEST. Thanks,
Quoted 1 line
Alex
Kconfig dependency is what I had in mind, yes. The X86 specific code in virtio_pci_modern.c can be moved to a separate file then use makefile tricks to skip it on other platforms.
Signature · 1 line
MST
Yishai Hadas
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 217 lines · Michael S. Tsirkin
Quoted 216 lines · Alex Williamson
Quoted 197 lines · On Thu, 14 Dec 2023 11:37:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 168 lines · Michael S. Tsirkin
Quoted 167 lines · Yishai Hadas
Quoted 135 lines · Michael S. Tsirkin
Quoted 134 lines · Yishai Hadas
Introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
Background, from the virtio spec [1].
--------------------------------------------------------------------
In some systems, there is a need to support a virtio legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.
--------------------------------------------------------------------Specifically, this driver adds support for a virtio-net VF to be exposed as a transitional device to a guest driver and allows the legacy IO BAR functionality on top.
This allows a VM which uses a legacy virtio-net driver in the guest to work transparently over a VF which its driver in the host is that new driver.
The driver can be extended easily to support some other types of virtio devices (e.g virtio-blk), by adding in a few places the specific type properties as was done for virtio-net.
For now, only the virtio-net use case was tested and as such we introduce the support only for such a device.
Practically,
Upon probing a VF for a virtio-net device, in case its PF supports legacy access over the virtio admin commands and the VF doesn't have BAR 0, we set some specific 'vfio_device_ops' to be able to simulate in SW a transitional device with I/O BAR in BAR 0.The existence of the simulated I/O bar is reported later on by overwriting the
VFIO_DEVICE_GET_REGION_INFOcommand and the device exposes itself as a transitional device by overwriting some properties upon reading its config space.Once we report the existence of I/O BAR as BAR 0 a legacy driver in the guest may use it via read/write calls according to the virtio specification.
Any read/write towards the control parts of the BAR will be captured by the new driver and will be translated into admin commands towards the device.
Any data path read/write access (i.e. virtio driver notifications) will be forwarded to the physical BAR which its properties were supplied by the admin command
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFOupon the probing/init flow.With that code in place a legacy driver in the guest has the look and feel as if having a transitional device with legacy support for both its control and data path flows.
[1]
https://github.com/oasis-tcs/virtio-spec/commit/03c2d32e5093ca9f2a17797242fbef88efe94b8cReviewed-by Jason Gunthorpe <[email protected]>Signed-off-by Yishai Hadas <[email protected]>--- MAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/virtio/Kconfig | 16 + drivers/vfio/pci/virtio/Makefile | 4 + drivers/vfio/pci/virtio/main.c | 567 +++++++++++++++++++++++++++++++ 6 files changed, 598 insertions(+)create mode 100644
drivers/vfio/pci/virtio/Kconfigcreate mode 100644drivers/vfio/pci/virtio/Makefilecreate mode 100644drivers/vfio/pci/virtio/main.cMAINTAINERS +7
-0@@ -22872,6 +22872,13 @@ L: [email protected] S: Maintained F: drivers/vfio/pci/mlx5/ +VFIO VIRTIO PCI DRIVER +M: Yishai Hadas <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +F: drivers/vfio/pci/virtio + VFIO PCI DEVICE SPECIFIC DRIVERS R: Jason Gunthorpe <[email protected]> R: Yishai Hadas <[email protected]>drivers/vfio/pci/Kconfig +2
-0@@ -65,4 +65,6 @@ source "drivers/vfio/pci/hisilicon/Kconfig" source "drivers/vfio/pci/pds/Kconfig" +source "drivers/vfio/pci/virtio/Kconfig" + endmenudrivers/vfio/pci/Makefile +2
-0@@ -13,3 +13,5 @@ obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ obj-$(CONFIG_PDS_VFIO_PCI) += pds/ + +obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/drivers/vfio/pci/virtio/Kconfig new file +12
-0@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VIRTIO_VFIO_PCI + tristate "VFIO support for VIRTIO NET PCI devices" + depends on VIRTIO_PCI + select VFIO_PCI_CORE + help + This provides support for exposing VIRTIO NET VF devices which support + legacy IO access, using the VFIO framework that can work with a legacy + virtio driver in the guest. + Based on PCIe spec, VFs do not support I/O Space; thus, VF BARs shall + not indicate I/O Space. + As of that this driver emulated I/O BAR in software to let a VF beQuoted 9 lines
drivers/vfio/pci/virtio/Kconfig line 16
Quoted 4 lines
drivers/vfio/pci/virtio/Kconfig new file +4
-0+ seen as a transitional device in the guest and let it work with + a legacy driver. + + If you don't know what to do here, say N.BTW shouldn't this driver be limited to X86? Things like lack of memory barriers will make legacy virtio racy on e.g. ARM will they not? And endian-ness will be broken on PPC ...
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;Quoted 5 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef
CONFIG_X86return false).However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index
37a0035f8381..53e29824d404100644drivers/virtio/virtio_pci_modern.c +1
-0@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO* commands are supported
drivers/virtio/virtio_pci_modern.c +0
-0@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)drivers/virtio/virtio_pci_modern.c bool virtio_pci_admin_has_legacy_io(struct pci_dev · line 818
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +6
-0return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);Doesn't this also raise the question of the purpose of virtio-vfio-pci on non-x86? Without any other features it offers nothing over vfio-pci and we should likely adjust the
Kconfigfor x86 orCOMPILE_TEST. Thanks,
Quoted 6 lines
Quoted 1 line
Alex
Kconfigdependency is what I had in mind, yes. The X86 specific code invirtio_pci_modern.ccan be moved to a separate file then use makefile tricks to skip it on other platforms.
The next feature for that driver will be the live migration support over virtio, once the specification which is WIP those day will be accepted.
The migration functionality is not X86 dependent and doesn't have the legacy virtio driver limitations that enforced us to run only on X86.
So, by that time we may need to enable in VFIO the loading of virtio-vfio-pci driver and put back the ifdef X86 inside VIRTIO, only on the legacy IO API, as I did already in V8.
So using a KCONFIG solution in VFIO is a short term one, which will be reverted just later on.
In addition, the virtio_pci_admin_has_legacy_io() API can be used in the future not only by VFIO, this was one of the reasons to put it inside VIRTIO.
As of that, we may expect that VIRTIO will consider its limitation and return false in systems that are not X86 and won't expose its concerns/knowledge to other modules.
As of the above, I believe that better staying with the solution inside VIRTIO as was suggested here and is part of V8 already.
Makes sense ?
Yishai
Re: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 82 lines · On Thu, 14 Dec 2023 18:03:30 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 68 lines · Michael S. Tsirkin
Quoted 67 lines · Alex Williamson
Quoted 49 lines · On Thu, 14 Dec 2023 11:37:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 21 lines
Quoted 21 lines
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;Quoted 5 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef
CONFIG_X86return false).However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index
37a0035f8381..53e29824d404100644drivers/virtio/virtio_pci_modern.c +1
-0@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO* commands are supported
drivers/virtio/virtio_pci_modern.c +0
-0@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)drivers/virtio/virtio_pci_modern.c bool virtio_pci_admin_has_legacy_io(struct pci_dev · line 818
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +6
-0return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);Doesn't this also raise the question of the purpose of virtio-vfio-pci on non-x86? Without any other features it offers nothing over vfio-pci and we should likely adjust the
Kconfigfor x86 orCOMPILE_TEST. Thanks,Quoted 6 lines
Quoted 1 line
Alex
Kconfigdependency is what I had in mind, yes. The X86 specific code invirtio_pci_modern.ccan be moved to a separate file then use makefile tricks to skip it on other platforms.The next feature for that driver will be the live migration support over virtio, once the specification which is WIP those day will be accepted.
The migration functionality is not X86 dependent and doesn't have the legacy virtio driver limitations that enforced us to run only on X86.
Quoted 6 lines
So, by that time we may need to enable in VFIO the loading of virtio-vfio-pci driver and put back the ifdef X86 inside VIRTIO, only on the legacy IO API, as I did already in V8.
So using a KCONFIG solution in VFIO is a short term one, which will be reverted just later on.
I understand the intent, but I don't think that justifies building a driver that serves no purpose in the interim. IF and when migration support becomes a reality, it's trivial to update the depends line.
Quoted 3 lines
In addition, the
virtio_pci_admin_has_legacy_io()API can be used in the future not only by VFIO, this was one of the reasons to put it inside VIRTIO.
Maybe this should be governed by a new Kconfig option which would be selected by drivers like this. Thanks,
Alex
Yishai Hadas
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 89 lines · Alex Williamson
Quoted 86 lines · On Thu, 14 Dec 2023 18:03:30 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 68 lines · Michael S. Tsirkin
Quoted 67 lines · Alex Williamson
Quoted 49 lines · On Thu, 14 Dec 2023 11:37:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 21 lines
Quoted 21 lines
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;Quoted 5 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef
CONFIG_X86return false).However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index
37a0035f8381..53e29824d404100644drivers/virtio/virtio_pci_modern.c +1
-0@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO* commands are supported
drivers/virtio/virtio_pci_modern.c +0
-0@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)drivers/virtio/virtio_pci_modern.c bool virtio_pci_admin_has_legacy_io(struct pci_dev · line 818
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +6
-0return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);Doesn't this also raise the question of the purpose of virtio-vfio-pci on non-x86? Without any other features it offers nothing over vfio-pci and we should likely adjust the
Kconfigfor x86 orCOMPILE_TEST. Thanks,Quoted 6 lines
Quoted 1 line
Alex
Kconfigdependency is what I had in mind, yes. The X86 specific code invirtio_pci_modern.ccan be moved to a separate file then use makefile tricks to skip it on other platforms.The next feature for that driver will be the live migration support over virtio, once the specification which is WIP those day will be accepted.
The migration functionality is not X86 dependent and doesn't have the legacy virtio driver limitations that enforced us to run only on X86.
So, by that time we may need to enable in VFIO the loading of virtio-vfio-pci driver and put back the ifdef X86 inside VIRTIO, only on the legacy IO API, as I did already in V8.
Quoted 7 lines
Quoted 2 lines
So using a KCONFIG solution in VFIO is a short term one, which will be reverted just later on.
I understand the intent, but I don't think that justifies building a driver that serves no purpose in the interim. IF and when migration support becomes a reality, it's trivial to update the depends line.
OK, so I'll add a KCONFIG dependency on X86 as you suggested as part of V9 inside VFIO.
Quoted 7 lines
Quoted 3 lines
In addition, the
virtio_pci_admin_has_legacy_io()API can be used in the future not only by VFIO, this was one of the reasons to put it inside VIRTIO.Maybe this should be governed by a new
Kconfigoption which would be selected by drivers like this. Thanks,
We can still keep the simple ifdef X86 inside VIRTIO for future users/usage which is not only VFIO.
Michael,
Can that work for you ?
Yishai
Quoted 2 lines
Alex
Michael S. Tsirkin
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 107 lines · Yishai Hadas
Quoted 87 lines · Alex Williamson
Quoted 84 lines · On Thu, 14 Dec 2023 18:03:30 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 67 lines · Michael S. Tsirkin
Quoted 66 lines · Alex Williamson
Quoted 48 lines · On Thu, 14 Dec 2023 11:37:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 21 lines
Quoted 21 lines
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;Quoted 4 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef
CONFIG_X86return false).However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index
37a0035f8381..53e29824d404100644drivers/virtio/virtio_pci_modern.c +1
-0@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO* commands are supported
drivers/virtio/virtio_pci_modern.c +0
-0@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)drivers/virtio/virtio_pci_modern.c bool virtio_pci_admin_has_legacy_io(struct pci_dev · line 818
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +6
-0return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);Doesn't this also raise the question of the purpose of virtio-vfio-pci on non-x86? Without any other features it offers nothing over vfio-pci and we should likely adjust the
Kconfigfor x86 orCOMPILE_TEST. Thanks,Quoted 5 lines
Quoted 1 line
Alex
Kconfigdependency is what I had in mind, yes. The X86 specific code invirtio_pci_modern.ccan be moved to a separate file then use makefile tricks to skip it on other platforms.The next feature for that driver will be the live migration support over virtio, once the specification which is WIP those day will be accepted.
The migration functionality is not X86 dependent and doesn't have the legacy virtio driver limitations that enforced us to run only on X86.
So, by that time we may need to enable in VFIO the loading of virtio-vfio-pci driver and put back the ifdef X86 inside VIRTIO, only on the legacy IO API, as I did already in V8.
Quoted 7 lines
Quoted 2 lines
So using a KCONFIG solution in VFIO is a short term one, which will be reverted just later on.
I understand the intent, but I don't think that justifies building a driver that serves no purpose in the interim. IF and when migration support becomes a reality, it's trivial to update the depends line.
OK, so I'll add a KCONFIG dependency on X86 as you suggested as part of V9 inside VFIO.
Quoted 7 lines
Quoted 3 lines
In addition, the
virtio_pci_admin_has_legacy_io()API can be used in the future not only by VFIO, this was one of the reasons to put it inside VIRTIO.Maybe this should be governed by a new
Kconfigoption which would be selected by drivers like this. Thanks,
Quoted 10 lines
We can still keep the simple ifdef X86 inside VIRTIO for future users/usage which is not only VFIO.
Michael,
Can that work for you ?Yishai
Quoted 2 lines
Alex
I am not sure what is proposed exactly. General admin q infrastructure can be kept as is. The legacy things however can never work outside X86. Best way to limit it to x86 is to move it to a separate file and only build that on X86. This way the only ifdef we need is where we set the flags to enable legacy commands.
Signature · 1 line
MST
Yishai Hadas
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 116 lines · Michael S. Tsirkin
Quoted 115 lines · Yishai Hadas
Quoted 87 lines · Alex Williamson
Quoted 84 lines · On Thu, 14 Dec 2023 18:03:30 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 67 lines · Michael S. Tsirkin
Quoted 66 lines · Alex Williamson
Quoted 48 lines · On Thu, 14 Dec 2023 11:37:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 21 lines
Quoted 21 lines
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;Quoted 4 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef
CONFIG_X86return false).However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index
37a0035f8381..53e29824d404100644drivers/virtio/virtio_pci_modern.c +1
-0@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO* commands are supported
drivers/virtio/virtio_pci_modern.c +0
-0@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)drivers/virtio/virtio_pci_modern.c bool virtio_pci_admin_has_legacy_io(struct pci_dev · line 818
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +6
-0return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);Doesn't this also raise the question of the purpose of virtio-vfio-pci on non-x86? Without any other features it offers nothing over vfio-pci and we should likely adjust the
Kconfigfor x86 orCOMPILE_TEST. Thanks,Quoted 5 lines
Quoted 1 line
Alex
Kconfigdependency is what I had in mind, yes. The X86 specific code invirtio_pci_modern.ccan be moved to a separate file then use makefile tricks to skip it on other platforms.The next feature for that driver will be the live migration support over virtio, once the specification which is WIP those day will be accepted.
The migration functionality is not X86 dependent and doesn't have the legacy virtio driver limitations that enforced us to run only on X86.
So, by that time we may need to enable in VFIO the loading of virtio-vfio-pci driver and put back the ifdef X86 inside VIRTIO, only on the legacy IO API, as I did already in V8.
Quoted 7 lines
Quoted 2 lines
So using a KCONFIG solution in VFIO is a short term one, which will be reverted just later on.
I understand the intent, but I don't think that justifies building a driver that serves no purpose in the interim. IF and when migration support becomes a reality, it's trivial to update the depends line.
OK, so I'll add a KCONFIG dependency on X86 as you suggested as part of V9 inside VFIO.
Quoted 7 lines
Quoted 3 lines
In addition, the
virtio_pci_admin_has_legacy_io()API can be used in the future not only by VFIO, this was one of the reasons to put it inside VIRTIO.Maybe this should be governed by a new
Kconfigoption which would be selected by drivers like this. Thanks,We can still keep the simple ifdef X86 inside VIRTIO for future users/usage which is not only VFIO.
Michael,
Can that work for you ?Yishai
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Alex
I am not sure what is proposed exactly. General admin q infrastructure can be kept as is. The legacy things however can never work outside X86. Best way to limit it to x86 is to move it to a separate file and only build that on X86. This way the only ifdef we need is where we set the flags to enable legacy commands.
In VFIO we already agreed to add a dependency on X86 [1] as Alex asked.
As VIRTIO should be ready for other clients and be self contained, I thought to keep things simple and just return false from virtio_pci_admin_has_legacy_io() in non X86 systems as was sent in V8.
However, we can go with your approach as well and compile out all the legacy IO stuff in non X86 systems by moving its code to a separate file (i.e. virtio_pci_admin_legacy_io.c) and control this file upon the Makefile. In addition, you suggested to control the 'supported_cmds' by an ifdef. This will let the device know that we don't support legacy IO as well on non X86 systems.
Please be aware that the above approach requires another ifdef on the H file which exposes the 6 exported symbols and some further changes inside virtio as of making vp_modern_admin_cmd_exec() non static as now we move the legacy IO stuff to another C file, etc.
Please see below how [2] it will look like.
If you prefer that, so OK, it will be part of V9.
Please let me know.
[1] diff --git a/drivers/vfio/pci/virtio/Kconfig b/drivers/vfio/pci/virtio/Kconfig index 050473b0e5df..a3e5d8ea22a0 100644
drivers/vfio/pci/virtio/Kconfig +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only config VIRTIO_VFIO_PCI tristate "VFIO support for VIRTIO NET PCI devices" - depends on VIRTIO_PCI + depends on X86 && VIRTIO_PCI select VFIO_PCI_CORE help This provides support for exposing VIRTIO NET VF devices
which support
[2] diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile index 8e98d24917cc..a73358bb4ebb 100644
drivers/virtio/Makefile +1 -0
@@ -7,6 +7,7 @@ obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o +virtio_pci-$(CONFIG_X86) += virtio_pci_admin_legacy_io.o obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.o
drivers/virtio/virtio_pci_common.h +10 -0
@@ -158,4 +158,14 @@ void virtio_pci_modern_remove(struct virtio_pci_device *);struct virtio_device *virtio_pci_vf_get_pf_dev (struct pci_dev *pdev); +#define VIRTIO_LEGACY_ADMIN_CMD_BITMAP \ + (BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) + +int vp_modern_admin_cmd_exec (struct virtio_device *vdev, +struct virtio_admin_cmd *cmd); +#endif
drivers/virtio/virtio_pci_modern.c +5 -27
@@ -75,8 +75,8 @@ static int virtqueue_exec_admin_cmd(struct virtio_pci_admin_vq *admin_vq,return 0 ; } -static int vp_modern_admin_cmd_exec (struct virtio_device *vdev, -struct virtio_admin_cmd *cmd) +int vp_modern_admin_cmd_exec (struct virtio_device *vdev, +struct virtio_admin_cmd *cmd) {struct scatterlist *sgs[VIRTIO_AVQ_SGS_MAX ], hdr, stat;struct virtio_pci_device *vp_dev =to_vp_device (vdev);
@@ -172,6 +172,9 @@ static void virtio_pci_admin_cmd_list_init(struct virtio_device *virtio_dev)if (ret) goto end; +#ifndef CONFIG_X86 + *data &= ~(cpu_to_le64 (VIRTIO_LEGACY_ADMIN_CMD_BITMAP )); +#endif sg_init_one (&data_sg, data,sizeof (*data)); cmd.opcode =cpu_to_le16 (VIRTIO_ADMIN_CMD_LIST_USE ); cmd.data_sg = &data_sg;
@@ -775,257 +778,6 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) vp_dev->del_vq (&vp_dev->admin_vq.info); } -#define VIRTIO_LEGACY_ADMIN_CMD_BITMAP \ - (BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE) | \ - BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ) | \ - BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE) | \ - BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ - BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) - -#ifdef CONFIG_X86 -/* -* virtio_pci_admin_has_legacy_io - Checks whether the legacy IO -* commands are supported -* @dev: VF pci_dev -* -* Returns true on success. -*/ -bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) -{ -struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev); -struct virtio_pci_device *vp_dev; - -if (!virtio_dev) -return false; - -if (!virtio_has_feature (virtio_dev,VIRTIO_F_ADMIN_VQ )) -return false;
<other deletion to the new file>
<other deletion to the new file>
..
..
drivers/virtio/virtio_pci_admin_legacy_io.c new file +33 -0
@@ -0,0 +1,244 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved +*/ + +#include "virtio_pci_common.h" + +/* +* virtio_pci_admin_has_legacy_io - Checks whether the legacy IO +* commands are supported +* @dev: VF pci_dev +* +* Returns true on success. +*/ +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev); +struct virtio_pci_device *vp_dev; + +if (!virtio_dev) +return false; + +if (!virtio_has_feature (virtio_dev,VIRTIO_F_ADMIN_VQ )) +return false; + + vp_dev =to_vp_device (virtio_dev); + +if ((vp_dev->admin_vq.supported_cmds &VIRTIO_LEGACY_ADMIN_CMD_BITMAP ) == +VIRTIO_LEGACY_ADMIN_CMD_BITMAP ) +return true; +return false; +} +EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);
<other legacy IO code>
<other legacy IO code>
...
...
include/linux/virtio_pci_admin.h +1 -0
@@ -5,6 +5,7 @@#include <linux/types.h> #include <linux/pci.h> +#ifdef CONFIG_X86 bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev); int virtio_pci_admin_legacy_common_io_write(struct pci_dev *pdev, u8
offset,
u8 size, u8 *buf);
@@ -17,5 +18,6 @@ int virtio_pci_admin_legacy_device_io_read(struct pci_dev *pdev, u8 offset,
int virtio_pci_admin_legacy_io_notify_info(struct pci_dev *pdev,
u8 req_bar_flags, u8 *bar,
u64 *bar_offset);+#endif
Yishai
Michael S. Tsirkin
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 187 lines · Yishai Hadas
Quoted 116 lines · Michael S. Tsirkin
Quoted 115 lines · Yishai Hadas
Quoted 87 lines · Alex Williamson
Quoted 84 lines · On Thu, 14 Dec 2023 18:03:30 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 67 lines · Michael S. Tsirkin
Quoted 66 lines · Alex Williamson
Quoted 48 lines · On Thu, 14 Dec 2023 11:37:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 21 lines
Quoted 21 lines
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;Quoted 4 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef
CONFIG_X86return false).However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index
37a0035f8381..53e29824d404100644drivers/virtio/virtio_pci_modern.c +1
-0@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO* commands are supported
drivers/virtio/virtio_pci_modern.c +0
-0@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)drivers/virtio/virtio_pci_modern.c bool virtio_pci_admin_has_legacy_io(struct pci_dev · line 818
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +6
-0return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);Doesn't this also raise the question of the purpose of virtio-vfio-pci on non-x86? Without any other features it offers nothing over vfio-pci and we should likely adjust the
Kconfigfor x86 orCOMPILE_TEST. Thanks,Quoted 5 lines
Quoted 1 line
Alex
Kconfigdependency is what I had in mind, yes. The X86 specific code invirtio_pci_modern.ccan be moved to a separate file then use makefile tricks to skip it on other platforms.The next feature for that driver will be the live migration support over virtio, once the specification which is WIP those day will be accepted.
The migration functionality is not X86 dependent and doesn't have the legacy virtio driver limitations that enforced us to run only on X86.
So, by that time we may need to enable in VFIO the loading of virtio-vfio-pci driver and put back the ifdef X86 inside VIRTIO, only on the legacy IO API, as I did already in V8.
Quoted 7 lines
Quoted 2 lines
So using a KCONFIG solution in VFIO is a short term one, which will be reverted just later on.
I understand the intent, but I don't think that justifies building a driver that serves no purpose in the interim. IF and when migration support becomes a reality, it's trivial to update the depends line.
OK, so I'll add a KCONFIG dependency on X86 as you suggested as part of V9 inside VFIO.
Quoted 7 lines
Quoted 3 lines
In addition, the
virtio_pci_admin_has_legacy_io()API can be used in the future not only by VFIO, this was one of the reasons to put it inside VIRTIO.Maybe this should be governed by a new
Kconfigoption which would be selected by drivers like this. Thanks,We can still keep the simple ifdef X86 inside VIRTIO for future users/usage which is not only VFIO.
Michael,
Can that work for you ?Yishai
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Alex
I am not sure what is proposed exactly. General admin q infrastructure can be kept as is. The legacy things however can never work outside X86. Best way to limit it to x86 is to move it to a separate file and only build that on X86. This way the only ifdef we need is where we set the flags to enable legacy commands.
In VFIO we already agreed to add a dependency on X86 [1] as Alex asked.
As VIRTIO should be ready for other clients and be self contained, I thought to keep things simple and just return false from
virtio_pci_admin_has_legacy_io()in non X86 systems as was sent in V8.However, we can go with your approach as well and compile out all the legacy IO stuff in non X86 systems by moving its code to a separate file (i.e.
virtio_pci_admin_legacy_io.c) and control this file upon theMakefile. In addition, you suggested to control the 'supported_cmds' by an ifdef. This will let the device know that we don't support legacy IO as well on non X86 systems.Please be aware that the above approach requires another ifdef on the H file which exposes the 6 exported symbols and some further changes inside virtio as of making
vp_modern_admin_cmd_exec()non static as now we move the legacy IO stuff to another C file, etc.Please see below how [2] it will look like.
If you prefer that, so OK, it will be part of V9.
Please let me know.[1] diff --git a/drivers/vfio/pci/virtio/Kconfig
b/drivers/vfio/pci/virtio/Kconfig
index050473b0e5df..a3e5d8ea22a0100644drivers/vfio/pci/virtio/Kconfig +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only config VIRTIO_VFIO_PCI tristate "VFIO support for VIRTIO NET PCI devices" - depends on VIRTIO_PCI + depends on X86 && VIRTIO_PCI select VFIO_PCI_CORE help This provides support for exposing VIRTIO NET VF devices whichsupport
[2] diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile index
8e98d24917cc..a73358bb4ebb100644drivers/virtio/Makefile +1
-0@@ -7,6 +7,7 @@ obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o +virtio_pci-$(CONFIG_X86) += virtio_pci_admin_legacy_io.o obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.odrivers/virtio/virtio_pci_common.h +0
-0@@ -158,4 +158,14 @@ void virtio_pci_modern_remove(struct virtio_pci_device *);
drivers/virtio/virtio_pci_common.h void virtio_pci_modern_remove(struct virtio_pci_device · line 168
Quoted 10 lines
drivers/virtio/virtio_pci_common.h +7
-0struct virtio_device *virtio_pci_vf_get_pf_dev (struct pci_dev *pdev); +#define VIRTIO_LEGACY_ADMIN_CMD_BITMAP \ + (BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +
I'd add something like:
#ifdef CONFIG_X86
#define VIRTIO_ADMIN_CMD_BITMAP VIRTIO_LEGACY_ADMIN_CMD_BITMAP #else
#define VIRTIO_ADMIN_CMD_BITMAP 0
#endif
Add a comment explaining why, please.
Quoted 22 lines
drivers/virtio/virtio_pci_common.h +3
-0+int vp_modern_admin_cmd_exec (struct virtio_device *vdev, +struct virtio_admin_cmd *cmd); +#endif
drivers/virtio/virtio_pci_modern.c +2
-2@@ -75,8 +75,8 @@ static int virtqueue_exec_admin_cmd(structvirtio_pci_admin_vq *admin_vq, return 0; } -static int vp_modern_admin_cmd_exec (struct virtio_device *vdev, -struct virtio_admin_cmd *cmd) +int vp_modern_admin_cmd_exec (struct virtio_device *vdev, +struct virtio_admin_cmd *cmd) {struct scatterlist *sgs[VIRTIO_AVQ_SGS_MAX ], hdr, stat;struct virtio_pci_device *vp_dev = to_vp_device(vdev);
drivers/virtio/virtio_pci_modern.c static void virtio_pci_admin_cmd_list_init(struct · line 178
Quoted 7 lines
drivers/virtio/virtio_pci_modern.c +3
-0@@ -172,6 +172,9 @@ static void virtio_pci_admin_cmd_list_init(struct virtio_device *virtio_dev)if (ret) goto end; +#ifndef CONFIG_X86 + *data &= ~(cpu_to_le64 (VIRTIO_LEGACY_ADMIN_CMD_BITMAP )); +#endif
Then here we don't need an ifdef just use VIRTIO_ADMIN_CMD_BITMAP.
Quoted 111 lines
drivers/virtio/virtio_pci_modern.c +0
-0sg_init_one (&data_sg, data,sizeof (*data)); cmd.opcode =cpu_to_le16 (VIRTIO_ADMIN_CMD_LIST_USE );cmd.data_sg = &data_sg;drivers/virtio/virtio_pci_modern.c +0
-25@@ -775,257 +778,6 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) vp_dev->del_vq (&vp_dev->admin_vq.info); } -#define VIRTIO_LEGACY_ADMIN_CMD_BITMAP \ - (BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE) | \ - BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ) | \ - BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE) | \ - BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ - BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) - -#ifdef CONFIG_X86 -/* -* virtio_pci_admin_has_legacy_io - Checks whether the legacy IO -* commands are supported -* @dev: VF pci_dev -* -* Returns true on success. -*/ -bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) -{ -struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev); -struct virtio_pci_device *vp_dev; - -if (!virtio_dev) -return false; - -if (!virtio_has_feature (virtio_dev,VIRTIO_F_ADMIN_VQ )) -return false;<other deletion to the new file>
<other deletion to the new file>
..
..drivers/virtio/virtio_pci_admin_legacy_io.c new file +28
-0@@ -0,0 +1,244 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved +*/ + +#include "virtio_pci_common.h" + +/* +* virtio_pci_admin_has_legacy_io - Checks whether the legacy IO +* commands are supported +* @dev: VF pci_dev +* +* Returns true on success. +*/ +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev); +struct virtio_pci_device *vp_dev; + +if (!virtio_dev) +return false; + +if (!virtio_has_feature (virtio_dev,VIRTIO_F_ADMIN_VQ )) +return false; + + vp_dev =to_vp_device (virtio_dev); + + if ((vp_dev->admin_vq.supported_cmds &
VIRTIO_LEGACY_ADMIN_CMD_BITMAP) ==drivers/virtio/virtio_pci_admin_legacy_io.c new file +5
-0+ VIRTIO_LEGACY_ADMIN_CMD_BITMAP) +return true; +return false; +} +EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);<other legacy IO code>
<other legacy IO code>
...
...include/linux/virtio_pci_admin.h +1
-0@@ -5,6 +5,7 @@#include <linux/types.h> #include <linux/pci.h> +#ifdef CONFIG_X86 bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev); int virtio_pci_admin_legacy_common_io_write(struct pci_dev *pdev, u8 offset,u8 size, u8 *buf);include/linux/virtio_pci_admin.h +1
-0@@ -17,5 +18,6 @@ int virtio_pci_admin_legacy_device_io_read(struct pci_dev *pdev, u8 offset, int virtio_pci_admin_legacy_io_notify_info(struct pci_dev *pdev, u8 req_bar_flags, u8 *bar, u64 *bar_offset); +#endif Yishai
Yishai Hadas
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 202 lines · Michael S. Tsirkin
Quoted 187 lines · Yishai Hadas
Quoted 116 lines · Michael S. Tsirkin
Quoted 115 lines · Yishai Hadas
Quoted 87 lines · Alex Williamson
Quoted 84 lines · On Thu, 14 Dec 2023 18:03:30 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 67 lines · Michael S. Tsirkin
Quoted 66 lines · Alex Williamson
Quoted 48 lines · On Thu, 14 Dec 2023 11:37:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 21 lines
Quoted 21 lines
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;Quoted 4 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef
CONFIG_X86return false).However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index
37a0035f8381..53e29824d404100644drivers/virtio/virtio_pci_modern.c +1
-0@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO* commands are supported
drivers/virtio/virtio_pci_modern.c +0
-0@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)drivers/virtio/virtio_pci_modern.c bool virtio_pci_admin_has_legacy_io(struct pci_dev · line 818
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +6
-0return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);Doesn't this also raise the question of the purpose of virtio-vfio-pci on non-x86? Without any other features it offers nothing over vfio-pci and we should likely adjust the
Kconfigfor x86 orCOMPILE_TEST. Thanks,Quoted 5 lines
Quoted 1 line
Alex
Kconfigdependency is what I had in mind, yes. The X86 specific code invirtio_pci_modern.ccan be moved to a separate file then use makefile tricks to skip it on other platforms.The next feature for that driver will be the live migration support over virtio, once the specification which is WIP those day will be accepted.
The migration functionality is not X86 dependent and doesn't have the legacy virtio driver limitations that enforced us to run only on X86.
So, by that time we may need to enable in VFIO the loading of virtio-vfio-pci driver and put back the ifdef X86 inside VIRTIO, only on the legacy IO API, as I did already in V8.
Quoted 7 lines
Quoted 2 lines
So using a KCONFIG solution in VFIO is a short term one, which will be reverted just later on.
I understand the intent, but I don't think that justifies building a driver that serves no purpose in the interim. IF and when migration support becomes a reality, it's trivial to update the depends line.
OK, so I'll add a KCONFIG dependency on X86 as you suggested as part of V9 inside VFIO.
Quoted 7 lines
Quoted 3 lines
In addition, the
virtio_pci_admin_has_legacy_io()API can be used in the future not only by VFIO, this was one of the reasons to put it inside VIRTIO.Maybe this should be governed by a new
Kconfigoption which would be selected by drivers like this. Thanks,We can still keep the simple ifdef X86 inside VIRTIO for future users/usage which is not only VFIO.
Michael,
Can that work for you ?Yishai
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Alex
I am not sure what is proposed exactly. General admin q infrastructure can be kept as is. The legacy things however can never work outside X86. Best way to limit it to x86 is to move it to a separate file and only build that on X86. This way the only ifdef we need is where we set the flags to enable legacy commands.
In VFIO we already agreed to add a dependency on X86 [1] as Alex asked.
As VIRTIO should be ready for other clients and be self contained, I thought to keep things simple and just return false from
virtio_pci_admin_has_legacy_io()in non X86 systems as was sent in V8.However, we can go with your approach as well and compile out all the legacy IO stuff in non X86 systems by moving its code to a separate file (i.e.
virtio_pci_admin_legacy_io.c) and control this file upon theMakefile. In addition, you suggested to control the 'supported_cmds' by an ifdef. This will let the device know that we don't support legacy IO as well on non X86 systems.Please be aware that the above approach requires another ifdef on the H file which exposes the 6 exported symbols and some further changes inside virtio as of making
vp_modern_admin_cmd_exec()non static as now we move the legacy IO stuff to another C file, etc.Please see below how [2] it will look like.
If you prefer that, so OK, it will be part of V9.
Please let me know.[1] diff --git a/drivers/vfio/pci/virtio/Kconfig
b/drivers/vfio/pci/virtio/Kconfig
index050473b0e5df..a3e5d8ea22a0100644drivers/vfio/pci/virtio/Kconfig +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only config VIRTIO_VFIO_PCI tristate "VFIO support for VIRTIO NET PCI devices" - depends on VIRTIO_PCI + depends on X86 && VIRTIO_PCI select VFIO_PCI_CORE help This provides support for exposing VIRTIO NET VF devices whichsupport
[2] diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile index
8e98d24917cc..a73358bb4ebb100644drivers/virtio/Makefile +1
-0@@ -7,6 +7,7 @@ obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o +virtio_pci-$(CONFIG_X86) += virtio_pci_admin_legacy_io.o obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.odrivers/virtio/virtio_pci_common.h +0
-0@@ -158,4 +158,14 @@ void virtio_pci_modern_remove(struct virtio_pci_device *);drivers/virtio/virtio_pci_common.h void virtio_pci_modern_remove(struct virtio_pci_device · line 168
Quoted 10 lines
drivers/virtio/virtio_pci_common.h +7
-0struct virtio_device *virtio_pci_vf_get_pf_dev (struct pci_dev *pdev); +#define VIRTIO_LEGACY_ADMIN_CMD_BITMAP \ + (BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +I'd add something like:
Quoted 6 lines
#ifdef
CONFIG_X86
#defineVIRTIO_ADMIN_CMD_BITMAPVIRTIO_LEGACY_ADMIN_CMD_BITMAP#else
#defineVIRTIO_ADMIN_CMD_BITMAP0
#endif
This new macro (i.e. VIRTIO_ADMIN_CMD_BITMAP) will hold in the future another sets of supported bits (e.g. Live migration), right ?
Quoted 1 line
Add a comment explaining why, please.
OK
How about the below ?
"As of some limitations in the legacy driver (e.g. lack of memory barriers in ARM, endian-ness is broken in PPC) the supported legacy IO commands are masked out on non x86 systems."
Quoted 28 lines
Quoted 28 lines
drivers/virtio/virtio_pci_common.h +3
-0+int vp_modern_admin_cmd_exec (struct virtio_device *vdev, +struct virtio_admin_cmd *cmd); +#endifdrivers/virtio/virtio_pci_modern.c +2
-2@@ -75,8 +75,8 @@ static int virtqueue_exec_admin_cmd(structvirtio_pci_admin_vq *admin_vq, return 0; } -static int vp_modern_admin_cmd_exec (struct virtio_device *vdev, -struct virtio_admin_cmd *cmd) +int vp_modern_admin_cmd_exec (struct virtio_device *vdev, +struct virtio_admin_cmd *cmd) {struct scatterlist *sgs[VIRTIO_AVQ_SGS_MAX ], hdr, stat;struct virtio_pci_device *vp_dev = to_vp_device(vdev);drivers/virtio/virtio_pci_modern.c +0
-0@@ -172,6 +172,9 @@ static void virtio_pci_admin_cmd_list_init(struct virtio_device *virtio_dev)if (ret) goto end;
Quoted 6 lines
drivers/virtio/virtio_pci_modern.c static void virtio_pci_admin_cmd_list_init(struct virtio_device *virtio_dev) · line 177
Quoted 3 lines
drivers/virtio/virtio_pci_modern.c +3
-0+#ifndef CONFIG_X86 + *data &= ~(cpu_to_le64 (VIRTIO_LEGACY_ADMIN_CMD_BITMAP )); +#endif Then here we don't need an ifdef just use
VIRTIO_ADMIN_CMD_BITMAP.
Following your suggestion, this will become,
*data &= cpu_to_le64(VIRTIO_ADMIN_CMD_BITMAP);
(no ifdef, and no ~).
Yishai
Michael S. Tsirkin
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 210 lines · Yishai Hadas
Quoted 202 lines · Michael S. Tsirkin
Quoted 187 lines · Yishai Hadas
Quoted 116 lines · Michael S. Tsirkin
Quoted 115 lines · Yishai Hadas
Quoted 87 lines · Alex Williamson
Quoted 84 lines · On Thu, 14 Dec 2023 18:03:30 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 67 lines · Michael S. Tsirkin
Quoted 66 lines · Alex Williamson
Quoted 48 lines · On Thu, 14 Dec 2023 11:37:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 21 lines
Quoted 21 lines
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
drivers/virtio/virtio_pci_modern.c +3
-0@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)struct virtio_device *virtio_dev =virtio_pci_vf_get_pf_dev (pdev);struct virtio_pci_device *vp_dev; +#ifndef CONFIG_X86 +return false; +#endif if (!virtio_dev)return false;Quoted 4 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef
CONFIG_X86return false).However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index
37a0035f8381..53e29824d404100644drivers/virtio/virtio_pci_modern.c +1
-0@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO* commands are supported
drivers/virtio/virtio_pci_modern.c +0
-0@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)drivers/virtio/virtio_pci_modern.c bool virtio_pci_admin_has_legacy_io(struct pci_dev · line 818
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +6
-0return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);Doesn't this also raise the question of the purpose of virtio-vfio-pci on non-x86? Without any other features it offers nothing over vfio-pci and we should likely adjust the
Kconfigfor x86 orCOMPILE_TEST. Thanks,Quoted 5 lines
Quoted 1 line
Alex
Kconfigdependency is what I had in mind, yes. The X86 specific code invirtio_pci_modern.ccan be moved to a separate file then use makefile tricks to skip it on other platforms.The next feature for that driver will be the live migration support over virtio, once the specification which is WIP those day will be accepted.
The migration functionality is not X86 dependent and doesn't have the legacy virtio driver limitations that enforced us to run only on X86.
So, by that time we may need to enable in VFIO the loading of virtio-vfio-pci driver and put back the ifdef X86 inside VIRTIO, only on the legacy IO API, as I did already in V8.
Quoted 7 lines
Quoted 2 lines
So using a KCONFIG solution in VFIO is a short term one, which will be reverted just later on.
I understand the intent, but I don't think that justifies building a driver that serves no purpose in the interim. IF and when migration support becomes a reality, it's trivial to update the depends line.
OK, so I'll add a KCONFIG dependency on X86 as you suggested as part of V9 inside VFIO.
Quoted 7 lines
Quoted 3 lines
In addition, the
virtio_pci_admin_has_legacy_io()API can be used in the future not only by VFIO, this was one of the reasons to put it inside VIRTIO.Maybe this should be governed by a new
Kconfigoption which would be selected by drivers like this. Thanks,We can still keep the simple ifdef X86 inside VIRTIO for future users/usage which is not only VFIO.
Michael,
Can that work for you ?Yishai
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Alex
I am not sure what is proposed exactly. General admin q infrastructure can be kept as is. The legacy things however can never work outside X86. Best way to limit it to x86 is to move it to a separate file and only build that on X86. This way the only ifdef we need is where we set the flags to enable legacy commands.
In VFIO we already agreed to add a dependency on X86 [1] as Alex asked.
As VIRTIO should be ready for other clients and be self contained, I thought to keep things simple and just return false from
virtio_pci_admin_has_legacy_io()in non X86 systems as was sent in V8.However, we can go with your approach as well and compile out all the legacy IO stuff in non X86 systems by moving its code to a separate file (i.e.
virtio_pci_admin_legacy_io.c) and control this file upon theMakefile. In addition, you suggested to control the 'supported_cmds' by an ifdef. This will let the device know that we don't support legacy IO as well on non X86 systems.Please be aware that the above approach requires another ifdef on the H file which exposes the 6 exported symbols and some further changes inside virtio as of making
vp_modern_admin_cmd_exec()non static as now we move the legacy IO stuff to another C file, etc.Please see below how [2] it will look like.
If you prefer that, so OK, it will be part of V9.
Please let me know.[1] diff --git a/drivers/vfio/pci/virtio/Kconfig
b/drivers/vfio/pci/virtio/Kconfig
index050473b0e5df..a3e5d8ea22a0100644drivers/vfio/pci/virtio/Kconfig +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only config VIRTIO_VFIO_PCI tristate "VFIO support for VIRTIO NET PCI devices" - depends on VIRTIO_PCI + depends on X86 && VIRTIO_PCI select VFIO_PCI_CORE help This provides support for exposing VIRTIO NET VF devices whichsupport
[2] diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile index
8e98d24917cc..a73358bb4ebb100644drivers/virtio/Makefile +1
-0@@ -7,6 +7,7 @@ obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o +virtio_pci-$(CONFIG_X86) += virtio_pci_admin_legacy_io.o obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.odrivers/virtio/virtio_pci_common.h +0
-0@@ -158,4 +158,14 @@ void virtio_pci_modern_remove(struct virtio_pci_device *);drivers/virtio/virtio_pci_common.h void virtio_pci_modern_remove(struct virtio_pci_device · line 168
Quoted 10 lines
drivers/virtio/virtio_pci_common.h +7
-0struct virtio_device *virtio_pci_vf_get_pf_dev (struct pci_dev *pdev); +#define VIRTIO_LEGACY_ADMIN_CMD_BITMAP \ + (BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +I'd add something like:
Quoted 6 lines
#ifdef
CONFIG_X86
#defineVIRTIO_ADMIN_CMD_BITMAPVIRTIO_LEGACY_ADMIN_CMD_BITMAP#else
#defineVIRTIO_ADMIN_CMD_BITMAP0
#endif
Quoted 2 lines
This new macro (i.e.
VIRTIO_ADMIN_CMD_BITMAP) will hold in the future another sets of supported bits (e.g. Live migration), right ?
That's my idea, yes.
Quoted 9 lines
Quoted 1 line
Add a comment explaining why, please.
OK
How about the below ?
"As of some limitations in the legacy driver (e.g. lack of memory barriers in ARM, endian-ness is broken in PPC) the supported legacy IO commands are masked out on non x86 systems."
Some grammar corrections and clarifications:
Unlike modern drivers which support hardware virtio devices, legacy drivers assume software-based devices: e.g. they don't use proper memory barriers on ARM, use big endian on PPC, etc. X86 drivers are mostly ok though, more or less by chance. For now, only support legacy IO on x86.
Quoted 35 lines
Quoted 28 lines
Quoted 28 lines
drivers/virtio/virtio_pci_common.h +3
-0+int vp_modern_admin_cmd_exec (struct virtio_device *vdev, +struct virtio_admin_cmd *cmd); +#endifdrivers/virtio/virtio_pci_modern.c +2
-2@@ -75,8 +75,8 @@ static int virtqueue_exec_admin_cmd(structvirtio_pci_admin_vq *admin_vq, return 0; } -static int vp_modern_admin_cmd_exec (struct virtio_device *vdev, -struct virtio_admin_cmd *cmd) +int vp_modern_admin_cmd_exec (struct virtio_device *vdev, +struct virtio_admin_cmd *cmd) {struct scatterlist *sgs[VIRTIO_AVQ_SGS_MAX ], hdr, stat;struct virtio_pci_device *vp_dev = to_vp_device(vdev);drivers/virtio/virtio_pci_modern.c +0
-0@@ -172,6 +172,9 @@ static void virtio_pci_admin_cmd_list_init(struct virtio_device *virtio_dev)if (ret) goto end;Quoted 6 lines
drivers/virtio/virtio_pci_modern.c static void virtio_pci_admin_cmd_list_init(struct virtio_device *virtio_dev) · line 177
Quoted 3 lines
drivers/virtio/virtio_pci_modern.c +3
-0+#ifndef CONFIG_X86 + *data &= ~(cpu_to_le64 (VIRTIO_LEGACY_ADMIN_CMD_BITMAP )); +#endif Then here we don't need an ifdef just use
VIRTIO_ADMIN_CMD_BITMAP.
Quoted 6 lines
Following your suggestion, this will become,
*data &=cpu_to_le64(VIRTIO_ADMIN_CMD_BITMAP);(no ifdef, and no ~).
Yishai
right
Yishai Hadas
maintainerRe: [PATCH V7 vfio 9/9] vfio/virtio: Introduce a vfio driver over virtio devices
Quoted 227 lines · Michael S. Tsirkin
Quoted 210 lines · Yishai Hadas
Quoted 202 lines · Michael S. Tsirkin
Quoted 187 lines · Yishai Hadas
Quoted 116 lines · Michael S. Tsirkin
Quoted 115 lines · Yishai Hadas
Quoted 87 lines · Alex Williamson
Quoted 84 lines · On Thu, 14 Dec 2023 18:03:30 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 67 lines · Michael S. Tsirkin
Quoted 66 lines · Alex Williamson
Quoted 48 lines · On Thu, 14 Dec 2023 11:37:10 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 21 lines
Quoted 21 lines
OK, if so, we can come with the below extra code.
Makes sense ?I'll squash it as part of V8 to the relevant patch.
diff --git a/drivers/virtio/virtio_pci_modern.c
b/drivers/virtio/virtio_pci_modern.c
index 37a0035f8381..b652e91b9df4 100644
--- a/drivers/virtio/virtio_pci_modern.c
+++ b/drivers/virtio/virtio_pci_modern.c
@@ -794,6 +794,9 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev
*pdev)
struct virtio_device *virtio_dev = virtio_pci_vf_get_pf_dev(pdev);
struct virtio_pci_device *vp_dev;+#ifndef CONFIG_X86
+ return false;
+#endif
if (!virtio_dev)
return false;Quoted 4 lines
Quoted 1 line
Yishai
Isn't there going to be a bunch more dead code that compiler won't be able to elide?
On my setup the compiler didn't complain about dead-code (I simulated it by using ifdef
CONFIG_X86return false).However, if we suspect that some compiler might complain, we can come with the below instead.
Do you prefer that ?
index
37a0035f8381..53e29824d404100644drivers/virtio/virtio_pci_modern.c +1
-0@@ -782,6 +782,7 @@ static void vp_modern_destroy_avq(struct virtio_device *vdev) BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +#ifdef CONFIG_X86 /* * virtio_pci_admin_has_legacy_io - Checks whether the legacy IO* commands are supported
drivers/virtio/virtio_pci_modern.c +0
-0@@ -807,6 +808,12 @@ bool virtio_pci_admin_has_legacy_io(struct pci_dev *pdev)drivers/virtio/virtio_pci_modern.c bool virtio_pci_admin_has_legacy_io(struct pci_dev · line 818
Quoted 10 lines
drivers/virtio/virtio_pci_modern.c +6
-0return true; return false; } +#else +bool virtio_pci_admin_has_legacy_io (struct pci_dev *pdev) +{ +return false; +} +#endif EXPORT_SYMBOL_GPL (virtio_pci_admin_has_legacy_io);Doesn't this also raise the question of the purpose of virtio-vfio-pci on non-x86? Without any other features it offers nothing over vfio-pci and we should likely adjust the
Kconfigfor x86 orCOMPILE_TEST. Thanks,Quoted 5 lines
Quoted 1 line
Alex
Kconfigdependency is what I had in mind, yes. The X86 specific code invirtio_pci_modern.ccan be moved to a separate file then use makefile tricks to skip it on other platforms.The next feature for that driver will be the live migration support over virtio, once the specification which is WIP those day will be accepted.
The migration functionality is not X86 dependent and doesn't have the legacy virtio driver limitations that enforced us to run only on X86.
So, by that time we may need to enable in VFIO the loading of virtio-vfio-pci driver and put back the ifdef X86 inside VIRTIO, only on the legacy IO API, as I did already in V8.
Quoted 7 lines
Quoted 2 lines
So using a KCONFIG solution in VFIO is a short term one, which will be reverted just later on.
I understand the intent, but I don't think that justifies building a driver that serves no purpose in the interim. IF and when migration support becomes a reality, it's trivial to update the depends line.
OK, so I'll add a KCONFIG dependency on X86 as you suggested as part of V9 inside VFIO.
Quoted 7 lines
Quoted 3 lines
In addition, the
virtio_pci_admin_has_legacy_io()API can be used in the future not only by VFIO, this was one of the reasons to put it inside VIRTIO.Maybe this should be governed by a new
Kconfigoption which would be selected by drivers like this. Thanks,We can still keep the simple ifdef X86 inside VIRTIO for future users/usage which is not only VFIO.
Michael,
Can that work for you ?Yishai
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Alex
I am not sure what is proposed exactly. General admin q infrastructure can be kept as is. The legacy things however can never work outside X86. Best way to limit it to x86 is to move it to a separate file and only build that on X86. This way the only ifdef we need is where we set the flags to enable legacy commands.
In VFIO we already agreed to add a dependency on X86 [1] as Alex asked.
As VIRTIO should be ready for other clients and be self contained, I thought to keep things simple and just return false from
virtio_pci_admin_has_legacy_io()in non X86 systems as was sent in V8.However, we can go with your approach as well and compile out all the legacy IO stuff in non X86 systems by moving its code to a separate file (i.e.
virtio_pci_admin_legacy_io.c) and control this file upon theMakefile. In addition, you suggested to control the 'supported_cmds' by an ifdef. This will let the device know that we don't support legacy IO as well on non X86 systems.Please be aware that the above approach requires another ifdef on the H file which exposes the 6 exported symbols and some further changes inside virtio as of making
vp_modern_admin_cmd_exec()non static as now we move the legacy IO stuff to another C file, etc.Please see below how [2] it will look like.
If you prefer that, so OK, it will be part of V9.
Please let me know.[1] diff --git a/drivers/vfio/pci/virtio/Kconfig
b/drivers/vfio/pci/virtio/Kconfig
index050473b0e5df..a3e5d8ea22a0100644drivers/vfio/pci/virtio/Kconfig +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only config VIRTIO_VFIO_PCI tristate "VFIO support for VIRTIO NET PCI devices" - depends on VIRTIO_PCI + depends on X86 && VIRTIO_PCI select VFIO_PCI_CORE help This provides support for exposing VIRTIO NET VF devices whichsupport
[2] diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile index
8e98d24917cc..a73358bb4ebb100644drivers/virtio/Makefile +1
-0@@ -7,6 +7,7 @@ obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o +virtio_pci-$(CONFIG_X86) += virtio_pci_admin_legacy_io.o obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.odrivers/virtio/virtio_pci_common.h +0
-0@@ -158,4 +158,14 @@ void virtio_pci_modern_remove(struct virtio_pci_device *);drivers/virtio/virtio_pci_common.h void virtio_pci_modern_remove(struct virtio_pci_device · line 168
Quoted 10 lines
drivers/virtio/virtio_pci_common.h +7
-0struct virtio_device *virtio_pci_vf_get_pf_dev (struct pci_dev *pdev); +#define VIRTIO_LEGACY_ADMIN_CMD_BITMAP \ + (BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ) | \ + BIT_ULL(VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO)) +I'd add something like:
Quoted 6 lines
#ifdef
CONFIG_X86
#defineVIRTIO_ADMIN_CMD_BITMAPVIRTIO_LEGACY_ADMIN_CMD_BITMAP#else
#defineVIRTIO_ADMIN_CMD_BITMAP0
#endifQuoted 2 lines
This new macro (i.e.
VIRTIO_ADMIN_CMD_BITMAP) will hold in the future another sets of supported bits (e.g. Live migration), right ?That's my idea, yes.
Quoted 9 lines
Quoted 1 line
Add a comment explaining why, please.
OK
How about the below ?
"As of some limitations in the legacy driver (e.g. lack of memory barriers in ARM, endian-ness is broken in PPC) the supported legacy IO commands are masked out on non x86 systems."
Quoted 6 lines
Some grammar corrections and clarifications:
Unlike modern drivers which support hardware virtio devices, legacy drivers assume software-based devices: e.g. they don't use proper memory barriers on ARM, use big endian on PPC, etc. X86 drivers are mostly ok though, more or less by chance. For now, only support legacy IO on x86.
OK, thanks, I'll use the above.
Note:
I may use it also as part of the commit log (next patch) where we'll introduce the legacy IO APIs only on X86 systems, to let it be claer why it was done.
Yishai
Yishai Hadas
maintainerQuoted 200 lines · Yishai Hadas
This series introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
Background, from the virtio spec [1].
--------------------------------------------------------------------
In some systems, there is a need to support a virtio legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.
--------------------------------------------------------------------The first 6 patches are in the virtio area and handle the below:
- Introduce the admin virtqueue infrastcture.
- Expose the layout of the commands that should be used for supporting the legacy access.
- Expose APIs to enable upper layers as of vfio, net, etc to execute admin commands.The above follows the virtio spec that was lastly accepted in that area [1].
The last 3 patches are in the vfio area and handle the below:
- Expose some APIs from vfio/pci to be used by the vfio/virtio driver.
- Introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.The series was tested successfully over virtio-net VFs in the host, while running in the guest both modern and legacy drivers.
[1]
https://github.com/oasis-tcs/virtio-spec/commit/03c2d32e5093ca9f2a17797242fbef88efe94b8cChanges from V6: https://lore.kernel.org/kvm/[email protected]/
Vfio:
- Put thepm_runtimestuff intotranslate_io_bar_to_mem_bar()and
organize the callers to be more success oriented, as suggested by Jason.
- Add missing 'ops' (i.e. 'detach_ioas' and 'device_feature'), as mentioned by Jason.
- Cleanvirtiovf_bar0_exists()to cast to bool automatically, as
suggested by Jason.
- Add Reviewed-by: Jason Gunthorpe <[email protected]>.Changes from V5: https://lore.kernel.org/kvm/[email protected]/
Vfio:
- Renamevfio_pci_iowrite64tovfio_pci_core_iowrite64as was mentioned
by Alex.Changes from V4: https://lore.kernel.org/all/[email protected]/T/
Virtio:
- Drop the unused macro 'VIRTIO_ADMIN_MAX_CMD_OPCODE' as was asked by Michael.
- Add Acked-by: Michael S. Tsirkin <[email protected]>
Vfio:
- Exportvfio_pci_core_setup_barmap()in place and rename vfio_pci_iowrite/read<xxx> to have the 'core' prefix as part of the functions names, as was discussed with Alex.
- Improve packing of structvirtiovf_pci_core_device, as was suggested
by Alex.
- Upon reset, set 'pci_cmd' back to zero, in addition, if
the user didn't set the 'PCI_COMMAND_IO' bit, return-EIOupon any read/write towards the IO bar, as was suggested by Alex.
- Enforce byBUILD_BUG_ONthat 'bar0_virtual_buf_size' is power of 2 as part ofvirtiovf_pci_init_device()and clean the 'sizing calculation' code accordingly, as was suggested by Alex.Changes from V3: https://www.spinics.net/lists/kvm/msg333008.html
Virtio:
- Rebase on top of 6.7 rc3.
Vfio:
- Fix a typo, drop 'acc' from 'virtiovf_acc_vfio_pci_tran_ops'.Changes from V2: https://lore.kernel.org/all/[email protected]/T/
Virtio:
- Rebase on top of 6.7 rc1.
- Add a mutex to serialize admin commands execution and virtqueue
deletion, as was suggested by Michael.
- Remove the 'ref_count' usage which is not needed any more.
- Reduce the depth of the admin vq to match a single command at a given time.
- Add a supported check upon command execution and move to use a single flow ofvirtqueue_exec_admin_cmd().
- Improve the description of the exported commands to better match the specification and the expected usage as was asked by Michael.Vfio:
- Upon calling to virtio_pci_admin_legacy/common_device_io_read/write() supply the 'offset' within the relevant configuration area, following the virtio exported APIs.Changes from V1: https://lore.kernel.org/all/[email protected]/T/
Virtio:
- Drop its first patch, it was accepted upstream already.
- Add a new patch (#6) which initializes the supported admin commands
upon admin queue activation as was suggested by Michael.
- Split the legacy_io_read/write commands per common/device
configuration as was asked by Michael.
- Don't expose any more the list query/used APIs outside of virtio.
- Instead, expose an API to check whether the legacy io functionality is
supported as was suggested by Michael.
- Fix some Krobot's note by adding the missing include file.Vfio:
- Refer specifically to virtio-net as part of the driver/module description as Alex asked.
- Change to check MSIX enablement based on the irq type of the given vfio core device. In addition, drop its capable checking from the probe flow as was asked by Alex.
- Adapt to use the new virtio exposed APIs and clean some code accordingly.
- Adapt to some cleaner style code in some places (if/else) as was suggested by Alex.
- Fix therange_intersect_range()function and adapt its usage as was pointed by Alex.
- Make structvirtiovf_pci_core_devicebetter packed.
- Overwrite the subsystem vendor ID to be0x1af4as was discussed in the ML.
- Add support for the 'bar sizing negotiation' as was asked by Alex.
- Drop the 'acc' from the 'ops' as Alex asked.Changes from V0: https://www.spinics.net/lists/linux-virtualization/msg63802.html
Virtio:
- Fix the common config map size issue that was reported by Michael Tsirkin.
- Do not use vp_dev->vqs[] array uponvp_del_vqs()as was asked by Michael, instead skip the AQ specifically.
- Move admin vq implementation intovirtio_pci_modern.cas was asked by Michael.
- Rename structurevirtio_avqtovirtio_pci_admin_vqand some extra corresponding renames.
- Remove exported symbolsvirtio_pci_vf_get_pf_dev(),virtio_admin_cmd_exec()as now callers are local to the module.
- Handle inflight commands as part of the device reset flow.
- Introduce APIs per admin command in virtio-pci as was asked by Michael.Vfio:
- Change to useEXPORT_SYMBOL_GPLinstead ofEXPORT_SYMBOLforvfio_pci_core_setup_barmap()and vfio_pci_iowrite#xxx() as pointed by Alex.
- Drop the intermediate patch which prepares the commands and calls the generic virtio admin command API (i.e.virtio_admin_cmd_exec()).
- Instead, call directly to the new APIs per admin command that are exported from Virtio - based on Michael's request.
- Enable only virtio-net as part of thepci_device_idtable to enforce upon binding only what is supported as suggested by Alex.
- Add support for byte-wise access (read/write) over the device config region as was asked by Alex.
- Consider whether MSIX is practically enabled/disabled to choose the right opcode upon issuing read/write admin command, as mentioned by Michael.
- Move to useVIRTIO_PCI_CONFIG_OFFinstead of adding some new defines as was suggested by Michael.
- Set the '.close_device' op tovfio_pci_core_close_device()as was pointed by Alex.
- Adapt to Vfio multi-line comment style in a few places.
- Add [email protected] in theMAINTAINERSfile to be CCed for the new driver as was suggested by Jason.Yishai
Feng Liu (4):
virtio: Define feature bit for administration virtqueue
virtio-pci: Introduce admin virtqueue
virtio-pci: Introduce admin command sending function
virtio-pci: Introduce admin commandsYishai Hadas (5):
virtio-pci: Initialize the supported admin commands
virtio-pci: Introduce APIs to execute legacy IO admin commands vfio/pci: Exposevfio_pci_core_setup_barmap()vfio/pci: Expose vfio_pci_core_iowrite/read##size() vfio/virtio: Introduce a vfio driver over virtio devicesMAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/vfio_pci_rdwr.c | 57 +-- drivers/vfio/pci/virtio/Kconfig | 16 + drivers/vfio/pci/virtio/Makefile | 4 + drivers/vfio/pci/virtio/main.c | 567 +++++++++++++++++++++++++ drivers/virtio/virtio.c | 37 +- drivers/virtio/virtio_pci_common.c | 14 + drivers/virtio/virtio_pci_common.h | 21 +- drivers/virtio/virtio_pci_modern.c | 503 +++++++++++++++++++++- drivers/virtio/virtio_pci_modern_dev.c | 24 +- include/linux/vfio_pci_core.h | 20 + include/linux/virtio.h | 8 + include/linux/virtio_config.h | 4 + include/linux/virtio_pci_admin.h | 21 + include/linux/virtio_pci_modern.h | 2 + include/uapi/linux/virtio_config.h | 8 +- include/uapi/linux/virtio_pci.h | 68 +++ 19 files changed, 1349 insertions(+), 36 deletions(-)
Quoted 5 lines
create mode 100644
drivers/vfio/pci/virtio/Kconfigcreate mode 100644drivers/vfio/pci/virtio/Makefilecreate mode 100644drivers/vfio/pci/virtio/main.ccreate mode 100644include/linux/virtio_pci_admin.h
Hi Michael and Alex,
It seems that we are done with all the last notes here.
Michael,
Based on Alex's note here [1] "the preferred merge approach would be that virtio maintainers take patches 1-6 and provide a branch or tag I can merge to bring 7-9 in through the vfio tree"
Can that please be done to proceed here ?
[1] https://patchwork.kernel.org/project/kvm/patch/[email protected]/
Thanks,
Yishai
Michael S. Tsirkin
maintainerQuoted 209 lines · Yishai Hadas
Quoted 200 lines · Yishai Hadas
This series introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.
Background, from the virtio spec [1].
--------------------------------------------------------------------
In some systems, there is a need to support a virtio legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.
--------------------------------------------------------------------The first 6 patches are in the virtio area and handle the below:
- Introduce the admin virtqueue infrastcture.
- Expose the layout of the commands that should be used for supporting the legacy access.
- Expose APIs to enable upper layers as of vfio, net, etc to execute admin commands.The above follows the virtio spec that was lastly accepted in that area [1].
The last 3 patches are in the vfio area and handle the below:
- Expose some APIs from vfio/pci to be used by the vfio/virtio driver.
- Introduce a vfio driver over virtio devices to support the legacy interface functionality for VFs.The series was tested successfully over virtio-net VFs in the host, while running in the guest both modern and legacy drivers.
[1]
https://github.com/oasis-tcs/virtio-spec/commit/03c2d32e5093ca9f2a17797242fbef88efe94b8cChanges from V6: https://lore.kernel.org/kvm/[email protected]/
Vfio:
- Put thepm_runtimestuff intotranslate_io_bar_to_mem_bar()and
organize the callers to be more success oriented, as suggested by Jason.
- Add missing 'ops' (i.e. 'detach_ioas' and 'device_feature'), as mentioned by Jason.
- Cleanvirtiovf_bar0_exists()to cast to bool automatically, as
suggested by Jason.
- Add Reviewed-by: Jason Gunthorpe <[email protected]>.Changes from V5: https://lore.kernel.org/kvm/[email protected]/
Vfio:
- Renamevfio_pci_iowrite64tovfio_pci_core_iowrite64as was mentioned
by Alex.Changes from V4: https://lore.kernel.org/all/[email protected]/T/
Virtio:
- Drop the unused macro 'VIRTIO_ADMIN_MAX_CMD_OPCODE' as was asked by Michael.
- Add Acked-by: Michael S. Tsirkin <[email protected]>
Vfio:
- Exportvfio_pci_core_setup_barmap()in place and rename vfio_pci_iowrite/read<xxx> to have the 'core' prefix as part of the functions names, as was discussed with Alex.
- Improve packing of structvirtiovf_pci_core_device, as was suggested
by Alex.
- Upon reset, set 'pci_cmd' back to zero, in addition, if
the user didn't set the 'PCI_COMMAND_IO' bit, return-EIOupon any read/write towards the IO bar, as was suggested by Alex.
- Enforce byBUILD_BUG_ONthat 'bar0_virtual_buf_size' is power of 2 as part ofvirtiovf_pci_init_device()and clean the 'sizing calculation' code accordingly, as was suggested by Alex.Changes from V3: https://www.spinics.net/lists/kvm/msg333008.html
Virtio:
- Rebase on top of 6.7 rc3.
Vfio:
- Fix a typo, drop 'acc' from 'virtiovf_acc_vfio_pci_tran_ops'.Changes from V2: https://lore.kernel.org/all/[email protected]/T/
Virtio:
- Rebase on top of 6.7 rc1.
- Add a mutex to serialize admin commands execution and virtqueue
deletion, as was suggested by Michael.
- Remove the 'ref_count' usage which is not needed any more.
- Reduce the depth of the admin vq to match a single command at a given time.
- Add a supported check upon command execution and move to use a single flow ofvirtqueue_exec_admin_cmd().
- Improve the description of the exported commands to better match the specification and the expected usage as was asked by Michael.Vfio:
- Upon calling to virtio_pci_admin_legacy/common_device_io_read/write() supply the 'offset' within the relevant configuration area, following the virtio exported APIs.Changes from V1: https://lore.kernel.org/all/[email protected]/T/
Virtio:
- Drop its first patch, it was accepted upstream already.
- Add a new patch (#6) which initializes the supported admin commands
upon admin queue activation as was suggested by Michael.
- Split the legacy_io_read/write commands per common/device
configuration as was asked by Michael.
- Don't expose any more the list query/used APIs outside of virtio.
- Instead, expose an API to check whether the legacy io functionality is
supported as was suggested by Michael.
- Fix some Krobot's note by adding the missing include file.Vfio:
- Refer specifically to virtio-net as part of the driver/module description as Alex asked.
- Change to check MSIX enablement based on the irq type of the given vfio core device. In addition, drop its capable checking from the probe flow as was asked by Alex.
- Adapt to use the new virtio exposed APIs and clean some code accordingly.
- Adapt to some cleaner style code in some places (if/else) as was suggested by Alex.
- Fix therange_intersect_range()function and adapt its usage as was pointed by Alex.
- Make structvirtiovf_pci_core_devicebetter packed.
- Overwrite the subsystem vendor ID to be0x1af4as was discussed in the ML.
- Add support for the 'bar sizing negotiation' as was asked by Alex.
- Drop the 'acc' from the 'ops' as Alex asked.Changes from V0: https://www.spinics.net/lists/linux-virtualization/msg63802.html
Virtio:
- Fix the common config map size issue that was reported by Michael Tsirkin.
- Do not use vp_dev->vqs[] array uponvp_del_vqs()as was asked by Michael, instead skip the AQ specifically.
- Move admin vq implementation intovirtio_pci_modern.cas was asked by Michael.
- Rename structurevirtio_avqtovirtio_pci_admin_vqand some extra corresponding renames.
- Remove exported symbolsvirtio_pci_vf_get_pf_dev(),virtio_admin_cmd_exec()as now callers are local to the module.
- Handle inflight commands as part of the device reset flow.
- Introduce APIs per admin command in virtio-pci as was asked by Michael.Vfio:
- Change to useEXPORT_SYMBOL_GPLinstead ofEXPORT_SYMBOLforvfio_pci_core_setup_barmap()and vfio_pci_iowrite#xxx() as pointed by Alex.
- Drop the intermediate patch which prepares the commands and calls the generic virtio admin command API (i.e.virtio_admin_cmd_exec()).
- Instead, call directly to the new APIs per admin command that are exported from Virtio - based on Michael's request.
- Enable only virtio-net as part of thepci_device_idtable to enforce upon binding only what is supported as suggested by Alex.
- Add support for byte-wise access (read/write) over the device config region as was asked by Alex.
- Consider whether MSIX is practically enabled/disabled to choose the right opcode upon issuing read/write admin command, as mentioned by Michael.
- Move to useVIRTIO_PCI_CONFIG_OFFinstead of adding some new defines as was suggested by Michael.
- Set the '.close_device' op tovfio_pci_core_close_device()as was pointed by Alex.
- Adapt to Vfio multi-line comment style in a few places.
- Add [email protected] in theMAINTAINERSfile to be CCed for the new driver as was suggested by Jason.Yishai
Feng Liu (4):
virtio: Define feature bit for administration virtqueue
virtio-pci: Introduce admin virtqueue
virtio-pci: Introduce admin command sending function
virtio-pci: Introduce admin commandsYishai Hadas (5):
virtio-pci: Initialize the supported admin commands
virtio-pci: Introduce APIs to execute legacy IO admin commands vfio/pci: Exposevfio_pci_core_setup_barmap()vfio/pci: Expose vfio_pci_core_iowrite/read##size() vfio/virtio: Introduce a vfio driver over virtio devicesMAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/vfio_pci_rdwr.c | 57 +-- drivers/vfio/pci/virtio/Kconfig | 16 + drivers/vfio/pci/virtio/Makefile | 4 + drivers/vfio/pci/virtio/main.c | 567 +++++++++++++++++++++++++ drivers/virtio/virtio.c | 37 +- drivers/virtio/virtio_pci_common.c | 14 + drivers/virtio/virtio_pci_common.h | 21 +- drivers/virtio/virtio_pci_modern.c | 503 +++++++++++++++++++++- drivers/virtio/virtio_pci_modern_dev.c | 24 +- include/linux/vfio_pci_core.h | 20 + include/linux/virtio.h | 8 + include/linux/virtio_config.h | 4 + include/linux/virtio_pci_admin.h | 21 + include/linux/virtio_pci_modern.h | 2 + include/uapi/linux/virtio_config.h | 8 +- include/uapi/linux/virtio_pci.h | 68 +++ 19 files changed, 1349 insertions(+), 36 deletions(-)Quoted 5 lines
create mode 100644
drivers/vfio/pci/virtio/Kconfigcreate mode 100644drivers/vfio/pci/virtio/Makefilecreate mode 100644drivers/vfio/pci/virtio/main.ccreate mode 100644include/linux/virtio_pci_admin.hHi Michael and Alex,
Quoted 8 lines
It seems that we are done with all the last notes here.
Michael,
Based on Alex's note here [1] "the preferred merge approach would be that virtio maintainers take patches 1-6 and provide a branch or tag I can merge to bring 7-9 in through the vfio tree"Can that please be done to proceed here ?
Hmm ok. Next week-ish, too busy this week.
Quoted 4 lines
[1] https://patchwork.kernel.org/project/kvm/patch/[email protected]/
Thanks,
Yishai