Add device DMA logging support for mlx5 driver
Subsystems (8)
Yishai Hadas
maintainerv7 cover Add device DMA logging support for mlx5 driver
This series adds device DMA logging uAPIs and their implementation as part of mlx5 driver.
DMA logging allows a device to internally record what DMAs the device is initiating and report them back to userspace. It is part of the VFIO migration infrastructure that allows implementing dirty page tracking during the pre copy phase of live migration. Only DMA WRITEs are logged, and this API is not connected to VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE.
The uAPIs are based on the FEATURE ioctl as were introduced earlier by the below RFC [1] and follows the notes that were discussed in the mailing list.
It includes:
- A PROBE option to detect if the device supports DMA logging.
- A SET option to start device DMA logging in given IOVAs ranges.
- A GET option to read back and clear the device DMA log.
- A SET option to stop device DMA logging that was previously started.
Extra details exist as part of relevant patches in the series.
In addition, the series adds some infrastructure support for managing an IOVA bitmap done by Joao Martins.
It abstracts how an IOVA range is represented in a bitmap that is granulated by a given page_size. So it translates all the lifting of dealing with user pointers into its corresponding kernel addresses. This new functionality abstracts the complexity of user/kernel bitmap pointer usage and finally enables an API to set some bits.
This functionality will be used as part of IOMMUFD series for the system IOMMU tracking.
Finally, we come with mlx5 implementation based on its device specification for the DMA logging APIs.
The matching qemu changes can be previewed here [2].
They come on top of the v2 migration protocol patches that were sent already to the mailing list.
Note:
- As this series touched mlx5_core parts we may need to send the net/mlx5 patches as a pull request format to VFIO to avoid conflicts before acceptance.
[1] https://lore.kernel.org/all/[email protected]/T/
[2] https://github.com/avihai1122/qemu/commits/device_dirty_tracking
Changes from V6: https://lore.kernel.org/all/[email protected]/T/
- Use the first two patches from the PR that was sent by Leon, no code change was involved.
- Patch #5:
* Add a documentation note near vfio_log_ops as Alex suggested.
Changes from V5: https://lore.kernel.org/netdev/[email protected]/T/
- Rebase on top of the latest code from vfio/next.
- Patch #4:
Improvements from Joao for his IOVA bitmap patch, it includes the below:
* Make sure documentation refers to struct page pointer size
* Some rephrasing and grammar fixes in the documentation.
* Make iova_bitmap_set() return void
* Turn the iterator callback function pointer into a type as part of the header file
* Use max_t instead of max as part of iova_bitmap_set() to address krobot note
Changes from V4: https://lore.kernel.org/netdev/[email protected]/T/
Patch #4:
Improvements from Joao for his IOVA bitmap patch to be aligned with Alex suggestions. It includes the below:
* Simplify API by removing iterator helpers to introduce a iova_bitmap_for_each() helper
* Simplify API by making struct iova_bitmap private and make it the
argument of all API callers instead of having the iterator mode.
* Rename iova_bitmap_init() into _alloc() given that now it allocates
the iova_bitmap structure
* Free the object in iova_bitmap_free(), given the previous change
* Rename struct iova_bitmap_iter into struct iova_bitmap
* Rename struct iova_bitmap into struct iova_bitmap_map
* Rename iova_bitmap_map::dirty into ::mapped and all its references
* Rename iova_bitmap_map::start_offset into ::pgoff
* Rename iova_bitmap::data into ::bitmap
* Rename iova_bitmap::offset into ::mapped_base_index
* Rename iova_bitmap::count into ::mapped_total_index
* Change ::mapped_base_index and ::mapped_total_index type to unsigned
long
* Change ::length type to size_t
* Rename iova_bitmap_iova_to_index into iova_bitmap_offset_to_index()
* Rename iova_bitmap_index_to_iova into iova_bitmap_index_to_offset()
* Rename iova_bitmap_iter_remaining() to iova_bitmap_mapped_remaining()
* Rename iova_bitmap_iova() to iova_bitmap_mapped_iova()
* Rename iova_bitmap_length() to iova_bitmap_mapped_length()
* Drop _iter_ suffix and rename @iter into @bitmap in variables/arguments
* Make a whole bunch of API calls static, given that the iteration is
now private
* Move kdoc into the source file rather than the header
Patch #5:
* Adapt to use the new IOVA bitmap API
* Upon the report UAPI enforce at least 4K page size and not the system PAGE_SIZE as pointed out by Alex.
* Add an extra overflow checking in both the report/start UAPIs for the
input iova and length as pointed out by Jason.
Changes from V3: https://lore.kernel.org/all/[email protected]/t/
Rebase on top of v6.0 rc1.
Patch #3:
- Drop the documentation note regarding the 'atomic OR' usage of the bitmap as part of VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT.
This deletion was missed as part of V3 to match kernel code.
To better clarify, as part of testing V3, we could see a big penalty in performance (*2 in some cases) when the iova bitmap patch used atomic bit operations. As QEMU doesn't really share bitmaps between multiple trackers we saw no reason to use atomics and get a bad performance. If in the future, will be a real use case that will justify it, we can come with VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT_ATOMIC new option with the matching kernel code.
Patch #4:
- The rename patch from vfio.c to vfio_main.c was accepted already, not part of this series anymore.
Changes from V2: https://lore.kernel.org/netdev/[email protected]/t/
Patch #1
- Add some reserved fields that were missed.
Patch #3:
- Improve the UAPI documentation in few places as was asked by Alex and Kevin, based on the discussion in the mailing list.
Patch #5:
- Improvements from Joao for his IOVA bitmap patch to be cleaner/simpler as was asked by Alex. It includes the below:
* Make iova_to_index and index_to_iova fully symmetrical.
* Use 'sizeof(*iter->data) * BITS_PER_BYTE' in both index_to_iova and iova_to_index.
* Remove iova_bitmap_init() and just stay with iova_bitmap_iter_init().
* s/left/remaining/
* To not use @remaining variable for both index and iova/length.
* Remove stale comment on max dirty bitmap bits.
* Remove DIV_ROUNDUP from iova_to_index() helper and replace with a division.
* Use iova rather than length where appropriate, while noting with commentary the usage of length as next relative IOVA.
* Rework pinning to be internal and remove that from the iova iter API caller.
* get() and put() now teardown iova_bitmap::dirty npages.
* Move unnecessary includes into the C file.
* Add theory of operation and theory of usage in the header file.
* Add more comments on private helpers on less obvious logic
* Add documentation on all public APIs.
* Change commit to reflect new usage of APIs.
Patch #6:
- Drop the hard-coded 1024 for LOG_MAX_RANGES and replace to consider PAGE_SIZE as was suggested by Jason.
- Return -E2BIG as Alex suggested.
- Adapt the loop upon logging report to new IOVA bit map stuff.
Changes from V1: https://lore.kernel.org/netdev/[email protected]/T/
- Patch #6: Fix a note given by krobot, select INTERVAL_TREE for VFIO.
Changes from V0: https://lore.kernel.org/netdev/[email protected]/T/
- Drop the first 2 patches that Alex merged already.
- Fix a note given by krobot, based on Jason's suggestion.
- Some improvements from Joao for his IOVA bitmap patch to be cleaner/simpler. It includes the below:
* Rename iova_bitmap_array_length to iova_bitmap_iova_to_index.
* Rename iova_bitmap_index_to_length to iova_bitmap_index_to_iova.
* Change iova_bitmap_iova_to_index to take an iova_bitmap_iter as an argument to pair with iova_bitmap_index_to_length.
* Make iova_bitmap_iter_done() use >= instead of substraction+comparison. This fixes iova_bitmap_iter_done() return as it was previously returning when !done.
* Remove iova_bitmap_iter_length().
* Simplify iova_bitmap_length() overcomplicated trailing end check
* Convert all sizeof(u64) into sizeof(*iter->data).
* Use u64 __user for ::data instead of void in both struct and initialization of iova_bitmap.
Yishai
Joao Martins (1):
vfio: Add an IOVA bitmap support
Yishai Hadas (9):
net/mlx5: Introduce ifc bits for page tracker net/mlx5: Query ADV_VIRTUALIZATION capabilities
vfio: Introduce DMA logging uAPIs
vfio: Introduce the DMA logging feature support vfio/mlx5: Init QP based resources for dirty tracking vfio/mlx5: Create and destroy page tracker object vfio/mlx5: Report dirty pages from tracker vfio/mlx5: Manage error scenarios on tracker vfio/mlx5: Set the driver DMA logging callbacks
drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 + .../net/ethernet/mellanox/mlx5/core/main.c | 1 + drivers/vfio/Kconfig | 1 + drivers/vfio/Makefile | 6 +- drivers/vfio/iova_bitmap.c | 422 ++++++++ drivers/vfio/pci/mlx5/cmd.c | 995 +++++++++++++++++- drivers/vfio/pci/mlx5/cmd.h | 63 +- drivers/vfio/pci/mlx5/main.c | 9 +- drivers/vfio/pci/vfio_pci_core.c | 5 + drivers/vfio/vfio_main.c | 175 +++ include/linux/iova_bitmap.h | 26 + include/linux/mlx5/device.h | 9 + include/linux/mlx5/mlx5_ifc.h | 83 +- include/linux/vfio.h | 28 +- include/uapi/linux/vfio.h | 86 ++ 15 files changed, 1895 insertions(+), 20 deletions(-) create mode 100644 drivers/vfio/iova_bitmap.c create mode 100644 include/linux/iova_bitmap.h
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH v7 1/10 net/mlx5: Introduce ifc bits for page tracker
Introduce ifc related stuff to enable using page tracker.
A page tracker is a dirty page tracking object used by the device to report the tracking log.
--- include/linux/mlx5/mlx5_ifc.h | 83 ++++++++++++++++++++++++++++++++++- 1 file changed, 82 insertions(+), 1 deletion(-)
include/linux/mlx5/mlx5_ifc.h +82 -1
@@ -89,6 +89,7 @@ enum {MLX5_OBJ_TYPE_VIRTIO_NET_Q =0x000d ,MLX5_OBJ_TYPE_VIRTIO_Q_COUNTERS =0x001c ,MLX5_OBJ_TYPE_MATCH_DEFINER =0x0018 , +MLX5_OBJ_TYPE_PAGE_TRACK =0x46 ,MLX5_OBJ_TYPE_MKEY =0xff01 ,MLX5_OBJ_TYPE_QP =0xff02 , MLX5_OBJ_TYPE_PSV = 0xff03,
@@ -1733,7 +1734,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {u8 max_geneve_tlv_options[0x8 ];u8 reserved_at_568[0x3 ];u8 max_geneve_tlv_option_data_len[0x5 ]; -u8 reserved_at_570[0x10 ]; +u8 reserved_at_570[0x9 ]; +u8 adv_virtualization[0x1 ]; +u8 reserved_at_57a[0x6 ];u8 reserved_at_580[0xb ];u8 log_max_dci_stream_channels[0x5 ];
@@ -11818,4 +11821,82 @@ struct mlx5_ifc_load_vhca_state_out_bits {u8 reserved_at_40[0x40 ]; }; +struct mlx5_ifc_adv_virtualization_cap_bits { +u8 reserved_at_0[0x3 ]; +u8 pg_track_log_max_num[0x5 ]; +u8 pg_track_max_num_range[0x8 ]; +u8 pg_track_log_min_addr_space[0x8 ]; +u8 pg_track_log_max_addr_space[0x8 ]; + +u8 reserved_at_20[0x3 ]; +u8 pg_track_log_min_msg_size[0x5 ]; +u8 reserved_at_28[0x3 ]; +u8 pg_track_log_max_msg_size[0x5 ]; +u8 reserved_at_30[0x3 ]; +u8 pg_track_log_min_page_size[0x5 ]; +u8 reserved_at_38[0x3 ]; +u8 pg_track_log_max_page_size[0x5 ]; + +u8 reserved_at_40[0x7c0 ]; +}; + +struct mlx5_ifc_page_track_report_entry_bits { +u8 dirty_address_high[0x20 ]; + +u8 dirty_address_low[0x20 ]; +}; + +enum { +MLX5_PAGE_TRACK_STATE_TRACKING , +MLX5_PAGE_TRACK_STATE_REPORTING , +MLX5_PAGE_TRACK_STATE_ERROR , +}; + +struct mlx5_ifc_page_track_range_bits { +u8 start_address[0x40 ]; + +u8 length[0x40 ]; +}; + +struct mlx5_ifc_page_track_bits { +u8 modify_field_select[0x40 ]; + +u8 reserved_at_40[0x10 ]; +u8 vhca_id[0x10 ]; + +u8 reserved_at_60[0x20 ]; + +u8 state[0x4 ]; +u8 track_type[0x4 ]; +u8 log_addr_space_size[0x8 ]; +u8 reserved_at_90[0x3 ]; +u8 log_page_size[0x5 ]; +u8 reserved_at_98[0x3 ]; +u8 log_msg_size[0x5 ]; + +u8 reserved_at_a0[0x8 ]; +u8 reporting_qpn[0x18 ]; + +u8 reserved_at_c0[0x18 ]; +u8 num_ranges[0x8 ]; + +u8 reserved_at_e0[0x20 ]; + +u8 range_start_address[0x40 ]; + +u8 length[0x40 ]; + +struct mlx5_ifc_page_track_range_bits track_range[0 ]; +}; + +struct mlx5_ifc_create_page_track_obj_in_bits { +struct mlx5_ifc_general_obj_in_cmd_hdr_bits general_obj_in_cmd_hdr; +struct mlx5_ifc_page_track_bits obj_context; +}; + +struct mlx5_ifc_modify_page_track_obj_in_bits { +struct mlx5_ifc_general_obj_in_cmd_hdr_bits general_obj_in_cmd_hdr; +struct mlx5_ifc_page_track_bits obj_context; +}; +#endif /* MLX5_IFC_H */
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH v7 2/10 net/mlx5: Query ADV_VIRTUALIZATION capabilities
In reply to Yishai HadasQuery ADV_VIRTUALIZATION capabilities which provide information for advanced virtualization related features.
Current capabilities refer to the page tracker object which is used for tracking the pages that are dirtied by the device.
--- drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 ++++++ drivers/net/ethernet/mellanox/mlx5/core/main.c | 1 + include/linux/mlx5/device.h | 9 +++++++++ 3 files changed, 16 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/fw.c +6 -0
@@ -273,6 +273,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)return err; } +if (MLX5_CAP_GEN (dev, adv_virtualization)) { + err =mlx5_core_get_caps (dev,MLX5_CAP_ADV_VIRTUALIZATION ); +if (err) +return err; + } +return 0 ; }
drivers/net/ethernet/mellanox/mlx5/core/main.c +1 -0
@@ -1488,6 +1488,7 @@ static const int types[] = {MLX5_CAP_IPSEC ,MLX5_CAP_PORT_SELECTION ,MLX5_CAP_DEV_SHAMPO , + MLX5_CAP_ADV_VIRTUALIZATION, };static void mlx5_hca_caps_free (struct mlx5_core_dev *dev)
include/linux/mlx5/device.h +9 -0
@@ -1200,6 +1200,7 @@ enum mlx5_cap_type {MLX5_CAP_DEV_SHAMPO =0x1d ,MLX5_CAP_GENERAL_2 =0x20 ,MLX5_CAP_PORT_SELECTION =0x25 , +MLX5_CAP_ADV_VIRTUALIZATION =0x26 ,/* NUM OF CAP Types */ MLX5_CAP_NUM };
@@ -1365,6 +1366,14 @@ enum mlx5_qcam_feature_groups {MLX5_GET (port_selection_cap, \ mdev->caps.hca[MLX5_CAP_PORT_SELECTION]->max, cap) +#define MLX5_CAP_ADV_VIRTUALIZATION (mdev, cap) \ + MLX5_GET(adv_virtualization_cap, \ + mdev->caps.hca[MLX5_CAP_ADV_VIRTUALIZATION]->cur, cap) + +#define MLX5_CAP_ADV_VIRTUALIZATION_MAX (mdev, cap) \ + MLX5_GET(adv_virtualization_cap, \ + mdev->caps.hca[MLX5_CAP_ADV_VIRTUALIZATION]->max, cap) +#define MLX5_CAP_FLOWTABLE_PORT_SELECTION (mdev, cap) \ MLX5_CAP_PORT_SELECTION(mdev, flow_table_properties_port_selection.cap)
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH v7 3/10 vfio: Introduce DMA logging uAPIs
In reply to Yishai HadasDMA logging allows a device to internally record what DMAs the device is initiating and report them back to userspace. It is part of the VFIO migration infrastructure that allows implementing dirty page tracking during the pre copy phase of live migration. Only DMA WRITEs are logged, and this API is not connected to VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE.
This patch introduces the DMA logging involved uAPIs.
It uses the FEATURE ioctl with its GET/SET/PROBE options as of below.
It exposes a PROBE option to detect if the device supports DMA logging. It exposes a SET option to start device DMA logging in given IOVAs ranges.
It exposes a SET option to stop device DMA logging that was previously started.
It exposes a GET option to read back and clear the device DMA log.
Extra details exist as part of vfio.h per a specific option.
--- include/uapi/linux/vfio.h | 86 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+)
include/uapi/linux/vfio.h +86 -0
@@ -1042,6 +1042,92 @@ struct vfio_device_low_power_entry_with_wakeup {*/ #define VFIO_DEVICE_FEATURE_LOW_POWER_EXIT 5 +/* +* Upon VFIO_DEVICE_FEATURE_SET start/stop device DMA logging. +* VFIO_DEVICE_FEATURE_PROBE can be used to detect if the device supports +* DMA logging. +* +* DMA logging allows a device to internally record what DMAs the device is +* initiating and report them back to userspace. It is part of the VFIO +* migration infrastructure that allows implementing dirty page tracking +* during the pre copy phase of live migration. Only DMA WRITEs are logged, +* and this API is not connected to VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE. +* +* When DMA logging is started a range of IOVAs to monitor is provided and the +* device can optimize its logging to cover only the IOVA range given. Each +* DMA that the device initiates inside the range will be logged by the device +* for later retrieval. +* +* page_size is an input that hints what tracking granularity the device +* should try to achieve. If the device cannot do the hinted page size then +* it's the driver choice which page size to pick based on its support. +* On output the device will return the page size it selected. +* +* ranges is a pointer to an array of +* struct vfio_device_feature_dma_logging_range. +* +* The core kernel code guarantees to support by minimum num_ranges that fit +* into a single kernel page. User space can try higher values but should give +* up if the above can't be achieved as of some driver limitations. +* +* A single call to start device DMA logging can be issued and a matching stop +* should follow at the end. Another start is not allowed in the meantime. +*/ +struct vfio_device_feature_dma_logging_control { +__aligned_u64 page_size; +__u32 num_ranges; +__u32 __reserved; +__aligned_u64 ranges; +}; + +struct vfio_device_feature_dma_logging_range { +__aligned_u64 iova; +__aligned_u64 length; +}; + +#define VFIO_DEVICE_FEATURE_DMA_LOGGING_START 6 + +/* +* Upon VFIO_DEVICE_FEATURE_SET stop device DMA logging that was started +* by VFIO_DEVICE_FEATURE_DMA_LOGGING_START +*/ +#define VFIO_DEVICE_FEATURE_DMA_LOGGING_STOP 7 + +/* +* Upon VFIO_DEVICE_FEATURE_GET read back and clear the device DMA log +* +* Query the device's DMA log for written pages within the given IOVA range. +* During querying the log is cleared for the IOVA range. +* +* bitmap is a pointer to an array of u64s that will hold the output bitmap +* with 1 bit reporting a page_size unit of IOVA. The mapping of IOVA to bits +* is given by: +* bitmap[(addr - iova)/page_size] & (1ULL << (addr % 64)) +* +* The input page_size can be any power of two value and does not have to +* match the value given to VFIO_DEVICE_FEATURE_DMA_LOGGING_START. The driver +* will format its internal logging to match the reporting page size, possibly +* by replicating bits if the internal page size is lower than requested. +* +* The LOGGING_REPORT will only set bits in the bitmap and never clear or +* perform any initialization of the user provided bitmap. +* +* If any error is returned userspace should assume that the dirty log is +* corrupted. Error recovery is to consider all memory dirty and try to +* restart the dirty tracking, or to abort/restart the whole migration. +* +* If DMA logging is not enabled, an error will be returned. +* +*/ +struct vfio_device_feature_dma_logging_report { +__aligned_u64 iova; +__aligned_u64 length; +__aligned_u64 page_size; +__aligned_u64 bitmap; +}; + +#define VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT 8 +/* -------- API for Type1 VFIO IOMMU -------- */ /**
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH v7 4/10 vfio: Add an IOVA bitmap support
In reply to Yishai HadasFrom: Joao Martins <[email protected]>
The new facility adds a bunch of wrappers that abstract how an IOVA range is represented in a bitmap that is granulated by a given page_size. So it translates all the lifting of dealing with user pointers into its corresponding kernel addresses backing said user memory into doing finally the (non-atomic) bitmap ops to change various bits.
The formula for the bitmap is:
data[(iova / page_size) / 64] & (1ULL << (iova % 64))
Where 64 is the number of bits in a unsigned long (depending on arch)
It introduces an IOVA iterator that uses a windowing scheme to minimize the pinning overhead, as opposed to pinning it on demand 4K at a time. Assuming a 4K kernel page and 4K requested page size, we can use a single kernel page to hold 512 page pointers, mapping 2M of bitmap, representing 64G of IOVA space.
An example usage of these helpers for a given @base_iova, @page_size, @length and __user @data:
bitmap = iova_bitmap_alloc(base_iova, page_size, length, data);
if (IS_ERR(bitmap))
return -ENOMEM;
ret = iova_bitmap_for_each(bitmap, arg, dirty_reporter_fn);
iova_bitmap_free(bitmap);Each iteration of the @dirty_reporter_fn is called with a unique @iova and @length argument, indicating the current range available through the iova_bitmap. The @dirty_reporter_fn uses iova_bitmap_set() to mark dirty areas (@iova_length) within that provided range, as following:
iova_bitmap_set(bitmap, iova, iova_length);
The facility is intended to be used for user bitmaps representing dirtied IOVAs by IOMMU (via IOMMUFD) and PCI Devices (via vfio-pci).
--- drivers/vfio/Makefile | 6 +- drivers/vfio/iova_bitmap.c | 422 ++++++++++++++++++++++++++++++++++++ include/linux/iova_bitmap.h | 26 +++ 3 files changed, 452 insertions(+), 2 deletions(-) create mode 100644 drivers/vfio/iova_bitmap.c create mode 100644 include/linux/iova_bitmap.h
drivers/vfio/Makefile +4 -2
@@ -1,9 +1,11 @@ # SPDX-License-Identifier: GPL-2.0 vfio_virqfd-y := virqfd.o -vfio-y += vfio_main.o - obj-$(CONFIG_VFIO) += vfio.o + +vfio-y += vfio_main.o \ + iova_bitmap.o \ + obj-$(CONFIG_VFIO_VIRQFD) += vfio_virqfd.o obj-$(CONFIG_VFIO_IOMMU_TYPE1) += vfio_iommu_type1.o obj-$(CONFIG_VFIO_IOMMU_SPAPR_TCE) += vfio_iommu_spapr_tce.o
drivers/vfio/iova_bitmap.c new file +422 -0
422 lines not on this page · raw message
include/linux/iova_bitmap.h new file +26 -0
@@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2022, Oracle and/or its affiliates. +* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved +*/ +#ifndef _IOVA_BITMAP_H_ +#define _IOVA_BITMAP_H_ + +#include <linux/types.h> + +struct iova_bitmap ; + +typedef int (*iova_bitmap_fn_t )(struct iova_bitmap *bitmap, +unsigned long iova,size_t length, +void *opaque); + +struct iova_bitmap *iova_bitmap_alloc (unsigned long iova,size_t length, +unsigned long page_size, +u64 __user *data); +void iova_bitmap_free (struct iova_bitmap *bitmap); +int iova_bitmap_for_each (struct iova_bitmap *bitmap,void *opaque, +iova_bitmap_fn_t fn); +void iova_bitmap_set (struct iova_bitmap *bitmap, +unsigned long iova,size_t length); + +#endif
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH v7 5/10 vfio: Introduce the DMA logging feature support
In reply to Yishai HadasIntroduce the DMA logging feature support in the vfio core layer.
It includes the processing of the device start/stop/report DMA logging UAPIs and calling the relevant driver 'op' to do the work.
Specifically,
Upon start, the core translates the given input ranges into an interval tree, checks for unexpected overlapping, non aligned ranges and then pass the translated input to the driver for start tracking the given ranges.
Upon report, the core translates the given input user space bitmap and page size into an IOVA kernel bitmap iterator. Then it iterates it and call the driver to set the corresponding bits for the dirtied pages in a specific IOVA range.
Upon stop, the driver is called to stop the previous started tracking.
The next patches from the series will introduce the mlx5 driver implementation for the logging ops.
--- drivers/vfio/Kconfig | 1 + drivers/vfio/pci/vfio_pci_core.c | 5 + drivers/vfio/vfio_main.c | 175 +++++++++++++++++++++++++++++++ include/linux/vfio.h | 28 ++++- 4 files changed, 207 insertions(+), 2 deletions(-)
drivers/vfio/Kconfig +1 -0
@@ -3,6 +3,7 @@ menuconfig VFIO tristate "VFIO Non-Privileged userspace driver framework" select IOMMU_API select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64) + select INTERVAL_TREE help VFIO provides a framework for secure userspace device drivers. See Documentation/driver-api/vfio.rst for more details.
drivers/vfio/pci/vfio_pci_core.c +5 -0
@@ -2128,6 +2128,11 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev)return -EINVAL ; } +if (vdev->vdev.log_ops && !(vdev->vdev.log_ops->log_start && + vdev->vdev.log_ops->log_stop && + vdev->vdev.log_ops->log_read_and_clear)) +return -EINVAL ; +/* * Prevent binding to PFs with VFs enabled, the VFs might be in use * by the host or other users. We cannot capture the VFs if they
drivers/vfio/vfio_main.c +175 -0
193 lines not on this page · raw message
include/linux/vfio.h +26 -2
@@ -14,6 +14,7 @@#include <linux/workqueue.h> #include <linux/poll.h> #include <uapi/linux/vfio.h> +#include <linux/iova_bitmap.h> struct kvm ;
@@ -33,10 +34,11 @@ struct vfio_device {struct device *dev;const struct vfio_device_ops *ops;/* -* mig_ops is a static property of the vfio_device which must be set -* prior to registering the vfio_device. +* mig_ops/log_ops is a static property of the vfio_device which must +* be set prior to registering the vfio_device. */ const struct vfio_migration_ops *mig_ops; +const struct vfio_log_ops *log_ops;struct vfio_group *group;struct vfio_device_set *dev_set;struct list_head dev_set_list;
@@ -108,6 +110,28 @@ struct vfio_migration_ops { enum vfio_device_mig_state *curr_state); }; +/** +* @log_start: Optional callback to ask the device start DMA logging. +* @log_stop: Optional callback to ask the device stop DMA logging. +* @log_read_and_clear: Optional callback to ask the device read +* and clear the dirty DMAs in some given range. +* +* The vfio core implementation of the DEVICE_FEATURE_DMA_LOGGING_ set +* of features does not track logging state relative to the device, +* therefore the device implementation of vfio_log_ops must handle +* arbitrary user requests. This includes rejecting subsequent calls +* to log_start without an intervening log_stop, as well as graceful +* handling of log_stop and log_read_and_clear from invalid states. +*/ +struct vfio_log_ops { +int (*log_start)(struct vfio_device *device, +struct rb_root_cached *ranges,u32 nnodes,u64 *page_size); +int (*log_stop)(struct vfio_device *device); +int (*log_read_and_clear)(struct vfio_device *device, +unsigned long iova,unsigned long length, +struct iova_bitmap *dirty); +}; +/** * vfio_check_feature - Validate user input for the VFIO_DEVICE_FEATURE ioctl * @flags: Arg from the device_feature op
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH v7 6/10 vfio/mlx5: Init QP based resources for dirty tracking
In reply to Yishai HadasInit QP based resources for dirty tracking to be used upon start logging.
It includes:
Creating the host and firmware RC QPs, move each of them to its expected state based on the device specification, etc.
Creating the relevant resources which are needed by both QPs as of UAR, PD, etc.
Creating the host receive side resources as of MKEY, CQ, receive WQEs, etc.
The above resources are cleaned-up upon stop logging.
The tracker object that will be introduced by next patches will use those resources.
--- drivers/vfio/pci/mlx5/cmd.c | 595 +++++++++++++++++++++++++++++++++++- drivers/vfio/pci/mlx5/cmd.h | 53 ++++ 2 files changed, 636 insertions(+), 12 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +583 -12
@@ -7,6 +7,8 @@static int mlx5vf_cmd_get_vhca_id (struct mlx5_core_dev *mdev,u16 function_id,u16 *vhca_id); +static void +_mlx5vf_free_page_tracker_resources (struct mlx5vf_pci_core_device *mvdev);int mlx5vf_cmd_suspend_vhca (struct mlx5vf_pci_core_device *mvdev,u16 op_mod) {
@@ -72,19 +74,22 @@ static int mlx5fv_vf_event(struct notifier_block *nb,struct mlx5vf_pci_core_device *mvdev = container_of(nb, struct mlx5vf_pci_core_device, nb); -mutex_lock (&mvdev->state_mutex);switch (event) {case MLX5_PF_NOTIFY_ENABLE_VF : +mutex_lock (&mvdev->state_mutex); mvdev->mdev_detach = false; +mlx5vf_state_mutex_unlock (mvdev);break ;case MLX5_PF_NOTIFY_DISABLE_VF : -mlx5vf_disable_fds (mvdev); +mlx5vf_cmd_close_migratable (mvdev); +mutex_lock (&mvdev->state_mutex); mvdev->mdev_detach = true; +mlx5vf_state_mutex_unlock (mvdev);break ;default :break ; } -mlx5vf_state_mutex_unlock (mvdev); +return 0 ; }
@@ -95,6 +100,7 @@ void mlx5vf_cmd_close_migratable(struct mlx5vf_pci_core_device *mvdev)mutex_lock (&mvdev->state_mutex);mlx5vf_disable_fds (mvdev); +_mlx5vf_free_page_tracker_resources (mvdev);mlx5vf_state_mutex_unlock (mvdev); }
@@ -188,11 +194,13 @@ static int mlx5vf_cmd_get_vhca_id(struct mlx5_core_dev *mdev, u16 function_id,return ret; } -static int _create_state_mkey (struct mlx5_core_dev *mdev,u32 pdn, -struct mlx5_vf_migration_file *migf,u32 *mkey) +static int _create_mkey (struct mlx5_core_dev *mdev,u32 pdn, +struct mlx5_vf_migration_file *migf, +struct mlx5_vhca_recv_buf *recv_buf, +u32 *mkey) { -size_t npages =DIV_ROUND_UP (migf->total_length,PAGE_SIZE ); -struct sg_dma_page_iter dma_iter; +size_t npages = migf ?DIV_ROUND_UP (migf->total_length,PAGE_SIZE ) : + recv_buf->npages;int err =0 , inlen;__be64 *mtt;void *mkc;
@@ -209,8 +217,17 @@ static int _create_state_mkey(struct mlx5_core_dev *mdev, u32 pdn, DIV_ROUND_UP(npages, 2)); mtt = (__be64 *)MLX5_ADDR_OF (create_mkey_in, in, klm_pas_mtt); -for_each_sgtable_dma_page (&migf->table.sgt, &dma_iter,0 ) - *mtt++ =cpu_to_be64 (sg_page_iter_dma_address (&dma_iter)); +if (migf) { +struct sg_dma_page_iter dma_iter; + +for_each_sgtable_dma_page (&migf->table.sgt, &dma_iter,0 ) + *mtt++ =cpu_to_be64 (sg_page_iter_dma_address (&dma_iter)); + }else { +int i; + +for (i =0 ; i < npages; i++) + *mtt++ =cpu_to_be64 (recv_buf->dma_addrs[i]); + } mkc =MLX5_ADDR_OF (create_mkey_in, in, memory_key_mkey_entry);MLX5_SET (mkc, mkc, access_mode_1_0,MLX5_MKC_ACCESS_MODE_MTT );
@@ -223,7 +240,8 @@ static int _create_state_mkey(struct mlx5_core_dev *mdev, u32 pdn,MLX5_SET (mkc, mkc, qpn,0xffffff );MLX5_SET (mkc, mkc, log_page_size,PAGE_SHIFT );MLX5_SET (mkc, mkc, translations_octword_size,DIV_ROUND_UP (npages,2 )); -MLX5_SET64 (mkc, mkc, len, migf->total_length); +MLX5_SET64 (mkc, mkc, len, + migf ? migf->total_length : (npages *PAGE_SIZE )); err =mlx5_core_create_mkey (mdev, mkey, in, inlen);kvfree (in);return err;
@@ -297,7 +315,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,if (err) goto err_dma_map; - err =_create_state_mkey (mdev, pdn, migf, &mkey); + err =_create_mkey (mdev, pdn, migf,NULL , &mkey);if (err) goto err_create_mkey;
@@ -369,7 +387,7 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,if (err) goto err_reg; - err =_create_state_mkey (mdev, pdn, migf, &mkey); + err =_create_mkey (mdev, pdn, migf,NULL , &mkey);if (err) goto err_mkey;
@@ -391,3 +409,556 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,mutex_unlock (&migf->lock);return err; } + +static int alloc_cq_frag_buf (struct mlx5_core_dev *mdev, +struct mlx5_vhca_cq_buf *buf,int nent, +int cqe_size) +{ +struct mlx5_frag_buf *frag_buf = &buf->frag_buf; +u8 log_wq_stride =6 + (cqe_size ==128 ?1 :0 ); +u8 log_wq_sz =ilog2 (cqe_size); +int err; + + err =mlx5_frag_buf_alloc_node (mdev, nent * cqe_size, frag_buf, + mdev->priv.numa_node); +if (err) +return err; + +mlx5_init_fbc (frag_buf->frags, log_wq_stride, log_wq_sz, &buf->fbc); + buf->cqe_size = cqe_size; + buf->nent = nent; +return 0 ; +} + +static void init_cq_frag_buf (struct mlx5_vhca_cq_buf *buf) +{ +struct mlx5_cqe64 *cqe64; +void *cqe; +int i; + +for (i =0 ; i < buf->nent; i++) { + cqe =mlx5_frag_buf_get_wqe (&buf->fbc, i); + cqe64 = buf->cqe_size ==64 ? cqe : cqe +64 ; + cqe64->op_own =MLX5_CQE_INVALID <<4 ; + } +} + +static void mlx5vf_destroy_cq (struct mlx5_core_dev *mdev, +struct mlx5_vhca_cq *cq) +{ +mlx5_core_destroy_cq (mdev, &cq->mcq); +mlx5_frag_buf_free (mdev, &cq->buf.frag_buf); +mlx5_db_free (mdev, &cq->db); +} + +static int mlx5vf_create_cq (struct mlx5_core_dev *mdev, +struct mlx5_vhca_page_tracker *tracker, +size_t ncqe) +{ +int cqe_size =cache_line_size () ==128 ?128 :64 ; +u32 out[MLX5_ST_SZ_DW (create_cq_out)]; +struct mlx5_vhca_cq *cq; +int inlen, err, eqn; +void *cqc, *in; +__be64 *pas; +int vector; + + cq = &tracker->cq; + ncqe =roundup_pow_of_two (ncqe); + err =mlx5_db_alloc_node (mdev, &cq->db, mdev->priv.numa_node); +if (err) +return err; + + cq->ncqe = ncqe; + cq->mcq.set_ci_db = cq->db.db; + cq->mcq.arm_db = cq->db.db +1 ; + cq->mcq.cqe_sz = cqe_size; + err =alloc_cq_frag_buf (mdev, &cq->buf, ncqe, cqe_size); +if (err) + goto err_db_free; + +init_cq_frag_buf (&cq->buf); + inlen =MLX5_ST_SZ_BYTES (create_cq_in) + +MLX5_FLD_SZ_BYTES (create_cq_in, pas[0 ]) * + cq->buf.frag_buf.npages; + in =kvzalloc (inlen,GFP_KERNEL ); +if (!in) { + err = -ENOMEM ; + goto err_buff; + } + + vector =raw_smp_processor_id () %mlx5_comp_vectors_count (mdev); + err =mlx5_vector2eqn (mdev, vector, &eqn); +if (err) + goto err_vec; + + cqc =MLX5_ADDR_OF (create_cq_in, in, cq_context); +MLX5_SET (cqc, cqc, log_cq_size,ilog2 (ncqe)); +MLX5_SET (cqc, cqc, c_eqn_or_apu_element, eqn); +MLX5_SET (cqc, cqc, uar_page, tracker->uar->index); +MLX5_SET (cqc, cqc, log_page_size, cq->buf.frag_buf.page_shift - +MLX5_ADAPTER_PAGE_SHIFT ); +MLX5_SET64 (cqc, cqc, dbr_addr, cq->db.dma); + pas = (__be64 *)MLX5_ADDR_OF (create_cq_in, in, pas); +mlx5_fill_page_frag_array (&cq->buf.frag_buf, pas); + err =mlx5_core_create_cq (mdev, &cq->mcq, in, inlen, out,sizeof (out)); +if (err) + goto err_vec; + +kvfree (in); +return 0 ; + +err_vec: +kvfree (in); +err_buff: +mlx5_frag_buf_free (mdev, &cq->buf.frag_buf); +err_db_free: +mlx5_db_free (mdev, &cq->db); +return err; +} + +static struct mlx5_vhca_qp * +mlx5vf_create_rc_qp (struct mlx5_core_dev *mdev, +struct mlx5_vhca_page_tracker *tracker,u32 max_recv_wr) +{ +u32 out[MLX5_ST_SZ_DW (create_qp_out)] = {}; +struct mlx5_vhca_qp *qp; +u8 log_rq_stride; +u8 log_rq_sz; +void *qpc; +int inlen; +void *in; +int err; + + qp =kzalloc (sizeof (*qp),GFP_KERNEL ); +if (!qp) +return ERR_PTR (-ENOMEM ); + + qp->rq.wqe_cnt =roundup_pow_of_two (max_recv_wr); + log_rq_stride =ilog2 (MLX5_SEND_WQE_DS ); + log_rq_sz =ilog2 (qp->rq.wqe_cnt); + err =mlx5_db_alloc_node (mdev, &qp->db, mdev->priv.numa_node); +if (err) + goto err_free; + +if (max_recv_wr) { + err =mlx5_frag_buf_alloc_node (mdev, +wq_get_byte_sz (log_rq_sz, log_rq_stride), + &qp->buf, mdev->priv.numa_node); +if (err) + goto err_db_free; +mlx5_init_fbc (qp->buf.frags, log_rq_stride, log_rq_sz, &qp->rq.fbc); + } + + qp->rq.db = &qp->db.db[MLX5_RCV_DBR ]; + inlen =MLX5_ST_SZ_BYTES (create_qp_in) + +MLX5_FLD_SZ_BYTES (create_qp_in, pas[0 ]) * + qp->buf.npages; + in =kvzalloc (inlen,GFP_KERNEL ); +if (!in) { + err = -ENOMEM ; + goto err_in; + } + + qpc =MLX5_ADDR_OF (create_qp_in, in, qpc); +MLX5_SET (qpc, qpc, st,MLX5_QP_ST_RC ); +MLX5_SET (qpc, qpc, pm_state,MLX5_QP_PM_MIGRATED ); +MLX5_SET (qpc, qpc, pd, tracker->pdn); +MLX5_SET (qpc, qpc, uar_page, tracker->uar->index); +MLX5_SET (qpc, qpc, log_page_size, + qp->buf.page_shift -MLX5_ADAPTER_PAGE_SHIFT ); +MLX5_SET (qpc, qpc, ts_format,mlx5_get_qp_default_ts (mdev)); +if (MLX5_CAP_GEN (mdev, cqe_version) ==1 ) +MLX5_SET (qpc, qpc, user_index,0xFFFFFF ); +MLX5_SET (qpc, qpc, no_sq,1 ); +if (max_recv_wr) { +MLX5_SET (qpc, qpc, cqn_rcv, tracker->cq.mcq.cqn); +MLX5_SET (qpc, qpc, log_rq_stride, log_rq_stride -4 ); +MLX5_SET (qpc, qpc, log_rq_size, log_rq_sz); +MLX5_SET (qpc, qpc, rq_type,MLX5_NON_ZERO_RQ ); +MLX5_SET64 (qpc, qpc, dbr_addr, qp->db.dma); +mlx5_fill_page_frag_array (&qp->buf, + (__be64 *)MLX5_ADDR_OF (create_qp_in, + in, pas)); + }else { +MLX5_SET (qpc, qpc, rq_type,MLX5_ZERO_LEN_RQ ); + } + +MLX5_SET (create_qp_in, in, opcode,MLX5_CMD_OP_CREATE_QP ); + err =mlx5_cmd_exec (mdev, in, inlen, out,sizeof (out)); +kvfree (in); +if (err) + goto err_in; + + qp->qpn =MLX5_GET (create_qp_out, out, qpn); +return qp; + +err_in: +if (max_recv_wr) +mlx5_frag_buf_free (mdev, &qp->buf); +err_db_free: +mlx5_db_free (mdev, &qp->db); +err_free: +kfree (qp); +return ERR_PTR (err); +} + +static void mlx5vf_post_recv (struct mlx5_vhca_qp *qp) +{ +struct mlx5_wqe_data_seg *data; +unsigned int ix; + +WARN_ON (qp->rq.pc - qp->rq.cc >= qp->rq.wqe_cnt); + ix = qp->rq.pc & (qp->rq.wqe_cnt -1 ); + data =mlx5_frag_buf_get_wqe (&qp->rq.fbc, ix); + data->byte_count =cpu_to_be32 (qp->max_msg_size); + data->lkey =cpu_to_be32 (qp->recv_buf.mkey); + data->addr =cpu_to_be64 (qp->recv_buf.next_rq_offset); + qp->rq.pc++; +/* Make sure that descriptors are written before doorbell record. */ +dma_wmb (); + *qp->rq.db =cpu_to_be32 (qp->rq.pc &0xffff ); +} + +static int mlx5vf_activate_qp (struct mlx5_core_dev *mdev, +struct mlx5_vhca_qp *qp,u32 remote_qpn, +bool host_qp) +{ +u32 init_in[MLX5_ST_SZ_DW (rst2init_qp_in)] = {}; +u32 rtr_in[MLX5_ST_SZ_DW (init2rtr_qp_in)] = {}; +u32 rts_in[MLX5_ST_SZ_DW (rtr2rts_qp_in)] = {}; +void *qpc; +int ret; + +/* Init */ + qpc =MLX5_ADDR_OF (rst2init_qp_in, init_in, qpc); +MLX5_SET (qpc, qpc, primary_address_path.vhca_port_num,1 ); +MLX5_SET (qpc, qpc, pm_state,MLX5_QPC_PM_STATE_MIGRATED ); +MLX5_SET (qpc, qpc, rre,1 ); +MLX5_SET (qpc, qpc, rwe,1 ); +MLX5_SET (rst2init_qp_in, init_in, opcode,MLX5_CMD_OP_RST2INIT_QP ); +MLX5_SET (rst2init_qp_in, init_in, qpn, qp->qpn); + ret =mlx5_cmd_exec_in (mdev, rst2init_qp, init_in); +if (ret) +return ret; + +if (host_qp) { +struct mlx5_vhca_recv_buf *recv_buf = &qp->recv_buf; +int i; + +for (i =0 ; i < qp->rq.wqe_cnt; i++) { +mlx5vf_post_recv (qp); + recv_buf->next_rq_offset += qp->max_msg_size; + } + } + +/* RTR */ + qpc =MLX5_ADDR_OF (init2rtr_qp_in, rtr_in, qpc); +MLX5_SET (init2rtr_qp_in, rtr_in, qpn, qp->qpn); +MLX5_SET (qpc, qpc, mtu,IB_MTU_4096 ); +MLX5_SET (qpc, qpc, log_msg_max,MLX5_CAP_GEN (mdev, log_max_msg)); +MLX5_SET (qpc, qpc, remote_qpn, remote_qpn); +MLX5_SET (qpc, qpc, primary_address_path.vhca_port_num,1 ); +MLX5_SET (qpc, qpc, primary_address_path.fl,1 ); +MLX5_SET (qpc, qpc, min_rnr_nak,1 ); +MLX5_SET (init2rtr_qp_in, rtr_in, opcode,MLX5_CMD_OP_INIT2RTR_QP ); +MLX5_SET (init2rtr_qp_in, rtr_in, qpn, qp->qpn); + ret =mlx5_cmd_exec_in (mdev, init2rtr_qp, rtr_in); +if (ret || host_qp) +return ret; + +/* RTS */ + qpc =MLX5_ADDR_OF (rtr2rts_qp_in, rts_in, qpc); +MLX5_SET (rtr2rts_qp_in, rts_in, qpn, qp->qpn); +MLX5_SET (qpc, qpc, retry_count,7 ); +MLX5_SET (qpc, qpc, rnr_retry,7 );/* Infinite retry if RNR NACK */ +MLX5_SET (qpc, qpc, primary_address_path.ack_timeout,0x8 );/* ~1ms */ +MLX5_SET (rtr2rts_qp_in, rts_in, opcode,MLX5_CMD_OP_RTR2RTS_QP ); +MLX5_SET (rtr2rts_qp_in, rts_in, qpn, qp->qpn); + +return mlx5_cmd_exec_in (mdev, rtr2rts_qp, rts_in); +} + +static void mlx5vf_destroy_qp (struct mlx5_core_dev *mdev, +struct mlx5_vhca_qp *qp) +{ +u32 in[MLX5_ST_SZ_DW (destroy_qp_in)] = {}; + +MLX5_SET (destroy_qp_in, in, opcode,MLX5_CMD_OP_DESTROY_QP ); +MLX5_SET (destroy_qp_in, in, qpn, qp->qpn); +mlx5_cmd_exec_in (mdev, destroy_qp, in); + +mlx5_frag_buf_free (mdev, &qp->buf); +mlx5_db_free (mdev, &qp->db); +kfree (qp); +} + +static void free_recv_pages (struct mlx5_vhca_recv_buf *recv_buf) +{ +int i; + +/* Undo alloc_pages_bulk_array() */ +for (i =0 ; i < recv_buf->npages; i++) +__free_page (recv_buf->page_list[i]); + +kvfree (recv_buf->page_list); +} + +static int alloc_recv_pages (struct mlx5_vhca_recv_buf *recv_buf, +unsigned int npages) +{ +unsigned int filled =0 , done =0 ; +int i; + + recv_buf->page_list =kvcalloc (npages,sizeof (*recv_buf->page_list), +GFP_KERNEL ); +if (!recv_buf->page_list) +return -ENOMEM ; + +for (;;) { + filled =alloc_pages_bulk_array (GFP_KERNEL , npages - done, + recv_buf->page_list + done); +if (!filled) + goto err; + + done += filled; +if (done == npages) +break ; + } + + recv_buf->npages = npages; +return 0 ; + +err: +for (i =0 ; i < npages; i++) { +if (recv_buf->page_list[i]) +__free_page (recv_buf->page_list[i]); + } + +kvfree (recv_buf->page_list); +return -ENOMEM ; +} + +static int register_dma_recv_pages (struct mlx5_core_dev *mdev, +struct mlx5_vhca_recv_buf *recv_buf) +{ +int i, j; + + recv_buf->dma_addrs =kvcalloc (recv_buf->npages, +sizeof (*recv_buf->dma_addrs), +GFP_KERNEL ); +if (!recv_buf->dma_addrs) +return -ENOMEM ; + +for (i =0 ; i < recv_buf->npages; i++) { + recv_buf->dma_addrs[i] =dma_map_page (mdev->device, + recv_buf->page_list[i], +0 ,PAGE_SIZE , +DMA_FROM_DEVICE ); +if (dma_mapping_error (mdev->device, recv_buf->dma_addrs[i])) + goto error; + } +return 0 ; + +error: +for (j =0 ; j < i; j++) +dma_unmap_single (mdev->device, recv_buf->dma_addrs[j], +PAGE_SIZE ,DMA_FROM_DEVICE ); + +kvfree (recv_buf->dma_addrs); +return -ENOMEM ; +} + +static void unregister_dma_recv_pages (struct mlx5_core_dev *mdev, +struct mlx5_vhca_recv_buf *recv_buf) +{ +int i; + +for (i =0 ; i < recv_buf->npages; i++) +dma_unmap_single (mdev->device, recv_buf->dma_addrs[i], +PAGE_SIZE ,DMA_FROM_DEVICE ); + +kvfree (recv_buf->dma_addrs); +} + +static void mlx5vf_free_qp_recv_resources (struct mlx5_core_dev *mdev, +struct mlx5_vhca_qp *qp) +{ +struct mlx5_vhca_recv_buf *recv_buf = &qp->recv_buf; + +mlx5_core_destroy_mkey (mdev, recv_buf->mkey); +unregister_dma_recv_pages (mdev, recv_buf); +free_recv_pages (&qp->recv_buf); +} + +static int mlx5vf_alloc_qp_recv_resources (struct mlx5_core_dev *mdev, +struct mlx5_vhca_qp *qp,u32 pdn, +u64 rq_size) +{ +unsigned int npages =DIV_ROUND_UP_ULL (rq_size,PAGE_SIZE ); +struct mlx5_vhca_recv_buf *recv_buf = &qp->recv_buf; +int err; + + err =alloc_recv_pages (recv_buf, npages); +if (err <0 ) +return err; + + err =register_dma_recv_pages (mdev, recv_buf); +if (err) + goto end; + + err =_create_mkey (mdev, pdn,NULL , recv_buf, &recv_buf->mkey); +if (err) + goto err_create_mkey; + +return 0 ; + +err_create_mkey: +unregister_dma_recv_pages (mdev, recv_buf); +end: +free_recv_pages (recv_buf); +return err; +} + +static void +_mlx5vf_free_page_tracker_resources (struct mlx5vf_pci_core_device *mvdev) +{ +struct mlx5_vhca_page_tracker *tracker = &mvdev->tracker; +struct mlx5_core_dev *mdev = mvdev->mdev; + +lockdep_assert_held (&mvdev->state_mutex); + +if (!mvdev->log_active) +return ; + +WARN_ON (mvdev->mdev_detach); + +mlx5vf_destroy_qp (mdev, tracker->fw_qp); +mlx5vf_free_qp_recv_resources (mdev, tracker->host_qp); +mlx5vf_destroy_qp (mdev, tracker->host_qp); +mlx5vf_destroy_cq (mdev, &tracker->cq); +mlx5_core_dealloc_pd (mdev, tracker->pdn); +mlx5_put_uars_page (mdev, tracker->uar); + mvdev->log_active = false; +} + +int mlx5vf_stop_page_tracker (struct vfio_device *vdev) +{ +struct mlx5vf_pci_core_device *mvdev =container_of ( + vdev, struct mlx5vf_pci_core_device, core_device.vdev); + +mutex_lock (&mvdev->state_mutex); +if (!mvdev->log_active) + goto end; + +_mlx5vf_free_page_tracker_resources (mvdev); + mvdev->log_active = false; +end: +mlx5vf_state_mutex_unlock (mvdev); +return 0 ; +} + +int mlx5vf_start_page_tracker (struct vfio_device *vdev, +struct rb_root_cached *ranges,u32 nnodes, +u64 *page_size) +{ +struct mlx5vf_pci_core_device *mvdev =container_of ( + vdev, struct mlx5vf_pci_core_device, core_device.vdev); +struct mlx5_vhca_page_tracker *tracker = &mvdev->tracker; +u8 log_tracked_page =ilog2 (*page_size); +struct mlx5_vhca_qp *host_qp; +struct mlx5_vhca_qp *fw_qp; +struct mlx5_core_dev *mdev; +u32 max_msg_size =PAGE_SIZE ; +u64 rq_size =SZ_2M ; +u32 max_recv_wr; +int err; + +mutex_lock (&mvdev->state_mutex); +if (mvdev->mdev_detach) { + err = -ENOTCONN ; + goto end; + } + +if (mvdev->log_active) { + err = -EINVAL ; + goto end; + } + + mdev = mvdev->mdev; +memset (tracker,0 ,sizeof (*tracker)); + tracker->uar =mlx5_get_uars_page (mdev); +if (IS_ERR (tracker->uar)) { + err =PTR_ERR (tracker->uar); + goto end; + } + + err =mlx5_core_alloc_pd (mdev, &tracker->pdn); +if (err) + goto err_uar; + + max_recv_wr =DIV_ROUND_UP_ULL (rq_size, max_msg_size); + err =mlx5vf_create_cq (mdev, tracker, max_recv_wr); +if (err) + goto err_dealloc_pd; + + host_qp =mlx5vf_create_rc_qp (mdev, tracker, max_recv_wr); +if (IS_ERR (host_qp)) { + err =PTR_ERR (host_qp); + goto err_cq; + } + + host_qp->max_msg_size = max_msg_size; +if (log_tracked_page <MLX5_CAP_ADV_VIRTUALIZATION (mdev, + pg_track_log_min_page_size)) { + log_tracked_page =MLX5_CAP_ADV_VIRTUALIZATION (mdev, + pg_track_log_min_page_size); + }else if (log_tracked_page >MLX5_CAP_ADV_VIRTUALIZATION (mdev, + pg_track_log_max_page_size)) { + log_tracked_page =MLX5_CAP_ADV_VIRTUALIZATION (mdev, + pg_track_log_max_page_size); + } + + host_qp->tracked_page_size = (1ULL << log_tracked_page); + err =mlx5vf_alloc_qp_recv_resources (mdev, host_qp, tracker->pdn, + rq_size); +if (err) + goto err_host_qp; + + fw_qp =mlx5vf_create_rc_qp (mdev, tracker,0 ); +if (IS_ERR (fw_qp)) { + err =PTR_ERR (fw_qp); + goto err_recv_resources; + } + + err =mlx5vf_activate_qp (mdev, host_qp, fw_qp->qpn, true); +if (err) + goto err_activate; + + err =mlx5vf_activate_qp (mdev, fw_qp, host_qp->qpn, false); +if (err) + goto err_activate; + + tracker->host_qp = host_qp; + tracker->fw_qp = fw_qp; + *page_size = host_qp->tracked_page_size; + mvdev->log_active = true; +mlx5vf_state_mutex_unlock (mvdev); +return 0 ; + +err_activate: +mlx5vf_destroy_qp (mdev, fw_qp); +err_recv_resources: +mlx5vf_free_qp_recv_resources (mdev, host_qp); +err_host_qp: +mlx5vf_destroy_qp (mdev, host_qp); +err_cq: +mlx5vf_destroy_cq (mdev, &tracker->cq); +err_dealloc_pd: +mlx5_core_dealloc_pd (mdev, tracker->pdn); +err_uar: +mlx5_put_uars_page (mdev, tracker->uar); +end: +mlx5vf_state_mutex_unlock (mvdev); +return err; +}
drivers/vfio/pci/mlx5/cmd.h +53 -0
81 lines not on this page · raw message
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH v7 7/10 vfio/mlx5: Create and destroy page tracker object
In reply to Yishai HadasAdd support for creating and destroying page tracker object.
This object is used to control/report the device dirty pages.
As part of creating the tracker need to consider the device capabilities for max ranges and adapt/combine ranges accordingly.
--- drivers/vfio/pci/mlx5/cmd.c | 147 ++++++++++++++++++++++++++++++++++++ drivers/vfio/pci/mlx5/cmd.h | 1 + 2 files changed, 148 insertions(+)
drivers/vfio/pci/mlx5/cmd.c +147 -0
@@ -410,6 +410,148 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,return err; } +static void combine_ranges (struct rb_root_cached *root,u32 cur_nodes, +u32 req_nodes) +{ +struct interval_tree_node *prev, *curr, *comb_start, *comb_end; +unsigned long min_gap; +unsigned long curr_gap; + +/* Special shortcut when a single range is required */ +if (req_nodes ==1 ) { +unsigned long last; + + curr = comb_start =interval_tree_iter_first (root,0 ,ULONG_MAX ); +while (curr) { + last = curr->last; + prev = curr; + curr =interval_tree_iter_next (curr,0 ,ULONG_MAX ); +if (prev != comb_start) +interval_tree_remove (prev, root); + } + comb_start->last = last; +return ; + } + +/* Combine ranges which have the smallest gap */ +while (cur_nodes > req_nodes) { + prev =NULL ; + min_gap =ULONG_MAX ; + curr =interval_tree_iter_first (root,0 ,ULONG_MAX ); +while (curr) { +if (prev) { + curr_gap = curr->start - prev->last; +if (curr_gap < min_gap) { + min_gap = curr_gap; + comb_start = prev; + comb_end = curr; + } + } + prev = curr; + curr =interval_tree_iter_next (curr,0 ,ULONG_MAX ); + } + comb_start->last = comb_end->last; +interval_tree_remove (comb_end, root); + cur_nodes--; + } +} + +static int mlx5vf_create_tracker (struct mlx5_core_dev *mdev, +struct mlx5vf_pci_core_device *mvdev, +struct rb_root_cached *ranges,u32 nnodes) +{ +int max_num_range = +MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_max_num_range); +struct mlx5_vhca_page_tracker *tracker = &mvdev->tracker; +int record_size =MLX5_ST_SZ_BYTES (page_track_range); +u32 out[MLX5_ST_SZ_DW (general_obj_out_cmd_hdr)] = {}; +struct interval_tree_node *node =NULL ; +u64 total_ranges_len =0 ; +u32 num_ranges = nnodes; +u8 log_addr_space_size; +void *range_list_ptr; +void *obj_context; +void *cmd_hdr; +int inlen; +void *in; +int err; +int i; + +if (num_ranges > max_num_range) { +combine_ranges (ranges, nnodes, max_num_range); + num_ranges = max_num_range; + } + + inlen =MLX5_ST_SZ_BYTES (create_page_track_obj_in) + + record_size * num_ranges; + in =kzalloc (inlen,GFP_KERNEL ); +if (!in) +return -ENOMEM ; + + cmd_hdr =MLX5_ADDR_OF (create_page_track_obj_in, in, + general_obj_in_cmd_hdr); +MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, opcode, +MLX5_CMD_OP_CREATE_GENERAL_OBJECT ); +MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, obj_type, +MLX5_OBJ_TYPE_PAGE_TRACK ); + obj_context =MLX5_ADDR_OF (create_page_track_obj_in, in, obj_context); +MLX5_SET (page_track, obj_context, vhca_id, mvdev->vhca_id); +MLX5_SET (page_track, obj_context, track_type,1 ); +MLX5_SET (page_track, obj_context, log_page_size, +ilog2 (tracker->host_qp->tracked_page_size)); +MLX5_SET (page_track, obj_context, log_msg_size, +ilog2 (tracker->host_qp->max_msg_size)); +MLX5_SET (page_track, obj_context, reporting_qpn, tracker->fw_qp->qpn); +MLX5_SET (page_track, obj_context, num_ranges, num_ranges); + + range_list_ptr =MLX5_ADDR_OF (page_track, obj_context, track_range); + node =interval_tree_iter_first (ranges,0 ,ULONG_MAX ); +for (i =0 ; i < num_ranges; i++) { +void *addr_range_i_base = range_list_ptr + record_size * i; +unsigned long length = node->last - node->start; + +MLX5_SET64 (page_track_range, addr_range_i_base, start_address, + node->start); +MLX5_SET64 (page_track_range, addr_range_i_base, length, length); + total_ranges_len += length; + node =interval_tree_iter_next (node,0 ,ULONG_MAX ); + } + +WARN_ON (node); + log_addr_space_size =ilog2 (total_ranges_len); +if (log_addr_space_size < + (MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_log_min_addr_space)) || + log_addr_space_size > + (MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_log_max_addr_space))) { + err = -EOPNOTSUPP ; + goto out; + } + +MLX5_SET (page_track, obj_context, log_addr_space_size, + log_addr_space_size); + err =mlx5_cmd_exec (mdev, in, inlen, out,sizeof (out)); +if (err) + goto out; + + tracker->id =MLX5_GET (general_obj_out_cmd_hdr, out, obj_id); +out: +kfree (in); +return err; +} + +static int mlx5vf_cmd_destroy_tracker (struct mlx5_core_dev *mdev, +u32 tracker_id) +{ +u32 in[MLX5_ST_SZ_DW (general_obj_in_cmd_hdr)] = {}; +u32 out[MLX5_ST_SZ_DW (general_obj_out_cmd_hdr)] = {}; + +MLX5_SET (general_obj_in_cmd_hdr, in, opcode,MLX5_CMD_OP_DESTROY_GENERAL_OBJECT ); +MLX5_SET (general_obj_in_cmd_hdr, in, obj_type,MLX5_OBJ_TYPE_PAGE_TRACK ); +MLX5_SET (general_obj_in_cmd_hdr, in, obj_id, tracker_id); + +return mlx5_cmd_exec (mdev, in,sizeof (in), out,sizeof (out)); +} +static int alloc_cq_frag_buf (struct mlx5_core_dev *mdev,struct mlx5_vhca_cq_buf *buf,int nent,int cqe_size)
@@ -833,6 +975,7 @@ _mlx5vf_free_page_tracker_resources(struct mlx5vf_pci_core_device *mvdev)WARN_ON (mvdev->mdev_detach); +mlx5vf_cmd_destroy_tracker (mdev, tracker->id);mlx5vf_destroy_qp (mdev, tracker->fw_qp);mlx5vf_free_qp_recv_resources (mdev, tracker->host_qp);mlx5vf_destroy_qp (mdev, tracker->host_qp);
@@ -941,6 +1084,10 @@ int mlx5vf_start_page_tracker(struct vfio_device *vdev, tracker->host_qp = host_qp; tracker->fw_qp = fw_qp; + err =mlx5vf_create_tracker (mdev, mvdev, ranges, nnodes); +if (err) + goto err_activate; + *page_size = host_qp->tracked_page_size; mvdev->log_active = true;mlx5vf_state_mutex_unlock (mvdev);
drivers/vfio/pci/mlx5/cmd.h +1 -0
@@ -80,6 +80,7 @@ struct mlx5_vhca_qp { };struct mlx5_vhca_page_tracker { +u32 id;u32 pdn;struct mlx5_uars_page *uar;struct mlx5_vhca_cq cq;
Signature · 1 line
2.18.1
Re: [PATCH V7 vfio 07/10] vfio/mlx5: Create and destroy page tracker object
Hello,
Quoted 127 lines · Yishai Hadas
Add support for creating and destroying page tracker object.
This object is used to control/report the device dirty pages.
As part of creating the tracker need to consider the device capabilities for max ranges and adapt/combine ranges accordingly.
Signed-off-by Yishai Hadas <[email protected]>--- drivers/vfio/pci/mlx5/cmd.c | 147 ++++++++++++++++++++++++++++++++++++ drivers/vfio/pci/mlx5/cmd.h | 1 + 2 files changed, 148 insertions(+)drivers/vfio/pci/mlx5/cmd.c +106
-0@@ -410,6 +410,148 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,return err; } +static void combine_ranges (struct rb_root_cached *root,u32 cur_nodes, +u32 req_nodes) +{ +struct interval_tree_node *prev, *curr, *comb_start, *comb_end; +unsigned long min_gap; +unsigned long curr_gap; + +/* Special shortcut when a single range is required */ +if (req_nodes ==1 ) { +unsigned long last; + + curr = comb_start =interval_tree_iter_first (root,0 ,ULONG_MAX ); +while (curr) { + last = curr->last; + prev = curr; + curr =interval_tree_iter_next (curr,0 ,ULONG_MAX ); +if (prev != comb_start) +interval_tree_remove (prev, root); + } + comb_start->last = last; +return ; + } + +/* Combine ranges which have the smallest gap */ +while (cur_nodes > req_nodes) { + prev =NULL ; + min_gap =ULONG_MAX ; + curr =interval_tree_iter_first (root,0 ,ULONG_MAX ); +while (curr) { +if (prev) { + curr_gap = curr->start - prev->last; +if (curr_gap < min_gap) { + min_gap = curr_gap; + comb_start = prev; + comb_end = curr; + } + } + prev = curr; + curr =interval_tree_iter_next (curr,0 ,ULONG_MAX ); + } + comb_start->last = comb_end->last; +interval_tree_remove (comb_end, root); + cur_nodes--; + } +} + +static int mlx5vf_create_tracker (struct mlx5_core_dev *mdev, +struct mlx5vf_pci_core_device *mvdev, +struct rb_root_cached *ranges,u32 nnodes) +{ +int max_num_range = +MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_max_num_range); +struct mlx5_vhca_page_tracker *tracker = &mvdev->tracker; +int record_size =MLX5_ST_SZ_BYTES (page_track_range); +u32 out[MLX5_ST_SZ_DW (general_obj_out_cmd_hdr)] = {}; +struct interval_tree_node *node =NULL ; +u64 total_ranges_len =0 ; +u32 num_ranges = nnodes; +u8 log_addr_space_size; +void *range_list_ptr; +void *obj_context; +void *cmd_hdr; +int inlen; +void *in; +int err; +int i; + +if (num_ranges > max_num_range) { +combine_ranges (ranges, nnodes, max_num_range); + num_ranges = max_num_range; + } + + inlen =MLX5_ST_SZ_BYTES (create_page_track_obj_in) + + record_size * num_ranges; + in =kzalloc (inlen,GFP_KERNEL ); +if (!in) +return -ENOMEM ; + + cmd_hdr =MLX5_ADDR_OF (create_page_track_obj_in, in, + general_obj_in_cmd_hdr); +MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, opcode, +MLX5_CMD_OP_CREATE_GENERAL_OBJECT ); +MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, obj_type, +MLX5_OBJ_TYPE_PAGE_TRACK ); + obj_context =MLX5_ADDR_OF (create_page_track_obj_in, in, obj_context); +MLX5_SET (page_track, obj_context, vhca_id, mvdev->vhca_id); +MLX5_SET (page_track, obj_context, track_type,1 ); +MLX5_SET (page_track, obj_context, log_page_size, +ilog2 (tracker->host_qp->tracked_page_size)); +MLX5_SET (page_track, obj_context, log_msg_size, +ilog2 (tracker->host_qp->max_msg_size)); +MLX5_SET (page_track, obj_context, reporting_qpn, tracker->fw_qp->qpn); +MLX5_SET (page_track, obj_context, num_ranges, num_ranges); + + range_list_ptr =MLX5_ADDR_OF (page_track, obj_context, track_range); + node =interval_tree_iter_first (ranges,0 ,ULONG_MAX ); +for (i =0 ; i < num_ranges; i++) { +void *addr_range_i_base = range_list_ptr + record_size * i; +unsigned long length = node->last - node->start; + +MLX5_SET64 (page_track_range, addr_range_i_base, start_address, + node->start); +MLX5_SET64 (page_track_range, addr_range_i_base, length, length); + total_ranges_len += length; + node =interval_tree_iter_next (node,0 ,ULONG_MAX ); + }
drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 528
Quoted 10 lines
drivers/vfio/pci/mlx5/cmd.c +10
-0+ +WARN_ON (node); + log_addr_space_size =ilog2 (total_ranges_len); +if (log_addr_space_size < + (MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_log_min_addr_space)) || + log_addr_space_size > + (MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_log_max_addr_space))) { + err = -EOPNOTSUPP ; + goto out; + }
We are seeing an issue with dirty page tracking when doing migration of an OVMF VM guest. The vfio-pci variant driver for the MLX5 VF device complains when dirty page tracking is initialized from QEMU :
qemu-kvm: 0000:b1:00.2: Failed to start DMA logging, err -95 (Operation not supported)
The 64-bit computed range is :
vfio_device_dirty_tracking_start nr_ranges 2 32:[0x0 - 0x807fffff], 64:[0x100000000 - 0x3838000fffff]
which seems to be too large for the HW. AFAICT, the MLX5 HW has a 42 bits address space limitation for dirty tracking (min is 12). Is it a FW tunable or a strict limitation ?
We should probably introduce more ranges to overcome the issue.
Thanks,
C.
Quoted 59 lines
drivers/vfio/pci/mlx5/cmd.c +30
-0+MLX5_SET (page_track, obj_context, log_addr_space_size, + log_addr_space_size); + err =mlx5_cmd_exec (mdev, in, inlen, out,sizeof (out)); +if (err) + goto out; + + tracker->id =MLX5_GET (general_obj_out_cmd_hdr, out, obj_id); +out: +kfree (in); +return err; +} + +static int mlx5vf_cmd_destroy_tracker (struct mlx5_core_dev *mdev, +u32 tracker_id) +{ +u32 in[MLX5_ST_SZ_DW (general_obj_in_cmd_hdr)] = {}; +u32 out[MLX5_ST_SZ_DW (general_obj_out_cmd_hdr)] = {}; + +MLX5_SET (general_obj_in_cmd_hdr, in, opcode,MLX5_CMD_OP_DESTROY_GENERAL_OBJECT ); +MLX5_SET (general_obj_in_cmd_hdr, in, obj_type,MLX5_OBJ_TYPE_PAGE_TRACK ); +MLX5_SET (general_obj_in_cmd_hdr, in, obj_id, tracker_id); + +return mlx5_cmd_exec (mdev, in,sizeof (in), out,sizeof (out)); +} +static int alloc_cq_frag_buf (struct mlx5_core_dev *mdev,struct mlx5_vhca_cq_buf *buf,int nent,int cqe_size)@@ -833,6 +975,7 @@ _mlx5vf_free_page_tracker_resources(struct mlx5vf_pci_core_device *mvdev)WARN_ON (mvdev->mdev_detach); +mlx5vf_cmd_destroy_tracker (mdev, tracker->id);mlx5vf_destroy_qp (mdev, tracker->fw_qp);mlx5vf_free_qp_recv_resources (mdev, tracker->host_qp);mlx5vf_destroy_qp (mdev, tracker->host_qp);@@ -941,6 +1084,10 @@ int mlx5vf_start_page_tracker(struct vfio_device *vdev, tracker->host_qp = host_qp; tracker->fw_qp = fw_qp; + err =mlx5vf_create_tracker (mdev, mvdev, ranges, nnodes); +if (err) + goto err_activate; + *page_size = host_qp->tracked_page_size; mvdev->log_active = true;mlx5vf_state_mutex_unlock (mvdev);drivers/vfio/pci/mlx5/cmd.h +1
-0@@ -80,6 +80,7 @@ struct mlx5_vhca_qp { };struct mlx5_vhca_page_tracker { +u32 id;u32 pdn;struct mlx5_uars_page *uar;struct mlx5_vhca_cq cq;
Yishai Hadas
maintainerRe: [PATCH V7 vfio 07/10] vfio/mlx5: Create and destroy page tracker object
Quoted 155 lines · Cédric Le Goater
Hello,
Quoted 136 lines · Yishai Hadas
Add support for creating and destroying page tracker object.
This object is used to control/report the device dirty pages.
As part of creating the tracker need to consider the device capabilities for max ranges and adapt/combine ranges accordingly.
Signed-off-by Yishai Hadas <[email protected]>---
drivers/vfio/pci/mlx5/cmd.c| 147 ++++++++++++++++++++++++++++++++++++drivers/vfio/pci/mlx5/cmd.h| 1 +2 files changed, 148 insertions(+)drivers/vfio/pci/mlx5/cmd.c +91
-0@@ -410,6 +410,148 @@ int mlx5vf_cmd_load_vhca_state(structmlx5vf_pci_core_device *mvdev, return err; } +static void combine_ranges(struct rb_root_cached *root, u32 cur_nodes, + u32 req_nodes) +{ + struct interval_tree_node *prev, *curr, *comb_start, *comb_end; + unsigned long min_gap; + unsigned long curr_gap; + + /* Special shortcut when a single range is required */ + if (req_nodes == 1) { + unsigned long last; + + curr = comb_start = interval_tree_iter_first(root, 0,ULONG_MAX ); + while (curr) { + last = curr->last; + prev = curr; + curr = interval_tree_iter_next(curr, 0, ULONG_MAX); + if (prev != comb_start) + interval_tree_remove(prev, root); + } + comb_start->last = last; + return; + } + + /* Combine ranges which have the smallest gap */ + while (cur_nodes > req_nodes) { + prev = NULL; + min_gap = ULONG_MAX; + curr = interval_tree_iter_first(root, 0, ULONG_MAX); + while (curr) { + if (prev) { + curr_gap = curr->start - prev->last; + if (curr_gap < min_gap) { + min_gap = curr_gap; + comb_start = prev; + comb_end = curr; + } + } + prev = curr; + curr = interval_tree_iter_next(curr, 0, ULONG_MAX); + } + comb_start->last = comb_end->last; + interval_tree_remove(comb_end, root); + cur_nodes--; + } +} + +static int mlx5vf_create_tracker (struct mlx5_core_dev *mdev, + struct mlx5vf_pci_core_device *mvdev, + struct rb_root_cached *ranges, u32 nnodes) +{ + int max_num_range = + MLX5_CAP_ADV_VIRTUALIZATION(mdev, pg_track_max_num_range); + struct mlx5_vhca_page_tracker *tracker = &mvdev->tracker; + int record_size = MLX5_ST_SZ_BYTES(page_track_range); + u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {}; + struct interval_tree_node *node = NULL; + u64 total_ranges_len = 0; + u32 num_ranges = nnodes; + u8 log_addr_space_size; + void *range_list_ptr; + void *obj_context; + void *cmd_hdr; + int inlen; + void *in; + int err; + int i; + + if (num_ranges > max_num_range) { + combine_ranges(ranges, nnodes, max_num_range); + num_ranges = max_num_range; + } + + inlen = MLX5_ST_SZ_BYTES(create_page_track_obj_in) + + record_size * num_ranges; + in = kzalloc(inlen, GFP_KERNEL); + if (!in) + return -ENOMEM; + + cmd_hdr = MLX5_ADDR_OF(create_page_track_obj_in, in, + general_obj_in_cmd_hdr); + MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, opcode, + MLX5_CMD_OP_CREATE_GENERAL_OBJECT); + MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, obj_type, + MLX5_OBJ_TYPE_PAGE_TRACK); + obj_context = MLX5_ADDR_OF(create_page_track_obj_in, in, obj_context); + MLX5_SET(page_track, obj_context, vhca_id, mvdev->vhca_id); + MLX5_SET(page_track, obj_context, track_type, 1); + MLX5_SET(page_track, obj_context, log_page_size, + ilog2(tracker->host_qp->tracked_page_size)); + MLX5_SET(page_track, obj_context, log_msg_size, + ilog2(tracker->host_qp->max_msg_size)); + MLX5_SET(page_track, obj_context, reporting_qpn,
tracker->fw_qp->qpn);
+MLX5_SET(page_track, obj_context, num_ranges, num_ranges); +
+range_list_ptr=MLX5_ADDR_OF(page_track, obj_context, track_range);
+ node =interval_tree_iter_first(ranges, 0, ULONG_MAX); + for (i = 0; i <num_ranges; i++) {
+ void*addr_range_i_base=range_list_ptr+record_size* i; + unsigned long length =node->last-node->start;
+
+MLX5_SET64(page_track_range, addr_range_i_base, start_address, + node->start);
+MLX5_SET64(page_track_range, addr_range_i_base, length, length);
+total_ranges_len+= length;
+ node =interval_tree_iter_next(node, 0, ULONG_MAX); + }
+
+WARN_ON(node);
+log_addr_space_size=ilog2(total_ranges_len);
+ if (log_addr_space_size<Quoted 8 lines
+ (MLX5_CAP_ADV_VIRTUALIZATION(mdev,
pg_track_log_min_addr_space)) ||
+log_addr_space_size>
+ (MLX5_CAP_ADV_VIRTUALIZATION(mdev,pg_track_log_max_addr_space))) {
+ err =-EOPNOTSUPP;
+ goto out;
+ }We are seeing an issue with dirty page tracking when doing migration of an OVMF VM guest. The vfio-pci variant driver for the MLX5 VF device complains when dirty page tracking is initialized from QEMU :
qemu-kvm: 0000:b1:00.2: Failed to start DMA logging, err -95 (Operation not supported)
Quoted 7 lines
The 64-bit computed range is :
vfio_device_dirty_tracking_startnr_ranges2 32:[0x0 -0x807fffff], 64:[0x100000000 -0x3838000fffff]which seems to be too large for the HW. AFAICT, the MLX5 HW has a 42 bits address space limitation for dirty tracking (min is 12). Is it a FW tunable or a strict limitation ?
It's mainly a FW limitation.
Tracking larger address space than 2^42 might take a lot of time in FW to allocate the required resources which might end-up in command timeout, etc.
Quoted 2 lines
We should probably introduce more ranges to overcome the issue.
More ranges can help only if the total address space of the given ranges is < 2^42.
So, if there are some areas that don't require tracking (why?), breaking into more ranges with smaller total size can help.
Yishai
Jason Gunthorpe
reviewerRe: [PATCH V7 vfio 07/10] vfio/mlx5: Create and destroy page tracker object
In reply to Cédric Le GoaterQuoted 17 lines · Cédric Le Goater
drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 528
Quoted 9 lines
drivers/vfio/pci/mlx5/cmd.c +9
-0+WARN_ON (node); + log_addr_space_size =ilog2 (total_ranges_len); +if (log_addr_space_size < + (MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_log_min_addr_space)) || + log_addr_space_size > + (MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_log_max_addr_space))) { + err = -EOPNOTSUPP ; + goto out; + }We are seeing an issue with dirty page tracking when doing migration of an OVMF VM guest. The vfio-pci variant driver for the MLX5 VF device complains when dirty page tracking is initialized from QEMU :
qemu-kvm: 0000:b1:00.2: Failed to start DMA logging, err -95 (Operation not supported)
Quoted 7 lines
The 64-bit computed range is :
vfio_device_dirty_tracking_startnr_ranges2 32:[0x0 -0x807fffff], 64:[0x100000000 -0x3838000fffff]which seems to be too large for the HW. AFAICT, the MLX5 HW has a 42 bits address space limitation for dirty tracking (min is 12). Is it a FW tunable or a strict limitation ?
It would be good to explain where this is coming from, all devices need to make some decision on what address space ranges to track and I would say 2^42 is already pretty generous limit..
Can we go the other direction and reduce the ranges qemu is interested in?
Jason
Re: [PATCH V7 vfio 07/10] vfio/mlx5: Create and destroy page tracker object
Quoted 27 lines · Jason Gunthorpe
Quoted 17 lines · Cédric Le Goater
drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 528
Quoted 9 lines
drivers/vfio/pci/mlx5/cmd.c +9
-0+WARN_ON (node); + log_addr_space_size =ilog2 (total_ranges_len); +if (log_addr_space_size < + (MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_log_min_addr_space)) || + log_addr_space_size > + (MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_log_max_addr_space))) { + err = -EOPNOTSUPP ; + goto out; + }We are seeing an issue with dirty page tracking when doing migration of an OVMF VM guest. The vfio-pci variant driver for the MLX5 VF device complains when dirty page tracking is initialized from QEMU :
qemu-kvm: 0000:b1:00.2: Failed to start DMA logging, err -95 (Operation not supported)
Quoted 7 lines
The 64-bit computed range is :
vfio_device_dirty_tracking_startnr_ranges2 32:[0x0 -0x807fffff], 64:[0x100000000 -0x3838000fffff]which seems to be too large for the HW. AFAICT, the MLX5 HW has a 42 bits address space limitation for dirty tracking (min is 12). Is it a FW tunable or a strict limitation ?
Quoted 6 lines
It would be good to explain where this is coming from, all devices need to make some decision on what address space ranges to track and I would say 2^42 is already pretty generous limit..
Can we go the other direction and reduce the ranges qemu is interested in?
There's also a chance that this are those 16x-32x socket Intel machines with 48T-64T of memory (judging from the ranges alone). Meaning that these ranges even if reduced wouldn't remove much of the aggregate address space width.
Re: [PATCH V7 vfio 07/10] vfio/mlx5: Create and destroy page tracker object
In reply to Jason GunthorpeQuoted 23 lines · Jason Gunthorpe
Quoted 21 lines · Cédric Le Goater
drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 528
Quoted 9 lines
drivers/vfio/pci/mlx5/cmd.c +9
-0+WARN_ON (node); + log_addr_space_size =ilog2 (total_ranges_len); +if (log_addr_space_size < + (MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_log_min_addr_space)) || + log_addr_space_size > + (MLX5_CAP_ADV_VIRTUALIZATION (mdev, pg_track_log_max_addr_space))) { + err = -EOPNOTSUPP ; + goto out; + }We are seeing an issue with dirty page tracking when doing migration of an OVMF VM guest. The vfio-pci variant driver for the MLX5 VF device complains when dirty page tracking is initialized from QEMU :
qemu-kvm: 0000:b1:00.2: Failed to start DMA logging, err -95 (Operation not supported)
The 64-bit computed range is :
vfio_device_dirty_tracking_start nr_ranges 2 32:[0x0 - 0x807fffff], 64:[0x100000000 - 0x3838000fffff]
Quoted 7 lines
Quoted 3 lines
which seems to be too large for the HW. AFAICT, the MLX5 HW has a 42 bits address space limitation for dirty tracking (min is 12). Is it a FW tunable or a strict limitation ?
It would be good to explain where this is coming from, all devices need to make some decision on what address space ranges to track and I would say 2^42 is already pretty generous limit..
QEMU computes the DMA logging ranges for two predefined ranges: 32-bit and 64-bit. In the OVMF case, QEMU includes in the 64-bit range, RAM (at the lower part) and device RAM regions (at the top of the address space). The size of that range can be bigger than the 2^42 limit of the MLX5 HW for dirty tracking. QEMU is not making much effort to be smart. There is room for improvement.
Quoted 2 lines
Can we go the other direction and reduce the ranges qemu is interested in?
We can not exclude the device RAM regions since we don't know what the HW could do with them. Correct me here. But we can certainly avoid the huge gap in the 64-bit range by adding support for more than 2 ranges in QEMU.
Then, if the overall size exceeds what HW supports, the vfio-pci variant driver will report an error, as of today.
Thanks,
C.
Re: [PATCH V7 vfio 07/10] vfio/mlx5: Create and destroy page tracker object
Quoted 35 lines · Cédric Le Goater
Quoted 25 lines · Jason Gunthorpe
Quoted 23 lines · Cédric Le Goater
Quoted 9 lines
+
WARN_ON(node);
+log_addr_space_size=ilog2(total_ranges_len);
+ if (log_addr_space_size<
+ (MLX5_CAP_ADV_VIRTUALIZATION(mdev, pg_track_log_min_addr_space)) || +log_addr_space_size>
+ (MLX5_CAP_ADV_VIRTUALIZATION(mdev, pg_track_log_max_addr_space))) { + err =-EOPNOTSUPP;
+ goto out;
+ }We are seeing an issue with dirty page tracking when doing migration of an OVMF VM guest. The vfio-pci variant driver for the MLX5 VF device complains when dirty page tracking is initialized from QEMU :
qemu-kvm: 0000:b1:00.2: Failed to start DMA logging, err -95 (Operation not supported)
The 64-bit computed range is :
vfio_device_dirty_tracking_startnr_ranges2 32:[0x0 -0x807fffff], 64:[0x100000000 -0x3838000fffff]Quoted 7 lines
Quoted 3 lines
which seems to be too large for the HW. AFAICT, the MLX5 HW has a 42 bits address space limitation for dirty tracking (min is 12). Is it a FW tunable or a strict limitation ?
It would be good to explain where this is coming from, all devices need to make some decision on what address space ranges to track and I would say 2^42 is already pretty generous limit..
Quoted 7 lines
QEMU computes the DMA logging ranges for two predefined ranges: 32-bit and 64-bit. In the OVMF case, QEMU includes in the 64-bit range, RAM (at the lower part) and device RAM regions (at the top of the address space). The size of that range can be bigger than the 2^42 limit of the MLX5 HW for dirty tracking. QEMU is not making much effort to be smart. There is room for improvement.
Interesting, we haven't reproduced this in our testing with OVMF multi-TB configs with these VFs. Could you share the OVMF base version you were using? or maybe we didn't triggered it considering the total device RAM regions would be small enough to fit the 32G PCI hole64 that is advertised that avoids a hypothetical relocation.
We could use do more than 2 ranges (or going back to sharing all ranges), or add a set of ranges that represents the device RAM without computing a min/max there (not sure we can figure that out from within the memory listener does all this logic); it would perhaps a bit too BIOS specific if we start looking at specific parts of the address space (e.g. phys-bits-1) to compute these ranges.
Joao
Re: [PATCH V7 vfio 07/10] vfio/mlx5: Create and destroy page tracker object
Quoted 44 lines · Joao Martins
Quoted 35 lines · Cédric Le Goater
Quoted 25 lines · Jason Gunthorpe
Quoted 23 lines · Cédric Le Goater
Quoted 9 lines
+
WARN_ON(node);
+log_addr_space_size=ilog2(total_ranges_len);
+ if (log_addr_space_size<
+ (MLX5_CAP_ADV_VIRTUALIZATION(mdev, pg_track_log_min_addr_space)) || +log_addr_space_size>
+ (MLX5_CAP_ADV_VIRTUALIZATION(mdev, pg_track_log_max_addr_space))) { + err =-EOPNOTSUPP;
+ goto out;
+ }We are seeing an issue with dirty page tracking when doing migration of an OVMF VM guest. The vfio-pci variant driver for the MLX5 VF device complains when dirty page tracking is initialized from QEMU :
qemu-kvm: 0000:b1:00.2: Failed to start DMA logging, err -95 (Operation
not supported)The 64-bit computed range is :
vfio_device_dirty_tracking_startnr_ranges2 32:[0x0 -0x807fffff],
64:[0x100000000 -0x3838000fffff]Quoted 7 lines
Quoted 3 lines
which seems to be too large for the HW. AFAICT, the MLX5 HW has a 42 bits address space limitation for dirty tracking (min is 12). Is it a FW tunable or a strict limitation ?
It would be good to explain where this is coming from, all devices need to make some decision on what address space ranges to track and I would say 2^42 is already pretty generous limit..
Quoted 7 lines
QEMU computes the DMA logging ranges for two predefined ranges: 32-bit and 64-bit. In the OVMF case, QEMU includes in the 64-bit range, RAM (at the lower part) and device RAM regions (at the top of the address space). The size of that range can be bigger than the 2^42 limit of the MLX5 HW for dirty tracking. QEMU is not making much effort to be smart. There is room for improvement.
Quoted 2 lines
Interesting, we haven't reproduced this in our testing with OVMF multi-TB configs with these VFs. Could you share the OVMF base version you were using?
edk2-ovmf-20230524-3.el9.noarch
host is a :
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHzQuoted 4 lines
or
maybe we didn't triggered it considering the total device RAM regions would be small enough to fit the 32G PCI hole64 that is advertised that avoids a hypothetical relocation.
You need RAM above 4G in the guest :
100000000-27fffffff : System RAM
237800000-2387fffff : Kernel code
238800000-23932cfff : Kernel rodata
239400000-239977cff : Kernel data
23a202000-23b3fffff : Kernel bss
380000000000-3807ffffffff : PCI Bus 0000:00
380000000000-3800000fffff : 0000:00:03.0
380000000000-3800000fffff : mlx5_coreActivating the QEMU trace events shows quickly the issue :
vfio_device_dirty_tracking_update section 0x0 - 0x9ffff -> update [0x0 - 0x9ffff]
vfio_device_dirty_tracking_update section 0xa0000 - 0xaffff -> update [0x0 - 0xaffff]
vfio_device_dirty_tracking_update section 0xc0000 - 0xc3fff -> update [0x0 - 0xc3fff]
vfio_device_dirty_tracking_update section 0xc4000 - 0xdffff -> update [0x0 - 0xdffff]
vfio_device_dirty_tracking_update section 0xe0000 - 0xfffff -> update [0x0 - 0xfffff]
vfio_device_dirty_tracking_update section 0x100000 - 0x7fffffff -> update [0x0 - 0x7fffffff]
vfio_device_dirty_tracking_update section 0x80000000 - 0x807fffff -> update [0x0 - 0x807fffff]
vfio_device_dirty_tracking_update section 0x100000000 - 0x27fffffff -> update [0x100000000 - 0x27fffffff]
vfio_device_dirty_tracking_update section 0x383800000000 - 0x383800001fff -> update [0x100000000 - 0x383800001fff]
vfio_device_dirty_tracking_update section 0x383800003000 - 0x3838000fffff -> update [0x100000000 - 0x3838000fffff]So that's nice. And with less RAM in the VM, 2G, migration should work though.
Quoted 4 lines
We could use do more than 2 ranges (or going back to sharing all ranges), or add a set of ranges that represents the device RAM without computing a min/max there (not sure we can figure that out from within the memory listener does all this logic);
The listener is container based. May we could add one range per device if we can identify a different owner per memory section.
C.
Quoted 4 lines
it would perhaps a bit too BIOS specific if we start looking at specific parts of the address space (e.g. phys-bits-1) to compute these ranges.
Joao
Re: [PATCH V7 vfio 07/10] vfio/mlx5: Create and destroy page tracker object
Quoted 66 lines · Cédric Le Goater
Quoted 44 lines · Joao Martins
Quoted 35 lines · Cédric Le Goater
Quoted 25 lines · Jason Gunthorpe
Quoted 23 lines · Cédric Le Goater
Quoted 9 lines
+
WARN_ON(node);
+log_addr_space_size=ilog2(total_ranges_len);
+ if (log_addr_space_size<
+ (MLX5_CAP_ADV_VIRTUALIZATION(mdev, pg_track_log_min_addr_space)) || +log_addr_space_size>
+ (MLX5_CAP_ADV_VIRTUALIZATION(mdev, pg_track_log_max_addr_space))) { + err =-EOPNOTSUPP;
+ goto out;
+ }We are seeing an issue with dirty page tracking when doing migration of an OVMF VM guest. The vfio-pci variant driver for the MLX5 VF device complains when dirty page tracking is initialized from QEMU :
qemu-kvm: 0000:b1:00.2: Failed to start DMA logging, err -95 (Operation not supported)
The 64-bit computed range is :
vfio_device_dirty_tracking_startnr_ranges2 32:[0x0 -0x807fffff], 64:[0x100000000 -0x3838000fffff]Quoted 7 lines
Quoted 3 lines
which seems to be too large for the HW. AFAICT, the MLX5 HW has a 42 bits address space limitation for dirty tracking (min is 12). Is it a FW tunable or a strict limitation ?
It would be good to explain where this is coming from, all devices need to make some decision on what address space ranges to track and I would say 2^42 is already pretty generous limit..
Quoted 7 lines
QEMU computes the DMA logging ranges for two predefined ranges: 32-bit and 64-bit. In the OVMF case, QEMU includes in the 64-bit range, RAM (at the lower part) and device RAM regions (at the top of the address space). The size of that range can be bigger than the 2^42 limit of the MLX5 HW for dirty tracking. QEMU is not making much effort to be smart. There is room for improvement.
Quoted 2 lines
Interesting, we haven't reproduced this in our testing with OVMF multi-TB configs with these VFs. Could you share the OVMF base version you were using?
edk2-ovmf-20230524-3.el9.noarch
host is a :
Architecture:x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHzQuoted 4 lines
or
maybe we didn't triggered it considering the total device RAM regions would be small enough to fit the 32G PCI hole64 that is advertised that avoids a hypothetical relocation.
Quoted 9 lines
You need RAM above 4G in the guest :
100000000-27fffffff : System RAM
237800000-2387fffff : Kernel code
238800000-23932cfff : Kernel rodata
239400000-239977cff : Kernel data
23a202000-23b3fffff : Kernel bss
380000000000-3807ffffffff : PCI Bus 0000:00
380000000000-3800000fffff : 0000:00:03.0
380000000000-3800000fffff :mlx5_core
Similar machine to yours, but in my 32G guests with older OVMF it's putting the PCI area after max-ram:
vfio_device_dirty_tracking_update section 0x0 - 0x9ffff -> update [0x0 - 0x9ffff] vfio_device_dirty_tracking_update section 0xc0000 - 0xcafff -> update [0x0 - 0xcafff]vfio_device_dirty_tracking_update section 0xcb000 - 0xcdfff -> update [0x0 - 0xcdfff]vfio_device_dirty_tracking_update section 0xce000 - 0xe7fff -> update [0x0 - 0xe7fff]vfio_device_dirty_tracking_update section 0xe8000 - 0xeffff -> update [0x0 - 0xeffff]vfio_device_dirty_tracking_update section 0xf0000 - 0xfffff -> update [0x0 - 0xfffff]vfio_device_dirty_tracking_update section 0x100000 - 0x7fffffff -> update [0x0 - 0x7fffffff]vfio_device_dirty_tracking_update section 0xfd000000 - 0xfdffffff -> update [0x0
- 0xfdffffff]vfio_device_dirty_tracking_update section 0xfffc0000 - 0xffffffff -> update [0x0
- 0xffffffff]vfio_device_dirty_tracking_update section 0x100000000 - 0x87fffffff -> update [0x100000000 - 0x87fffffff]vfio_device_dirty_tracking_update section 0x880000000 - 0x880001fff -> update [0x100000000 - 0x880001fff]vfio_device_dirty_tracking_update section 0x880003000 - 0x8ffffffff -> update [0x100000000 - 0x8ffffffff]
Quoted 26 lines
Activating the QEMU trace events shows quickly the issue :
vfio_device_dirty_tracking_updatesection0x0-0x9ffff-> update [0x0-0x9ffff]
vfio_device_dirty_tracking_updatesection0xa0000-0xaffff-> update [0x0-0xaffff]
vfio_device_dirty_tracking_updatesection0xc0000-0xc3fff-> update [0x0-0xc3fff]
vfio_device_dirty_tracking_updatesection0xc4000-0xdffff-> update [0x0-0xdffff]
vfio_device_dirty_tracking_updatesection0xe0000-0xfffff-> update [0x0-0xfffff]
vfio_device_dirty_tracking_updatesection0x100000-0x7fffffff-> update [0x0-0x7fffffff]
vfio_device_dirty_tracking_updatesection0x80000000-0x807fffff-> update [0x0-0x807fffff]
vfio_device_dirty_tracking_updatesection0x100000000-0x27fffffff-> update [0x100000000-0x27fffffff]vfio_device_dirty_tracking_updatesection0x383800000000-0x383800001fff-> update [0x100000000-0x383800001fff]vfio_device_dirty_tracking_updatesection0x383800003000-0x3838000fffff-> update [0x100000000-0x3838000fffff]So that's nice. And with less RAM in the VM, 2G, migration should work though.
Quoted 8 lines
Quoted 4 lines
We could use do more than 2 ranges (or going back to sharing all ranges), or add a set of ranges that represents the device RAM without computing a min/max there (not sure we can figure that out from within the memory listener does all this logic);
The listener is container based. May we could add one range per device if we can identify a different owner per memory section.
For brainstorm purposes ... Maybe something like this below. Should make your case work. As mentioned earlier in my case it's placed always at maxram+1, so makes no difference in having the "pci" range
------>8-------
From: Joao Martins <[email protected]>
Date: Thu, 7 Sep 2023 09:23:38 -0700
Subject: [PATCH] vfio/common: Separate vfio-pci ranges
--- hw/vfio/common.c | 48 ++++++++++++++++++++++++++++++++++++++++---- hw/vfio/trace-events | 2 +- 2 files changed, 45 insertions(+), 5 deletions(-)
hw/vfio/common.c +28 -0
@@ -27,6 +27,7 @@#include "hw/vfio/vfio-common.h" #include "hw/vfio/vfio.h" +#include "hw/vfio/pci.h" #include "exec/address-spaces.h" #include "exec/memory.h" #include "exec/ram_addr.h"
@@ -1424,6 +1425,8 @@ typedef struct VFIODirtyRanges {hwaddr max32;hwaddr min64;hwaddr max64; +hwaddr minpci; +hwaddr maxpci; } VFIODirtyRanges; typedef struct VFIODirtyRangesListener {
@@ -1432,6 +1435,31 @@ typedef struct VFIODirtyRangesListener {MemoryListener listener; } VFIODirtyRangesListener; +static bool vfio_section_is_vfio_pci (MemoryRegionSection *section, +VFIOContainer *container) +{ +VFIOPCIDevice *pcidev; +VFIODevice *vbasedev; +VFIOGroup *group; +Object *owner; + + owner =memory_region_owner (section->mr); + + QLIST_FOREACH(group, &container->group_list, container_next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { +if (vbasedev->type !=VFIO_DEVICE_TYPE_PCI ) { +continue ; + } + pcidev =container_of (vbasedev, VFIOPCIDevice, vbasedev); +if (OBJECT (pcidev) == owner) { +return true; + } + } + } + +return false; +} +static void vfio_dirty_tracking_update (MemoryListener *listener,MemoryRegionSection *section) {
@@ -1458,8 +1486,13 @@ static void vfio_dirty_tracking_update(MemoryListener
*listener,
* would be an IOVATree but that has a much bigger runtime overhead and
* unnecessary complexity.
*/
- min = (end <= UINT32_MAX) ? &range->min32 : &range->min64;
- max = (end <= UINT32_MAX) ? &range->max32 : &range->max64;
+ if (!vfio_section_is_vfio_pci(section, dirty->container)) {+ min = (end <= UINT32_MAX) ? &range->min32 : &range->min64;
+ max = (end <= UINT32_MAX) ? &range->max32 : &range->max64;
+ } else {
+ min = &range->minpci;
+ max = &range->maxpci;
+ }
if (*min > iova) {
*min = iova;
@@ -1485,6 +1518,7 @@ static void vfio_dirty_tracking_init(VFIOContainer *container,
memset(&dirty, 0, sizeof(dirty));
dirty.ranges.min32 = UINT32_MAX;
dirty.ranges.min64 = UINT64_MAX;
+ dirty.ranges.minpci = UINT64_MAX;
dirty.listener = vfio_dirty_tracking_listener;
dirty.container = container;@@ -1555,7 +1589,7 @@ vfio_device_feature_dma_logging_start_create(VFIOContainer *container,
* DMA logging uAPI guarantees to support at least a number of ranges that
* fits into a single host kernel base page.
*/
- control->num_ranges = !!tracking->max32 + !!tracking->max64;
+ control->num_ranges = !!tracking->max32 + !!tracking->max64 +
!!tracking->maxpci;
ranges = g_try_new0(struct vfio_device_feature_dma_logging_range,
control->num_ranges);
if (!ranges) {
@@ -1574,11 +1608,17 @@
vfio_device_feature_dma_logging_start_create(VFIOContainer *container,
if (tracking->max64) {
ranges->iova = tracking->min64;
ranges->length = (tracking->max64 - tracking->min64) + 1;+ ranges++;
+ }
+ if (tracking->maxpci) {
+ ranges->iova = tracking->minpci;
+ ranges->length = (tracking->maxpci - tracking->minpci) + 1;
}
trace_vfio_device_dirty_tracking_start(control->num_ranges,
tracking->min32, tracking->max32,
- tracking->min64, tracking->max64);
+ tracking->min64, tracking->max64,
+ tracking->minpci, tracking->maxpci);return feature;
}
hw/vfio/trace-events +0 -0
@@ -104,7 +104,7 @@ vfio_known_safe_misalignment(const char *name, uint64_t
iova, uint64_t offset_wi
vfio_listener_region_add_no_dma_map(const char *name, uint64_t iova, uint64_t
size, uint64_t page_size) "Region \"%s\" 0x%"PRIx64" size=0x%"PRIx64" is not aligned to 0x%"PRIx64" and cannot be mapped for DMA"
vfio_listener_region_del(uint64_t start, uint64_t end) "region_del 0x%"PRIx64"
- 0x%"PRIx64
vfio_device_dirty_tracking_update(uint64_t start, uint64_t end, uint64_t min,uint64_t max) "section 0x%"PRIx64" - 0x%"PRIx64" -> update [0x%"PRIx64" - 0x%"PRIx64"]"
-vfio_device_dirty_tracking_start(int nr_ranges, uint64_t min32, uint64_t max32, uint64_t min64, uint64_t max64) "nr_ranges %d 32:[0x%"PRIx64" - 0x%"PRIx64"], 64:[0x%"PRIx64" - 0x%"PRIx64"]" +vfio_device_dirty_tracking_start(int nr_ranges, uint64_t min32, uint64_t max32, uint64_t min64, uint64_t max64, uint64_t minpci, uint64_t maxpci) "nr_ranges %d 32:[0x%"PRIx64" - 0x%"PRIx64"], 64:[0x%"PRIx64" - 0x%"PRIx64"], pci:[0x%"PRIx64"
- 0x%"PRIx64"]"
vfio_disconnect_container(int fd) "close container->fd=%d" vfio_put_group(int fd) "close group->fd=%d"
vfio_get_device(const char * name, unsigned int flags, unsigned intnum_regions, unsigned int num_irqs) "Device %s flags: %u, regions: %u, irqs: %u" --
2.39.3
Re: [PATCH V7 vfio 07/10] vfio/mlx5: Create and destroy page tracker object
Quoted 95 lines · Joao Martins
Quoted 66 lines · Cédric Le Goater
Quoted 44 lines · Joao Martins
Quoted 35 lines · Cédric Le Goater
Quoted 25 lines · Jason Gunthorpe
Quoted 23 lines · Cédric Le Goater
Quoted 9 lines
+
WARN_ON(node);
+log_addr_space_size=ilog2(total_ranges_len);
+ if (log_addr_space_size<
+ (MLX5_CAP_ADV_VIRTUALIZATION(mdev, pg_track_log_min_addr_space)) || +log_addr_space_size>
+ (MLX5_CAP_ADV_VIRTUALIZATION(mdev, pg_track_log_max_addr_space))) { + err =-EOPNOTSUPP;
+ goto out;
+ }We are seeing an issue with dirty page tracking when doing migration of an OVMF VM guest. The vfio-pci variant driver for the MLX5 VF device complains when dirty page tracking is initialized from QEMU :
qemu-kvm: 0000:b1:00.2: Failed to start DMA logging, err -95 (Operation
not supported)The 64-bit computed range is :
vfio_device_dirty_tracking_startnr_ranges2 32:[0x0 -0x807fffff],
64:[0x100000000 -0x3838000fffff]Quoted 7 lines
Quoted 3 lines
which seems to be too large for the HW. AFAICT, the MLX5 HW has a 42 bits address space limitation for dirty tracking (min is 12). Is it a FW tunable or a strict limitation ?
It would be good to explain where this is coming from, all devices need to make some decision on what address space ranges to track and I would say 2^42 is already pretty generous limit..
Quoted 7 lines
QEMU computes the DMA logging ranges for two predefined ranges: 32-bit and 64-bit. In the OVMF case, QEMU includes in the 64-bit range, RAM (at the lower part) and device RAM regions (at the top of the address space). The size of that range can be bigger than the 2^42 limit of the MLX5 HW for dirty tracking. QEMU is not making much effort to be smart. There is room for improvement.
Quoted 2 lines
Interesting, we haven't reproduced this in our testing with OVMF multi-TB configs with these VFs. Could you share the OVMF base version you were using?
edk2-ovmf-20230524-3.el9.noarch
host is a :
Architecture:x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHzQuoted 4 lines
or
maybe we didn't triggered it considering the total device RAM regions would be small enough to fit the 32G PCI hole64 that is advertised that avoids a hypothetical relocation.Quoted 9 lines
You need RAM above 4G in the guest :
100000000-27fffffff : System RAM
237800000-2387fffff : Kernel code
238800000-23932cfff : Kernel rodata
239400000-239977cff : Kernel data
23a202000-23b3fffff : Kernel bss
380000000000-3807ffffffff : PCI Bus 0000:00
380000000000-3800000fffff : 0000:00:03.0
380000000000-3800000fffff :mlx5_coreSimilar machine to yours, but in my 32G guests with older OVMF it's putting the PCI area after max-ram:
vfio_device_dirty_tracking_updatesection0x0-0x9ffff-> update [0x0-0x9ffff]vfio_device_dirty_tracking_updatesection0xc0000-0xcafff-> update [0x0-0xcafff]vfio_device_dirty_tracking_updatesection0xcb000-0xcdfff-> update [0x0-0xcdfff]vfio_device_dirty_tracking_updatesection0xce000-0xe7fff-> update [0x0-0xe7fff]vfio_device_dirty_tracking_updatesection0xe8000-0xeffff-> update [0x0-0xeffff]vfio_device_dirty_tracking_updatesection0xf0000-0xfffff-> update [0x0-0xfffff]vfio_device_dirty_tracking_updatesection0x100000-0x7fffffff-> update [0x0-0x7fffffff]vfio_device_dirty_tracking_updatesection0xfd000000-0xfdffffff-> update [0x0
-0xfdffffff]
Quoted 9 lines
vfio_device_dirty_tracking_updatesection0xfffc0000-0xffffffff-> update [0x0
-0xffffffff]vfio_device_dirty_tracking_updatesection0x100000000-0x87fffffff-> update [0x100000000-0x87fffffff]vfio_device_dirty_tracking_updatesection0x880000000-0x880001fff-> update [0x100000000-0x880001fff]vfio_device_dirty_tracking_updatesection0x880003000-0x8ffffffff-> update [0x100000000-0x8ffffffff]
and so the range is smaller.
The latest version of OVMF enables the dynamic mmio window which causes the issue.
Quoted 31 lines
Quoted 31 lines
Activating the QEMU trace events shows quickly the issue :
vfio_device_dirty_tracking_updatesection0x0-0x9ffff-> update [0x0-0x9ffff]
vfio_device_dirty_tracking_updatesection0xa0000-0xaffff-> update [0x0-0xaffff]
vfio_device_dirty_tracking_updatesection0xc0000-0xc3fff-> update [0x0-0xc3fff]
vfio_device_dirty_tracking_updatesection0xc4000-0xdffff-> update [0x0-0xdffff]
vfio_device_dirty_tracking_updatesection0xe0000-0xfffff-> update [0x0-0xfffff]
vfio_device_dirty_tracking_updatesection0x100000-0x7fffffff-> update
[0x0-0x7fffffff]
vfio_device_dirty_tracking_updatesection0x80000000-0x807fffff-> update
[0x0-0x807fffff]
vfio_device_dirty_tracking_updatesection0x100000000-0x27fffffff->
update [0x100000000-0x27fffffff]
vfio_device_dirty_tracking_updatesection0x383800000000-0x383800001fff->
update [0x100000000-0x383800001fff]
vfio_device_dirty_tracking_updatesection0x383800003000-0x3838000fffff->
update [0x100000000-0x3838000fffff]So that's nice. And with less RAM in the VM, 2G, migration should work though.
Quoted 4 lines
We could use do more than 2 ranges (or going back to sharing all ranges), or add a set of ranges that represents the device RAM without computing a min/max there (not sure we can figure that out from within the memory listener does all this logic);
Quoted 7 lines
Quoted 3 lines
The listener is container based. May we could add one range per device if we can identify a different owner per memory section.
For brainstorm purposes ... Maybe something like this below. Should make your case work. As mentioned earlier in my case it's placed always at maxram+1, so makes no difference in having the "pci" range
yes.
That said, it looks good and it does handle the useless gap in the 64-bit range. Thanks for doing it. We should explore the non OVMF case with this patch also.
C.
Quoted 149 lines
------>8-------
From: Joao Martins <[email protected]>
Date: Thu, 7 Sep 2023 09:23:38 -0700
Subject: [PATCH] vfio/common: Separate vfio-pci rangesSigned-off-by Joao Martins <[email protected]>--- hw/vfio/common.c | 48 ++++++++++++++++++++++++++++++++++++++++---- hw/vfio/trace-events | 2 +- 2 files changed, 45 insertions(+), 5 deletions(-)hw/vfio/common.c +35
-2@@ -27,6 +27,7 @@#include "hw/vfio/vfio-common.h" #include "hw/vfio/vfio.h" +#include "hw/vfio/pci.h" #include "exec/address-spaces.h" #include "exec/memory.h" #include "exec/ram_addr.h" @@ -1424,6 +1425,8 @@ typedef struct VFIODirtyRanges {hwaddr max32;hwaddr min64;hwaddr max64; +hwaddr minpci; +hwaddr maxpci; } VFIODirtyRanges; typedef struct VFIODirtyRangesListener {@@ -1432,6 +1435,31 @@ typedef struct VFIODirtyRangesListener {MemoryListener listener; } VFIODirtyRangesListener; +static bool vfio_section_is_vfio_pci (MemoryRegionSection *section, +VFIOContainer *container) +{ +VFIOPCIDevice *pcidev; +VFIODevice *vbasedev; +VFIOGroup *group; +Object *owner; + + owner =memory_region_owner (section->mr); + + QLIST_FOREACH(group, &container->group_list, container_next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { +if (vbasedev->type !=VFIO_DEVICE_TYPE_PCI ) { +continue ; + } + pcidev =container_of (vbasedev, VFIOPCIDevice, vbasedev); +if (OBJECT (pcidev) == owner) { +return true; + } + } + } + +return false; +} +static void vfio_dirty_tracking_update (MemoryListener *listener,MemoryRegionSection *section) {@@ -1458,8 +1486,13 @@ static void vfio_dirty_tracking_update(MemoryListener*listener, * would be an IOVATree but that has a much bigger runtime overhead and * unnecessary complexity. */ - min = (end <=UINT32_MAX ) ? &range->min32 : &range->min64; - max = (end <=UINT32_MAX ) ? &range->max32 : &range->max64; +if (!vfio_section_is_vfio_pci (section, dirty->container)) { + min = (end <=UINT32_MAX ) ? &range->min32 : &range->min64; + max = (end <=UINT32_MAX ) ? &range->max32 : &range->max64; + }else { + min = &range->minpci; + max = &range->maxpci; + }if (*min > iova) {*min = iova;hw/vfio/common.c +2
-1@@ -1485,6 +1518,7 @@ static void vfio_dirty_tracking_init(VFIOContainer *container,memset (&dirty,0 ,sizeof (dirty)); dirty.ranges.min32 =UINT32_MAX ; dirty.ranges.min64 =UINT64_MAX ; + dirty.ranges.minpci =UINT64_MAX ; dirty.listener = vfio_dirty_tracking_listener; dirty.container = container;@@ -1555,7 +1589,7 @@ vfio_device_feature_dma_logging_start_create(VFIOContainer*container, * DMA logging uAPI guarantees to support at least a number of ranges that * fits into a single host kernel base page. */ - control->num_ranges = !!tracking->max32 + !!tracking->max64; + control->num_ranges = !!tracking->max32 + !!tracking->max64 + !!tracking->maxpci; ranges = g_try_new0(struct vfio_device_feature_dma_logging_range,control->num_ranges); if (!ranges) {hw/vfio/common.c +7
-1@@ -1574,11 +1608,17 @@vfio_device_feature_dma_logging_start_create (VFIOContainer *container, if (tracking->max64) { ranges->iova = tracking->min64; ranges->length = (tracking->max64 - tracking->min64) +1 ; + ranges++; + } + if (tracking->maxpci) { + ranges->iova = tracking->minpci; + ranges->length = (tracking->maxpci - tracking->minpci) +1 ; }trace_vfio_device_dirty_tracking_start (control->num_ranges, tracking->min32, tracking->max32, - tracking->min64, tracking->max64); + tracking->min64, tracking->max64, + tracking->minpci, tracking->maxpci);return feature;}hw/vfio/trace-events +0
-0@@ -104,7 +104,7 @@ vfio_known_safe_misalignment(const char *name, uint64_tiova,
uint64_toffset_wi
vfio_listener_region_add_no_dma_map(const char *name,uint64_tiova,uint64_t
size,uint64_tpage_size) "Region \"%s\" 0x%"PRIx64" size=0x%"PRIx64" is not aligned to 0x%"PRIx64" and cannot be mapped for DMA"
vfio_listener_region_del(uint64_t start, uint64_t end)"region_del0x%"PRIx64"
- 0x%"PRIx64
vfio_device_dirty_tracking_update(uint64_t start,uint64_tend,uint64_tmin,uint64_tmax) "section 0x%"PRIx64" - 0x%"PRIx64" -> update [0x%"PRIx64" - 0x%"PRIx64"]"
-vfio_device_dirty_tracking_start(intnr_ranges,uint64_tmin32,uint64_tmax32,uint64_tmin64,uint64_tmax64) "nr_ranges%d 32:[0x%"PRIx64" - 0x%"PRIx64"], 64:[0x%"PRIx64" - 0x%"PRIx64"]" +vfio_device_dirty_tracking_start(intnr_ranges,uint64_tmin32,uint64_tmax32,uint64_tmin64,uint64_tmax64,uint64_tminpci,uint64_tmaxpci) "nr_ranges%d 32:[0x%"PRIx64" - 0x%"PRIx64"], 64:[0x%"PRIx64" - 0x%"PRIx64"], pci:[0x%"PRIx64"
- 0x%"PRIx64"]"
vfio_disconnect_container(int fd)"close container->fd=%d"vfio_put_group(int fd)"close group->fd=%d"
vfio_get_device(const char * name, unsigned int flags, unsigned intnum_regions, unsigned intnum_irqs) "Device %s flags: %u, regions: %u, irqs: %u" --
2.39.3
Yishai Hadas
maintainerPATCH v7 8/10 vfio/mlx5: Report dirty pages from tracker
In reply to Yishai HadasReport dirty pages from tracker.
It includes:
Querying for dirty pages in a given IOVA range, this is done by modifying the tracker into the reporting state and supplying the required range.
Using the CQ event completion mechanism to be notified once data is ready on the CQ/QP to be processed.
Once data is available turn on the corresponding bits in the bit map.
This functionality will be used as part of the 'log_read_and_clear' driver callback in the next patches.
--- drivers/vfio/pci/mlx5/cmd.c | 191 ++++++++++++++++++++++++++++++++++++ drivers/vfio/pci/mlx5/cmd.h | 4 + 2 files changed, 195 insertions(+)
drivers/vfio/pci/mlx5/cmd.c +191 -0
@@ -5,6 +5,8 @@#include "cmd.h" +enum {CQ_OK =0 ,CQ_EMPTY =-1 ,CQ_POLL_ERR =-2 }; +static int mlx5vf_cmd_get_vhca_id (struct mlx5_core_dev *mdev,u16 function_id,u16 *vhca_id); static void
@@ -157,6 +159,7 @@ void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev,VFIO_MIGRATION_STOP_COPY |VFIO_MIGRATION_P2P ; mvdev->core_device.vdev.mig_ops = mig_ops; +init_completion (&mvdev->tracker_comp); end:mlx5_vf_put_core_dev (mvdev->mdev);
@@ -552,6 +555,29 @@ static int mlx5vf_cmd_destroy_tracker(struct mlx5_core_dev *mdev,return mlx5_cmd_exec (mdev, in,sizeof (in), out,sizeof (out)); } +static int mlx5vf_cmd_modify_tracker (struct mlx5_core_dev *mdev, +u32 tracker_id,unsigned long iova, +unsigned long length,u32 tracker_state) +{ +u32 in[MLX5_ST_SZ_DW (modify_page_track_obj_in)] = {}; +u32 out[MLX5_ST_SZ_DW (general_obj_out_cmd_hdr)] = {}; +void *obj_context; +void *cmd_hdr; + + cmd_hdr =MLX5_ADDR_OF (modify_page_track_obj_in, in, general_obj_in_cmd_hdr); +MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, opcode,MLX5_CMD_OP_MODIFY_GENERAL_OBJECT ); +MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, obj_type,MLX5_OBJ_TYPE_PAGE_TRACK ); +MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, obj_id, tracker_id); + + obj_context =MLX5_ADDR_OF (modify_page_track_obj_in, in, obj_context); +MLX5_SET64 (page_track, obj_context, modify_field_select,0x3 ); +MLX5_SET64 (page_track, obj_context, range_start_address, iova); +MLX5_SET64 (page_track, obj_context, length, length); +MLX5_SET (page_track, obj_context, state, tracker_state); + +return mlx5_cmd_exec (mdev, in,sizeof (in), out,sizeof (out)); +} +static int alloc_cq_frag_buf (struct mlx5_core_dev *mdev,struct mlx5_vhca_cq_buf *buf,int nent,int cqe_size)
@@ -593,6 +619,16 @@ static void mlx5vf_destroy_cq(struct mlx5_core_dev *mdev,mlx5_db_free (mdev, &cq->db); } +static void mlx5vf_cq_complete (struct mlx5_core_cq *mcq, +struct mlx5_eqe *eqe) +{ +struct mlx5vf_pci_core_device *mvdev = + container_of(mcq, struct mlx5vf_pci_core_device, + tracker.cq.mcq); + +complete (&mvdev->tracker_comp); +} +static int mlx5vf_create_cq (struct mlx5_core_dev *mdev,struct mlx5_vhca_page_tracker *tracker,size_t ncqe)
@@ -643,10 +679,13 @@ static int mlx5vf_create_cq(struct mlx5_core_dev *mdev,MLX5_SET64 (cqc, cqc, dbr_addr, cq->db.dma); pas = (__be64 *)MLX5_ADDR_OF (create_cq_in, in, pas);mlx5_fill_page_frag_array (&cq->buf.frag_buf, pas); + cq->mcq.comp = mlx5vf_cq_complete; err =mlx5_core_create_cq (mdev, &cq->mcq, in, inlen, out,sizeof (out));if (err) goto err_vec; +mlx5_cq_arm (&cq->mcq,MLX5_CQ_DB_REQ_NOT , tracker->uar->map, + cq->mcq.cons_index);kvfree (in);return 0 ;
@@ -1109,3 +1148,155 @@ int mlx5vf_start_page_tracker(struct vfio_device *vdev,mlx5vf_state_mutex_unlock (mvdev);return err; } + +static void +set_report_output (u32 size,int index,struct mlx5_vhca_qp *qp, +struct iova_bitmap *dirty) +{ +u32 entry_size =MLX5_ST_SZ_BYTES (page_track_report_entry); +u32 nent = size / entry_size; +struct page *page; +u64 addr; +u64 *buf; +int i; + +if (WARN_ON (index >= qp->recv_buf.npages || + (nent > qp->max_msg_size / entry_size))) +return ; + + page = qp->recv_buf.page_list[index]; + buf =kmap_local_page (page); +for (i =0 ; i < nent; i++) { + addr =MLX5_GET (page_track_report_entry, buf + i, + dirty_address_low); + addr |= (u64 )MLX5_GET (page_track_report_entry, buf + i, + dirty_address_high) <<32 ; +iova_bitmap_set (dirty, addr, qp->tracked_page_size); + } +kunmap_local (buf); +} + +static void +mlx5vf_rq_cqe (struct mlx5_vhca_qp *qp,struct mlx5_cqe64 *cqe, +struct iova_bitmap *dirty,int *tracker_status) +{ +u32 size; +int ix; + + qp->rq.cc++; + *tracker_status =be32_to_cpu (cqe->immediate) >>28 ; + size =be32_to_cpu (cqe->byte_cnt); + ix =be16_to_cpu (cqe->wqe_counter) & (qp->rq.wqe_cnt -1 ); + +/* zero length CQE, no data */ +WARN_ON (!size && *tracker_status ==MLX5_PAGE_TRACK_STATE_REPORTING ); +if (size) +set_report_output (size, ix, qp, dirty); + + qp->recv_buf.next_rq_offset = ix * qp->max_msg_size; +mlx5vf_post_recv (qp); +} + +static void *get_cqe (struct mlx5_vhca_cq *cq,int n) +{ +return mlx5_frag_buf_get_wqe (&cq->buf.fbc, n); +} + +static struct mlx5_cqe64 *get_sw_cqe (struct mlx5_vhca_cq *cq,int n) +{ +void *cqe =get_cqe (cq, n & (cq->ncqe -1 )); +struct mlx5_cqe64 *cqe64; + + cqe64 = (cq->mcq.cqe_sz ==64 ) ? cqe : cqe +64 ; + +if (likely (get_cqe_opcode (cqe64) !=MLX5_CQE_INVALID ) && + !((cqe64->op_own &MLX5_CQE_OWNER_MASK ) ^ !!(n & (cq->ncqe)))) { +return cqe64; + }else { +return NULL ; + } +} + +static int +mlx5vf_cq_poll_one (struct mlx5_vhca_cq *cq,struct mlx5_vhca_qp *qp, +struct iova_bitmap *dirty,int *tracker_status) +{ +struct mlx5_cqe64 *cqe; +u8 opcode; + + cqe =get_sw_cqe (cq, cq->mcq.cons_index); +if (!cqe) +return CQ_EMPTY ; + + ++cq->mcq.cons_index; +/* +* Make sure we read CQ entry contents after we've checked the +* ownership bit. +*/ +rmb (); + opcode =get_cqe_opcode (cqe); +switch (opcode) { +case MLX5_CQE_RESP_SEND_IMM : +mlx5vf_rq_cqe (qp, cqe, dirty, tracker_status); +return CQ_OK ; +default : +return CQ_POLL_ERR ; + } +} + +int mlx5vf_tracker_read_and_clear (struct vfio_device *vdev,unsigned long iova, +unsigned long length, +struct iova_bitmap *dirty) +{ +struct mlx5vf_pci_core_device *mvdev =container_of ( + vdev, struct mlx5vf_pci_core_device, core_device.vdev); +struct mlx5_vhca_page_tracker *tracker = &mvdev->tracker; +struct mlx5_vhca_cq *cq = &tracker->cq; +struct mlx5_core_dev *mdev; +int poll_err, err; + +mutex_lock (&mvdev->state_mutex); +if (!mvdev->log_active) { + err = -EINVAL ; + goto end; + } + +if (mvdev->mdev_detach) { + err = -ENOTCONN ; + goto end; + } + + mdev = mvdev->mdev; + err =mlx5vf_cmd_modify_tracker (mdev, tracker->id, iova, length, +MLX5_PAGE_TRACK_STATE_REPORTING ); +if (err) + goto end; + + tracker->status =MLX5_PAGE_TRACK_STATE_REPORTING ; +while (tracker->status ==MLX5_PAGE_TRACK_STATE_REPORTING ) { + poll_err =mlx5vf_cq_poll_one (cq, tracker->host_qp, dirty, + &tracker->status); +if (poll_err ==CQ_EMPTY ) { +mlx5_cq_arm (&cq->mcq,MLX5_CQ_DB_REQ_NOT , tracker->uar->map, + cq->mcq.cons_index); + poll_err =mlx5vf_cq_poll_one (cq, tracker->host_qp, + dirty, &tracker->status); +if (poll_err ==CQ_EMPTY ) { +wait_for_completion (&mvdev->tracker_comp); +continue ; + } + } +if (poll_err ==CQ_POLL_ERR ) { + err = -EIO ; + goto end; + } +mlx5_cq_set_ci (&cq->mcq); + } + +if (tracker->status ==MLX5_PAGE_TRACK_STATE_ERROR ) + err = -EIO ; + +end: +mlx5vf_state_mutex_unlock (mvdev); +return err; +}
drivers/vfio/pci/mlx5/cmd.h +4 -0
@@ -86,6 +86,7 @@ struct mlx5_vhca_page_tracker {struct mlx5_vhca_cq cq;struct mlx5_vhca_qp *host_qp;struct mlx5_vhca_qp *fw_qp; +int status; };struct mlx5vf_pci_core_device {
@@ -96,6 +97,7 @@ struct mlx5vf_pci_core_device { u8 deferred_reset:1; u8 mdev_detach:1; u8 log_active:1; +struct completion tracker_comp;/* protect migration state */ struct mutex state_mutex;enum vfio_device_mig_state mig_state;
@@ -127,4 +129,6 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work);int mlx5vf_start_page_tracker (struct vfio_device *vdev,struct rb_root_cached *ranges,u32 nnodes,u64 *page_size);int mlx5vf_stop_page_tracker (struct vfio_device *vdev); +int mlx5vf_tracker_read_and_clear (struct vfio_device *vdev,unsigned long iova, +unsigned long length,struct iova_bitmap *dirty);#endif /* MLX5_VFIO_CMD_H */
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH v7 9/10 vfio/mlx5: Manage error scenarios on tracker
In reply to Yishai HadasHandle async error events and health/recovery flow to safely stop the tracker upon error scenarios.
--- drivers/vfio/pci/mlx5/cmd.c | 61 +++++++++++++++++++++++++++++++++++-- drivers/vfio/pci/mlx5/cmd.h | 2 ++ 2 files changed, 61 insertions(+), 2 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +59 -2
@@ -70,6 +70,13 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev,return 0 ; } +static void set_tracker_error (struct mlx5vf_pci_core_device *mvdev) +{ +/* Mark the tracker under an error and wake it up if it's running */ + mvdev->tracker.is_err = true; +complete (&mvdev->tracker_comp); +} +static int mlx5fv_vf_event (struct notifier_block *nb,unsigned long event,void *data) {
@@ -100,6 +107,8 @@ void mlx5vf_cmd_close_migratable(struct mlx5vf_pci_core_device *mvdev)if (!mvdev->migrate_cap)return ; +/* Must be done outside the lock to let it progress */ +set_tracker_error (mvdev);mutex_lock (&mvdev->state_mutex);mlx5vf_disable_fds (mvdev);_mlx5vf_free_page_tracker_resources (mvdev);
@@ -619,6 +628,47 @@ static void mlx5vf_destroy_cq(struct mlx5_core_dev *mdev,mlx5_db_free (mdev, &cq->db); } +static void mlx5vf_cq_event (struct mlx5_core_cq *mcq,enum mlx5_event type) +{ +if (type !=MLX5_EVENT_TYPE_CQ_ERROR ) +return ; + +set_tracker_error (container_of(mcq ,struct mlx5vf_pci_core_device , + tracker.cq.mcq)); +} + +static int mlx5vf_event_notifier (struct notifier_block *nb,unsigned long type, +void *data) +{ +struct mlx5_vhca_page_tracker *tracker = + mlx5_nb_cof(nb, struct mlx5_vhca_page_tracker, nb); +struct mlx5vf_pci_core_device *mvdev =container_of ( + tracker, struct mlx5vf_pci_core_device, tracker); +struct mlx5_eqe *eqe = data; +u8 event_type = (u8 )type; +u8 queue_type; +int qp_num; + +switch (event_type) { +case MLX5_EVENT_TYPE_WQ_CATAS_ERROR : +case MLX5_EVENT_TYPE_WQ_ACCESS_ERROR : +case MLX5_EVENT_TYPE_WQ_INVAL_REQ_ERROR : + queue_type = eqe->data.qp_srq.type; +if (queue_type !=MLX5_EVENT_QUEUE_TYPE_QP ) +break ; + qp_num =be32_to_cpu (eqe->data.qp_srq.qp_srq_n) &0xffffff ; +if (qp_num != tracker->host_qp->qpn && + qp_num != tracker->fw_qp->qpn) +break ; +set_tracker_error (mvdev); +break ; +default : +break ; + } + +return NOTIFY_OK ; +} +static void mlx5vf_cq_complete (struct mlx5_core_cq *mcq,struct mlx5_eqe *eqe) {
@@ -680,6 +730,7 @@ static int mlx5vf_create_cq(struct mlx5_core_dev *mdev, pas = (__be64 *)MLX5_ADDR_OF (create_cq_in, in, pas);mlx5_fill_page_frag_array (&cq->buf.frag_buf, pas); cq->mcq.comp = mlx5vf_cq_complete; + cq->mcq.event = mlx5vf_cq_event; err =mlx5_core_create_cq (mdev, &cq->mcq, in, inlen, out,sizeof (out));if (err) goto err_vec;
@@ -1014,6 +1065,7 @@ _mlx5vf_free_page_tracker_resources(struct mlx5vf_pci_core_device *mvdev)WARN_ON (mvdev->mdev_detach); +mlx5_eq_notifier_unregister (mdev, &tracker->nb);mlx5vf_cmd_destroy_tracker (mdev, tracker->id);mlx5vf_destroy_qp (mdev, tracker->fw_qp);mlx5vf_free_qp_recv_resources (mdev, tracker->host_qp);
@@ -1127,6 +1179,8 @@ int mlx5vf_start_page_tracker(struct vfio_device *vdev,if (err) goto err_activate; +MLX5_NB_INIT (&tracker->nb, mlx5vf_event_notifier,NOTIFY_ANY ); +mlx5_eq_notifier_register (mdev, &tracker->nb); *page_size = host_qp->tracked_page_size; mvdev->log_active = true;mlx5vf_state_mutex_unlock (mvdev);
@@ -1273,7 +1327,8 @@ int mlx5vf_tracker_read_and_clear(struct vfio_device *vdev, unsigned long iova, goto end; tracker->status =MLX5_PAGE_TRACK_STATE_REPORTING ; -while (tracker->status ==MLX5_PAGE_TRACK_STATE_REPORTING ) { +while (tracker->status ==MLX5_PAGE_TRACK_STATE_REPORTING && + !tracker->is_err) { poll_err =mlx5vf_cq_poll_one (cq, tracker->host_qp, dirty, &tracker->status);if (poll_err ==CQ_EMPTY ) {
@@ -1294,8 +1349,10 @@ int mlx5vf_tracker_read_and_clear(struct vfio_device *vdev, unsigned long iova, }if (tracker->status ==MLX5_PAGE_TRACK_STATE_ERROR ) - err = -EIO ; + tracker->is_err = true; +if (tracker->is_err) + err = -EIO ; end:mlx5vf_state_mutex_unlock (mvdev);return err;
drivers/vfio/pci/mlx5/cmd.h +2 -0
@@ -82,10 +82,12 @@ struct mlx5_vhca_qp {struct mlx5_vhca_page_tracker {u32 id;u32 pdn; +u8 is_err:1 ;struct mlx5_uars_page *uar;struct mlx5_vhca_cq cq;struct mlx5_vhca_qp *host_qp;struct mlx5_vhca_qp *fw_qp; +struct mlx5_nb nb;int status; };
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH v7 10/10 vfio/mlx5: Set the driver DMA logging callbacks
In reply to Yishai HadasNow that everything is ready set the driver DMA logging callbacks if supported by the device.
--- drivers/vfio/pci/mlx5/cmd.c | 5 ++++- drivers/vfio/pci/mlx5/cmd.h | 3 ++- drivers/vfio/pci/mlx5/main.c | 9 ++++++++- 3 files changed, 14 insertions(+), 3 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +4 -1
@@ -126,7 +126,8 @@ void mlx5vf_cmd_remove_migratable(struct mlx5vf_pci_core_device *mvdev) }void mlx5vf_cmd_set_migratable (struct mlx5vf_pci_core_device *mvdev, -const struct vfio_migration_ops *mig_ops) +const struct vfio_migration_ops *mig_ops, +const struct vfio_log_ops *log_ops) {struct pci_dev *pdev = mvdev->core_device.pdev;int ret;
@@ -169,6 +170,8 @@ void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev,VFIO_MIGRATION_P2P ; mvdev->core_device.vdev.mig_ops = mig_ops;init_completion (&mvdev->tracker_comp); +if (MLX5_CAP_GEN (mvdev->mdev, adv_virtualization)) + mvdev->core_device.vdev.log_ops = log_ops; end:mlx5_vf_put_core_dev (mvdev->mdev);
drivers/vfio/pci/mlx5/cmd.h +2 -1
@@ -118,7 +118,8 @@ int mlx5vf_cmd_resume_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod);int mlx5vf_cmd_query_vhca_migration_state (struct mlx5vf_pci_core_device *mvdev,size_t *state_size);void mlx5vf_cmd_set_migratable (struct mlx5vf_pci_core_device *mvdev, -const struct vfio_migration_ops *mig_ops); +const struct vfio_migration_ops *mig_ops, +const struct vfio_log_ops *log_ops);void mlx5vf_cmd_remove_migratable (struct mlx5vf_pci_core_device *mvdev);void mlx5vf_cmd_close_migratable (struct mlx5vf_pci_core_device *mvdev); int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
drivers/vfio/pci/mlx5/main.c +8 -1
@@ -579,6 +579,12 @@ static const struct vfio_migration_ops mlx5vf_pci_mig_ops = { .migration_get_state = mlx5vf_pci_get_device_state, }; +static const struct vfio_log_ops mlx5vf_pci_log_ops = { + .log_start = mlx5vf_start_page_tracker, + .log_stop = mlx5vf_stop_page_tracker, + .log_read_and_clear = mlx5vf_tracker_read_and_clear, +}; +static const struct vfio_device_ops mlx5vf_pci_ops = { .name ="mlx5-vfio-pci" , .open_device = mlx5vf_pci_open_device,
@@ -602,7 +608,8 @@ static int mlx5vf_pci_probe(struct pci_dev *pdev,if (!mvdev)return -ENOMEM ;vfio_pci_core_init_device (&mvdev->core_device, pdev, &mlx5vf_pci_ops); -mlx5vf_cmd_set_migratable (mvdev, &mlx5vf_pci_mig_ops); +mlx5vf_cmd_set_migratable (mvdev, &mlx5vf_pci_mig_ops, + &mlx5vf_pci_log_ops);dev_set_drvdata (&pdev->dev, &mvdev->core_device); ret =vfio_pci_core_register_device (&mvdev->core_device);if (ret)
Signature · 1 line
2.18.1
Quoted 45 lines · On Thu, 8 Sep 2022 21:34:38 +0300 Yishai Hadas <[email protected]> wrote:
This series adds device DMA logging uAPIs and their implementation as part of mlx5 driver.
DMA logging allows a device to internally record what DMAs the device is initiating and report them back to userspace. It is part of the VFIO migration infrastructure that allows implementing dirty page tracking during the pre copy phase of live migration. Only DMA WRITEs are logged, and this API is not connected to
VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE.The uAPIs are based on the FEATURE ioctl as were introduced earlier by the below RFC [1] and follows the notes that were discussed in the mailing list.
It includes:
- A PROBE option to detect if the device supports DMA logging.
- A SET option to start device DMA logging in given IOVAs ranges.
- A GET option to read back and clear the device DMA log.
- A SET option to stop device DMA logging that was previously started.Extra details exist as part of relevant patches in the series.
In addition, the series adds some infrastructure support for managing an IOVA bitmap done by Joao Martins.
It abstracts how an IOVA range is represented in a bitmap that is granulated by a given
page_size. So it translates all the lifting of dealing with user pointers into its corresponding kernel addresses. This new functionality abstracts the complexity of user/kernel bitmap pointer usage and finally enables an API to set some bits.This functionality will be used as part of IOMMUFD series for the system IOMMU tracking.
Finally, we come with mlx5 implementation based on its device specification for the DMA logging APIs.
The matching qemu changes can be previewed here [2].
They come on top of the v2 migration protocol patches that were sent already to the mailing list.Note:
- As this series touchedmlx5_coreparts we may need to send the net/mlx5 patches as a pull request format to VFIO to avoid conflicts before acceptance.
Quoted 8 lines
[1] https://lore.kernel.org/all/[email protected]/T/
[2] https://github.com/avihai1122/qemu/commits/device_dirty_trackingChanges from V6: https://lore.kernel.org/all/[email protected]/T/
- Use the first two patches from the PR that was sent by Leon, no code change was involved.
- Patch #5:
* Add a documentation note nearvfio_log_opsas Alex suggested.
Pulled topic branch from Leon and applied 3-10 to the vfio next branch for v6.1. Thanks,
Alex