Add UAPIs to support migration for VFIO devices
Subsystems (2)
v20 cover Add UAPIs to support migration for VFIO devices
Hi,
This patch set adds:
* IOCTL VFIO_IOMMU_DIRTY_PAGES to get dirty pages bitmap with respect to IOMMU container rather than per device. All pages pinned by vendor driver through vfio_pin_pages external API has to be marked as dirty during migration. When IOMMU capable device is present in the container and all pages are pinned and mapped, then all pages are marked dirty.
When there are CPU writes, CPU dirty page tracking can identify dirtied pages, but any page pinned by vendor driver can also be written by device. As of now there is no device which has hardware support for dirty page tracking. So all pages which are pinned should be considered as dirty.
This ioctl is also used to start/stop dirty pages tracking for pinned and unpinned pages while migration is active.
* Updated IOCTL VFIO_IOMMU_UNMAP_DMA to get dirty pages bitmap before unmapping IO virtual address range.
With vIOMMU, during pre-copy phase of migration, while CPUs are still running, IO virtual address unmap can happen while device still keeping reference of guest pfns. Those pages should be reported as dirty before unmap, so that VFIO user space application can copy content of those pages from source to destination.
* Patch 8 detect if IOMMU capable device driver is smart to report pages to be marked dirty by pinning pages using vfio_pin_pages() API.
Yet TODO:
Since there is no device which has hardware support for system memmory dirty bitmap tracking, right now there is no other API from vendor driver to VFIO IOMMU module to report dirty pages. In future, when such hardware support will be implemented, an API will be required such that vendor driver could report dirty pages to VFIO module during migration phases.
Adding revision history from previous QEMU patch set to understand KABI changes done till now
v19 -> v20
- Fixed ioctl to get dirty bitmap to get bitmap of multiple vfio_dmas
- Fixed unmap ioctl to get dirty bitmap of multiple vfio_dmas.
- Removed flag definition from migration capability.
v18 -> v19
- Updated migration capability with supported page sizes bitmap for dirty page tracking and maximum bitmap size supported by kernel module.
- Added patch to calculate and cache pgsize_bitmap when iommu->domain_list is updated.
- Removed extra buffers added in previous version for bitmap manipulation and optimised the code.
v17 -> v18
- Add migration capability to the capability chain for VFIO_IOMMU_GET_INFO ioctl
- Updated UMAP_DMA ioctl to return bitmap of multiple vfio_dma
v16 -> v17
- Fixed errors reported by kbuild test robot <[email protected]> on i386
v15 -> v16
- Minor edits and nit picks (Auger Eric)
- On copying bitmap to user, re-populated bitmap only for pinned pages, excluding unmapped pages and CPU dirtied pages.
- Patches are on tag: next-20200318 and 1-3 patches from Yan's series
https://lkml.org/lkml/2020/3/12/1255
v14 -> v15
- Minor edits and nit picks.
- In the verification of user allocated bitmap memory, added check of maximum size.
- Patches are on tag: next-20200318 and 1-3 patches from Yan's series
https://lkml.org/lkml/2020/3/12/1255
v13 -> v14
- Added struct vfio_bitmap to kabi. updated structure vfio_iommu_type1_dirty_bitmap_get and vfio_iommu_type1_dma_unmap.
- All small changes suggested by Alex.
- Patches are on tag: next-20200318 and 1-3 patches from Yan's series
https://lkml.org/lkml/2020/3/12/1255
v12 -> v13
- Changed bitmap allocation in vfio_iommu_type1 to per vfio_dma
- Changed VFIO_IOMMU_DIRTY_PAGES ioctl behaviour to be per vfio_dma range.
- Changed vfio_iommu_type1_dirty_bitmap structure to have separate data field.
v11 -> v12
- Changed bitmap allocation in vfio_iommu_type1.
- Remove atomicity of ref_count.
- Updated comments for migration device state structure about error reporting.
- Nit picks from v11 reviews
v10 -> v11
- Fix pin pages API to free vpfn if it is marked as unpinned tracking page.
- Added proposal to detect if IOMMU capable device calls external pin pages API to mark pages dirty.
- Nit picks from v10 reviews
v9 -> v10:
- Updated existing VFIO_IOMMU_UNMAP_DMA ioctl to get dirty pages bitmap during unmap while migration is active
- Added flag in VFIO_IOMMU_GET_INFO to indicate driver support dirty page tracking.
- If iommu_mapped, mark all pages dirty.
- Added unpinned pages tracking while migration is active.
- Updated comments for migration device state structure with bit combination table and state transition details.
v8 -> v9:
- Split patch set in 2 sets, Kernel and QEMU.
- Dirty pages bitmap is queried from IOMMU container rather than from vendor driver for per device. Added 2 ioctls to achieve this.
v7 -> v8:
- Updated comments for KABI
- Added BAR address validation check during PCI device's config space load as suggested by Dr. David Alan Gilbert.
- Changed vfio_migration_set_state() to set or clear device state flags.
- Some nit fixes.
v6 -> v7:
- Fix build failures.
v5 -> v6:
- Fix build failure.
v4 -> v5:
- Added decriptive comment about the sequence of access of members of structure vfio_device_migration_info to be followed based on Alex's suggestion
- Updated get dirty pages sequence.
- As per Cornelia Huck's suggestion, added callbacks to VFIODeviceOps to get_object, save_config and load_config.
- Fixed multiple nit picks.
- Tested live migration with multiple vfio device assigned to a VM.
v3 -> v4:
- Added one more bit for _RESUMING flag to be set explicitly.
- data_offset field is read-only for user space application.
- data_size is read for every iteration before reading data from migration, that is removed assumption that data will be till end of migration region.
- If vendor driver supports mappable sparsed region, map those region during setup state of save/load, similarly unmap those from cleanup routines.
- Handles race condition that causes data corruption in migration region during save device state by adding mutex and serialiaing save_buffer and get_dirty_pages routines.
- Skip called get_dirty_pages routine for mapped MMIO region of device.
- Added trace events.
- Split into multiple functional patches.
v2 -> v3:
- Removed enum of VFIO device states. Defined VFIO device state with 2 bits.
- Re-structured vfio_device_migration_info to keep it minimal and defined action on read and write access on its members.
v1 -> v2:
- Defined MIGRATION region type and sub-type which should be used with region type capability.
- Re-structured vfio_device_migration_info. This structure will be placed at 0th offset of migration region.
- Replaced ioctl with read/write for trapped part of migration region.
- Added both type of access support, trapped or mmapped, for data section of the region.
- Moved PCI device functions to pci file.
- Added iteration to get dirty page bitmap until bitmap for all requested pages are copied.
Thanks,
Kirti
Kirti Wankhede (8):
vfio: UAPI for migration interface for device state
vfio iommu: Remove atomicity of ref_count of pinned pages vfio iommu: Cache pgsize_bitmap in struct vfio_iommu
vfio iommu: Add ioctl definition for dirty pages tracking vfio iommu: Implementation of ioctl for dirty pages tracking vfio iommu: Update UNMAP_DMA ioctl to get dirty bitmap before unmap vfio iommu: Add migration capability to report supported features
vfio: Selective dirty page tracking if IOMMU backed device pins pages
drivers/vfio/vfio.c | 13 +- drivers/vfio/vfio_iommu_type1.c | 565 ++++++++++++++++++++++++++++++++++++---- include/linux/vfio.h | 4 +- include/uapi/linux/vfio.h | 315 ++++++++++++++++++++++ 4 files changed, 838 insertions(+), 59 deletions(-)
Signature · 1 line
2.7.0
PATCH v20 1/8 vfio: UAPI for migration interface for device state
- Defined MIGRATION region type and sub-type.
- Defined vfio_device_migration_info structure which will be placed at the 0th offset of migration region to get/set VFIO device related information. Defined members of structure and usage on read/write access.
- Defined device states and state transition details.
- Defined sequence to be followed while saving and resuming VFIO device.
--- include/uapi/linux/vfio.h | 228 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 228 insertions(+)
include/uapi/linux/vfio.h +228 -0
@@ -305,6 +305,7 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)#define VFIO_REGION_TYPE_GFX (1)#define VFIO_REGION_TYPE_CCW (2) +#define VFIO_REGION_TYPE_MIGRATION (3)/* sub-types for VFIO_REGION_TYPE_PCI_* */
@@ -379,6 +380,233 @@ struct vfio_region_gfx_edid {/* sub-types for VFIO_REGION_TYPE_CCW */ #define VFIO_REGION_SUBTYPE_CCW_ASYNC_CMD (1) +/* sub-types for VFIO_REGION_TYPE_MIGRATION */ +#define VFIO_REGION_SUBTYPE_MIGRATION (1) + +/* +* The structure vfio_device_migration_info is placed at the 0th offset of +* the VFIO_REGION_SUBTYPE_MIGRATION region to get and set VFIO device related +* migration information. Field accesses from this structure are only supported +* at their native width and alignment. Otherwise, the result is undefined and +* vendor drivers should return an error. +* +* device_state: (read/write) +* - The user application writes to this field to inform the vendor driver +* about the device state to be transitioned to. +* - The vendor driver should take the necessary actions to change the +* device state. After successful transition to a given state, the +* vendor driver should return success on write(device_state, state) +* system call. If the device state transition fails, the vendor driver +* should return an appropriate -errno for the fault condition. +* - On the user application side, if the device state transition fails, +* that is, if write(device_state, state) returns an error, read +* device_state again to determine the current state of the device from +* the vendor driver. +* - The vendor driver should return previous state of the device unless +* the vendor driver has encountered an internal error, in which case +* the vendor driver may report the device_state VFIO_DEVICE_STATE_ERROR. +* - The user application must use the device reset ioctl to recover the +* device from VFIO_DEVICE_STATE_ERROR state. If the device is +* indicated to be in a valid device state by reading device_state, the +* user application may attempt to transition the device to any valid +* state reachable from the current state or terminate itself. +* +* device_state consists of 3 bits: +* - If bit 0 is set, it indicates the _RUNNING state. If bit 0 is clear, +* it indicates the _STOP state. When the device state is changed to +* _STOP, driver should stop the device before write() returns. +* - If bit 1 is set, it indicates the _SAVING state, which means that the +* driver should start gathering device state information that will be +* provided to the VFIO user application to save the device's state. +* - If bit 2 is set, it indicates the _RESUMING state, which means that +* the driver should prepare to resume the device. Data provided through +* the migration region should be used to resume the device. +* Bits 3 - 31 are reserved for future use. To preserve them, the user +* application should perform a read-modify-write operation on this +* field when modifying the specified bits. +* +* +------- _RESUMING +* |+------ _SAVING +* ||+----- _RUNNING +* ||| +* 000b => Device Stopped, not saving or resuming +* 001b => Device running, which is the default state +* 010b => Stop the device & save the device state, stop-and-copy state +* 011b => Device running and save the device state, pre-copy state +* 100b => Device stopped and the device state is resuming +* 101b => Invalid state +* 110b => Error state +* 111b => Invalid state +* +* State transitions: +* +* _RESUMING _RUNNING Pre-copy Stop-and-copy _STOP +* (100b) (001b) (011b) (010b) (000b) +* 0. Running or default state +* | +* +* 1. Normal Shutdown (optional) +* |------------------------------------->| +* +* 2. Save the state or suspend +* |------------------------->|---------->| +* +* 3. Save the state during live migration +* |----------->|------------>|---------->| +* +* 4. Resuming +* |<---------| +* +* 5. Resumed +* |--------->| +* +* 0. Default state of VFIO device is _RUNNNG when the user application starts. +* 1. During normal shutdown of the user application, the user application may +* optionally change the VFIO device state from _RUNNING to _STOP. This +* transition is optional. The vendor driver must support this transition but +* must not require it. +* 2. When the user application saves state or suspends the application, the +* device state transitions from _RUNNING to stop-and-copy and then to _STOP. +* On state transition from _RUNNING to stop-and-copy, driver must stop the +* device, save the device state and send it to the application through the +* migration region. The sequence to be followed for such transition is given +* below. +* 3. In live migration of user application, the state transitions from _RUNNING +* to pre-copy, to stop-and-copy, and to _STOP. +* On state transition from _RUNNING to pre-copy, the driver should start +* gathering the device state while the application is still running and send +* the device state data to application through the migration region. +* On state transition from pre-copy to stop-and-copy, the driver must stop +* the device, save the device state and send it to the user application +* through the migration region. +* Vendor drivers must support the pre-copy state even for implementations +* where no data is provided to the user before the stop-and-copy state. The +* user must not be required to consume all migration data before the device +* transitions to a new state, including the stop-and-copy state. +* The sequence to be followed for above two transitions is given below. +* 4. To start the resuming phase, the device state should be transitioned from +* the _RUNNING to the _RESUMING state. +* In the _RESUMING state, the driver should use the device state data +* received through the migration region to resume the device. +* 5. After providing saved device data to the driver, the application should +* change the state from _RESUMING to _RUNNING. +* +* reserved: +* Reads on this field return zero and writes are ignored. +* +* pending_bytes: (read only) +* The number of pending bytes still to be migrated from the vendor driver. +* +* data_offset: (read only) +* The user application should read data_offset field from the migration +* region. The user application should read the device data from this +* offset within the migration region during the _SAVING state or write +* the device data during the _RESUMING state. See below for details of +* sequence to be followed. +* +* data_size: (read/write) +* The user application should read data_size to get the size in bytes of +* the data copied in the migration region during the _SAVING state and +* write the size in bytes of the data copied in the migration region +* during the _RESUMING state. +* +* The format of the migration region is as follows: +* ------------------------------------------------------------------ +* |vfio_device_migration_info| data section | +* | | /////////////////////////////// | +* ------------------------------------------------------------------ +* ^ ^ +* offset 0-trapped part data_offset +* +* The structure vfio_device_migration_info is always followed by the data +* section in the region, so data_offset will always be nonzero. The offset +* from where the data is copied is decided by the kernel driver. The data +* section can be trapped, mmapped, or partitioned, depending on how the kernel +* driver defines the data section. The data section partition can be defined +* as mapped by the sparse mmap capability. If mmapped, data_offset must be +* page aligned, whereas initial section which contains the +* vfio_device_migration_info structure, might not end at the offset, which is +* page aligned. The user is not required to access through mmap regardless +* of the capabilities of the region mmap. +* The vendor driver should determine whether and how to partition the data +* section. The vendor driver should return data_offset accordingly. +* +* The sequence to be followed while in pre-copy state and stop-and-copy state +* is as follows: +* a. Read pending_bytes, indicating the start of a new iteration to get device +* data. Repeated read on pending_bytes at this stage should have no side +* effects. +* If pending_bytes == 0, the user application should not iterate to get data +* for that device. +* If pending_bytes > 0, perform the following steps. +* b. Read data_offset, indicating that the vendor driver should make data +* available through the data section. The vendor driver should return this +* read operation only after data is available from (region + data_offset) +* to (region + data_offset + data_size). +* c. Read data_size, which is the amount of data in bytes available through +* the migration region. +* Read on data_offset and data_size should return the offset and size of +* the current buffer if the user application reads data_offset and +* data_size more than once here. +* d. Read data_size bytes of data from (region + data_offset) from the +* migration region. +* e. Process the data. +* f. Read pending_bytes, which indicates that the data from the previous +* iteration has been read. If pending_bytes > 0, go to step b. +* +* The user application can transition from the _SAVING|_RUNNING +* (pre-copy state) to the _SAVING (stop-and-copy) state regardless of the +* number of pending bytes. The user application should iterate in _SAVING +* (stop-and-copy) until pending_bytes is 0. +* +* The sequence to be followed while _RESUMING device state is as follows: +* While data for this device is available, repeat the following steps: +* a. Read data_offset from where the user application should write data. +* b. Write migration data starting at the migration region + data_offset for +* the length determined by data_size from the migration source. +* c. Write data_size, which indicates to the vendor driver that data is +* written in the migration region. Vendor driver must return this write +* operations on consuming data. Vendor driver should apply the +* user-provided migration region data to the device resume state. +* +* If an error occurs during the above sequences, the vendor driver can return +* an error code for next read() or write() operation, which will terminate the +* loop. The user application should then take the next necessary action, for +* example, failing migration or terminating the user application. +* +* For the user application, data is opaque. The user application should write +* data in the same order as the data is received and the data should be of +* same transaction size at the source. +*/ + +struct vfio_device_migration_info { +__u32 device_state;/* VFIO device state */ +#define VFIO_DEVICE_STATE_STOP (0) +#define VFIO_DEVICE_STATE_RUNNING (1 << 0) +#define VFIO_DEVICE_STATE_SAVING (1 << 1) +#define VFIO_DEVICE_STATE_RESUMING (1 << 2) +#define VFIO_DEVICE_STATE_MASK (VFIO_DEVICE_STATE_RUNNING | \ + VFIO_DEVICE_STATE_SAVING | \ + VFIO_DEVICE_STATE_RESUMING) + +#define VFIO_DEVICE_STATE_VALID (state) \ + (state & VFIO_DEVICE_STATE_RESUMING ? \ + (state & VFIO_DEVICE_STATE_MASK) == VFIO_DEVICE_STATE_RESUMING : 1) + +#define VFIO_DEVICE_STATE_IS_ERROR (state) \ + ((state & VFIO_DEVICE_STATE_MASK) == (VFIO_DEVICE_STATE_SAVING | \ + VFIO_DEVICE_STATE_RESUMING)) + +#define VFIO_DEVICE_STATE_SET_ERROR (state) \ + ((state & ~VFIO_DEVICE_STATE_MASK) | VFIO_DEVICE_SATE_SAVING | \ + VFIO_DEVICE_STATE_RESUMING) + +__u32 reserved; +__u64 pending_bytes; +__u64 data_offset; +__u64 data_size; +}; +/* * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped * which allows direct access to non-MSIX registers which happened to be within
Signature · 1 line
2.7.0
PATCH v20 2/8 vfio iommu: Remove atomicity of ref_count of pinned pages
In reply to Kirti Wankhedevfio_pfn.ref_count is always updated while holding iommu->lock, using atomic variable is overkill.
--- drivers/vfio/vfio_iommu_type1.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/vfio/vfio_iommu_type1.c +5 -4
@@ -112,7 +112,7 @@ struct vfio_pfn {struct rb_node node;dma_addr_t iova;/* Device address */ unsigned long pfn;/* Host pfn */ -atomic_t ref_count; +unsigned int ref_count; };struct vfio_regions {
@@ -233,7 +233,7 @@ static int vfio_add_to_pfn_list(struct vfio_dma *dma, dma_addr_t iova, vpfn->iova = iova; vpfn->pfn = pfn; -atomic_set (&vpfn->ref_count,1 ); + vpfn->ref_count =1 ;vfio_link_pfn (dma, vpfn);return 0 ; }
@@ -251,7 +251,7 @@ static struct vfio_pfn *vfio_iova_get_vfio_pfn(struct vfio_dma *dma,struct vfio_pfn *vpfn =vfio_find_vpfn (dma, iova);if (vpfn) -atomic_inc (&vpfn->ref_count); + vpfn->ref_count++;return vpfn; }
@@ -259,7 +259,8 @@ static int vfio_iova_put_vfio_pfn(struct vfio_dma *dma, struct vfio_pfn *vpfn) {int ret =0 ; -if (atomic_dec_and_test (&vpfn->ref_count)) { + vpfn->ref_count--; +if (!vpfn->ref_count) { ret =put_pfn (vpfn->pfn, dma->prot);vfio_remove_from_pfn_list (dma, vpfn); }
Signature · 1 line
2.7.0
PATCH v20 3/8 vfio iommu: Cache pgsize_bitmap in struct vfio_iommu
In reply to Kirti WankhedeCalculate and cache pgsize_bitmap when iommu->domain_list is updated and iommu->external_domain is set for mdev device.
Add iommu->lock protection when cached pgsize_bitmap is accessed.
--- drivers/vfio/vfio_iommu_type1.c | 88 +++++++++++++++++++++++------------------ 1 file changed, 49 insertions(+), 39 deletions(-)
drivers/vfio/vfio_iommu_type1.c +49 -39
@@ -69,6 +69,7 @@ struct vfio_iommu {struct rb_root dma_list;struct blocking_notifier_head notifier;unsigned int dma_avail; +uint64_t pgsize_bitmap;bool v2;bool nesting; };
@@ -805,15 +806,14 @@ static void vfio_remove_dma(struct vfio_iommu *iommu, struct vfio_dma *dma) iommu->dma_avail++; } -static unsigned long vfio_pgsize_bitmap (struct vfio_iommu *iommu) +static void vfio_pgsize_bitmap (struct vfio_iommu *iommu) {struct vfio_domain *domain; -unsigned long bitmap =ULONG_MAX ; -mutex_lock (&iommu->lock); + iommu->pgsize_bitmap =ULONG_MAX ; +list_for_each_entry (domain, &iommu->domain_list, next) - bitmap &= domain->domain->pgsize_bitmap; -mutex_unlock (&iommu->lock); + iommu->pgsize_bitmap &= domain->domain->pgsize_bitmap;/* * In case the IOMMU supports page sizes smaller than PAGE_SIZE
@@ -823,12 +823,10 @@ static unsigned long vfio_pgsize_bitmap(struct vfio_iommu *iommu)* granularity while iommu driver can use the sub-PAGE_SIZE size * to map the buffer. */ -if (bitmap & ~PAGE_MASK ) { - bitmap &=PAGE_MASK ; - bitmap |=PAGE_SIZE ; +if (iommu->pgsize_bitmap & ~PAGE_MASK ) { + iommu->pgsize_bitmap &=PAGE_MASK ; + iommu->pgsize_bitmap |=PAGE_SIZE ; } - -return bitmap; } static int vfio_dma_do_unmap(struct vfio_iommu *iommu,
@@ -839,19 +837,28 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu,size_t unmapped =0 ;int ret =0 , retries =0 ; - mask = ((uint64_t )1 <<__ffs (vfio_pgsize_bitmap (iommu))) -1 ; +mutex_lock (&iommu->lock); + + mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + +if (unmap->iova & mask) { + ret = -EINVAL ; + goto unlock; + } + +if (!unmap->size || unmap->size & mask) { + ret = -EINVAL ; + goto unlock; + } -if (unmap->iova & mask) -return -EINVAL ; -if (!unmap->size || unmap->size & mask) -return -EINVAL ;if (unmap->iova + unmap->size -1 < unmap->iova || - unmap->size >SIZE_MAX ) -return -EINVAL ; + unmap->size >SIZE_MAX ) { + ret = -EINVAL ; + goto unlock; + }WARN_ON (mask &PAGE_MASK ); again: -mutex_lock (&iommu->lock);/* * vfio-iommu-type1 (v1) - User mappings were coalesced together to
@@ -930,6 +937,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu,blocking_notifier_call_chain (&iommu->notifier,VFIO_IOMMU_NOTIFY_DMA_UNMAP , &nb_unmap); +mutex_lock (&iommu->lock); goto again; } unmapped += dma->size;
@@ -1045,24 +1053,28 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,if (map->size != size || map->vaddr != vaddr || map->iova != iova)return -EINVAL ; - mask = ((uint64_t )1 <<__ffs (vfio_pgsize_bitmap (iommu))) -1 ; - -WARN_ON (mask &PAGE_MASK ); -/* READ/WRITE from device perspective */ if (map->flags &VFIO_DMA_MAP_FLAG_WRITE ) prot |=IOMMU_WRITE ;if (map->flags &VFIO_DMA_MAP_FLAG_READ ) prot |=IOMMU_READ ; -if (!prot || !size || (size | iova | vaddr) & mask) -return -EINVAL ; +mutex_lock (&iommu->lock); -/* Don't allow IOVA or virtual address wrap */ -if (iova + size -1 < iova || vaddr + size -1 < vaddr) -return -EINVAL ; + mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; -mutex_lock (&iommu->lock); +WARN_ON (mask &PAGE_MASK ); + +if (!prot || !size || (size | iova | vaddr) & mask) { + ret = -EINVAL ; + goto out_unlock; + } + +/* Don't allow IOVA or virtual address wrap */ +if (iova + size -1 < iova || vaddr + size -1 < vaddr) { + ret = -EINVAL ; + goto out_unlock; + }if (vfio_find_dma (iommu, iova, size)) { ret = -EEXIST ;
@@ -1668,6 +1680,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,if (!iommu->external_domain) {INIT_LIST_HEAD (&domain->group_list); iommu->external_domain = domain; +vfio_pgsize_bitmap (iommu); }else {kfree (domain); }
@@ -1793,6 +1806,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data, }list_add (&domain->next, &iommu->domain_list); +vfio_pgsize_bitmap (iommu); done:/* Delete the old one and insert new iova list */ vfio_iommu_iova_insert_copy (iommu, &iova_copy);
@@ -2004,6 +2018,7 @@ static void vfio_iommu_type1_detach_group(void *iommu_data,list_del (&domain->next);kfree (domain);vfio_iommu_aper_expand (iommu, &iova_copy); +vfio_pgsize_bitmap (iommu); }break ; }
@@ -2136,8 +2151,6 @@ static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu,size_t size;int iovas =0 , i =0 , ret; -mutex_lock (&iommu->lock); -list_for_each_entry (iova, &iommu->iova_list, list) iovas++;
@@ -2146,17 +2159,14 @@ static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu,* Return 0 as a container with a single mdev device * will have an empty list */ - ret =0 ; - goto out_unlock; +return 0 ; } size =sizeof (*cap_iovas) + (iovas *sizeof (*cap_iovas->iova_ranges)); cap_iovas =kzalloc (size,GFP_KERNEL ); -if (!cap_iovas) { - ret = -ENOMEM ; - goto out_unlock; - } +if (!cap_iovas) +return -ENOMEM ; cap_iovas->nr_iovas = iovas;
@@ -2169,8 +2179,6 @@ static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu, ret =vfio_iommu_iova_add_cap (caps, cap_iovas, size);kfree (cap_iovas); -out_unlock: -mutex_unlock (&iommu->lock);return ret; }
@@ -2215,11 +2223,13 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, info.cap_offset =0 ;/* output, no-recopy necessary */ } +mutex_lock (&iommu->lock); info.flags =VFIO_IOMMU_INFO_PGSIZES ; - info.iova_pgsizes =vfio_pgsize_bitmap (iommu); + info.iova_pgsizes = iommu->pgsize_bitmap; ret =vfio_iommu_iova_build_caps (iommu, &caps); +mutex_unlock (&iommu->lock);if (ret)return ret;
Signature · 1 line
2.7.0
PATCH v20 4/8 vfio iommu: Add ioctl definition for dirty pages tracking
In reply to Kirti WankhedeIOMMU container maintains a list of all pages pinned by vfio_pin_pages API. All pages pinned by vendor driver through this API should be considered as dirty during migration. When container consists of IOMMU capable device and all pages are pinned and mapped, then all pages are marked dirty. Added support to start/stop dirtied pages tracking and to get bitmap of all dirtied pages for requested IO virtual address range.
--- include/uapi/linux/vfio.h | 55 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)
include/uapi/linux/vfio.h +55 -0
@@ -1033,6 +1033,12 @@ struct vfio_iommu_type1_dma_map {#define VFIO_IOMMU_MAP_DMA _IO(VFIO_TYPE, VFIO_BASE + 13) +struct vfio_bitmap { +__u64 pgsize;/* page size for bitmap in bytes */ +__u64 size;/* in bytes */ +__u64 __user *data;/* one bit per page */ +}; +/** * VFIO_IOMMU_UNMAP_DMA - _IOWR(VFIO_TYPE, VFIO_BASE + 14, * struct vfio_dma_unmap)
@@ -1059,6 +1065,55 @@ struct vfio_iommu_type1_dma_unmap {#define VFIO_IOMMU_ENABLE _IO(VFIO_TYPE, VFIO_BASE + 15)#define VFIO_IOMMU_DISABLE _IO(VFIO_TYPE, VFIO_BASE + 16) +/** +* VFIO_IOMMU_DIRTY_PAGES - _IOWR(VFIO_TYPE, VFIO_BASE + 17, +* struct vfio_iommu_type1_dirty_bitmap) +* IOCTL is used for dirty pages tracking. +* Caller should set flag depending on which operation to perform, details as +* below: +* +* When IOCTL is called with VFIO_IOMMU_DIRTY_PAGES_FLAG_START set, indicates +* migration is active and IOMMU module should track pages which are dirtied or +* potentially dirtied by device. +* Dirty pages are tracked until tracking is stopped by user application by +* setting VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP flag. +* +* When IOCTL is called with VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP set, indicates +* IOMMU should stop tracking dirtied pages. +* +* When IOCTL is called with VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP flag set, +* IOCTL returns dirty pages bitmap for IOMMU container during migration for +* given IOVA range. User must provide data[] as the structure +* vfio_iommu_type1_dirty_bitmap_get through which user provides IOVA range and +* pgsize. This interface supports to get bitmap of smallest supported pgsize +* only and can be modified in future to get bitmap of specified pgsize. +* User must allocate memory for bitmap, zero the bitmap memory and set size +* of allocated memory in bitmap.size field. One bit is used to represent one +* page consecutively starting from iova offset. User should provide page size +* in bitmap.pgsize field. Bit set in bitmap indicates page at that offset from +* iova is dirty. Caller must set argsz including size of structure +* vfio_iommu_type1_dirty_bitmap_get. +* +* Only one of the flags _START, STOP and _GET may be specified at a time. +* +*/ +struct vfio_iommu_type1_dirty_bitmap { +__u32 argsz; +__u32 flags; +#define VFIO_IOMMU_DIRTY_PAGES_FLAG_START (1 << 0) +#define VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP (1 << 1) +#define VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP (1 << 2) +__u8 data[]; +}; + +struct vfio_iommu_type1_dirty_bitmap_get { +__u64 iova;/* IO virtual address */ +__u64 size;/* Size of iova range */ +struct vfio_bitmap bitmap; +}; + +#define VFIO_IOMMU_DIRTY_PAGES _IO(VFIO_TYPE, VFIO_BASE + 17) +/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */ /*
Signature · 1 line
2.7.0
Re: [PATCH Kernel v20 4/8] vfio iommu: Add ioctl definition for dirty pages tracking
Quoted 34 lines · On Fri, 15 May 2020 02:07:43 +0530 Kirti Wankhede <[email protected]> wrote:
IOMMU container maintains a list of all pages pinned by
vfio_pin_pagesAPI. All pages pinned by vendor driver through this API should be considered as dirty during migration. When container consists of IOMMU capable device and all pages are pinned and mapped, then all pages are marked dirty. Added support to start/stop dirtied pages tracking and to get bitmap of all dirtied pages for requested IO virtual address range.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- include/uapi/linux/vfio.h | 55 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)include/uapi/linux/vfio.h +6
-0@@ -1033,6 +1033,12 @@ struct vfio_iommu_type1_dma_map {#define VFIO_IOMMU_MAP_DMA _IO(VFIO_TYPE, VFIO_BASE + 13) +struct vfio_bitmap { +__u64 pgsize;/* page size for bitmap in bytes */ +__u64 size;/* in bytes */ +__u64 __user *data;/* one bit per page */ +}; +/** * VFIO_IOMMU_UNMAP_DMA - _IOWR(VFIO_TYPE, VFIO_BASE + 14, * struct vfio_dma_unmap) @@ -1059,6 +1065,55 @@ struct vfio_iommu_type1_dma_unmap {#define VFIO_IOMMU_ENABLE _IO(VFIO_TYPE, VFIO_BASE + 15)#define VFIO_IOMMU_DISABLE _IO(VFIO_TYPE, VFIO_BASE + 16)
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 1077
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+/** +* VFIO_IOMMU_DIRTY_PAGES - _IOWR(VFIO_TYPE, VFIO_BASE + 17, +* struct vfio_iommu_type1_dirty_bitmap) +* IOCTL is used for dirty pages tracking. +* Caller should set flag depending on which operation to perform, details as +* below: +* +* When IOCTL is called with VFIO_IOMMU_DIRTY_PAGES_FLAG_START set, indicates +* migration is active and IOMMU module should track pages which are dirtied or +* potentially dirtied by device.
"Calling the IOCTL with VFIO_IOMMU_DIRTY_PAGES_FLAG_START instructs the IOMMU driver to track pages that are dirtied or potentially dirtied by the device; designed to be used when a migration is in progress."
?
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 1079
Quoted 2 lines
include/uapi/linux/vfio.h +2
-0+* Dirty pages are tracked until tracking is stopped by user application by +* setting VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP flag.
"...by calling the IOCTL with VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP." ?
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 1082
Quoted 3 lines
include/uapi/linux/vfio.h +3
-0+* +* When IOCTL is called with VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP set, indicates +* IOMMU should stop tracking dirtied pages.
"Calling the IOCTL with VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP instructs the IOMMU driver to stop tracking dirtied pages."
?
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap {
Quoted 4 lines
include/uapi/linux/vfio.h +4
-0+* +* When IOCTL is called with VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP flag set, +* IOCTL returns dirty pages bitmap for IOMMU container during migration for +* given IOVA range.
"Calling the IOCTL with VFIO_IOMMU_DIRTY_PAGES_GET_BITMAP returns the dirty pages bitmap for the IOMMU container for a given IOVA range." ?
Q: How does this interact with the two other operations? I imagine getting an empty bitmap before _START and a bitmap-in-progress between _START and _STOP. After _STOP, will subsequent calls always give the same bitmap?
Quoted 3 lines
User must provide data[] as the structure
+ *vfio_iommu_type1_dirty_bitmap_getthrough which user provides IOVA range and
+ * pgsize.
"The user must specify the IOVA range and the pgsize through the vfio_iommu_type1_dirty_bitmap_get structure in the data[] portion."
?
Quoted 2 lines
This interface supports to get bitmap of smallest supported pgsize
+ * only and can be modified in future to get bitmap of specified pgsize.
That's a current restriction? How can the user find out whether it has been lifted (or, more generally, find out which pgsize values are supported)?
Quoted 2 lines
+ * User must allocate memory for bitmap, zero the bitmap memory and set size
+ * of allocated memory in bitmap.size field.
"The user must provide a zeroed memory area for the bitmap memory and specify its size in bitmap.size."
?
Quoted 3 lines
One bit is used to represent one
+ * page consecutively starting from iova offset. User should provide page size
+ * in bitmap.pgsize field.
s/User/The user/
Is that the 'pgsize' the comment above talks about?
Quoted 2 lines
Bit set in bitmap indicates page at that offset from
+ * iova is dirty.
"A bit set in the bitmap indicates that the page at that offset from iova is dirty." ?
Quoted 2 lines
Caller must set argsz including size of structure
+ *vfio_iommu_type1_dirty_bitmap_get.
s/Caller/The caller/
Does argz also include the size of the bitmap?
Quoted 2 lines
+ *
+ * Only one of the flags_START, STOP and_GETmay be specified at a time.
s/STOP/_STOP/
(just to be consistent)
Quoted 6 lines
+ *
+ */include/uapi/linux/vfio.h +4
-0+struct vfio_iommu_type1_dirty_bitmap { +__u32 argsz; +__u32 flags; +#define VFIO_IOMMU_DIRTY_PAGES_FLAG_START (1 << 0)
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 1113
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+#define VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP (1 << 1) +#define VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP (1 << 2) +__u8 data[]; +}; + +struct vfio_iommu_type1_dirty_bitmap_get { +__u64 iova;/* IO virtual address */ +__u64 size;/* Size of iova range */ +struct vfio_bitmap bitmap; +};
That's for type1 only, right?
Quoted 6 lines
include/uapi/linux/vfio.h +3
-0+ +#define VFIO_IOMMU_DIRTY_PAGES _IO(VFIO_TYPE, VFIO_BASE + 17) +/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */ /*
Re: [PATCH Kernel v20 4/8] vfio iommu: Add ioctl definition for dirty pages tracking
Quoted 52 lines · Cornelia Huck
Quoted 34 lines · On Fri, 15 May 2020 02:07:43 +0530 Kirti Wankhede <[email protected]> wrote:
IOMMU container maintains a list of all pages pinned by
vfio_pin_pagesAPI. All pages pinned by vendor driver through this API should be considered as dirty during migration. When container consists of IOMMU capable device and all pages are pinned and mapped, then all pages are marked dirty. Added support to start/stop dirtied pages tracking and to get bitmap of all dirtied pages for requested IO virtual address range.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- include/uapi/linux/vfio.h | 55 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)include/uapi/linux/vfio.h +6
-0@@ -1033,6 +1033,12 @@ struct vfio_iommu_type1_dma_map {#define VFIO_IOMMU_MAP_DMA _IO(VFIO_TYPE, VFIO_BASE + 13) +struct vfio_bitmap { +__u64 pgsize;/* page size for bitmap in bytes */ +__u64 size;/* in bytes */ +__u64 __user *data;/* one bit per page */ +}; +/** * VFIO_IOMMU_UNMAP_DMA - _IOWR(VFIO_TYPE, VFIO_BASE + 14, * struct vfio_dma_unmap) @@ -1059,6 +1065,55 @@ struct vfio_iommu_type1_dma_unmap {#define VFIO_IOMMU_ENABLE _IO(VFIO_TYPE, VFIO_BASE + 15)#define VFIO_IOMMU_DISABLE _IO(VFIO_TYPE, VFIO_BASE + 16)include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 1077
Quoted 10 lines
include/uapi/linux/vfio.h +10
-0+/** +* VFIO_IOMMU_DIRTY_PAGES - _IOWR(VFIO_TYPE, VFIO_BASE + 17, +* struct vfio_iommu_type1_dirty_bitmap) +* IOCTL is used for dirty pages tracking. +* Caller should set flag depending on which operation to perform, details as +* below: +* +* When IOCTL is called with VFIO_IOMMU_DIRTY_PAGES_FLAG_START set, indicates +* migration is active and IOMMU module should track pages which are dirtied or +* potentially dirtied by device. "Calling the IOCTL with
VFIO_IOMMU_DIRTY_PAGES_FLAG_STARTinstructs the IOMMU driver to track pages that are dirtied or potentially dirtied by the device; designed to be used when a migration is in progress."
Quoted 2 lines
?
Ok, updating.
Quoted 5 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 1079
Quoted 2 lines
include/uapi/linux/vfio.h +2
-0+* Dirty pages are tracked until tracking is stopped by user application by +* setting VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP flag. "...by calling the IOCTL with
VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP." ?
Quoted 9 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 1082
Quoted 3 lines
include/uapi/linux/vfio.h +3
-0+* +* When IOCTL is called with VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP set, indicates +* IOMMU should stop tracking dirtied pages. "Calling the IOCTL with
VFIO_IOMMU_DIRTY_PAGES_FLAG_STOPinstructs the IOMMU driver to stop tracking dirtied pages."?
Ok.
Quoted 10 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap {
Quoted 4 lines
include/uapi/linux/vfio.h +4
-0+* +* When IOCTL is called with VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP flag set, +* IOCTL returns dirty pages bitmap for IOMMU container during migration for +* given IOVA range. "Calling the IOCTL with
VFIO_IOMMU_DIRTY_PAGES_GET_BITMAPreturns the dirty pages bitmap for the IOMMU container for a given IOVA range." ?Q: How does this interact with the two other operations? I imagine getting an empty bitmap before
_START
No, if dirty page tracking is not started, get_bitmap IOCTL will fail with -EINVAL.
Quoted 4 lines
and a bitmap-in-progress between
_STARTand_STOP. > After_STOP, will subsequent calls always give the same bitmap?
No, return -EINVAL.
Quoted 7 lines
Quoted 3 lines
User must provide data[] as the structure
+ *vfio_iommu_type1_dirty_bitmap_getthrough which user provides IOVA range and
+ * pgsize."The user must specify the IOVA range and the pgsize through the
vfio_iommu_type1_dirty_bitmap_getstructure in the data[] portion."
Quoted 8 lines
?
Quoted 2 lines
This interface supports to get bitmap of smallest supported pgsize
+ * only and can be modified in future to get bitmap of specified pgsize.That's a current restriction? How can the user find out whether it has been lifted (or, more generally, find out which pgsize values are supported)?
Migration capability is added to IOMMU info chain. That gives supported pgsize bitmap by IOMMU driver.
Quoted 9 lines
Quoted 3 lines
+ * User must allocate memory for bitmap, zero the bitmap memory and set size
+ * of allocated memory in bitmap.size field."The user must provide a zeroed memory area for the bitmap memory and specify its size in bitmap.size."
?
Quoted 8 lines
Quoted 3 lines
One bit is used to represent one
+ * page consecutively starting from iova offset. User should provide page size
+ * in bitmap.pgsize field.s/User/The user/
Is that the 'pgsize' the comment above talks about?
By specifying pgsize here user can ask for bitmap of specific pgsize.
Quoted 6 lines
Quoted 2 lines
Bit set in bitmap indicates page at that offset from
+ * iova is dirty."A bit set in the bitmap indicates that the page at that offset from iova is dirty." ?
Quoted 6 lines
Quoted 2 lines
Caller must set argsz including size of structure
+ *vfio_iommu_type1_dirty_bitmap_get.s/Caller/The caller/
Does argz also include the size of the bitmap?
No.
Quoted 18 lines
Quoted 3 lines
+ *
+ * Only one of the flags_START, STOP and_GETmay be specified at a time.s/STOP/_STOP/
(just to be consistent)
Quoted 10 lines
+ *
+ */include/uapi/linux/vfio.h +8
-0+struct vfio_iommu_type1_dirty_bitmap { +__u32 argsz; +__u32 flags; +#define VFIO_IOMMU_DIRTY_PAGES_FLAG_START (1 << 0) +#define VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP (1 << 1) +#define VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP (1 << 2) +__u8 data[]; +};
Quoted 9 lines
include/uapi/linux/vfio.h struct vfio_iommu_type1_dma_unmap { · line 1113
Quoted 6 lines
include/uapi/linux/vfio.h +6
-0+ +struct vfio_iommu_type1_dirty_bitmap_get { +__u64 iova;/* IO virtual address */ +__u64 size;/* Size of iova range */ +struct vfio_bitmap bitmap; +};That's for type1 only, right?
Yes.
Thanks,
Kirti
Quoted 7 lines
Quoted 6 lines
include/uapi/linux/vfio.h +3
-0+ +#define VFIO_IOMMU_DIRTY_PAGES _IO(VFIO_TYPE, VFIO_BASE + 17) +/* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */ /*
Re: [PATCH Kernel v20 4/8] vfio iommu: Add ioctl definition for dirty pages tracking
Quoted 16 lines · On Fri, 15 May 2020 23:05:24 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 15 lines · Cornelia Huck
Quoted 12 lines · On Fri, 15 May 2020 02:07:43 +0530 Kirti Wankhede <[email protected]> wrote:
IOMMU container maintains a list of all pages pinned by
vfio_pin_pagesAPI. All pages pinned by vendor driver through this API should be considered as dirty during migration. When container consists of IOMMU capable device and all pages are pinned and mapped, then all pages are marked dirty. Added support to start/stop dirtied pages tracking and to get bitmap of all dirtied pages for requested IO virtual address range.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- include/uapi/linux/vfio.h | 55 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)
(...)
Quoted 13 lines
Quoted 9 lines
Quoted 3 lines
+ * When IOCTL is called with
VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAPflag set,
+ * IOCTL returns dirty pages bitmap for IOMMU container during migration for
+ * given IOVA range."Calling the IOCTL with
VFIO_IOMMU_DIRTY_PAGES_GET_BITMAPreturns the dirty pages bitmap for the IOMMU container for a given IOVA range." ?Q: How does this interact with the two other operations? I imagine getting an empty bitmap before
_STARTNo, if dirty page tracking is not started,
get_bitmapIOCTL will fail with-EINVAL.
Quoted 6 lines
Quoted 4 lines
and a bitmap-in-progress between
_STARTand_STOP. > After_STOP, will subsequent calls always give the same bitmap?No, return
-EINVAL.
Maybe add
"If the IOCTL has not yet been called with VFIO_IOMMU_DIRTY_PAGES_FLAG_START, or if it has been called with VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP, calling it with VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP will return -EINVAL." ?
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines
User must provide data[] as the structure
+ *vfio_iommu_type1_dirty_bitmap_getthrough which user provides IOVA range and
+ * pgsize."The user must specify the IOVA range and the pgsize through the
vfio_iommu_type1_dirty_bitmap_getstructure in the data[] portion."?
Quoted 2 lines
This interface supports to get bitmap of smallest supported pgsize
+ * only and can be modified in future to get bitmap of specified pgsize.
Quoted 6 lines
Quoted 3 lines
That's a current restriction? How can the user find out whether it has been lifted (or, more generally, find out which pgsize values are supported)?
Migration capability is added to IOMMU info chain. That gives supported pgsize bitmap by IOMMU driver.
Add that info?
"The supported pgsize values for this interface are reported via the migration capability in the IOMMU info chain."
Quoted 14 lines
Quoted 14 lines
Quoted 4 lines
+ * User must allocate memory for bitmap, zero the bitmap memory and set size
+ * of allocated memory in bitmap.size field."The user must provide a zeroed memory area for the bitmap memory and specify its size in bitmap.size."
?
Quoted 3 lines
One bit is used to represent one
+ * page consecutively starting from iova offset. User should provide page size
+ * in bitmap.pgsize field.
Quoted 6 lines
Quoted 4 lines
s/User/The user/
Is that the 'pgsize' the comment above talks about?
By specifying pgsize here user can ask for bitmap of specific pgsize.
"The user should provide the page size for the bitmap in the bitmap.pgsize field." ?
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Bit set in bitmap indicates page at that offset from
+ * iova is dirty."A bit set in the bitmap indicates that the page at that offset from iova is dirty." ?
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Caller must set argsz including size of structure
+ *vfio_iommu_type1_dirty_bitmap_get.s/Caller/The caller/
Does argz also include the size of the bitmap?
No.
"The caller must set argsz to a value including the size of stuct vfio_io_type1_dirty_bitmap_get, but excluding the size of the actual bitmap." ?
Re: [PATCH Kernel v20 4/8] vfio iommu: Add ioctl definition for dirty pages tracking
Quoted 40 lines · On Tue, 19 May 2020 17:35:07 +0200 Cornelia Huck <[email protected]> wrote:
Quoted 16 lines · On Fri, 15 May 2020 23:05:24 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 15 lines · Cornelia Huck
Quoted 12 lines · On Fri, 15 May 2020 02:07:43 +0530 Kirti Wankhede <[email protected]> wrote:
IOMMU container maintains a list of all pages pinned by
vfio_pin_pagesAPI. All pages pinned by vendor driver through this API should be considered as dirty during migration. When container consists of IOMMU capable device and all pages are pinned and mapped, then all pages are marked dirty. Added support to start/stop dirtied pages tracking and to get bitmap of all dirtied pages for requested IO virtual address range.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- include/uapi/linux/vfio.h | 55 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)(...)
Quoted 18 lines
Quoted 9 lines
Quoted 3 lines
+ * When IOCTL is called with
VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAPflag set,
+ * IOCTL returns dirty pages bitmap for IOMMU container during migration for
+ * given IOVA range."Calling the IOCTL with
VFIO_IOMMU_DIRTY_PAGES_GET_BITMAPreturns the dirty pages bitmap for the IOMMU container for a given IOVA range." ?Q: How does this interact with the two other operations? I imagine getting an empty bitmap before
_STARTNo, if dirty page tracking is not started,
get_bitmapIOCTL will fail with-EINVAL.Quoted 4 lines
and a bitmap-in-progress between
_STARTand_STOP. > After_STOP, will subsequent calls always give the same bitmap?
Quoted 8 lines
Quoted 1 line
No, return
-EINVAL.Maybe add
"If the IOCTL has not yet been called with
VFIO_IOMMU_DIRTY_PAGES_FLAG_START, or if it has been called withVFIO_IOMMU_DIRTY_PAGES_FLAG_STOP, calling it withVFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAPwill return-EINVAL." ?
Let's not specify ourselves into a corner, I think we can simply say that the dirty bitmap is only available while dirty logging is enabled. We certainly don't need to specify specific errno values that'll trip us up later.
Quoted 60 lines
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
User must provide data[] as the structure
+ *vfio_iommu_type1_dirty_bitmap_getthrough which user provides IOVA range and
+ * pgsize."The user must specify the IOVA range and the pgsize through the
vfio_iommu_type1_dirty_bitmap_getstructure in the data[] portion."?
Quoted 2 lines
This interface supports to get bitmap of smallest supported pgsize
+ * only and can be modified in future to get bitmap of specified pgsize.Quoted 6 lines
Quoted 3 lines
That's a current restriction? How can the user find out whether it has been lifted (or, more generally, find out which pgsize values are supported)?
Migration capability is added to IOMMU info chain. That gives supported pgsize bitmap by IOMMU driver.
Add that info?
"The supported pgsize values for this interface are reported via the migration capability in the IOMMU info chain."
Quoted 14 lines
Quoted 14 lines
Quoted 4 lines
+ * User must allocate memory for bitmap, zero the bitmap memory and set size
+ * of allocated memory in bitmap.size field."The user must provide a zeroed memory area for the bitmap memory and specify its size in bitmap.size."
?
Quoted 3 lines
One bit is used to represent one
+ * page consecutively starting from iova offset. User should provide page size
+ * in bitmap.pgsize field.Quoted 6 lines
Quoted 4 lines
s/User/The user/
Is that the 'pgsize' the comment above talks about?
By specifying pgsize here user can ask for bitmap of specific pgsize.
"The user should provide the page size for the bitmap in the bitmap.pgsize field." ?
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
Bit set in bitmap indicates page at that offset from
+ * iova is dirty."A bit set in the bitmap indicates that the page at that offset from iova is dirty." ?
Quoted 2 lines
Caller must set argsz including size of structure
+ *vfio_iommu_type1_dirty_bitmap_get.s/Caller/The caller/
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Does argz also include the size of the bitmap?
No.
"The caller must set argsz to a value including the size of stuct
vfio_io_type1_dirty_bitmap_get, but excluding the size of the actual bitmap." ?
Yes, it wouldn't make sense for argsz to include the size of the bitmap itself, that's accessed independently via a user provided pointer and we have a separate size field for that. Thanks,
Alex
Re: [PATCH Kernel v20 4/8] vfio iommu: Add ioctl definition for dirty pages tracking
Quoted 52 lines · On Tue, 19 May 2020 09:53:56 -0600 Alex Williamson…
Quoted 40 lines · On Tue, 19 May 2020 17:35:07 +0200 Cornelia Huck <[email protected]> wrote:
Quoted 16 lines · On Fri, 15 May 2020 23:05:24 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 15 lines · Cornelia Huck
Quoted 12 lines · On Fri, 15 May 2020 02:07:43 +0530 Kirti Wankhede <[email protected]> wrote:
IOMMU container maintains a list of all pages pinned by
vfio_pin_pagesAPI. All pages pinned by vendor driver through this API should be considered as dirty during migration. When container consists of IOMMU capable device and all pages are pinned and mapped, then all pages are marked dirty. Added support to start/stop dirtied pages tracking and to get bitmap of all dirtied pages for requested IO virtual address range.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- include/uapi/linux/vfio.h | 55 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)(...)
Quoted 18 lines
Quoted 9 lines
Quoted 3 lines
+ * When IOCTL is called with
VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAPflag set,
+ * IOCTL returns dirty pages bitmap for IOMMU container during migration for
+ * given IOVA range."Calling the IOCTL with
VFIO_IOMMU_DIRTY_PAGES_GET_BITMAPreturns the dirty pages bitmap for the IOMMU container for a given IOVA range." ?Q: How does this interact with the two other operations? I imagine getting an empty bitmap before
_STARTNo, if dirty page tracking is not started,
get_bitmapIOCTL will fail with-EINVAL.Quoted 4 lines
and a bitmap-in-progress between
_STARTand_STOP. > After_STOP, will subsequent calls always give the same bitmap?Quoted 8 lines
Quoted 1 line
No, return
-EINVAL.Maybe add
"If the IOCTL has not yet been called with
VFIO_IOMMU_DIRTY_PAGES_FLAG_START, or if it has been called withVFIO_IOMMU_DIRTY_PAGES_FLAG_STOP, calling it withVFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAPwill return-EINVAL." ?
Quoted 4 lines
Let's not specify ourselves into a corner, I think we can simply say that the dirty bitmap is only available while dirty logging is enabled. We certainly don't need to specify specific errno values that'll trip us up later.
"If dirty logging is not enabled, an error will be returned." ?
(...)
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Caller must set argsz including size of structure
+ *vfio_iommu_type1_dirty_bitmap_get.s/Caller/The caller/
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Does argz also include the size of the bitmap?
No.
"The caller must set argsz to a value including the size of stuct
vfio_io_type1_dirty_bitmap_get, but excluding the size of the actual bitmap." ?
Quoted 5 lines
Yes, it wouldn't make sense for argsz to include the size of the bitmap itself, that's accessed independently via a user provided pointer and we have a separate size field for that. Thanks,
Alex
Yes, I just wanted to make it as obvious as possible to make it easier for folks trying to interact with this interface.
PATCH v20 5/8 vfio iommu: Implementation of ioctl for dirty pages tracking
In reply to Kirti WankhedeVFIO_IOMMU_DIRTY_PAGES ioctl performs three operations:
- Start dirty pages tracking while migration is active
- Stop dirty pages tracking.
- Get dirty pages bitmap. Its user space application's responsibility to copy content of dirty pages from source to destination during migration.
To prevent DoS attack, memory for bitmap is allocated per vfio_dma structure. Bitmap size is calculated considering smallest supported page size. Bitmap is allocated for all vfio_dmas when dirty logging is enabled
Bitmap is populated for already pinned pages when bitmap is allocated for a vfio_dma with the smallest supported page size. Update bitmap from pinning functions when tracking is enabled. When user application queries bitmap, check if requested page size is same as page size used to populated bitmap. If it is equal, copy bitmap, but if not equal, return error.
Fixed error reported by build bot by changing pgsize type from uint64_t to size_t.
--- drivers/vfio/vfio_iommu_type1.c | 294 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 288 insertions(+), 6 deletions(-)
drivers/vfio/vfio_iommu_type1.c +288 -6
@@ -72,6 +72,7 @@ struct vfio_iommu {uint64_t pgsize_bitmap;bool v2;bool nesting; +bool dirty_page_tracking; };struct vfio_domain {
@@ -92,6 +93,7 @@ struct vfio_dma {bool lock_cap;/* capable(CAP_IPC_LOCK) */ struct task_struct *task;struct rb_root pfn_list;/* Ex-user pinned pfn list */ +unsigned long *bitmap; };struct vfio_group {
@@ -126,6 +128,19 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +#define DIRTY_BITMAP_BYTES (n) (ALIGN(n, BITS_PER_TYPE(u64)) / BITS_PER_BYTE) + +/* +* Input argument of number of bits to bitmap_set() is unsigned integer, which +* further casts to signed integer for unaligned multi-bit operation, +* __bitmap_set(). +* Then maximum bitmap size supported is 2^31 bits divided by 2^3 bits/byte, +* that is 2^28 (256 MB) which maps to 2^31 * 2^12 = 2^43 (8TB) on 4K page +* system. +*/ +#define DIRTY_BITMAP_PAGES_MAX ((u64)INT_MAX) +#define DIRTY_BITMAP_SIZE_MAX DIRTY_BITMAP_BYTES(DIRTY_BITMAP_PAGES_MAX) +static int put_pfn (unsigned long pfn,int prot);/*
@@ -176,6 +191,74 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)rb_erase (&old->node, &iommu->dma_list); } + +static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) +{ +uint64_t npages = dma->size / pgsize; + +if (npages >DIRTY_BITMAP_PAGES_MAX ) +return -EINVAL ; + + dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +if (!dma->bitmap) +return -ENOMEM ; + +return 0 ; +} + +static void vfio_dma_bitmap_free (struct vfio_dma *dma) +{ +kfree (dma->bitmap); + dma->bitmap =NULL ; +} + +static void vfio_dma_populate_bitmap (struct vfio_dma *dma,size_t pgsize) +{ +struct rb_node *p; + +for (p =rb_first (&dma->pfn_list); p; p =rb_next (p)) { + struct vfio_pfn *vpfn = rb_entry(p, struct vfio_pfn, node); + +bitmap_set (dma->bitmap, (vpfn->iova - dma->iova) / pgsize,1 ); + } +} + +static int vfio_dma_bitmap_alloc_all (struct vfio_iommu *iommu,size_t pgsize) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); +int ret; + + ret =vfio_dma_bitmap_alloc (dma, pgsize); +if (ret) { +struct rb_node *p =rb_prev (n); + +for (; p; p =rb_prev (p)) { +struct vfio_dma *dma =rb_entry (n, + struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +return ret; + } +vfio_dma_populate_bitmap (dma, pgsize); + } +return 0 ; +} + +static void vfio_dma_bitmap_free_all (struct vfio_iommu *iommu) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +} +/* * Helper Functions for host iova-pfn list */
@@ -568,6 +651,17 @@ static int vfio_iommu_type1_pin_pages(void *iommu_data,vfio_unpin_page_external (dma, iova, do_accounting); goto pin_unwind; } + +if (iommu->dirty_page_tracking) { +unsigned long pgshift =__ffs (iommu->pgsize_bitmap); + +/* +* Bitmap populated with the smallest supported page +* size +*/ +bitmap_set (dma->bitmap, + (vpfn->iova - dma->iova) >> pgshift,1 ); + } } ret = i;
@@ -802,6 +896,7 @@ static void vfio_remove_dma(struct vfio_iommu *iommu, struct vfio_dma *dma)vfio_unmap_unpin (iommu, dma, true);vfio_unlink_dma (iommu, dma);put_task_struct (dma->task); +vfio_dma_bitmap_free (dma);kfree (dma); iommu->dma_avail++; }
@@ -829,6 +924,80 @@ static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) } } +static int update_user_bitmap (u64 __user *bitmap,struct vfio_dma *dma, +dma_addr_t base_iova,size_t pgsize) +{ +unsigned long pgshift =__ffs (pgsize); +unsigned long nbits = dma->size >> pgshift; +unsigned long bit_offset = (dma->iova - base_iova) >> pgshift; +unsigned long copy_offset = bit_offset /BITS_PER_LONG ; +unsigned long shift = bit_offset %BITS_PER_LONG ; +unsigned long leftover; + +if (shift) { +bitmap_shift_left (dma->bitmap, dma->bitmap, shift, + nbits + shift); + +if (copy_from_user (&leftover, (u64 *)bitmap + copy_offset, +sizeof (leftover))) +return -EFAULT ; + +bitmap_or (dma->bitmap, dma->bitmap, &leftover, shift); + } + +if (copy_to_user ((u64 *)bitmap + copy_offset, dma->bitmap, +DIRTY_BITMAP_BYTES (nbits + shift))) +return -EFAULT ; + +return 0 ; +} + +static int vfio_iova_dirty_bitmap (u64 __user *bitmap,struct vfio_iommu *iommu, +dma_addr_t iova,size_t size,size_t pgsize) +{ +struct vfio_dma *dma; +dma_addr_t i = iova, limit = iova + size; +unsigned long pgshift =__ffs (pgsize); +size_t sz = size; +int ret; + +while ((dma =vfio_find_dma (iommu, i, sz))) { + +/* mark all pages dirty if all pages are pinned and mapped. */ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , dma->size >> pgshift); + + ret =update_user_bitmap (bitmap, dma, iova, pgsize); +if (ret) +return ret; + +/* +* Re-populate bitmap to include all pinned pages which are +* considered as dirty but exclude pages which are unpinned and +* pages which are marked dirty by vfio_dma_rw() +*/ +bitmap_clear (dma->bitmap,0 , dma->size >> pgshift); +vfio_dma_populate_bitmap (dma, pgsize); + + i = dma->iova + dma->size; +if (i >= limit) +break ; + + sz = limit - i; + } + +return 0 ; +} + +static int verify_bitmap_size (uint64_t npages,uint64_t bitmap_size) +{ +if (!npages || !bitmap_size || (bitmap_size >DIRTY_BITMAP_SIZE_MAX ) || + (bitmap_size <DIRTY_BITMAP_BYTES (npages))) +return -EINVAL ; + +return 0 ; +} +static int vfio_dma_do_unmap (struct vfio_iommu *iommu,struct vfio_iommu_type1_dma_unmap *unmap) {
@@ -1046,7 +1215,7 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,unsigned long vaddr = map->vaddr;size_t size = map->size;int ret =0 , prot =0 ; -uint64_t mask; +size_t pgsize;struct vfio_dma *dma;/* Verify that none of our __u64 fields overflow */
@@ -1061,11 +1230,11 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgsize = (size_t )1 <<__ffs (iommu->pgsize_bitmap); -WARN_ON (mask &PAGE_MASK ); +WARN_ON ((pgsize -1 ) &PAGE_MASK ); -if (!prot || !size || (size | iova | vaddr) & mask) { +if (!prot || !size || (size | iova | vaddr) & (pgsize -1 )) { ret = -EINVAL ; goto out_unlock; }
@@ -1142,6 +1311,12 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,else ret =vfio_pin_map_dma (iommu, dma, size); +if (!ret && iommu->dirty_page_tracking) { + ret =vfio_dma_bitmap_alloc (dma, pgsize); +if (ret) +vfio_remove_dma (iommu, dma); + } + out_unlock:mutex_unlock (&iommu->lock);return ret;
@@ -2288,6 +2463,104 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,return copy_to_user ((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + }else if (cmd ==VFIO_IOMMU_DIRTY_PAGES ) { +struct vfio_iommu_type1_dirty_bitmap dirty; +uint32_t mask =VFIO_IOMMU_DIRTY_PAGES_FLAG_START | +VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP | +VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP ; +int ret =0 ; + +if (!iommu->v2) +return -EACCES ; + + minsz = offsetofend(struct vfio_iommu_type1_dirty_bitmap, + flags); + +if (copy_from_user (&dirty, (void __user *)arg, minsz)) +return -EFAULT ; + +if (dirty.argsz < minsz || dirty.flags & ~mask) +return -EINVAL ; + +/* only one flag should be set at a time */ +if (__ffs (dirty.flags) !=__fls (dirty.flags)) +return -EINVAL ; + +if (dirty.flags &VFIO_IOMMU_DIRTY_PAGES_FLAG_START ) { +size_t pgsize; + +mutex_lock (&iommu->lock); + pgsize =1 <<__ffs (iommu->pgsize_bitmap); +if (!iommu->dirty_page_tracking) { + ret =vfio_dma_bitmap_alloc_all (iommu, pgsize); +if (!ret) + iommu->dirty_page_tracking = true; + } +mutex_unlock (&iommu->lock); +return ret; + }else if (dirty.flags &VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP ) { +mutex_lock (&iommu->lock); +if (iommu->dirty_page_tracking) { + iommu->dirty_page_tracking = false; +vfio_dma_bitmap_free_all (iommu); + } +mutex_unlock (&iommu->lock); +return 0 ; + }else if (dirty.flags & +VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP ) { +struct vfio_iommu_type1_dirty_bitmap_get range; +unsigned long pgshift; +size_t data_size = dirty.argsz - minsz; +size_t iommu_pgsize; + +if (!data_size || data_size <sizeof (range)) +return -EINVAL ; + +if (copy_from_user (&range, (void __user *)(arg + minsz), +sizeof (range))) +return -EFAULT ; + +if (range.iova + range.size < range.iova) +return -EINVAL ; +if (!access_ok ((void __user *)range.bitmap.data, + range.bitmap.size)) +return -EINVAL ; + + pgshift =__ffs (range.bitmap.pgsize); + ret =verify_bitmap_size (range.size >> pgshift, + range.bitmap.size); +if (ret) +return ret; + +mutex_lock (&iommu->lock); + + iommu_pgsize = (size_t )1 <<__ffs (iommu->pgsize_bitmap); + +/* allow only smallest supported pgsize */ +if (range.bitmap.pgsize != iommu_pgsize) { + ret = -EINVAL ; + goto out_unlock; + } +if (range.iova & (iommu_pgsize -1 )) { + ret = -EINVAL ; + goto out_unlock; + } +if (!range.size || range.size & (iommu_pgsize -1 )) { + ret = -EINVAL ; + goto out_unlock; + } + +if (iommu->dirty_page_tracking) + ret =vfio_iova_dirty_bitmap (range.bitmap.data, + iommu, range.iova, range.size, + range.bitmap.pgsize); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); + +return ret; + } }return -ENOTTY ;
@@ -2355,10 +2628,19 @@ static int vfio_iommu_type1_dma_rw_chunk(struct vfio_iommu *iommu, vaddr = dma->vaddr + offset; -if (write) +if (write) { *copied =copy_to_user ((void __user *)vaddr, data, count) ?0 : count; -else +if (*copied && iommu->dirty_page_tracking) { +unsigned long pgshift =__ffs (iommu->pgsize_bitmap); +/* +* Bitmap populated with the smallest supported page +* size +*/ +bitmap_set (dma->bitmap, offset >> pgshift, + *copied >> pgshift); + } + }else *copied =copy_from_user (data, (void __user *)vaddr, count) ?0 : count;if (kthread)
Signature · 1 line
2.7.0
Re: [PATCH Kernel v20 5/8] vfio iommu: Implementation of ioctl for dirty pages tracking
Quoted 200 lines · On Fri, 15 May 2020 02:07:44 +0530 Kirti Wankhede <[email protected]> wrote:
VFIO_IOMMU_DIRTY_PAGESioctl performs three operations:
- Start dirty pages tracking while migration is active
- Stop dirty pages tracking.
- Get dirty pages bitmap. Its user space application's responsibility to copy content of dirty pages from source to destination during migration.To prevent DoS attack, memory for bitmap is allocated per
vfio_dmastructure. Bitmap size is calculated considering smallest supported page size. Bitmap is allocated for allvfio_dmaswhen dirty logging is enabledBitmap is populated for already pinned pages when bitmap is allocated for a
vfio_dmawith the smallest supported page size. Update bitmap from pinning functions when tracking is enabled. When user application queries bitmap, check if requested page size is same as page size used to populated bitmap. If it is equal, copy bitmap, but if not equal, return error.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>Fixed error reported by build bot by changing pgsize type from
uint64_ttosize_t.Reported-by kbuild test robot <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 294 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 288 insertions(+), 6 deletions(-)drivers/vfio/vfio_iommu_type1.c +123
-0@@ -72,6 +72,7 @@ struct vfio_iommu {uint64_t pgsize_bitmap;bool v2;bool nesting; +bool dirty_page_tracking; };struct vfio_domain {@@ -92,6 +93,7 @@ struct vfio_dma {bool lock_cap;/* capable(CAP_IPC_LOCK) */ struct task_struct *task;struct rb_root pfn_list;/* Ex-user pinned pfn list */ +unsigned long *bitmap; };struct vfio_group {@@ -126,6 +128,19 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +#define DIRTY_BITMAP_BYTES (n) (ALIGN(n, BITS_PER_TYPE(u64)) / BITS_PER_BYTE) + +/* +* Input argument of number of bits to bitmap_set() is unsigned integer, which +* further casts to signed integer for unaligned multi-bit operation, +* __bitmap_set(). +* Then maximum bitmap size supported is 2^31 bits divided by 2^3 bits/byte, +* that is 2^28 (256 MB) which maps to 2^31 * 2^12 = 2^43 (8TB) on 4K page +* system. +*/ +#define DIRTY_BITMAP_PAGES_MAX ((u64)INT_MAX) +#define DIRTY_BITMAP_SIZE_MAX DIRTY_BITMAP_BYTES(DIRTY_BITMAP_PAGES_MAX) +static int put_pfn (unsigned long pfn,int prot);/* @@ -176,6 +191,74 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)rb_erase (&old->node, &iommu->dma_list); } + +static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) +{ +uint64_t npages = dma->size / pgsize; + +if (npages >DIRTY_BITMAP_PAGES_MAX ) +return -EINVAL ; + + dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +if (!dma->bitmap) +return -ENOMEM ; + +return 0 ; +} + +static void vfio_dma_bitmap_free (struct vfio_dma *dma) +{ +kfree (dma->bitmap); + dma->bitmap =NULL ; +} + +static void vfio_dma_populate_bitmap (struct vfio_dma *dma,size_t pgsize) +{ +struct rb_node *p; + +for (p =rb_first (&dma->pfn_list); p; p =rb_next (p)) { + struct vfio_pfn *vpfn = rb_entry(p, struct vfio_pfn, node); + +bitmap_set (dma->bitmap, (vpfn->iova - dma->iova) / pgsize,1 ); + } +} + +static int vfio_dma_bitmap_alloc_all (struct vfio_iommu *iommu,size_t pgsize) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); +int ret; + + ret =vfio_dma_bitmap_alloc (dma, pgsize); +if (ret) { +struct rb_node *p =rb_prev (n); + +for (; p; p =rb_prev (p)) { +struct vfio_dma *dma =rb_entry (n, + struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +return ret; + } +vfio_dma_populate_bitmap (dma, pgsize); + } +return 0 ; +} + +static void vfio_dma_bitmap_free_all (struct vfio_iommu *iommu) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +} +/* * Helper Functions for host iova-pfn list */ @@ -568,6 +651,17 @@ static int vfio_iommu_type1_pin_pages(void *iommu_data,vfio_unpin_page_external (dma, iova, do_accounting); goto pin_unwind; } + +if (iommu->dirty_page_tracking) { +unsigned long pgshift =__ffs (iommu->pgsize_bitmap); + +/* +* Bitmap populated with the smallest supported page +* size +*/ +bitmap_set (dma->bitmap, + (vpfn->iova - dma->iova) >> pgshift,1 ); + } } ret = i;@@ -802,6 +896,7 @@ static void vfio_remove_dma(struct vfio_iommu *iommu, struct vfio_dma *dma)vfio_unmap_unpin (iommu, dma, true);vfio_unlink_dma (iommu, dma);put_task_struct (dma->task); +vfio_dma_bitmap_free (dma);kfree (dma); iommu->dma_avail++; }@@ -829,6 +924,80 @@ static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) } } +static int update_user_bitmap (u64 __user *bitmap,struct vfio_dma *dma, +dma_addr_t base_iova,size_t pgsize) +{ +unsigned long pgshift =__ffs (pgsize); +unsigned long nbits = dma->size >> pgshift; +unsigned long bit_offset = (dma->iova - base_iova) >> pgshift; +unsigned long copy_offset = bit_offset /BITS_PER_LONG ; +unsigned long shift = bit_offset %BITS_PER_LONG ; +unsigned long leftover; + +if (shift) { +bitmap_shift_left (dma->bitmap, dma->bitmap, shift, + nbits + shift); + +if (copy_from_user (&leftover, (u64 *)bitmap + copy_offset, +sizeof (leftover))) +return -EFAULT ; + +bitmap_or (dma->bitmap, dma->bitmap, &leftover, shift); + } + +if (copy_to_user ((u64 *)bitmap + copy_offset, dma->bitmap, +DIRTY_BITMAP_BYTES (nbits + shift))) +return -EFAULT ; + +return 0 ; +} +
drivers/vfio/vfio_iommu_type1.c static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) · line 964
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+static int vfio_iova_dirty_bitmap (u64 __user *bitmap,struct vfio_iommu *iommu, +dma_addr_t iova,size_t size,size_t pgsize) +{ +struct vfio_dma *dma; +dma_addr_t i = iova, limit = iova + size; +unsigned long pgshift =__ffs (pgsize); +size_t sz = size; +int ret; + +while ((dma =vfio_find_dma (iommu, i, sz))) {
This would be a lot more efficient if we walked it as:
struct rb_node *n = rb_first(&iommu->dma_list);
for (; n; n = rb_next(n)) {
struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node);
if (dma->iova < iova)
continue;
if (dma->iova >= iova + size)
break;ie. there's no need to start from the beginning of the tree on each iteration. This I would accept as a follow-on internal implementation improvement. Thanks,
Alex
Quoted 203 lines
drivers/vfio/vfio_iommu_type1.c +155
-6+ +/* mark all pages dirty if all pages are pinned and mapped. */ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , dma->size >> pgshift); + + ret =update_user_bitmap (bitmap, dma, iova, pgsize); +if (ret) +return ret; + +/* +* Re-populate bitmap to include all pinned pages which are +* considered as dirty but exclude pages which are unpinned and +* pages which are marked dirty by vfio_dma_rw() +*/ +bitmap_clear (dma->bitmap,0 , dma->size >> pgshift); +vfio_dma_populate_bitmap (dma, pgsize); + + i = dma->iova + dma->size; +if (i >= limit) +break ; + + sz = limit - i; + } + +return 0 ; +} + +static int verify_bitmap_size (uint64_t npages,uint64_t bitmap_size) +{ +if (!npages || !bitmap_size || (bitmap_size >DIRTY_BITMAP_SIZE_MAX ) || + (bitmap_size <DIRTY_BITMAP_BYTES (npages))) +return -EINVAL ; + +return 0 ; +} +static int vfio_dma_do_unmap (struct vfio_iommu *iommu,struct vfio_iommu_type1_dma_unmap *unmap) {@@ -1046,7 +1215,7 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,unsigned long vaddr = map->vaddr;size_t size = map->size;int ret =0 , prot =0 ; -uint64_t mask; +size_t pgsize;struct vfio_dma *dma;/* Verify that none of our __u64 fields overflow */ @@ -1061,11 +1230,11 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgsize = (size_t )1 <<__ffs (iommu->pgsize_bitmap); -WARN_ON (mask &PAGE_MASK ); +WARN_ON ((pgsize -1 ) &PAGE_MASK ); -if (!prot || !size || (size | iova | vaddr) & mask) { +if (!prot || !size || (size | iova | vaddr) & (pgsize -1 )) { ret = -EINVAL ; goto out_unlock; }@@ -1142,6 +1311,12 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,else ret =vfio_pin_map_dma (iommu, dma, size); +if (!ret && iommu->dirty_page_tracking) { + ret =vfio_dma_bitmap_alloc (dma, pgsize); +if (ret) +vfio_remove_dma (iommu, dma); + } + out_unlock:mutex_unlock (&iommu->lock);return ret;@@ -2288,6 +2463,104 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,return copy_to_user ((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + }else if (cmd ==VFIO_IOMMU_DIRTY_PAGES ) { +struct vfio_iommu_type1_dirty_bitmap dirty; +uint32_t mask =VFIO_IOMMU_DIRTY_PAGES_FLAG_START | +VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP | +VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP ; +int ret =0 ; + +if (!iommu->v2) +return -EACCES ; + + minsz = offsetofend(struct vfio_iommu_type1_dirty_bitmap, + flags); + +if (copy_from_user (&dirty, (void __user *)arg, minsz)) +return -EFAULT ; + +if (dirty.argsz < minsz || dirty.flags & ~mask) +return -EINVAL ; + +/* only one flag should be set at a time */ +if (__ffs (dirty.flags) !=__fls (dirty.flags)) +return -EINVAL ; + +if (dirty.flags &VFIO_IOMMU_DIRTY_PAGES_FLAG_START ) { +size_t pgsize; + +mutex_lock (&iommu->lock); + pgsize =1 <<__ffs (iommu->pgsize_bitmap); +if (!iommu->dirty_page_tracking) { + ret =vfio_dma_bitmap_alloc_all (iommu, pgsize); +if (!ret) + iommu->dirty_page_tracking = true; + } +mutex_unlock (&iommu->lock); +return ret; + }else if (dirty.flags &VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP ) { +mutex_lock (&iommu->lock); +if (iommu->dirty_page_tracking) { + iommu->dirty_page_tracking = false; +vfio_dma_bitmap_free_all (iommu); + } +mutex_unlock (&iommu->lock); +return 0 ; + }else if (dirty.flags & +VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP ) { +struct vfio_iommu_type1_dirty_bitmap_get range; +unsigned long pgshift; +size_t data_size = dirty.argsz - minsz; +size_t iommu_pgsize; + +if (!data_size || data_size <sizeof (range)) +return -EINVAL ; + +if (copy_from_user (&range, (void __user *)(arg + minsz), +sizeof (range))) +return -EFAULT ; + +if (range.iova + range.size < range.iova) +return -EINVAL ; +if (!access_ok ((void __user *)range.bitmap.data, + range.bitmap.size)) +return -EINVAL ; + + pgshift =__ffs (range.bitmap.pgsize); + ret =verify_bitmap_size (range.size >> pgshift, + range.bitmap.size); +if (ret) +return ret; + +mutex_lock (&iommu->lock); + + iommu_pgsize = (size_t )1 <<__ffs (iommu->pgsize_bitmap); + +/* allow only smallest supported pgsize */ +if (range.bitmap.pgsize != iommu_pgsize) { + ret = -EINVAL ; + goto out_unlock; + } +if (range.iova & (iommu_pgsize -1 )) { + ret = -EINVAL ; + goto out_unlock; + } +if (!range.size || range.size & (iommu_pgsize -1 )) { + ret = -EINVAL ; + goto out_unlock; + } + +if (iommu->dirty_page_tracking) + ret =vfio_iova_dirty_bitmap (range.bitmap.data, + iommu, range.iova, range.size, + range.bitmap.pgsize); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); + +return ret; + } }return -ENOTTY ;@@ -2355,10 +2628,19 @@ static int vfio_iommu_type1_dma_rw_chunk(struct vfio_iommu *iommu, vaddr = dma->vaddr + offset; -if (write) +if (write) { *copied =copy_to_user ((void __user *)vaddr, data, count) ?0 : count; -else +if (*copied && iommu->dirty_page_tracking) { +unsigned long pgshift =__ffs (iommu->pgsize_bitmap); +/* +* Bitmap populated with the smallest supported page +* size +*/ +bitmap_set (dma->bitmap, offset >> pgshift, + *copied >> pgshift); + } + }else *copied =copy_from_user (data, (void __user *)vaddr, count) ?0 : count;if (kthread)
Re: [PATCH Kernel v20 5/8] vfio iommu: Implementation of ioctl for dirty pages tracking
Quoted 227 lines · On Thu, 14 May 2020 21:27:20 -0600 Alex Williamson…
Quoted 200 lines · On Fri, 15 May 2020 02:07:44 +0530 Kirti Wankhede <[email protected]> wrote:
VFIO_IOMMU_DIRTY_PAGESioctl performs three operations:
- Start dirty pages tracking while migration is active
- Stop dirty pages tracking.
- Get dirty pages bitmap. Its user space application's responsibility to copy content of dirty pages from source to destination during migration.To prevent DoS attack, memory for bitmap is allocated per
vfio_dmastructure. Bitmap size is calculated considering smallest supported page size. Bitmap is allocated for allvfio_dmaswhen dirty logging is enabledBitmap is populated for already pinned pages when bitmap is allocated for a
vfio_dmawith the smallest supported page size. Update bitmap from pinning functions when tracking is enabled. When user application queries bitmap, check if requested page size is same as page size used to populated bitmap. If it is equal, copy bitmap, but if not equal, return error.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>Fixed error reported by build bot by changing pgsize type from
uint64_ttosize_t.Reported-by kbuild test robot <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 294 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 288 insertions(+), 6 deletions(-)drivers/vfio/vfio_iommu_type1.c +123
-0@@ -72,6 +72,7 @@ struct vfio_iommu {uint64_t pgsize_bitmap;bool v2;bool nesting; +bool dirty_page_tracking; };struct vfio_domain {@@ -92,6 +93,7 @@ struct vfio_dma {bool lock_cap;/* capable(CAP_IPC_LOCK) */ struct task_struct *task;struct rb_root pfn_list;/* Ex-user pinned pfn list */ +unsigned long *bitmap; };struct vfio_group {@@ -126,6 +128,19 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +#define DIRTY_BITMAP_BYTES (n) (ALIGN(n, BITS_PER_TYPE(u64)) / BITS_PER_BYTE) + +/* +* Input argument of number of bits to bitmap_set() is unsigned integer, which +* further casts to signed integer for unaligned multi-bit operation, +* __bitmap_set(). +* Then maximum bitmap size supported is 2^31 bits divided by 2^3 bits/byte, +* that is 2^28 (256 MB) which maps to 2^31 * 2^12 = 2^43 (8TB) on 4K page +* system. +*/ +#define DIRTY_BITMAP_PAGES_MAX ((u64)INT_MAX) +#define DIRTY_BITMAP_SIZE_MAX DIRTY_BITMAP_BYTES(DIRTY_BITMAP_PAGES_MAX) +static int put_pfn (unsigned long pfn,int prot);/* @@ -176,6 +191,74 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)rb_erase (&old->node, &iommu->dma_list); } + +static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) +{ +uint64_t npages = dma->size / pgsize; + +if (npages >DIRTY_BITMAP_PAGES_MAX ) +return -EINVAL ; + + dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +if (!dma->bitmap) +return -ENOMEM ; + +return 0 ; +} + +static void vfio_dma_bitmap_free (struct vfio_dma *dma) +{ +kfree (dma->bitmap); + dma->bitmap =NULL ; +} + +static void vfio_dma_populate_bitmap (struct vfio_dma *dma,size_t pgsize) +{ +struct rb_node *p; + +for (p =rb_first (&dma->pfn_list); p; p =rb_next (p)) { + struct vfio_pfn *vpfn = rb_entry(p, struct vfio_pfn, node); + +bitmap_set (dma->bitmap, (vpfn->iova - dma->iova) / pgsize,1 ); + } +} + +static int vfio_dma_bitmap_alloc_all (struct vfio_iommu *iommu,size_t pgsize) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); +int ret; + + ret =vfio_dma_bitmap_alloc (dma, pgsize); +if (ret) { +struct rb_node *p =rb_prev (n); + +for (; p; p =rb_prev (p)) { +struct vfio_dma *dma =rb_entry (n, + struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +return ret; + } +vfio_dma_populate_bitmap (dma, pgsize); + } +return 0 ; +} + +static void vfio_dma_bitmap_free_all (struct vfio_iommu *iommu) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +} +/* * Helper Functions for host iova-pfn list */ @@ -568,6 +651,17 @@ static int vfio_iommu_type1_pin_pages(void *iommu_data,vfio_unpin_page_external (dma, iova, do_accounting); goto pin_unwind; } + +if (iommu->dirty_page_tracking) { +unsigned long pgshift =__ffs (iommu->pgsize_bitmap); + +/* +* Bitmap populated with the smallest supported page +* size +*/ +bitmap_set (dma->bitmap, + (vpfn->iova - dma->iova) >> pgshift,1 ); + } } ret = i;@@ -802,6 +896,7 @@ static void vfio_remove_dma(struct vfio_iommu *iommu, struct vfio_dma *dma)vfio_unmap_unpin (iommu, dma, true);vfio_unlink_dma (iommu, dma);put_task_struct (dma->task); +vfio_dma_bitmap_free (dma);kfree (dma); iommu->dma_avail++; }@@ -829,6 +924,80 @@ static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) } } +static int update_user_bitmap (u64 __user *bitmap,struct vfio_dma *dma, +dma_addr_t base_iova,size_t pgsize) +{ +unsigned long pgshift =__ffs (pgsize); +unsigned long nbits = dma->size >> pgshift; +unsigned long bit_offset = (dma->iova - base_iova) >> pgshift; +unsigned long copy_offset = bit_offset /BITS_PER_LONG ; +unsigned long shift = bit_offset %BITS_PER_LONG ; +unsigned long leftover; + +if (shift) { +bitmap_shift_left (dma->bitmap, dma->bitmap, shift, + nbits + shift); + +if (copy_from_user (&leftover, (u64 *)bitmap + copy_offset, +sizeof (leftover))) +return -EFAULT ; + +bitmap_or (dma->bitmap, dma->bitmap, &leftover, shift); + } + +if (copy_to_user ((u64 *)bitmap + copy_offset, dma->bitmap, +DIRTY_BITMAP_BYTES (nbits + shift))) +return -EFAULT ; + +return 0 ; +} +drivers/vfio/vfio_iommu_type1.c static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) · line 964
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+static int vfio_iova_dirty_bitmap (u64 __user *bitmap,struct vfio_iommu *iommu, +dma_addr_t iova,size_t size,size_t pgsize) +{ +struct vfio_dma *dma; +dma_addr_t i = iova, limit = iova + size; +unsigned long pgshift =__ffs (pgsize); +size_t sz = size; +int ret; + +while ((dma =vfio_find_dma (iommu, i, sz))) {This would be a lot more efficient if we walked it as:
struct rb_node *n = rb_first(&iommu->dma_list); for (; n; n = rb_next(n)) { struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); if (dma->iova < iova) continue; if (dma->iova >= iova + size) break;
Quoted 3 lines
ie. there's no need to start from the beginning of the tree on each iteration. This I would accept as a follow-on internal implementation improvement. Thanks,
Hmm, maybe I'm wrong here, vfio_find_dma() takes advantage of the tree layout to get to the relevant nodes versus the suggestion above where we're just walking every node in order. The best of both worlds would be to find the first node using the vfio_find_dma() approach and then walk the nodes forward from there until we've exceeded the range, but I won't ask for that. Thanks,
Alex
Quoted 204 lines
Quoted 203 lines
drivers/vfio/vfio_iommu_type1.c +155
-6+ +/* mark all pages dirty if all pages are pinned and mapped. */ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , dma->size >> pgshift); + + ret =update_user_bitmap (bitmap, dma, iova, pgsize); +if (ret) +return ret; + +/* +* Re-populate bitmap to include all pinned pages which are +* considered as dirty but exclude pages which are unpinned and +* pages which are marked dirty by vfio_dma_rw() +*/ +bitmap_clear (dma->bitmap,0 , dma->size >> pgshift); +vfio_dma_populate_bitmap (dma, pgsize); + + i = dma->iova + dma->size; +if (i >= limit) +break ; + + sz = limit - i; + } + +return 0 ; +} + +static int verify_bitmap_size (uint64_t npages,uint64_t bitmap_size) +{ +if (!npages || !bitmap_size || (bitmap_size >DIRTY_BITMAP_SIZE_MAX ) || + (bitmap_size <DIRTY_BITMAP_BYTES (npages))) +return -EINVAL ; + +return 0 ; +} +static int vfio_dma_do_unmap (struct vfio_iommu *iommu,struct vfio_iommu_type1_dma_unmap *unmap) {@@ -1046,7 +1215,7 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,unsigned long vaddr = map->vaddr;size_t size = map->size;int ret =0 , prot =0 ; -uint64_t mask; +size_t pgsize;struct vfio_dma *dma;/* Verify that none of our __u64 fields overflow */ @@ -1061,11 +1230,11 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgsize = (size_t )1 <<__ffs (iommu->pgsize_bitmap); -WARN_ON (mask &PAGE_MASK ); +WARN_ON ((pgsize -1 ) &PAGE_MASK ); -if (!prot || !size || (size | iova | vaddr) & mask) { +if (!prot || !size || (size | iova | vaddr) & (pgsize -1 )) { ret = -EINVAL ; goto out_unlock; }@@ -1142,6 +1311,12 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,else ret =vfio_pin_map_dma (iommu, dma, size); +if (!ret && iommu->dirty_page_tracking) { + ret =vfio_dma_bitmap_alloc (dma, pgsize); +if (ret) +vfio_remove_dma (iommu, dma); + } + out_unlock:mutex_unlock (&iommu->lock);return ret;@@ -2288,6 +2463,104 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,return copy_to_user ((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + }else if (cmd ==VFIO_IOMMU_DIRTY_PAGES ) { +struct vfio_iommu_type1_dirty_bitmap dirty; +uint32_t mask =VFIO_IOMMU_DIRTY_PAGES_FLAG_START | +VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP | +VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP ; +int ret =0 ; + +if (!iommu->v2) +return -EACCES ; + + minsz = offsetofend(struct vfio_iommu_type1_dirty_bitmap, + flags); + +if (copy_from_user (&dirty, (void __user *)arg, minsz)) +return -EFAULT ; + +if (dirty.argsz < minsz || dirty.flags & ~mask) +return -EINVAL ; + +/* only one flag should be set at a time */ +if (__ffs (dirty.flags) !=__fls (dirty.flags)) +return -EINVAL ; + +if (dirty.flags &VFIO_IOMMU_DIRTY_PAGES_FLAG_START ) { +size_t pgsize; + +mutex_lock (&iommu->lock); + pgsize =1 <<__ffs (iommu->pgsize_bitmap); +if (!iommu->dirty_page_tracking) { + ret =vfio_dma_bitmap_alloc_all (iommu, pgsize); +if (!ret) + iommu->dirty_page_tracking = true; + } +mutex_unlock (&iommu->lock); +return ret; + }else if (dirty.flags &VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP ) { +mutex_lock (&iommu->lock); +if (iommu->dirty_page_tracking) { + iommu->dirty_page_tracking = false; +vfio_dma_bitmap_free_all (iommu); + } +mutex_unlock (&iommu->lock); +return 0 ; + }else if (dirty.flags & +VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP ) { +struct vfio_iommu_type1_dirty_bitmap_get range; +unsigned long pgshift; +size_t data_size = dirty.argsz - minsz; +size_t iommu_pgsize; + +if (!data_size || data_size <sizeof (range)) +return -EINVAL ; + +if (copy_from_user (&range, (void __user *)(arg + minsz), +sizeof (range))) +return -EFAULT ; + +if (range.iova + range.size < range.iova) +return -EINVAL ; +if (!access_ok ((void __user *)range.bitmap.data, + range.bitmap.size)) +return -EINVAL ; + + pgshift =__ffs (range.bitmap.pgsize); + ret =verify_bitmap_size (range.size >> pgshift, + range.bitmap.size); +if (ret) +return ret; + +mutex_lock (&iommu->lock); + + iommu_pgsize = (size_t )1 <<__ffs (iommu->pgsize_bitmap); + +/* allow only smallest supported pgsize */ +if (range.bitmap.pgsize != iommu_pgsize) { + ret = -EINVAL ; + goto out_unlock; + } +if (range.iova & (iommu_pgsize -1 )) { + ret = -EINVAL ; + goto out_unlock; + } +if (!range.size || range.size & (iommu_pgsize -1 )) { + ret = -EINVAL ; + goto out_unlock; + } + +if (iommu->dirty_page_tracking) + ret =vfio_iova_dirty_bitmap (range.bitmap.data, + iommu, range.iova, range.size, + range.bitmap.pgsize); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); + +return ret; + } }return -ENOTTY ;@@ -2355,10 +2628,19 @@ static int vfio_iommu_type1_dma_rw_chunk(struct vfio_iommu *iommu, vaddr = dma->vaddr + offset; -if (write) +if (write) { *copied =copy_to_user ((void __user *)vaddr, data, count) ?0 : count; -else +if (*copied && iommu->dirty_page_tracking) { +unsigned long pgshift =__ffs (iommu->pgsize_bitmap); +/* +* Bitmap populated with the smallest supported page +* size +*/ +bitmap_set (dma->bitmap, offset >> pgshift, + *copied >> pgshift); + } + }else *copied =copy_from_user (data, (void __user *)vaddr, count) ?0 : count;if (kthread)
Re: [PATCH Kernel v20 5/8] vfio iommu: Implementation of ioctl for dirty pages tracking
In reply to Kirti WankhedeQuoted 200 lines · Kirti Wankhede
VFIO_IOMMU_DIRTY_PAGESioctl performs three operations:
- Start dirty pages tracking while migration is active
- Stop dirty pages tracking.
- Get dirty pages bitmap. Its user space application's responsibility to copy content of dirty pages from source to destination during migration.To prevent DoS attack, memory for bitmap is allocated per
vfio_dmastructure. Bitmap size is calculated considering smallest supported page size. Bitmap is allocated for allvfio_dmaswhen dirty logging is enabledBitmap is populated for already pinned pages when bitmap is allocated for a
vfio_dmawith the smallest supported page size. Update bitmap from pinning functions when tracking is enabled. When user application queries bitmap, check if requested page size is same as page size used to populated bitmap. If it is equal, copy bitmap, but if not equal, return error.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>Fixed error reported by build bot by changing pgsize type from
uint64_ttosize_t.Reported-by kbuild test robot <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 294 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 288 insertions(+), 6 deletions(-)drivers/vfio/vfio_iommu_type1.c +123
-0@@ -72,6 +72,7 @@ struct vfio_iommu {uint64_t pgsize_bitmap;bool v2;bool nesting; +bool dirty_page_tracking; };struct vfio_domain {@@ -92,6 +93,7 @@ struct vfio_dma {bool lock_cap;/* capable(CAP_IPC_LOCK) */ struct task_struct *task;struct rb_root pfn_list;/* Ex-user pinned pfn list */ +unsigned long *bitmap; };struct vfio_group {@@ -126,6 +128,19 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +#define DIRTY_BITMAP_BYTES (n) (ALIGN(n, BITS_PER_TYPE(u64)) / BITS_PER_BYTE) + +/* +* Input argument of number of bits to bitmap_set() is unsigned integer, which +* further casts to signed integer for unaligned multi-bit operation, +* __bitmap_set(). +* Then maximum bitmap size supported is 2^31 bits divided by 2^3 bits/byte, +* that is 2^28 (256 MB) which maps to 2^31 * 2^12 = 2^43 (8TB) on 4K page +* system. +*/ +#define DIRTY_BITMAP_PAGES_MAX ((u64)INT_MAX) +#define DIRTY_BITMAP_SIZE_MAX DIRTY_BITMAP_BYTES(DIRTY_BITMAP_PAGES_MAX) +static int put_pfn (unsigned long pfn,int prot);/* @@ -176,6 +191,74 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)rb_erase (&old->node, &iommu->dma_list); } + +static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) +{ +uint64_t npages = dma->size / pgsize; + +if (npages >DIRTY_BITMAP_PAGES_MAX ) +return -EINVAL ; + + dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +if (!dma->bitmap) +return -ENOMEM ; + +return 0 ; +} + +static void vfio_dma_bitmap_free (struct vfio_dma *dma) +{ +kfree (dma->bitmap); + dma->bitmap =NULL ; +} + +static void vfio_dma_populate_bitmap (struct vfio_dma *dma,size_t pgsize) +{ +struct rb_node *p; + +for (p =rb_first (&dma->pfn_list); p; p =rb_next (p)) { + struct vfio_pfn *vpfn = rb_entry(p, struct vfio_pfn, node); + +bitmap_set (dma->bitmap, (vpfn->iova - dma->iova) / pgsize,1 ); + } +} + +static int vfio_dma_bitmap_alloc_all (struct vfio_iommu *iommu,size_t pgsize) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); +int ret; + + ret =vfio_dma_bitmap_alloc (dma, pgsize); +if (ret) { +struct rb_node *p =rb_prev (n); + +for (; p; p =rb_prev (p)) { +struct vfio_dma *dma =rb_entry (n, + struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +return ret; + } +vfio_dma_populate_bitmap (dma, pgsize); + } +return 0 ; +} + +static void vfio_dma_bitmap_free_all (struct vfio_iommu *iommu) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +} +/* * Helper Functions for host iova-pfn list */ @@ -568,6 +651,17 @@ static int vfio_iommu_type1_pin_pages(void *iommu_data,vfio_unpin_page_external (dma, iova, do_accounting); goto pin_unwind; } + +if (iommu->dirty_page_tracking) { +unsigned long pgshift =__ffs (iommu->pgsize_bitmap); + +/* +* Bitmap populated with the smallest supported page +* size +*/ +bitmap_set (dma->bitmap, + (vpfn->iova - dma->iova) >> pgshift,1 ); + } } ret = i;@@ -802,6 +896,7 @@ static void vfio_remove_dma(struct vfio_iommu *iommu, struct vfio_dma *dma)vfio_unmap_unpin (iommu, dma, true);vfio_unlink_dma (iommu, dma);put_task_struct (dma->task); +vfio_dma_bitmap_free (dma);kfree (dma); iommu->dma_avail++; }@@ -829,6 +924,80 @@ static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) } } +static int update_user_bitmap (u64 __user *bitmap,struct vfio_dma *dma, +dma_addr_t base_iova,size_t pgsize) +{ +unsigned long pgshift =__ffs (pgsize); +unsigned long nbits = dma->size >> pgshift; +unsigned long bit_offset = (dma->iova - base_iova) >> pgshift; +unsigned long copy_offset = bit_offset /BITS_PER_LONG ; +unsigned long shift = bit_offset %BITS_PER_LONG ; +unsigned long leftover; + +if (shift) { +bitmap_shift_left (dma->bitmap, dma->bitmap, shift, + nbits + shift); + +if (copy_from_user (&leftover, (u64 *)bitmap + copy_offset, +sizeof (leftover))) +return -EFAULT ; + +bitmap_or (dma->bitmap, dma->bitmap, &leftover, shift); + } + +if (copy_to_user ((u64 *)bitmap + copy_offset, dma->bitmap, +DIRTY_BITMAP_BYTES (nbits + shift))) +return -EFAULT ; + +return 0 ; +} +
drivers/vfio/vfio_iommu_type1.c static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) · line 964
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+static int vfio_iova_dirty_bitmap (u64 __user *bitmap,struct vfio_iommu *iommu, +dma_addr_t iova,size_t size,size_t pgsize) +{ +struct vfio_dma *dma; +dma_addr_t i = iova, limit = iova + size; +unsigned long pgshift =__ffs (pgsize); +size_t sz = size; +int ret; + +while ((dma =vfio_find_dma (iommu, i, sz))) {
not quite get the logic here.
if (i, i + size) is intersecting with (dma->iova, dma->iova + dma->size), and a dma is found here, why the whole bitmap is cleared and copied?
Thanks
Yan
Quoted 206 lines
drivers/vfio/vfio_iommu_type1.c +155
-6+ +/* mark all pages dirty if all pages are pinned and mapped. */ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , dma->size >> pgshift); + + ret =update_user_bitmap (bitmap, dma, iova, pgsize); +if (ret) +return ret; + +/* +* Re-populate bitmap to include all pinned pages which are +* considered as dirty but exclude pages which are unpinned and +* pages which are marked dirty by vfio_dma_rw() +*/ +bitmap_clear (dma->bitmap,0 , dma->size >> pgshift); +vfio_dma_populate_bitmap (dma, pgsize); + + i = dma->iova + dma->size; +if (i >= limit) +break ; + + sz = limit - i; + } + +return 0 ; +} + +static int verify_bitmap_size (uint64_t npages,uint64_t bitmap_size) +{ +if (!npages || !bitmap_size || (bitmap_size >DIRTY_BITMAP_SIZE_MAX ) || + (bitmap_size <DIRTY_BITMAP_BYTES (npages))) +return -EINVAL ; + +return 0 ; +} +static int vfio_dma_do_unmap (struct vfio_iommu *iommu,struct vfio_iommu_type1_dma_unmap *unmap) {@@ -1046,7 +1215,7 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,unsigned long vaddr = map->vaddr;size_t size = map->size;int ret =0 , prot =0 ; -uint64_t mask; +size_t pgsize;struct vfio_dma *dma;/* Verify that none of our __u64 fields overflow */ @@ -1061,11 +1230,11 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgsize = (size_t )1 <<__ffs (iommu->pgsize_bitmap); -WARN_ON (mask &PAGE_MASK ); +WARN_ON ((pgsize -1 ) &PAGE_MASK ); -if (!prot || !size || (size | iova | vaddr) & mask) { +if (!prot || !size || (size | iova | vaddr) & (pgsize -1 )) { ret = -EINVAL ; goto out_unlock; }@@ -1142,6 +1311,12 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,else ret =vfio_pin_map_dma (iommu, dma, size); +if (!ret && iommu->dirty_page_tracking) { + ret =vfio_dma_bitmap_alloc (dma, pgsize); +if (ret) +vfio_remove_dma (iommu, dma); + } + out_unlock:mutex_unlock (&iommu->lock);return ret;@@ -2288,6 +2463,104 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,return copy_to_user ((void __user *)arg, &unmap, minsz) ? -EFAULT :0 ; + }else if (cmd ==VFIO_IOMMU_DIRTY_PAGES ) { +struct vfio_iommu_type1_dirty_bitmap dirty; +uint32_t mask =VFIO_IOMMU_DIRTY_PAGES_FLAG_START | +VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP | +VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP ; +int ret =0 ; + +if (!iommu->v2) +return -EACCES ; + + minsz = offsetofend(struct vfio_iommu_type1_dirty_bitmap, + flags); + +if (copy_from_user (&dirty, (void __user *)arg, minsz)) +return -EFAULT ; + +if (dirty.argsz < minsz || dirty.flags & ~mask) +return -EINVAL ; + +/* only one flag should be set at a time */ +if (__ffs (dirty.flags) !=__fls (dirty.flags)) +return -EINVAL ; + +if (dirty.flags &VFIO_IOMMU_DIRTY_PAGES_FLAG_START ) { +size_t pgsize; + +mutex_lock (&iommu->lock); + pgsize =1 <<__ffs (iommu->pgsize_bitmap); +if (!iommu->dirty_page_tracking) { + ret =vfio_dma_bitmap_alloc_all (iommu, pgsize); +if (!ret) + iommu->dirty_page_tracking = true; + } +mutex_unlock (&iommu->lock); +return ret; + }else if (dirty.flags &VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP ) { +mutex_lock (&iommu->lock); +if (iommu->dirty_page_tracking) { + iommu->dirty_page_tracking = false; +vfio_dma_bitmap_free_all (iommu); + } +mutex_unlock (&iommu->lock); +return 0 ; + }else if (dirty.flags & +VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP ) { +struct vfio_iommu_type1_dirty_bitmap_get range; +unsigned long pgshift; +size_t data_size = dirty.argsz - minsz; +size_t iommu_pgsize; + +if (!data_size || data_size <sizeof (range)) +return -EINVAL ; + +if (copy_from_user (&range, (void __user *)(arg + minsz), +sizeof (range))) +return -EFAULT ; + +if (range.iova + range.size < range.iova) +return -EINVAL ; +if (!access_ok ((void __user *)range.bitmap.data, + range.bitmap.size)) +return -EINVAL ; + + pgshift =__ffs (range.bitmap.pgsize); + ret =verify_bitmap_size (range.size >> pgshift, + range.bitmap.size); +if (ret) +return ret; + +mutex_lock (&iommu->lock); + + iommu_pgsize = (size_t )1 <<__ffs (iommu->pgsize_bitmap); + +/* allow only smallest supported pgsize */ +if (range.bitmap.pgsize != iommu_pgsize) { + ret = -EINVAL ; + goto out_unlock; + } +if (range.iova & (iommu_pgsize -1 )) { + ret = -EINVAL ; + goto out_unlock; + } +if (!range.size || range.size & (iommu_pgsize -1 )) { + ret = -EINVAL ; + goto out_unlock; + } + +if (iommu->dirty_page_tracking) + ret =vfio_iova_dirty_bitmap (range.bitmap.data, + iommu, range.iova, range.size, + range.bitmap.pgsize); +else + ret = -EINVAL ; +out_unlock: +mutex_unlock (&iommu->lock); + +return ret; + } }return -ENOTTY ;@@ -2355,10 +2628,19 @@ static int vfio_iommu_type1_dma_rw_chunk(struct vfio_iommu *iommu, vaddr = dma->vaddr + offset; -if (write) +if (write) { *copied =copy_to_user ((void __user *)vaddr, data, count) ?0 : count; -else +if (*copied && iommu->dirty_page_tracking) { +unsigned long pgshift =__ffs (iommu->pgsize_bitmap); +/* +* Bitmap populated with the smallest supported page +* size +*/ +bitmap_set (dma->bitmap, offset >> pgshift, + *copied >> pgshift); + } + }else *copied =copy_from_user (data, (void __user *)vaddr, count) ?0 : count;if (kthread)Signature · 1 line
2.7.0
Re: [PATCH Kernel v20 5/8] vfio iommu: Implementation of ioctl for dirty pages tracking
Quoted 206 lines · Yan Zhao
Quoted 205 lines · Kirti Wankhede
VFIO_IOMMU_DIRTY_PAGESioctl performs three operations:
- Start dirty pages tracking while migration is active
- Stop dirty pages tracking.
- Get dirty pages bitmap. Its user space application's responsibility to copy content of dirty pages from source to destination during migration.To prevent DoS attack, memory for bitmap is allocated per
vfio_dmastructure. Bitmap size is calculated considering smallest supported page size. Bitmap is allocated for allvfio_dmaswhen dirty logging is enabledBitmap is populated for already pinned pages when bitmap is allocated for a
vfio_dmawith the smallest supported page size. Update bitmap from pinning functions when tracking is enabled. When user application queries bitmap, check if requested page size is same as page size used to populated bitmap. If it is equal, copy bitmap, but if not equal, return error.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>Fixed error reported by build bot by changing pgsize type from
uint64_ttosize_t.Reported-by kbuild test robot <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 294 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 288 insertions(+), 6 deletions(-)drivers/vfio/vfio_iommu_type1.c +128
-0@@ -72,6 +72,7 @@ struct vfio_iommu {uint64_t pgsize_bitmap;bool v2;bool nesting; +bool dirty_page_tracking; };struct vfio_domain {@@ -92,6 +93,7 @@ struct vfio_dma {bool lock_cap;/* capable(CAP_IPC_LOCK) */ struct task_struct *task;struct rb_root pfn_list;/* Ex-user pinned pfn list */ +unsigned long *bitmap; };struct vfio_group {@@ -126,6 +128,19 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +#define DIRTY_BITMAP_BYTES (n) (ALIGN(n, BITS_PER_TYPE(u64)) / BITS_PER_BYTE) + +/* +* Input argument of number of bits to bitmap_set() is unsigned integer, which +* further casts to signed integer for unaligned multi-bit operation, +* __bitmap_set(). +* Then maximum bitmap size supported is 2^31 bits divided by 2^3 bits/byte, +* that is 2^28 (256 MB) which maps to 2^31 * 2^12 = 2^43 (8TB) on 4K page +* system. +*/ +#define DIRTY_BITMAP_PAGES_MAX ((u64)INT_MAX) +#define DIRTY_BITMAP_SIZE_MAX DIRTY_BITMAP_BYTES(DIRTY_BITMAP_PAGES_MAX) +static int put_pfn (unsigned long pfn,int prot);/* @@ -176,6 +191,74 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)rb_erase (&old->node, &iommu->dma_list); } + +static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) +{ +uint64_t npages = dma->size / pgsize; + +if (npages >DIRTY_BITMAP_PAGES_MAX ) +return -EINVAL ; + + dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +if (!dma->bitmap) +return -ENOMEM ; + +return 0 ; +} + +static void vfio_dma_bitmap_free (struct vfio_dma *dma) +{ +kfree (dma->bitmap); + dma->bitmap =NULL ; +} + +static void vfio_dma_populate_bitmap (struct vfio_dma *dma,size_t pgsize) +{ +struct rb_node *p; + +for (p =rb_first (&dma->pfn_list); p; p =rb_next (p)) { + struct vfio_pfn *vpfn = rb_entry(p, struct vfio_pfn, node); + +bitmap_set (dma->bitmap, (vpfn->iova - dma->iova) / pgsize,1 ); + } +} + +static int vfio_dma_bitmap_alloc_all (struct vfio_iommu *iommu,size_t pgsize) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); +int ret; + + ret =vfio_dma_bitmap_alloc (dma, pgsize); +if (ret) { +struct rb_node *p =rb_prev (n); + +for (; p; p =rb_prev (p)) { +struct vfio_dma *dma =rb_entry (n, + struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +return ret; + } +vfio_dma_populate_bitmap (dma, pgsize); + } +return 0 ; +} + +static void vfio_dma_bitmap_free_all (struct vfio_iommu *iommu) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +} +/* * Helper Functions for host iova-pfn list */ @@ -568,6 +651,17 @@ static int vfio_iommu_type1_pin_pages(void *iommu_data,vfio_unpin_page_external (dma, iova, do_accounting); goto pin_unwind; } + +if (iommu->dirty_page_tracking) { +unsigned long pgshift =__ffs (iommu->pgsize_bitmap); + +/* +* Bitmap populated with the smallest supported page +* size +*/ +bitmap_set (dma->bitmap, + (vpfn->iova - dma->iova) >> pgshift,1 ); + } } ret = i;@@ -802,6 +896,7 @@ static void vfio_remove_dma(struct vfio_iommu *iommu, struct vfio_dma *dma)vfio_unmap_unpin (iommu, dma, true);vfio_unlink_dma (iommu, dma);put_task_struct (dma->task); +vfio_dma_bitmap_free (dma);kfree (dma); iommu->dma_avail++; }@@ -829,6 +924,80 @@ static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) } } +static int update_user_bitmap (u64 __user *bitmap,struct vfio_dma *dma, +dma_addr_t base_iova,size_t pgsize) +{ +unsigned long pgshift =__ffs (pgsize); +unsigned long nbits = dma->size >> pgshift; +unsigned long bit_offset = (dma->iova - base_iova) >> pgshift; +unsigned long copy_offset = bit_offset /BITS_PER_LONG ; +unsigned long shift = bit_offset %BITS_PER_LONG ; +unsigned long leftover; + +if (shift) { +bitmap_shift_left (dma->bitmap, dma->bitmap, shift, + nbits + shift); + +if (copy_from_user (&leftover, (u64 *)bitmap + copy_offset, +sizeof (leftover))) +return -EFAULT ; + +bitmap_or (dma->bitmap, dma->bitmap, &leftover, shift); + } + +if (copy_to_user ((u64 *)bitmap + copy_offset, dma->bitmap, +DIRTY_BITMAP_BYTES (nbits + shift))) +return -EFAULT ; + +return 0 ; +} + +static int vfio_iova_dirty_bitmap (u64 __user *bitmap,struct vfio_iommu *iommu, +dma_addr_t iova,size_t size,size_t pgsize) +{ +struct vfio_dma *dma; +dma_addr_t i = iova, limit = iova + size;
Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) · line 964
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+unsigned long pgshift =__ffs (pgsize); +size_t sz = size; +int ret; + +while ((dma =vfio_find_dma (iommu, i, sz))) {not quite get the logic here.
if (i, i + size) is intersecting with (dma->iova,dma->iova+dma->size), and a dma is found here, why the whole bitmap is cleared and copied?
This works with multiple but full vfio_dma, not intersects of vfio_dma, similar to unmap ioctl.
Thanks,
Kirti
Quoted 2 lines
Thanks
Yan
Re: [PATCH Kernel v20 5/8] vfio iommu: Implementation of ioctl for dirty pages tracking
Quoted 217 lines · On Fri, 15 May 2020 16:44:38 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 206 lines · Yan Zhao
Quoted 205 lines · Kirti Wankhede
VFIO_IOMMU_DIRTY_PAGESioctl performs three operations:
- Start dirty pages tracking while migration is active
- Stop dirty pages tracking.
- Get dirty pages bitmap. Its user space application's responsibility to copy content of dirty pages from source to destination during migration.To prevent DoS attack, memory for bitmap is allocated per
vfio_dmastructure. Bitmap size is calculated considering smallest supported page size. Bitmap is allocated for allvfio_dmaswhen dirty logging is enabledBitmap is populated for already pinned pages when bitmap is allocated for a
vfio_dmawith the smallest supported page size. Update bitmap from pinning functions when tracking is enabled. When user application queries bitmap, check if requested page size is same as page size used to populated bitmap. If it is equal, copy bitmap, but if not equal, return error.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>Fixed error reported by build bot by changing pgsize type from
uint64_ttosize_t.Reported-by kbuild test robot <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 294 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 288 insertions(+), 6 deletions(-)drivers/vfio/vfio_iommu_type1.c +128
-0@@ -72,6 +72,7 @@ struct vfio_iommu {uint64_t pgsize_bitmap;bool v2;bool nesting; +bool dirty_page_tracking; };struct vfio_domain {@@ -92,6 +93,7 @@ struct vfio_dma {bool lock_cap;/* capable(CAP_IPC_LOCK) */ struct task_struct *task;struct rb_root pfn_list;/* Ex-user pinned pfn list */ +unsigned long *bitmap; };struct vfio_group {@@ -126,6 +128,19 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +#define DIRTY_BITMAP_BYTES (n) (ALIGN(n, BITS_PER_TYPE(u64)) / BITS_PER_BYTE) + +/* +* Input argument of number of bits to bitmap_set() is unsigned integer, which +* further casts to signed integer for unaligned multi-bit operation, +* __bitmap_set(). +* Then maximum bitmap size supported is 2^31 bits divided by 2^3 bits/byte, +* that is 2^28 (256 MB) which maps to 2^31 * 2^12 = 2^43 (8TB) on 4K page +* system. +*/ +#define DIRTY_BITMAP_PAGES_MAX ((u64)INT_MAX) +#define DIRTY_BITMAP_SIZE_MAX DIRTY_BITMAP_BYTES(DIRTY_BITMAP_PAGES_MAX) +static int put_pfn (unsigned long pfn,int prot);/* @@ -176,6 +191,74 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)rb_erase (&old->node, &iommu->dma_list); } + +static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) +{ +uint64_t npages = dma->size / pgsize; + +if (npages >DIRTY_BITMAP_PAGES_MAX ) +return -EINVAL ; + + dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +if (!dma->bitmap) +return -ENOMEM ; + +return 0 ; +} + +static void vfio_dma_bitmap_free (struct vfio_dma *dma) +{ +kfree (dma->bitmap); + dma->bitmap =NULL ; +} + +static void vfio_dma_populate_bitmap (struct vfio_dma *dma,size_t pgsize) +{ +struct rb_node *p; + +for (p =rb_first (&dma->pfn_list); p; p =rb_next (p)) { + struct vfio_pfn *vpfn = rb_entry(p, struct vfio_pfn, node); + +bitmap_set (dma->bitmap, (vpfn->iova - dma->iova) / pgsize,1 ); + } +} + +static int vfio_dma_bitmap_alloc_all (struct vfio_iommu *iommu,size_t pgsize) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); +int ret; + + ret =vfio_dma_bitmap_alloc (dma, pgsize); +if (ret) { +struct rb_node *p =rb_prev (n); + +for (; p; p =rb_prev (p)) { +struct vfio_dma *dma =rb_entry (n, + struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +return ret; + } +vfio_dma_populate_bitmap (dma, pgsize); + } +return 0 ; +} + +static void vfio_dma_bitmap_free_all (struct vfio_iommu *iommu) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +} +/* * Helper Functions for host iova-pfn list */ @@ -568,6 +651,17 @@ static int vfio_iommu_type1_pin_pages(void *iommu_data,vfio_unpin_page_external (dma, iova, do_accounting); goto pin_unwind; } + +if (iommu->dirty_page_tracking) { +unsigned long pgshift =__ffs (iommu->pgsize_bitmap); + +/* +* Bitmap populated with the smallest supported page +* size +*/ +bitmap_set (dma->bitmap, + (vpfn->iova - dma->iova) >> pgshift,1 ); + } } ret = i;@@ -802,6 +896,7 @@ static void vfio_remove_dma(struct vfio_iommu *iommu, struct vfio_dma *dma)vfio_unmap_unpin (iommu, dma, true);vfio_unlink_dma (iommu, dma);put_task_struct (dma->task); +vfio_dma_bitmap_free (dma);kfree (dma); iommu->dma_avail++; }@@ -829,6 +924,80 @@ static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) } } +static int update_user_bitmap (u64 __user *bitmap,struct vfio_dma *dma, +dma_addr_t base_iova,size_t pgsize) +{ +unsigned long pgshift =__ffs (pgsize); +unsigned long nbits = dma->size >> pgshift; +unsigned long bit_offset = (dma->iova - base_iova) >> pgshift; +unsigned long copy_offset = bit_offset /BITS_PER_LONG ; +unsigned long shift = bit_offset %BITS_PER_LONG ; +unsigned long leftover; + +if (shift) { +bitmap_shift_left (dma->bitmap, dma->bitmap, shift, + nbits + shift); + +if (copy_from_user (&leftover, (u64 *)bitmap + copy_offset, +sizeof (leftover))) +return -EFAULT ; + +bitmap_or (dma->bitmap, dma->bitmap, &leftover, shift); + } + +if (copy_to_user ((u64 *)bitmap + copy_offset, dma->bitmap, +DIRTY_BITMAP_BYTES (nbits + shift))) +return -EFAULT ; + +return 0 ; +} + +static int vfio_iova_dirty_bitmap (u64 __user *bitmap,struct vfio_iommu *iommu, +dma_addr_t iova,size_t size,size_t pgsize) +{ +struct vfio_dma *dma; +dma_addr_t i = iova, limit = iova + size;Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) · line 964
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+unsigned long pgshift =__ffs (pgsize); +size_t sz = size; +int ret; + +while ((dma =vfio_find_dma (iommu, i, sz))) {not quite get the logic here.
if (i, i + size) is intersecting with (dma->iova,dma->iova+dma->size), and a dma is found here, why the whole bitmap is cleared and copied?
Quoted 2 lines
This works with multiple but full
vfio_dma, not intersects ofvfio_dma, similar to unmap ioctl.
I don't see that the VFIO_IOMMU_DIRTY_PAGES ioctl validates that like VFIO_IOMMU_UNMAP_DMA does though. We should have the same test as vfio_dma_do_unmap() to verify that they user range doesn't bisect a mapping, otherwise Yan is right, it looks like we allow the user to specify an arbitrary range that might bisect a bitmap, but we clear and attempt to copy the entire bitmap to the user buffer regardless. Thanks,
Alex
Re: [PATCH Kernel v20 5/8] vfio iommu: Implementation of ioctl for dirty pages tracking
Quoted 223 lines · Alex Williamson
Quoted 217 lines · On Fri, 15 May 2020 16:44:38 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 206 lines · Yan Zhao
Quoted 205 lines · Kirti Wankhede
VFIO_IOMMU_DIRTY_PAGESioctl performs three operations:
- Start dirty pages tracking while migration is active
- Stop dirty pages tracking.
- Get dirty pages bitmap. Its user space application's responsibility to copy content of dirty pages from source to destination during migration.To prevent DoS attack, memory for bitmap is allocated per
vfio_dmastructure. Bitmap size is calculated considering smallest supported page size. Bitmap is allocated for allvfio_dmaswhen dirty logging is enabledBitmap is populated for already pinned pages when bitmap is allocated for a
vfio_dmawith the smallest supported page size. Update bitmap from pinning functions when tracking is enabled. When user application queries bitmap, check if requested page size is same as page size used to populated bitmap. If it is equal, copy bitmap, but if not equal, return error.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>Fixed error reported by build bot by changing pgsize type from
uint64_ttosize_t.Reported-by kbuild test robot <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 294 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 288 insertions(+), 6 deletions(-)drivers/vfio/vfio_iommu_type1.c +128
-0@@ -72,6 +72,7 @@ struct vfio_iommu {uint64_t pgsize_bitmap;bool v2;bool nesting; +bool dirty_page_tracking; };struct vfio_domain {@@ -92,6 +93,7 @@ struct vfio_dma {bool lock_cap;/* capable(CAP_IPC_LOCK) */ struct task_struct *task;struct rb_root pfn_list;/* Ex-user pinned pfn list */ +unsigned long *bitmap; };struct vfio_group {@@ -126,6 +128,19 @@ struct vfio_regions {#define IS_IOMMU_CAP_DOMAIN_IN_CONTAINER (iommu) \ (!list_empty(&iommu->domain_list)) +#define DIRTY_BITMAP_BYTES (n) (ALIGN(n, BITS_PER_TYPE(u64)) / BITS_PER_BYTE) + +/* +* Input argument of number of bits to bitmap_set() is unsigned integer, which +* further casts to signed integer for unaligned multi-bit operation, +* __bitmap_set(). +* Then maximum bitmap size supported is 2^31 bits divided by 2^3 bits/byte, +* that is 2^28 (256 MB) which maps to 2^31 * 2^12 = 2^43 (8TB) on 4K page +* system. +*/ +#define DIRTY_BITMAP_PAGES_MAX ((u64)INT_MAX) +#define DIRTY_BITMAP_SIZE_MAX DIRTY_BITMAP_BYTES(DIRTY_BITMAP_PAGES_MAX) +static int put_pfn (unsigned long pfn,int prot);/* @@ -176,6 +191,74 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)rb_erase (&old->node, &iommu->dma_list); } + +static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) +{ +uint64_t npages = dma->size / pgsize; + +if (npages >DIRTY_BITMAP_PAGES_MAX ) +return -EINVAL ; + + dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +if (!dma->bitmap) +return -ENOMEM ; + +return 0 ; +} + +static void vfio_dma_bitmap_free (struct vfio_dma *dma) +{ +kfree (dma->bitmap); + dma->bitmap =NULL ; +} + +static void vfio_dma_populate_bitmap (struct vfio_dma *dma,size_t pgsize) +{ +struct rb_node *p; + +for (p =rb_first (&dma->pfn_list); p; p =rb_next (p)) { + struct vfio_pfn *vpfn = rb_entry(p, struct vfio_pfn, node); + +bitmap_set (dma->bitmap, (vpfn->iova - dma->iova) / pgsize,1 ); + } +} + +static int vfio_dma_bitmap_alloc_all (struct vfio_iommu *iommu,size_t pgsize) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); +int ret; + + ret =vfio_dma_bitmap_alloc (dma, pgsize); +if (ret) { +struct rb_node *p =rb_prev (n); + +for (; p; p =rb_prev (p)) { +struct vfio_dma *dma =rb_entry (n, + struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +return ret; + } +vfio_dma_populate_bitmap (dma, pgsize); + } +return 0 ; +} + +static void vfio_dma_bitmap_free_all (struct vfio_iommu *iommu) +{ +struct rb_node *n =rb_first (&iommu->dma_list); + +for (; n; n =rb_next (n)) { + struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node); + +vfio_dma_bitmap_free (dma); + } +} +/* * Helper Functions for host iova-pfn list */ @@ -568,6 +651,17 @@ static int vfio_iommu_type1_pin_pages(void *iommu_data,vfio_unpin_page_external (dma, iova, do_accounting); goto pin_unwind; } + +if (iommu->dirty_page_tracking) { +unsigned long pgshift =__ffs (iommu->pgsize_bitmap); + +/* +* Bitmap populated with the smallest supported page +* size +*/ +bitmap_set (dma->bitmap, + (vpfn->iova - dma->iova) >> pgshift,1 ); + } } ret = i;@@ -802,6 +896,7 @@ static void vfio_remove_dma(struct vfio_iommu *iommu, struct vfio_dma *dma)vfio_unmap_unpin (iommu, dma, true);vfio_unlink_dma (iommu, dma);put_task_struct (dma->task); +vfio_dma_bitmap_free (dma);kfree (dma); iommu->dma_avail++; }@@ -829,6 +924,80 @@ static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) } } +static int update_user_bitmap (u64 __user *bitmap,struct vfio_dma *dma, +dma_addr_t base_iova,size_t pgsize) +{ +unsigned long pgshift =__ffs (pgsize); +unsigned long nbits = dma->size >> pgshift; +unsigned long bit_offset = (dma->iova - base_iova) >> pgshift; +unsigned long copy_offset = bit_offset /BITS_PER_LONG ; +unsigned long shift = bit_offset %BITS_PER_LONG ; +unsigned long leftover; + +if (shift) { +bitmap_shift_left (dma->bitmap, dma->bitmap, shift, + nbits + shift); + +if (copy_from_user (&leftover, (u64 *)bitmap + copy_offset, +sizeof (leftover))) +return -EFAULT ; + +bitmap_or (dma->bitmap, dma->bitmap, &leftover, shift); + } + +if (copy_to_user ((u64 *)bitmap + copy_offset, dma->bitmap, +DIRTY_BITMAP_BYTES (nbits + shift))) +return -EFAULT ; + +return 0 ; +} + +static int vfio_iova_dirty_bitmap (u64 __user *bitmap,struct vfio_iommu *iommu, +dma_addr_t iova,size_t size,size_t pgsize) +{ +struct vfio_dma *dma; +dma_addr_t i = iova, limit = iova + size;Quoted 9 lines
drivers/vfio/vfio_iommu_type1.c static void vfio_pgsize_bitmap(struct vfio_iommu *iommu) · line 964
Quoted 5 lines
drivers/vfio/vfio_iommu_type1.c +5
-0+unsigned long pgshift =__ffs (pgsize); +size_t sz = size; +int ret; + +while ((dma =vfio_find_dma (iommu, i, sz))) {not quite get the logic here.
if (i, i + size) is intersecting with (dma->iova,dma->iova+dma->size), and a dma is found here, why the whole bitmap is cleared and copied?Quoted 2 lines
This works with multiple but full
vfio_dma, not intersects ofvfio_dma, similar to unmap ioctl.
Quoted 6 lines
I don't see that the
VFIO_IOMMU_DIRTY_PAGESioctl validates that likeVFIO_IOMMU_UNMAP_DMAdoes though. We should have the same test asvfio_dma_do_unmap()to verify that they user range doesn't bisect a mapping, otherwise Yan is right, it looks like we allow the user to specify an arbitrary range that might bisect a bitmap, but we clear and attempt to copy the entire bitmap to the user buffer regardless.
Sure, updating patch. I'll send next version in some time.
Thanks,
Kirti
PATCH v20 6/8 vfio iommu: Update UNMAP_DMA ioctl to get dirty bitmap before unmap
In reply to Kirti WankhedeDMA mapped pages, including those pinned by mdev vendor drivers, might get unpinned and unmapped while migration is active and device is still running. For example, in pre-copy phase while guest driver could access those pages, host device or vendor driver can dirty these mapped pages. Such pages should be marked dirty so as to maintain memory consistency for a user making use of dirty page tracking.
To get bitmap during unmap, user should allocate memory for bitmap, set it all zeros, set size of allocated memory, set page size to be considered for bitmap and set flag VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP.
--- drivers/vfio/vfio_iommu_type1.c | 77 ++++++++++++++++++++++++++++++++++------- include/uapi/linux/vfio.h | 10 ++++++ 2 files changed, 75 insertions(+), 12 deletions(-)
drivers/vfio/vfio_iommu_type1.c +65 -12
@@ -195,11 +195,15 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) {uint64_t npages = dma->size / pgsize; +size_t bitmap_size;if (npages >DIRTY_BITMAP_PAGES_MAX )return -EINVAL ; - dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +/* Allocate extra 64 bits which are used for bitmap manipulation */ + bitmap_size =DIRTY_BITMAP_BYTES (npages) +sizeof (u64); + + dma->bitmap =kvzalloc (bitmap_size,GFP_KERNEL );if (!dma->bitmap)return -ENOMEM ;
@@ -999,23 +1003,25 @@ static int verify_bitmap_size(uint64_t npages, uint64_t bitmap_size) }static int vfio_dma_do_unmap (struct vfio_iommu *iommu, -struct vfio_iommu_type1_dma_unmap *unmap) +struct vfio_iommu_type1_dma_unmap *unmap, +struct vfio_bitmap *bitmap) { -uint64_t mask;struct vfio_dma *dma, *dma_last =NULL ; -size_t unmapped =0 ; +size_t unmapped =0 , pgsize;int ret =0 , retries =0 ; +unsigned long pgshift;mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgshift =__ffs (iommu->pgsize_bitmap); + pgsize = (size_t )1 << pgshift; -if (unmap->iova & mask) { +if (unmap->iova & (pgsize -1 )) { ret = -EINVAL ; goto unlock; } -if (!unmap->size || unmap->size & mask) { +if (!unmap->size || unmap->size & (pgsize -1 )) { ret = -EINVAL ; goto unlock; }
@@ -1026,9 +1032,15 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, goto unlock; } -WARN_ON (mask &PAGE_MASK ); -again: +/* When dirty tracking is enabled, allow only min supported pgsize */ +if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (!iommu->dirty_page_tracking || (bitmap->pgsize != pgsize))) { + ret = -EINVAL ; + goto unlock; + } +WARN_ON ((pgsize -1 ) &PAGE_MASK ); +again:/* * vfio-iommu-type1 (v1) - User mappings were coalesced together to * avoid tracking individual mappings. This means that the granularity
@@ -1066,6 +1078,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, ret = -EINVAL ; goto unlock; } + dma =vfio_find_dma (iommu, unmap->iova + unmap->size -1 ,0 );if (dma && dma->iova + dma->size != unmap->iova + unmap->size) { ret = -EINVAL ;
@@ -1083,6 +1096,23 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu,if (dma->task->mm != current->mm)break ; +if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (dma_last != dma)) { + +/* +* mark all pages dirty if all pages are pinned and +* mapped +*/ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , + dma->size >> pgshift); + + ret =update_user_bitmap (bitmap->data, dma, + unmap->iova, pgsize); +if (ret) +break ; + } +if (!RB_EMPTY_ROOT (&dma->pfn_list)) {struct vfio_iommu_type1_dma_unmap nb_unmap;
@@ -2447,17 +2477,40 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, }else if (cmd ==VFIO_IOMMU_UNMAP_DMA ) {struct vfio_iommu_type1_dma_unmap unmap; -long ret; +struct vfio_bitmap bitmap = {0 }; +int ret; minsz = offsetofend(struct vfio_iommu_type1_dma_unmap, size);if (copy_from_user (&unmap, (void __user *)arg, minsz))return -EFAULT ; -if (unmap.argsz < minsz || unmap.flags) +if (unmap.argsz < minsz || + unmap.flags & ~VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP )return -EINVAL ; - ret =vfio_dma_do_unmap (iommu, &unmap); +if (unmap.flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) { +unsigned long pgshift; + +if (unmap.argsz < (minsz +sizeof (bitmap))) +return -EINVAL ; + +if (copy_from_user (&bitmap, + (void __user *)(arg + minsz), +sizeof (bitmap))) +return -EFAULT ; + +if (!access_ok ((void __user *)bitmap.data, bitmap.size)) +return -EINVAL ; + + pgshift =__ffs (bitmap.pgsize); + ret =verify_bitmap_size (unmap.size >> pgshift, + bitmap.size); +if (ret) +return ret; + } + + ret =vfio_dma_do_unmap (iommu, &unmap, &bitmap);if (ret)return ret;
include/uapi/linux/vfio.h +10 -0
@@ -1048,12 +1048,22 @@ struct vfio_bitmap {* field. No guarantee is made to the user that arbitrary unmaps of iova * or size different from those used in the original mapping call will * succeed. +* VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP should be set to get dirty bitmap +* before unmapping IO virtual addresses. When this flag is set, user must +* provide data[] as structure vfio_bitmap. User must allocate memory to get +* bitmap, zero the bitmap memory and must set size of allocated memory in +* vfio_bitmap.size field. A bit in bitmap represents one page of user provided +* page size in 'pgsize', consecutively starting from iova offset. Bit set +* indicates page at that offset from iova is dirty. Bitmap of pages in the +* range of unmapped size is returned in vfio_bitmap.data */ struct vfio_iommu_type1_dma_unmap {__u32 argsz;__u32 flags; +#define VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP (1 << 0)__u64 iova;/* IO virtual address */ __u64 size;/* Size of mapping (bytes) */ +__u8 data[]; };#define VFIO_IOMMU_UNMAP_DMA _IO(VFIO_TYPE, VFIO_BASE + 14)
Signature · 1 line
2.7.0
Re: [PATCH Kernel v20 6/8] vfio iommu: Update UNMAP_DMA ioctl to get dirty bitmap before unmap
Quoted 101 lines · On Fri, 15 May 2020 02:07:45 +0530 Kirti Wankhede <[email protected]> wrote:
DMA mapped pages, including those pinned by mdev vendor drivers, might get unpinned and unmapped while migration is active and device is still running. For example, in pre-copy phase while guest driver could access those pages, host device or vendor driver can dirty these mapped pages. Such pages should be marked dirty so as to maintain memory consistency for a user making use of dirty page tracking.
To get bitmap during unmap, user should allocate memory for bitmap, set it all zeros, set size of allocated memory, set page size to be considered for bitmap and set flag
VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 77 ++++++++++++++++++++++++++++++++++------- include/uapi/linux/vfio.h | 10 ++++++ 2 files changed, 75 insertions(+), 12 deletions(-)drivers/vfio/vfio_iommu_type1.c +22
-9@@ -195,11 +195,15 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) {uint64_t npages = dma->size / pgsize; +size_t bitmap_size;if (npages >DIRTY_BITMAP_PAGES_MAX )return -EINVAL ; - dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +/* Allocate extra 64 bits which are used for bitmap manipulation */ + bitmap_size =DIRTY_BITMAP_BYTES (npages) +sizeof (u64); + + dma->bitmap =kvzalloc (bitmap_size,GFP_KERNEL );if (!dma->bitmap)return -ENOMEM ;@@ -999,23 +1003,25 @@ static int verify_bitmap_size(uint64_t npages, uint64_t bitmap_size) }static int vfio_dma_do_unmap (struct vfio_iommu *iommu, -struct vfio_iommu_type1_dma_unmap *unmap) +struct vfio_iommu_type1_dma_unmap *unmap, +struct vfio_bitmap *bitmap) { -uint64_t mask;struct vfio_dma *dma, *dma_last =NULL ; -size_t unmapped =0 ; +size_t unmapped =0 , pgsize;int ret =0 , retries =0 ; +unsigned long pgshift;mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgshift =__ffs (iommu->pgsize_bitmap); + pgsize = (size_t )1 << pgshift; -if (unmap->iova & mask) { +if (unmap->iova & (pgsize -1 )) { ret = -EINVAL ; goto unlock; } -if (!unmap->size || unmap->size & mask) { +if (!unmap->size || unmap->size & (pgsize -1 )) { ret = -EINVAL ; goto unlock; }@@ -1026,9 +1032,15 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, goto unlock; } -WARN_ON (mask &PAGE_MASK ); -again: +/* When dirty tracking is enabled, allow only min supported pgsize */ +if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (!iommu->dirty_page_tracking || (bitmap->pgsize != pgsize))) { + ret = -EINVAL ; + goto unlock; + } +WARN_ON ((pgsize -1 ) &PAGE_MASK ); +again:/* * vfio-iommu-type1 (v1) - User mappings were coalesced together to * avoid tracking individual mappings. This means that the granularity @@ -1066,6 +1078,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, ret = -EINVAL ; goto unlock; } + dma =vfio_find_dma (iommu, unmap->iova + unmap->size -1 ,0 );if (dma && dma->iova + dma->size != unmap->iova + unmap->size) { ret = -EINVAL ;@@ -1083,6 +1096,23 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu,if (dma->task->mm != current->mm)break ;
drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1108
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (dma_last != dma)) { + +/* +* mark all pages dirty if all pages are pinned and +* mapped +*/ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , + dma->size >> pgshift);
Nit, all the callers of update_user_bitmap() precede the call with this identical operation, we should probably push it into the function to do it.
drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1115
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+ + ret =update_user_bitmap (bitmap->data, dma, + unmap->iova, pgsize); +if (ret) +break ; + } +
As noted last time, the above is just busy work if pfn_list is not already empty. The entire code block above should be moved to after the block below. Thanks,
Alex
Quoted 74 lines
drivers/vfio/vfio_iommu_type1.c +26
-3if (!RB_EMPTY_ROOT (&dma->pfn_list)) {struct vfio_iommu_type1_dma_unmap nb_unmap;@@ -2447,17 +2477,40 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, }else if (cmd ==VFIO_IOMMU_UNMAP_DMA ) {struct vfio_iommu_type1_dma_unmap unmap; -long ret; +struct vfio_bitmap bitmap = {0 }; +int ret; minsz = offsetofend(struct vfio_iommu_type1_dma_unmap, size);if (copy_from_user (&unmap, (void __user *)arg, minsz))return -EFAULT ; -if (unmap.argsz < minsz || unmap.flags) +if (unmap.argsz < minsz || + unmap.flags & ~VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP )return -EINVAL ; - ret =vfio_dma_do_unmap (iommu, &unmap); +if (unmap.flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) { +unsigned long pgshift; + +if (unmap.argsz < (minsz +sizeof (bitmap))) +return -EINVAL ; + +if (copy_from_user (&bitmap, + (void __user *)(arg + minsz), +sizeof (bitmap))) +return -EFAULT ; + +if (!access_ok ((void __user *)bitmap.data, bitmap.size)) +return -EINVAL ; + + pgshift =__ffs (bitmap.pgsize); + ret =verify_bitmap_size (unmap.size >> pgshift, + bitmap.size); +if (ret) +return ret; + } + + ret =vfio_dma_do_unmap (iommu, &unmap, &bitmap);if (ret)return ret;include/uapi/linux/vfio.h +10
-0@@ -1048,12 +1048,22 @@ struct vfio_bitmap {* field. No guarantee is made to the user that arbitrary unmaps of iova * or size different from those used in the original mapping call will * succeed. +* VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP should be set to get dirty bitmap +* before unmapping IO virtual addresses. When this flag is set, user must +* provide data[] as structure vfio_bitmap. User must allocate memory to get +* bitmap, zero the bitmap memory and must set size of allocated memory in +* vfio_bitmap.size field. A bit in bitmap represents one page of user provided +* page size in 'pgsize', consecutively starting from iova offset. Bit set +* indicates page at that offset from iova is dirty. Bitmap of pages in the +* range of unmapped size is returned in vfio_bitmap.data */ struct vfio_iommu_type1_dma_unmap {__u32 argsz;__u32 flags; +#define VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP (1 << 0)__u64 iova;/* IO virtual address */ __u64 size;/* Size of mapping (bytes) */ +__u8 data[]; };#define VFIO_IOMMU_UNMAP_DMA _IO(VFIO_TYPE, VFIO_BASE + 14)
Re: [PATCH Kernel v20 6/8] vfio iommu: Update UNMAP_DMA ioctl to get dirty bitmap before unmap
Quoted 119 lines · Alex Williamson
Quoted 101 lines · On Fri, 15 May 2020 02:07:45 +0530 Kirti Wankhede <[email protected]> wrote:
DMA mapped pages, including those pinned by mdev vendor drivers, might get unpinned and unmapped while migration is active and device is still running. For example, in pre-copy phase while guest driver could access those pages, host device or vendor driver can dirty these mapped pages. Such pages should be marked dirty so as to maintain memory consistency for a user making use of dirty page tracking.
To get bitmap during unmap, user should allocate memory for bitmap, set it all zeros, set size of allocated memory, set page size to be considered for bitmap and set flag
VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 77 ++++++++++++++++++++++++++++++++++------- include/uapi/linux/vfio.h | 10 ++++++ 2 files changed, 75 insertions(+), 12 deletions(-)drivers/vfio/vfio_iommu_type1.c +22
-9@@ -195,11 +195,15 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) {uint64_t npages = dma->size / pgsize; +size_t bitmap_size;if (npages >DIRTY_BITMAP_PAGES_MAX )return -EINVAL ; - dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +/* Allocate extra 64 bits which are used for bitmap manipulation */ + bitmap_size =DIRTY_BITMAP_BYTES (npages) +sizeof (u64); + + dma->bitmap =kvzalloc (bitmap_size,GFP_KERNEL );if (!dma->bitmap)return -ENOMEM ;@@ -999,23 +1003,25 @@ static int verify_bitmap_size(uint64_t npages, uint64_t bitmap_size) }static int vfio_dma_do_unmap (struct vfio_iommu *iommu, -struct vfio_iommu_type1_dma_unmap *unmap) +struct vfio_iommu_type1_dma_unmap *unmap, +struct vfio_bitmap *bitmap) { -uint64_t mask;struct vfio_dma *dma, *dma_last =NULL ; -size_t unmapped =0 ; +size_t unmapped =0 , pgsize;int ret =0 , retries =0 ; +unsigned long pgshift;mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgshift =__ffs (iommu->pgsize_bitmap); + pgsize = (size_t )1 << pgshift; -if (unmap->iova & mask) { +if (unmap->iova & (pgsize -1 )) { ret = -EINVAL ; goto unlock; } -if (!unmap->size || unmap->size & mask) { +if (!unmap->size || unmap->size & (pgsize -1 )) { ret = -EINVAL ; goto unlock; }@@ -1026,9 +1032,15 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, goto unlock; } -WARN_ON (mask &PAGE_MASK ); -again: +/* When dirty tracking is enabled, allow only min supported pgsize */ +if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (!iommu->dirty_page_tracking || (bitmap->pgsize != pgsize))) { + ret = -EINVAL ; + goto unlock; + } +WARN_ON ((pgsize -1 ) &PAGE_MASK ); +again:/* * vfio-iommu-type1 (v1) - User mappings were coalesced together to * avoid tracking individual mappings. This means that the granularity @@ -1066,6 +1078,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, ret = -EINVAL ; goto unlock; } + dma =vfio_find_dma (iommu, unmap->iova + unmap->size -1 ,0 );if (dma && dma->iova + dma->size != unmap->iova + unmap->size) { ret = -EINVAL ;@@ -1083,6 +1096,23 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu,if (dma->task->mm != current->mm)break ;drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1108
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (dma_last != dma)) { + +/* +* mark all pages dirty if all pages are pinned and +* mapped +*/ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , + dma->size >> pgshift);Nit, all the callers of
update_user_bitmap()precede the call with this identical operation, we should probably push it into the function to do it.
Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1115
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+ + ret =update_user_bitmap (bitmap->data, dma, + unmap->iova, pgsize); +if (ret) +break ; + } +As noted last time, the above is just busy work if
pfn_listis not already empty. The entire code block above should be moved to after the block below. Thanks,
pfn_list will be empty for IOMMU backed devices where all pages are pinned and mapped, but those should be reported as dirty. So moved it back above empty pfn_list check.
Thanks,
Kirti
Quoted 77 lines
Alex
Quoted 74 lines
drivers/vfio/vfio_iommu_type1.c +26
-3if (!RB_EMPTY_ROOT (&dma->pfn_list)) {struct vfio_iommu_type1_dma_unmap nb_unmap;@@ -2447,17 +2477,40 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, }else if (cmd ==VFIO_IOMMU_UNMAP_DMA ) {struct vfio_iommu_type1_dma_unmap unmap; -long ret; +struct vfio_bitmap bitmap = {0 }; +int ret; minsz = offsetofend(struct vfio_iommu_type1_dma_unmap, size);if (copy_from_user (&unmap, (void __user *)arg, minsz))return -EFAULT ; -if (unmap.argsz < minsz || unmap.flags) +if (unmap.argsz < minsz || + unmap.flags & ~VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP )return -EINVAL ; - ret =vfio_dma_do_unmap (iommu, &unmap); +if (unmap.flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) { +unsigned long pgshift; + +if (unmap.argsz < (minsz +sizeof (bitmap))) +return -EINVAL ; + +if (copy_from_user (&bitmap, + (void __user *)(arg + minsz), +sizeof (bitmap))) +return -EFAULT ; + +if (!access_ok ((void __user *)bitmap.data, bitmap.size)) +return -EINVAL ; + + pgshift =__ffs (bitmap.pgsize); + ret =verify_bitmap_size (unmap.size >> pgshift, + bitmap.size); +if (ret) +return ret; + } + + ret =vfio_dma_do_unmap (iommu, &unmap, &bitmap);if (ret)return ret;include/uapi/linux/vfio.h +10
-0@@ -1048,12 +1048,22 @@ struct vfio_bitmap {* field. No guarantee is made to the user that arbitrary unmaps of iova * or size different from those used in the original mapping call will * succeed. +* VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP should be set to get dirty bitmap +* before unmapping IO virtual addresses. When this flag is set, user must +* provide data[] as structure vfio_bitmap. User must allocate memory to get +* bitmap, zero the bitmap memory and must set size of allocated memory in +* vfio_bitmap.size field. A bit in bitmap represents one page of user provided +* page size in 'pgsize', consecutively starting from iova offset. Bit set +* indicates page at that offset from iova is dirty. Bitmap of pages in the +* range of unmapped size is returned in vfio_bitmap.data */ struct vfio_iommu_type1_dma_unmap {__u32 argsz;__u32 flags; +#define VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP (1 << 0)__u64 iova;/* IO virtual address */ __u64 size;/* Size of mapping (bytes) */ +__u8 data[]; };#define VFIO_IOMMU_UNMAP_DMA _IO(VFIO_TYPE, VFIO_BASE + 14)
Re: [PATCH Kernel v20 6/8] vfio iommu: Update UNMAP_DMA ioctl to get dirty bitmap before unmap
Quoted 133 lines · On Fri, 15 May 2020 09:46:43 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 119 lines · Alex Williamson
Quoted 101 lines · On Fri, 15 May 2020 02:07:45 +0530 Kirti Wankhede <[email protected]> wrote:
DMA mapped pages, including those pinned by mdev vendor drivers, might get unpinned and unmapped while migration is active and device is still running. For example, in pre-copy phase while guest driver could access those pages, host device or vendor driver can dirty these mapped pages. Such pages should be marked dirty so as to maintain memory consistency for a user making use of dirty page tracking.
To get bitmap during unmap, user should allocate memory for bitmap, set it all zeros, set size of allocated memory, set page size to be considered for bitmap and set flag
VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 77 ++++++++++++++++++++++++++++++++++------- include/uapi/linux/vfio.h | 10 ++++++ 2 files changed, 75 insertions(+), 12 deletions(-)drivers/vfio/vfio_iommu_type1.c +22
-9@@ -195,11 +195,15 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) {uint64_t npages = dma->size / pgsize; +size_t bitmap_size;if (npages >DIRTY_BITMAP_PAGES_MAX )return -EINVAL ; - dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +/* Allocate extra 64 bits which are used for bitmap manipulation */ + bitmap_size =DIRTY_BITMAP_BYTES (npages) +sizeof (u64); + + dma->bitmap =kvzalloc (bitmap_size,GFP_KERNEL );if (!dma->bitmap)return -ENOMEM ;@@ -999,23 +1003,25 @@ static int verify_bitmap_size(uint64_t npages, uint64_t bitmap_size) }static int vfio_dma_do_unmap (struct vfio_iommu *iommu, -struct vfio_iommu_type1_dma_unmap *unmap) +struct vfio_iommu_type1_dma_unmap *unmap, +struct vfio_bitmap *bitmap) { -uint64_t mask;struct vfio_dma *dma, *dma_last =NULL ; -size_t unmapped =0 ; +size_t unmapped =0 , pgsize;int ret =0 , retries =0 ; +unsigned long pgshift;mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgshift =__ffs (iommu->pgsize_bitmap); + pgsize = (size_t )1 << pgshift; -if (unmap->iova & mask) { +if (unmap->iova & (pgsize -1 )) { ret = -EINVAL ; goto unlock; } -if (!unmap->size || unmap->size & mask) { +if (!unmap->size || unmap->size & (pgsize -1 )) { ret = -EINVAL ; goto unlock; }@@ -1026,9 +1032,15 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, goto unlock; } -WARN_ON (mask &PAGE_MASK ); -again: +/* When dirty tracking is enabled, allow only min supported pgsize */ +if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (!iommu->dirty_page_tracking || (bitmap->pgsize != pgsize))) { + ret = -EINVAL ; + goto unlock; + } +WARN_ON ((pgsize -1 ) &PAGE_MASK ); +again:/* * vfio-iommu-type1 (v1) - User mappings were coalesced together to * avoid tracking individual mappings. This means that the granularity @@ -1066,6 +1078,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, ret = -EINVAL ; goto unlock; } + dma =vfio_find_dma (iommu, unmap->iova + unmap->size -1 ,0 );if (dma && dma->iova + dma->size != unmap->iova + unmap->size) { ret = -EINVAL ;@@ -1083,6 +1096,23 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu,if (dma->task->mm != current->mm)break ;drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1108
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (dma_last != dma)) { + +/* +* mark all pages dirty if all pages are pinned and +* mapped +*/ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , + dma->size >> pgshift);Nit, all the callers of
update_user_bitmap()precede the call with this identical operation, we should probably push it into the function to do it.Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1115
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+ + ret =update_user_bitmap (bitmap->data, dma, + unmap->iova, pgsize); +if (ret) +break ; + } +As noted last time, the above is just busy work if
pfn_listis not already empty. The entire code block above should be moved to after the block below. Thanks,
Quoted 2 lines
pfn_listwill be empty for IOMMU backed devices where all pages are pinned and mapped,
Unless we're making use of the selective dirtying introduced in patch 8/8 or the container is shared with non-IOMMU backed mdevs.
Quoted 1 line
but those should be reported as dirty.
I'm confused how that justifies or requires this ordering.
Quoted 2 lines
So moved it
back above emptypfn_listcheck.
Sorry, it still doesn't make any sense to me, and with no discussion I can't differentiate ignored comments from discarded comments.
Pages in the pfn_list contribute to the dirty bitmap when they're pinned, we don't depend on pfn_list when reporting the dirty bitmap except for re-populating pfn_list dirtied pages after the bitmap has been cleared. We're unmapping the dma, so that's not the case here. Also since update_user_bitmap() shifts the bitmap in place now, any repetitive calls will give us incorrect results. Therefore, as I see it, we _can_ take the branch below and when we do any work we've done above is not only wasted but may lead to incorrect data copied to the user if we shift dma->bitmap in place more than once. Please explain in more detail if you believe this is still correct. Thanks,
Alex
Quoted 77 lines
Quoted 76 lines
Quoted 75 lines
drivers/vfio/vfio_iommu_type1.c +26
-3if (!RB_EMPTY_ROOT (&dma->pfn_list)) {struct vfio_iommu_type1_dma_unmap nb_unmap;@@ -2447,17 +2477,40 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, }else if (cmd ==VFIO_IOMMU_UNMAP_DMA ) {struct vfio_iommu_type1_dma_unmap unmap; -long ret; +struct vfio_bitmap bitmap = {0 }; +int ret; minsz = offsetofend(struct vfio_iommu_type1_dma_unmap, size);if (copy_from_user (&unmap, (void __user *)arg, minsz))return -EFAULT ; -if (unmap.argsz < minsz || unmap.flags) +if (unmap.argsz < minsz || + unmap.flags & ~VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP )return -EINVAL ; - ret =vfio_dma_do_unmap (iommu, &unmap); +if (unmap.flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) { +unsigned long pgshift; + +if (unmap.argsz < (minsz +sizeof (bitmap))) +return -EINVAL ; + +if (copy_from_user (&bitmap, + (void __user *)(arg + minsz), +sizeof (bitmap))) +return -EFAULT ; + +if (!access_ok ((void __user *)bitmap.data, bitmap.size)) +return -EINVAL ; + + pgshift =__ffs (bitmap.pgsize); + ret =verify_bitmap_size (unmap.size >> pgshift, + bitmap.size); +if (ret) +return ret; + } + + ret =vfio_dma_do_unmap (iommu, &unmap, &bitmap);if (ret)return ret;include/uapi/linux/vfio.h +10
-0@@ -1048,12 +1048,22 @@ struct vfio_bitmap {* field. No guarantee is made to the user that arbitrary unmaps of iova * or size different from those used in the original mapping call will * succeed. +* VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP should be set to get dirty bitmap +* before unmapping IO virtual addresses. When this flag is set, user must +* provide data[] as structure vfio_bitmap. User must allocate memory to get +* bitmap, zero the bitmap memory and must set size of allocated memory in +* vfio_bitmap.size field. A bit in bitmap represents one page of user provided +* page size in 'pgsize', consecutively starting from iova offset. Bit set +* indicates page at that offset from iova is dirty. Bitmap of pages in the +* range of unmapped size is returned in vfio_bitmap.data */ struct vfio_iommu_type1_dma_unmap {__u32 argsz;__u32 flags; +#define VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP (1 << 0)__u64 iova;/* IO virtual address */ __u64 size;/* Size of mapping (bytes) */ +__u8 data[]; };#define VFIO_IOMMU_UNMAP_DMA _IO(VFIO_TYPE, VFIO_BASE + 14)
Re: [PATCH Kernel v20 6/8] vfio iommu: Update UNMAP_DMA ioctl to get dirty bitmap before unmap
Quoted 139 lines · Alex Williamson
Quoted 133 lines · On Fri, 15 May 2020 09:46:43 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 119 lines · Alex Williamson
Quoted 101 lines · On Fri, 15 May 2020 02:07:45 +0530 Kirti Wankhede <[email protected]> wrote:
DMA mapped pages, including those pinned by mdev vendor drivers, might get unpinned and unmapped while migration is active and device is still running. For example, in pre-copy phase while guest driver could access those pages, host device or vendor driver can dirty these mapped pages. Such pages should be marked dirty so as to maintain memory consistency for a user making use of dirty page tracking.
To get bitmap during unmap, user should allocate memory for bitmap, set it all zeros, set size of allocated memory, set page size to be considered for bitmap and set flag
VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 77 ++++++++++++++++++++++++++++++++++------- include/uapi/linux/vfio.h | 10 ++++++ 2 files changed, 75 insertions(+), 12 deletions(-)drivers/vfio/vfio_iommu_type1.c +22
-9@@ -195,11 +195,15 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) {uint64_t npages = dma->size / pgsize; +size_t bitmap_size;if (npages >DIRTY_BITMAP_PAGES_MAX )return -EINVAL ; - dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +/* Allocate extra 64 bits which are used for bitmap manipulation */ + bitmap_size =DIRTY_BITMAP_BYTES (npages) +sizeof (u64); + + dma->bitmap =kvzalloc (bitmap_size,GFP_KERNEL );if (!dma->bitmap)return -ENOMEM ;@@ -999,23 +1003,25 @@ static int verify_bitmap_size(uint64_t npages, uint64_t bitmap_size) }static int vfio_dma_do_unmap (struct vfio_iommu *iommu, -struct vfio_iommu_type1_dma_unmap *unmap) +struct vfio_iommu_type1_dma_unmap *unmap, +struct vfio_bitmap *bitmap) { -uint64_t mask;struct vfio_dma *dma, *dma_last =NULL ; -size_t unmapped =0 ; +size_t unmapped =0 , pgsize;int ret =0 , retries =0 ; +unsigned long pgshift;mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgshift =__ffs (iommu->pgsize_bitmap); + pgsize = (size_t )1 << pgshift; -if (unmap->iova & mask) { +if (unmap->iova & (pgsize -1 )) { ret = -EINVAL ; goto unlock; } -if (!unmap->size || unmap->size & mask) { +if (!unmap->size || unmap->size & (pgsize -1 )) { ret = -EINVAL ; goto unlock; }@@ -1026,9 +1032,15 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, goto unlock; } -WARN_ON (mask &PAGE_MASK ); -again: +/* When dirty tracking is enabled, allow only min supported pgsize */ +if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (!iommu->dirty_page_tracking || (bitmap->pgsize != pgsize))) { + ret = -EINVAL ; + goto unlock; + } +WARN_ON ((pgsize -1 ) &PAGE_MASK ); +again:/* * vfio-iommu-type1 (v1) - User mappings were coalesced together to * avoid tracking individual mappings. This means that the granularity @@ -1066,6 +1078,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, ret = -EINVAL ; goto unlock; } + dma =vfio_find_dma (iommu, unmap->iova + unmap->size -1 ,0 );if (dma && dma->iova + dma->size != unmap->iova + unmap->size) { ret = -EINVAL ;@@ -1083,6 +1096,23 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu,if (dma->task->mm != current->mm)break ;drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1108
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (dma_last != dma)) { + +/* +* mark all pages dirty if all pages are pinned and +* mapped +*/ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , + dma->size >> pgshift);Nit, all the callers of
update_user_bitmap()precede the call with this identical operation, we should probably push it into the function to do it.Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1115
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+ + ret =update_user_bitmap (bitmap->data, dma, + unmap->iova, pgsize); +if (ret) +break ; + } +As noted last time, the above is just busy work if
pfn_listis not already empty. The entire code block above should be moved to after the block below. Thanks,Quoted 2 lines
pfn_listwill be empty for IOMMU backed devices where all pages are pinned and mapped,
Quoted 7 lines
Unless we're making use of the selective dirtying introduced in patch 8/8 or the container is shared with non-IOMMU backed mdevs.
Quoted 1 line
but those should be reported as dirty.
I'm confused how that justifies or requires this ordering.
1. non IOMMU mdev device:
- vendor driver pins pages
- pfn_list is not empty
- device dma or write to pinned pages
2. IOMMU backed mdev device or vfio device, but smart driver which pins required pages
- vendor driver pins pages
- pfn_list is not empty
- device dma or write to pinned pages
3. IOMMU backed mdev device or vfio device, driver is not smart
- pages are pinned and mapped during attach
- pfn_list is empty
- device dma or write to any of pinned pages
For case 3, here this function does bitmap_set(dma->bitmap), that is mark all pages dirty and then accordingly copy bitmap to user buffer. Copying dma->bitmap logic remains same.
Quoted 6 lines
Quoted 2 lines
So moved it
back above emptypfn_listcheck.Sorry, it still doesn't make any sense to me, and with no discussion I can't differentiate ignored comments from discarded comments.
Quoted 6 lines
Pages in the
pfn_listcontribute to the dirty bitmap when they're pinned, we don't depend onpfn_listwhen reporting the dirty bitmap except for re-populatingpfn_listdirtied pages after the bitmap has been cleared. We're unmapping the dma, so that's not the case here. Also sinceupdate_user_bitmap()shifts the bitmap in place now, any repetitive calls will give us incorrect results.
Right, but this is unmapping and freeing vfio_dma
Quoted 6 lines
Therefore, as I see
it, we _can_ take the branch below and when we do any work we've done above is not only wasted but may lead to incorrect data copied to the user if we shiftdma->bitmapin place more than once. Please explain in more detail if you believe this is still correct. Thanks,
In this case also bitmap copy to user happens once, (dma_last != dma) takes care of making sure that its called only once.
Thanks,
Kirti
Quoted 80 lines
Alex
Quoted 77 lines
Quoted 76 lines
Quoted 75 lines
drivers/vfio/vfio_iommu_type1.c +26
-3if (!RB_EMPTY_ROOT (&dma->pfn_list)) {struct vfio_iommu_type1_dma_unmap nb_unmap;@@ -2447,17 +2477,40 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, }else if (cmd ==VFIO_IOMMU_UNMAP_DMA ) {struct vfio_iommu_type1_dma_unmap unmap; -long ret; +struct vfio_bitmap bitmap = {0 }; +int ret; minsz = offsetofend(struct vfio_iommu_type1_dma_unmap, size);if (copy_from_user (&unmap, (void __user *)arg, minsz))return -EFAULT ; -if (unmap.argsz < minsz || unmap.flags) +if (unmap.argsz < minsz || + unmap.flags & ~VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP )return -EINVAL ; - ret =vfio_dma_do_unmap (iommu, &unmap); +if (unmap.flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) { +unsigned long pgshift; + +if (unmap.argsz < (minsz +sizeof (bitmap))) +return -EINVAL ; + +if (copy_from_user (&bitmap, + (void __user *)(arg + minsz), +sizeof (bitmap))) +return -EFAULT ; + +if (!access_ok ((void __user *)bitmap.data, bitmap.size)) +return -EINVAL ; + + pgshift =__ffs (bitmap.pgsize); + ret =verify_bitmap_size (unmap.size >> pgshift, + bitmap.size); +if (ret) +return ret; + } + + ret =vfio_dma_do_unmap (iommu, &unmap, &bitmap);if (ret)return ret;include/uapi/linux/vfio.h +10
-0@@ -1048,12 +1048,22 @@ struct vfio_bitmap {* field. No guarantee is made to the user that arbitrary unmaps of iova * or size different from those used in the original mapping call will * succeed. +* VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP should be set to get dirty bitmap +* before unmapping IO virtual addresses. When this flag is set, user must +* provide data[] as structure vfio_bitmap. User must allocate memory to get +* bitmap, zero the bitmap memory and must set size of allocated memory in +* vfio_bitmap.size field. A bit in bitmap represents one page of user provided +* page size in 'pgsize', consecutively starting from iova offset. Bit set +* indicates page at that offset from iova is dirty. Bitmap of pages in the +* range of unmapped size is returned in vfio_bitmap.data */ struct vfio_iommu_type1_dma_unmap {__u32 argsz;__u32 flags; +#define VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP (1 << 0)__u64 iova;/* IO virtual address */ __u64 size;/* Size of mapping (bytes) */ +__u8 data[]; };#define VFIO_IOMMU_UNMAP_DMA _IO(VFIO_TYPE, VFIO_BASE + 14)
Re: [PATCH Kernel v20 6/8] vfio iommu: Update UNMAP_DMA ioctl to get dirty bitmap before unmap
Quoted 159 lines · On Fri, 15 May 2020 12:17:03 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 139 lines · Alex Williamson
Quoted 133 lines · On Fri, 15 May 2020 09:46:43 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 119 lines · Alex Williamson
Quoted 101 lines · On Fri, 15 May 2020 02:07:45 +0530 Kirti Wankhede <[email protected]> wrote:
DMA mapped pages, including those pinned by mdev vendor drivers, might get unpinned and unmapped while migration is active and device is still running. For example, in pre-copy phase while guest driver could access those pages, host device or vendor driver can dirty these mapped pages. Such pages should be marked dirty so as to maintain memory consistency for a user making use of dirty page tracking.
To get bitmap during unmap, user should allocate memory for bitmap, set it all zeros, set size of allocated memory, set page size to be considered for bitmap and set flag
VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 77 ++++++++++++++++++++++++++++++++++------- include/uapi/linux/vfio.h | 10 ++++++ 2 files changed, 75 insertions(+), 12 deletions(-)drivers/vfio/vfio_iommu_type1.c +22
-9@@ -195,11 +195,15 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) {uint64_t npages = dma->size / pgsize; +size_t bitmap_size;if (npages >DIRTY_BITMAP_PAGES_MAX )return -EINVAL ; - dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +/* Allocate extra 64 bits which are used for bitmap manipulation */ + bitmap_size =DIRTY_BITMAP_BYTES (npages) +sizeof (u64); + + dma->bitmap =kvzalloc (bitmap_size,GFP_KERNEL );if (!dma->bitmap)return -ENOMEM ;@@ -999,23 +1003,25 @@ static int verify_bitmap_size(uint64_t npages, uint64_t bitmap_size) }static int vfio_dma_do_unmap (struct vfio_iommu *iommu, -struct vfio_iommu_type1_dma_unmap *unmap) +struct vfio_iommu_type1_dma_unmap *unmap, +struct vfio_bitmap *bitmap) { -uint64_t mask;struct vfio_dma *dma, *dma_last =NULL ; -size_t unmapped =0 ; +size_t unmapped =0 , pgsize;int ret =0 , retries =0 ; +unsigned long pgshift;mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgshift =__ffs (iommu->pgsize_bitmap); + pgsize = (size_t )1 << pgshift; -if (unmap->iova & mask) { +if (unmap->iova & (pgsize -1 )) { ret = -EINVAL ; goto unlock; } -if (!unmap->size || unmap->size & mask) { +if (!unmap->size || unmap->size & (pgsize -1 )) { ret = -EINVAL ; goto unlock; }@@ -1026,9 +1032,15 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, goto unlock; } -WARN_ON (mask &PAGE_MASK ); -again: +/* When dirty tracking is enabled, allow only min supported pgsize */ +if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (!iommu->dirty_page_tracking || (bitmap->pgsize != pgsize))) { + ret = -EINVAL ; + goto unlock; + } +WARN_ON ((pgsize -1 ) &PAGE_MASK ); +again:/* * vfio-iommu-type1 (v1) - User mappings were coalesced together to * avoid tracking individual mappings. This means that the granularity @@ -1066,6 +1078,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, ret = -EINVAL ; goto unlock; } + dma =vfio_find_dma (iommu, unmap->iova + unmap->size -1 ,0 );if (dma && dma->iova + dma->size != unmap->iova + unmap->size) { ret = -EINVAL ;@@ -1083,6 +1096,23 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu,if (dma->task->mm != current->mm)break ;drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1108
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (dma_last != dma)) { + +/* +* mark all pages dirty if all pages are pinned and +* mapped +*/ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , + dma->size >> pgshift);Nit, all the callers of
update_user_bitmap()precede the call with this identical operation, we should probably push it into the function to do it.Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1115
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+ + ret =update_user_bitmap (bitmap->data, dma, + unmap->iova, pgsize); +if (ret) +break ; + } +As noted last time, the above is just busy work if
pfn_listis not already empty. The entire code block above should be moved to after the block below. Thanks,Quoted 2 lines
pfn_listwill be empty for IOMMU backed devices where all pages are pinned and mapped,Quoted 7 lines
Unless we're making use of the selective dirtying introduced in patch 8/8 or the container is shared with non-IOMMU backed mdevs.
Quoted 1 line
but those should be reported as dirty.
I'm confused how that justifies or requires this ordering.
1. non IOMMU mdev device:
- vendor driver pins pages
-pfn_listis not empty
- device dma or write to pinned pages2. IOMMU backed mdev device or vfio device, but smart driver which pins required pages
- vendor driver pins pages
-pfn_listis not empty
- device dma or write to pinned pages
Quoted 8 lines
3. IOMMU backed mdev device or vfio device, driver is not smart
- pages are pinned and mapped during attach
-pfn_listis empty
- device dma or write to any of pinned pagesFor case 3, here this function does
bitmap_set(dma->bitmap), that is mark all pages dirty and then accordingly copy bitmap to user buffer. Copyingdma->bitmaplogic remains same.
What dependency does case 3 have on pfn_list? Why does it matter if that bitmap_set() occurs before or after we've drained pfn_list? How do we know we're in case 3 before we've looked at pfn_list?
Quoted 22 lines
Quoted 6 lines
Quoted 2 lines
So moved it
back above emptypfn_listcheck.Sorry, it still doesn't make any sense to me, and with no discussion I can't differentiate ignored comments from discarded comments.
Quoted 6 lines
Pages in the
pfn_listcontribute to the dirty bitmap when they're pinned, we don't depend onpfn_listwhen reporting the dirty bitmap except for re-populatingpfn_listdirtied pages after the bitmap has been cleared. We're unmapping the dma, so that's not the case here. Also sinceupdate_user_bitmap()shifts the bitmap in place now, any repetitive calls will give us incorrect results.Right, but this is unmapping and freeing
vfio_dmaQuoted 6 lines
Therefore, as I see
it, we _can_ take the branch below and when we do any work we've done above is not only wasted but may lead to incorrect data copied to the user if we shiftdma->bitmapin place more than once. Please explain in more detail if you believe this is still correct. Thanks,
Quoted 2 lines
In this case also bitmap copy to user happens once, (
dma_last!= dma) takes care of making sure that its called only once.
I did miss the dma_last check, so that prevents us from repeating this path, BUT we release iommu->lock if we enter the pfn_list !empty path. The moment we do that, we might unblock a vendor driver trying to pin or rw more pages. So the bitmap we've copied to the user might be stale and incorrect. I don't see why this is so contentious, what is the actual disadvantage to moving this until after we've drained pfn_list? The risks seem abundant to me. Thanks,
Alex
Quoted 79 lines
Quoted 78 lines
Quoted 77 lines
Quoted 76 lines
Quoted 75 lines
drivers/vfio/vfio_iommu_type1.c +26
-3if (!RB_EMPTY_ROOT (&dma->pfn_list)) {struct vfio_iommu_type1_dma_unmap nb_unmap;@@ -2447,17 +2477,40 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, }else if (cmd ==VFIO_IOMMU_UNMAP_DMA ) {struct vfio_iommu_type1_dma_unmap unmap; -long ret; +struct vfio_bitmap bitmap = {0 }; +int ret; minsz = offsetofend(struct vfio_iommu_type1_dma_unmap, size);if (copy_from_user (&unmap, (void __user *)arg, minsz))return -EFAULT ; -if (unmap.argsz < minsz || unmap.flags) +if (unmap.argsz < minsz || + unmap.flags & ~VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP )return -EINVAL ; - ret =vfio_dma_do_unmap (iommu, &unmap); +if (unmap.flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) { +unsigned long pgshift; + +if (unmap.argsz < (minsz +sizeof (bitmap))) +return -EINVAL ; + +if (copy_from_user (&bitmap, + (void __user *)(arg + minsz), +sizeof (bitmap))) +return -EFAULT ; + +if (!access_ok ((void __user *)bitmap.data, bitmap.size)) +return -EINVAL ; + + pgshift =__ffs (bitmap.pgsize); + ret =verify_bitmap_size (unmap.size >> pgshift, + bitmap.size); +if (ret) +return ret; + } + + ret =vfio_dma_do_unmap (iommu, &unmap, &bitmap);if (ret)return ret;include/uapi/linux/vfio.h +10
-0@@ -1048,12 +1048,22 @@ struct vfio_bitmap {* field. No guarantee is made to the user that arbitrary unmaps of iova * or size different from those used in the original mapping call will * succeed. +* VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP should be set to get dirty bitmap +* before unmapping IO virtual addresses. When this flag is set, user must +* provide data[] as structure vfio_bitmap. User must allocate memory to get +* bitmap, zero the bitmap memory and must set size of allocated memory in +* vfio_bitmap.size field. A bit in bitmap represents one page of user provided +* page size in 'pgsize', consecutively starting from iova offset. Bit set +* indicates page at that offset from iova is dirty. Bitmap of pages in the +* range of unmapped size is returned in vfio_bitmap.data */ struct vfio_iommu_type1_dma_unmap {__u32 argsz;__u32 flags; +#define VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP (1 << 0)__u64 iova;/* IO virtual address */ __u64 size;/* Size of mapping (bytes) */ +__u8 data[]; };#define VFIO_IOMMU_UNMAP_DMA _IO(VFIO_TYPE, VFIO_BASE + 14)
Re: [PATCH Kernel v20 6/8] vfio iommu: Update UNMAP_DMA ioctl to get dirty bitmap before unmap
Quoted 167 lines · Alex Williamson
Quoted 164 lines · On Fri, 15 May 2020 12:17:03 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 139 lines · Alex Williamson
Quoted 133 lines · On Fri, 15 May 2020 09:46:43 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 119 lines · Alex Williamson
Quoted 101 lines · On Fri, 15 May 2020 02:07:45 +0530 Kirti Wankhede <[email protected]> wrote:
DMA mapped pages, including those pinned by mdev vendor drivers, might get unpinned and unmapped while migration is active and device is still running. For example, in pre-copy phase while guest driver could access those pages, host device or vendor driver can dirty these mapped pages. Such pages should be marked dirty so as to maintain memory consistency for a user making use of dirty page tracking.
To get bitmap during unmap, user should allocate memory for bitmap, set it all zeros, set size of allocated memory, set page size to be considered for bitmap and set flag
VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 77 ++++++++++++++++++++++++++++++++++------- include/uapi/linux/vfio.h | 10 ++++++ 2 files changed, 75 insertions(+), 12 deletions(-)drivers/vfio/vfio_iommu_type1.c +22
-9@@ -195,11 +195,15 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) {uint64_t npages = dma->size / pgsize; +size_t bitmap_size;if (npages >DIRTY_BITMAP_PAGES_MAX )return -EINVAL ; - dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +/* Allocate extra 64 bits which are used for bitmap manipulation */ + bitmap_size =DIRTY_BITMAP_BYTES (npages) +sizeof (u64); + + dma->bitmap =kvzalloc (bitmap_size,GFP_KERNEL );if (!dma->bitmap)return -ENOMEM ;@@ -999,23 +1003,25 @@ static int verify_bitmap_size(uint64_t npages, uint64_t bitmap_size) }static int vfio_dma_do_unmap (struct vfio_iommu *iommu, -struct vfio_iommu_type1_dma_unmap *unmap) +struct vfio_iommu_type1_dma_unmap *unmap, +struct vfio_bitmap *bitmap) { -uint64_t mask;struct vfio_dma *dma, *dma_last =NULL ; -size_t unmapped =0 ; +size_t unmapped =0 , pgsize;int ret =0 , retries =0 ; +unsigned long pgshift;mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgshift =__ffs (iommu->pgsize_bitmap); + pgsize = (size_t )1 << pgshift; -if (unmap->iova & mask) { +if (unmap->iova & (pgsize -1 )) { ret = -EINVAL ; goto unlock; } -if (!unmap->size || unmap->size & mask) { +if (!unmap->size || unmap->size & (pgsize -1 )) { ret = -EINVAL ; goto unlock; }@@ -1026,9 +1032,15 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, goto unlock; } -WARN_ON (mask &PAGE_MASK ); -again: +/* When dirty tracking is enabled, allow only min supported pgsize */ +if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (!iommu->dirty_page_tracking || (bitmap->pgsize != pgsize))) { + ret = -EINVAL ; + goto unlock; + } +WARN_ON ((pgsize -1 ) &PAGE_MASK ); +again:/* * vfio-iommu-type1 (v1) - User mappings were coalesced together to * avoid tracking individual mappings. This means that the granularity @@ -1066,6 +1078,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, ret = -EINVAL ; goto unlock; } + dma =vfio_find_dma (iommu, unmap->iova + unmap->size -1 ,0 );if (dma && dma->iova + dma->size != unmap->iova + unmap->size) { ret = -EINVAL ;@@ -1083,6 +1096,23 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu,if (dma->task->mm != current->mm)break ;drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1108
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (dma_last != dma)) { + +/* +* mark all pages dirty if all pages are pinned and +* mapped +*/ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , + dma->size >> pgshift);Nit, all the callers of
update_user_bitmap()precede the call with this identical operation, we should probably push it into the function to do it.Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1115
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+ + ret =update_user_bitmap (bitmap->data, dma, + unmap->iova, pgsize); +if (ret) +break ; + } +As noted last time, the above is just busy work if
pfn_listis not already empty. The entire code block above should be moved to after the block below. Thanks,Quoted 2 lines
pfn_listwill be empty for IOMMU backed devices where all pages are pinned and mapped,Quoted 7 lines
Unless we're making use of the selective dirtying introduced in patch 8/8 or the container is shared with non-IOMMU backed mdevs.
Quoted 1 line
but those should be reported as dirty.
I'm confused how that justifies or requires this ordering.
1. non IOMMU mdev device:
- vendor driver pins pages
-pfn_listis not empty
- device dma or write to pinned pages2. IOMMU backed mdev device or vfio device, but smart driver which pins required pages
- vendor driver pins pages
-pfn_listis not empty
- device dma or write to pinned pages3. IOMMU backed mdev device or vfio device, driver is not smart
- pages are pinned and mapped during attach
-pfn_listis empty
- device dma or write to any of pinned pages
Quoted 8 lines
Quoted 3 lines
For case 3, here this function does
bitmap_set(dma->bitmap), that is mark all pages dirty and then accordingly copy bitmap to user buffer. Copyingdma->bitmaplogic remains same.What dependency does case 3 have on
pfn_list? Why does it matter if thatbitmap_set()occurs before or after we've drainedpfn_list? How do we know we're in case 3 before we've looked atpfn_list?
If update_user_bitmap() is moved in if block like:
if (!RB_EMPTY_ROOT(&dma->pfn_list)) {
update_user_bitmap()
...
}
then update_user_bitmap() would never get called for case 3.
Quoted 25 lines
Quoted 22 lines
Quoted 6 lines
Quoted 2 lines
So moved it
back above emptypfn_listcheck.Sorry, it still doesn't make any sense to me, and with no discussion I can't differentiate ignored comments from discarded comments.
Quoted 6 lines
Pages in the
pfn_listcontribute to the dirty bitmap when they're pinned, we don't depend onpfn_listwhen reporting the dirty bitmap except for re-populatingpfn_listdirtied pages after the bitmap has been cleared. We're unmapping the dma, so that's not the case here. Also sinceupdate_user_bitmap()shifts the bitmap in place now, any repetitive calls will give us incorrect results.Right, but this is unmapping and freeing
vfio_dmaQuoted 6 lines
Therefore, as I see
it, we _can_ take the branch below and when we do any work we've done above is not only wasted but may lead to incorrect data copied to the user if we shiftdma->bitmapin place more than once. Please explain in more detail if you believe this is still correct. Thanks,Quoted 2 lines
In this case also bitmap copy to user happens once, (
dma_last!= dma) takes care of making sure that its called only once.
Quoted 8 lines
I did miss the
dma_lastcheck, so that prevents us from repeating this path, BUT we releaseiommu->lockif we enter thepfn_list!empty path. The moment we do that, we might unblock a vendor driver trying to pin or rw more pages. So the bitmap we've copied to the user might be stale and incorrect. I don't see why this is so contentious, what is the actual disadvantage to moving this until after we've drainedpfn_list? The risks seem abundant to me. Thanks,
Sorry if I misinterpreted earlier, are you proposing to move update_user_bitmap() after this if block?
if (!RB_EMPTY_ROOT(&dma->pfn_list)) {
...
}
I thought you are asking to move it in block.
That makes sense. I'll update the patch in send next version in some time.
Thanks,
Kirti
Quoted 82 lines
Alex
Quoted 79 lines
Quoted 78 lines
Quoted 77 lines
Quoted 76 lines
Quoted 75 lines
drivers/vfio/vfio_iommu_type1.c +26
-3if (!RB_EMPTY_ROOT (&dma->pfn_list)) {struct vfio_iommu_type1_dma_unmap nb_unmap;@@ -2447,17 +2477,40 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, }else if (cmd ==VFIO_IOMMU_UNMAP_DMA ) {struct vfio_iommu_type1_dma_unmap unmap; -long ret; +struct vfio_bitmap bitmap = {0 }; +int ret; minsz = offsetofend(struct vfio_iommu_type1_dma_unmap, size);if (copy_from_user (&unmap, (void __user *)arg, minsz))return -EFAULT ; -if (unmap.argsz < minsz || unmap.flags) +if (unmap.argsz < minsz || + unmap.flags & ~VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP )return -EINVAL ; - ret =vfio_dma_do_unmap (iommu, &unmap); +if (unmap.flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) { +unsigned long pgshift; + +if (unmap.argsz < (minsz +sizeof (bitmap))) +return -EINVAL ; + +if (copy_from_user (&bitmap, + (void __user *)(arg + minsz), +sizeof (bitmap))) +return -EFAULT ; + +if (!access_ok ((void __user *)bitmap.data, bitmap.size)) +return -EINVAL ; + + pgshift =__ffs (bitmap.pgsize); + ret =verify_bitmap_size (unmap.size >> pgshift, + bitmap.size); +if (ret) +return ret; + } + + ret =vfio_dma_do_unmap (iommu, &unmap, &bitmap);if (ret)return ret;include/uapi/linux/vfio.h +10
-0@@ -1048,12 +1048,22 @@ struct vfio_bitmap {* field. No guarantee is made to the user that arbitrary unmaps of iova * or size different from those used in the original mapping call will * succeed. +* VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP should be set to get dirty bitmap +* before unmapping IO virtual addresses. When this flag is set, user must +* provide data[] as structure vfio_bitmap. User must allocate memory to get +* bitmap, zero the bitmap memory and must set size of allocated memory in +* vfio_bitmap.size field. A bit in bitmap represents one page of user provided +* page size in 'pgsize', consecutively starting from iova offset. Bit set +* indicates page at that offset from iova is dirty. Bitmap of pages in the +* range of unmapped size is returned in vfio_bitmap.data */ struct vfio_iommu_type1_dma_unmap {__u32 argsz;__u32 flags; +#define VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP (1 << 0)__u64 iova;/* IO virtual address */ __u64 size;/* Size of mapping (bytes) */ +__u8 data[]; };#define VFIO_IOMMU_UNMAP_DMA _IO(VFIO_TYPE, VFIO_BASE + 14)
Re: [PATCH Kernel v20 6/8] vfio iommu: Update UNMAP_DMA ioctl to get dirty bitmap before unmap
Quoted 222 lines · On Fri, 15 May 2020 21:00:32 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 167 lines · Alex Williamson
Quoted 164 lines · On Fri, 15 May 2020 12:17:03 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 139 lines · Alex Williamson
Quoted 133 lines · On Fri, 15 May 2020 09:46:43 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 119 lines · Alex Williamson
Quoted 101 lines · On Fri, 15 May 2020 02:07:45 +0530 Kirti Wankhede <[email protected]> wrote:
DMA mapped pages, including those pinned by mdev vendor drivers, might get unpinned and unmapped while migration is active and device is still running. For example, in pre-copy phase while guest driver could access those pages, host device or vendor driver can dirty these mapped pages. Such pages should be marked dirty so as to maintain memory consistency for a user making use of dirty page tracking.
To get bitmap during unmap, user should allocate memory for bitmap, set it all zeros, set size of allocated memory, set page size to be considered for bitmap and set flag
VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP.Signed-off-by Kirti Wankhede <[email protected]>Reviewed-by Neo Jia <[email protected]>--- drivers/vfio/vfio_iommu_type1.c | 77 ++++++++++++++++++++++++++++++++++------- include/uapi/linux/vfio.h | 10 ++++++ 2 files changed, 75 insertions(+), 12 deletions(-)drivers/vfio/vfio_iommu_type1.c +22
-9@@ -195,11 +195,15 @@ static void vfio_unlink_dma(struct vfio_iommu *iommu, struct vfio_dma *old)static int vfio_dma_bitmap_alloc (struct vfio_dma *dma,size_t pgsize) {uint64_t npages = dma->size / pgsize; +size_t bitmap_size;if (npages >DIRTY_BITMAP_PAGES_MAX )return -EINVAL ; - dma->bitmap =kvzalloc (DIRTY_BITMAP_BYTES (npages),GFP_KERNEL ); +/* Allocate extra 64 bits which are used for bitmap manipulation */ + bitmap_size =DIRTY_BITMAP_BYTES (npages) +sizeof (u64); + + dma->bitmap =kvzalloc (bitmap_size,GFP_KERNEL );if (!dma->bitmap)return -ENOMEM ;@@ -999,23 +1003,25 @@ static int verify_bitmap_size(uint64_t npages, uint64_t bitmap_size) }static int vfio_dma_do_unmap (struct vfio_iommu *iommu, -struct vfio_iommu_type1_dma_unmap *unmap) +struct vfio_iommu_type1_dma_unmap *unmap, +struct vfio_bitmap *bitmap) { -uint64_t mask;struct vfio_dma *dma, *dma_last =NULL ; -size_t unmapped =0 ; +size_t unmapped =0 , pgsize;int ret =0 , retries =0 ; +unsigned long pgshift;mutex_lock (&iommu->lock); - mask = ((uint64_t )1 <<__ffs (iommu->pgsize_bitmap)) -1 ; + pgshift =__ffs (iommu->pgsize_bitmap); + pgsize = (size_t )1 << pgshift; -if (unmap->iova & mask) { +if (unmap->iova & (pgsize -1 )) { ret = -EINVAL ; goto unlock; } -if (!unmap->size || unmap->size & mask) { +if (!unmap->size || unmap->size & (pgsize -1 )) { ret = -EINVAL ; goto unlock; }@@ -1026,9 +1032,15 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, goto unlock; } -WARN_ON (mask &PAGE_MASK ); -again: +/* When dirty tracking is enabled, allow only min supported pgsize */ +if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (!iommu->dirty_page_tracking || (bitmap->pgsize != pgsize))) { + ret = -EINVAL ; + goto unlock; + } +WARN_ON ((pgsize -1 ) &PAGE_MASK ); +again:/* * vfio-iommu-type1 (v1) - User mappings were coalesced together to * avoid tracking individual mappings. This means that the granularity @@ -1066,6 +1078,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, ret = -EINVAL ; goto unlock; } + dma =vfio_find_dma (iommu, unmap->iova + unmap->size -1 ,0 );if (dma && dma->iova + dma->size != unmap->iova + unmap->size) { ret = -EINVAL ;@@ -1083,6 +1096,23 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu,if (dma->task->mm != current->mm)break ;drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1108
Quoted 10 lines
drivers/vfio/vfio_iommu_type1.c +10
-0+if ((unmap->flags &VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP ) && + (dma_last != dma)) { + +/* +* mark all pages dirty if all pages are pinned and +* mapped +*/ +if (dma->iommu_mapped) +bitmap_set (dma->bitmap,0 , + dma->size >> pgshift);Nit, all the callers of
update_user_bitmap()precede the call with this identical operation, we should probably push it into the function to do it.Quoted 12 lines
drivers/vfio/vfio_iommu_type1.c static int vfio_dma_do_unmap(struct vfio_iommu *iommu, · line 1115
Quoted 7 lines
drivers/vfio/vfio_iommu_type1.c +7
-0+ + ret =update_user_bitmap (bitmap->data, dma, + unmap->iova, pgsize); +if (ret) +break ; + } +As noted last time, the above is just busy work if
pfn_listis not already empty. The entire code block above should be moved to after the block below. Thanks,Quoted 2 lines
pfn_listwill be empty for IOMMU backed devices where all pages are pinned and mapped,Quoted 7 lines
Unless we're making use of the selective dirtying introduced in patch 8/8 or the container is shared with non-IOMMU backed mdevs.
Quoted 1 line
but those should be reported as dirty.
I'm confused how that justifies or requires this ordering.
1. non IOMMU mdev device:
- vendor driver pins pages
-pfn_listis not empty
- device dma or write to pinned pages2. IOMMU backed mdev device or vfio device, but smart driver which pins required pages
- vendor driver pins pages
-pfn_listis not empty
- device dma or write to pinned pages3. IOMMU backed mdev device or vfio device, driver is not smart
- pages are pinned and mapped during attach
-pfn_listis empty
- device dma or write to any of pinned pagesQuoted 8 lines
Quoted 3 lines
For case 3, here this function does
bitmap_set(dma->bitmap), that is mark all pages dirty and then accordingly copy bitmap to user buffer. Copyingdma->bitmaplogic remains same.What dependency does case 3 have on
pfn_list? Why does it matter if thatbitmap_set()occurs before or after we've drainedpfn_list? How do we know we're in case 3 before we've looked atpfn_list?If
update_user_bitmap()is moved in if block like:
if (!RB_EMPTY_ROOT(&dma->pfn_list)) {
update_user_bitmap()
...
}then
update_user_bitmap()would never get called for case 3.Quoted 25 lines
Quoted 22 lines
Quoted 6 lines
Quoted 2 lines
So moved it
back above emptypfn_listcheck.Sorry, it still doesn't make any sense to me, and with no discussion I can't differentiate ignored comments from discarded comments.
Quoted 6 lines
Pages in the
pfn_listcontribute to the dirty bitmap when they're pinned, we don't depend onpfn_listwhen reporting the dirty bitmap except for re-populatingpfn_listdirtied pages after the bitmap has been cleared. We're unmapping the dma, so that's not the case here. Also sinceupdate_user_bitmap()shifts the bitmap in place now, any repetitive calls will give us incorrect results.Right, but this is unmapping and freeing
vfio_dmaQuoted 6 lines
Therefore, as I see
it, we _can_ take the branch below and when we do any work we've done above is not only wasted but may lead to incorrect data copied to the user if we shiftdma->bitmapin place more than once. Please explain in more detail if you believe this is still correct. Thanks,Quoted 2 lines
In this case also bitmap copy to user happens once, (
dma_last!= dma) takes care of making sure that its called only once.Quoted 8 lines
I did miss the
dma_lastcheck, so that prevents us from repeating this path, BUT we releaseiommu->lockif we enter thepfn_list!empty path. The moment we do that, we might unblock a vendor driver trying to pin or rw more pages. So the bitmap we've copied to the user might be stale and incorrect. I don't see why this is so contentious, what is the actual disadvantage to moving this until after we've drainedpfn_list? The risks seem abundant to me. Thanks,Sorry if I misinterpreted earlier, are you proposing to move
update_user_bitmap()after this if block?
Quoted 6 lines
if (!RB_EMPTY_ROOT(&dma->pfn_list)) {
...
}I thought you are asking to move it in block.
That makes sense. I'll update the patch in send next version in some time.
Yes, after not in. Thanks,
Alex
PATCH v20 7/8 vfio iommu: Add migration capability to report supported features
In reply to Kirti WankhedeAdded migration capability in IOMMU info chain.
User application should check IOMMU info chain for migration capability to use dirty page tracking feature provided by kernel module. User application must check page sizes supported and maximum dirty bitmap size returned by this capability structure for ioctls used to get dirty bitmap.
--- drivers/vfio/vfio_iommu_type1.c | 23 ++++++++++++++++++++++- include/uapi/linux/vfio.h | 22 ++++++++++++++++++++++ 2 files changed, 44 insertions(+), 1 deletion(-)
drivers/vfio/vfio_iommu_type1.c +22 -1
@@ -2387,6 +2387,22 @@ static int vfio_iommu_iova_build_caps(struct vfio_iommu *iommu,return ret; } +static int vfio_iommu_migration_build_caps (struct vfio_iommu *iommu, +struct vfio_info_cap *caps) +{ +struct vfio_iommu_type1_info_cap_migration cap_mig; + + cap_mig.header.id =VFIO_IOMMU_TYPE1_INFO_CAP_MIGRATION ; + cap_mig.header.version =1 ; + + cap_mig.flags =0 ; +/* support minimum pgsize */ + cap_mig.pgsize_bitmap = (size_t )1 <<__ffs (iommu->pgsize_bitmap); + cap_mig.max_dirty_bitmap_size =DIRTY_BITMAP_SIZE_MAX ; + +return vfio_info_add_capability (caps, &cap_mig.header,sizeof (cap_mig)); +} +static long vfio_iommu_type1_ioctl (void *iommu_data,unsigned int cmd,unsigned long arg) {
@@ -2433,8 +2449,13 @@ static long vfio_iommu_type1_ioctl(void *iommu_data, info.iova_pgsizes = iommu->pgsize_bitmap; - ret =vfio_iommu_iova_build_caps (iommu, &caps); + ret =vfio_iommu_migration_build_caps (iommu, &caps); + +if (!ret) + ret =vfio_iommu_iova_build_caps (iommu, &caps); +mutex_unlock (&iommu->lock); +if (ret)return ret;
include/uapi/linux/vfio.h +22 -0
@@ -1013,6 +1013,28 @@ struct vfio_iommu_type1_info_cap_iova_range {struct vfio_iova_range iova_ranges[]; }; +/* +* The migration capability allows to report supported features for migration. +* +* The structures below define version 1 of this capability. +* +* The existence of this capability indicates IOMMU kernel driver supports +* dirty page tracking. +* +* pgsize_bitmap: Kernel driver returns supported page sizes bitmap for dirty +* page tracking. +* max_dirty_bitmap_size: Kernel driver returns maximum supported dirty bitmap +* size in bytes to be used by user application for ioctls to get dirty bitmap. +*/ +#define VFIO_IOMMU_TYPE1_INFO_CAP_MIGRATION 1 + +struct vfio_iommu_type1_info_cap_migration { +struct vfio_info_cap_header header; +__u32 flags; +__u64 pgsize_bitmap; +__u64 max_dirty_bitmap_size;/* in bytes */ +}; +#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)/**
Signature · 1 line
2.7.0
PATCH v20 8/8 vfio: Selective dirty page tracking if IOMMU backed device pins pages
In reply to Kirti WankhedeAdded a check such that only singleton IOMMU groups can pin pages. From the point when vendor driver pins any pages, consider IOMMU group dirty page scope to be limited to pinned pages.
To optimize to avoid walking list often, added flag pinned_page_dirty_scope to indicate if all of the vfio_groups for each vfio_domain in the domain_list dirty page scope is limited to pinned pages. This flag is updated on first pinned pages request for that IOMMU group and on attaching/detaching group.
--- drivers/vfio/vfio.c | 13 +++-- drivers/vfio/vfio_iommu_type1.c | 104 ++++++++++++++++++++++++++++++++++++---- include/linux/vfio.h | 4 +- 3 files changed, 109 insertions(+), 12 deletions(-)
drivers/vfio/vfio.c +10 -3
@@ -85,6 +85,7 @@ struct vfio_group {atomic_t opened;wait_queue_head_t container_q;bool noiommu; +unsigned int dev_counter;struct kvm *kvm;struct blocking_notifier_head notifier; };
@@ -555,6 +556,7 @@ struct vfio_device *vfio_group_create_device(struct vfio_group *group,mutex_lock (&group->device_lock);list_add (&device->group_next, &group->device_list); + group->dev_counter++;mutex_unlock (&group->device_lock);return device;
@@ -567,6 +569,7 @@ static void vfio_device_release(struct kref *kref)struct vfio_group *group = device->group;list_del (&device->group_next); + group->dev_counter--;mutex_unlock (&group->device_lock);dev_set_drvdata (device->dev,NULL );
@@ -1945,6 +1948,9 @@ int vfio_pin_pages(struct device *dev, unsigned long *user_pfn, int npage,if (!group)return -ENODEV ; +if (group->dev_counter >1 ) +return -EINVAL ; + ret =vfio_group_add_container_user (group);if (ret) goto err_pin_pages;
@@ -1952,7 +1958,8 @@ int vfio_pin_pages(struct device *dev, unsigned long *user_pfn, int npage, container = group->container; driver = container->iommu_driver;if (likely (driver && driver->ops->pin_pages)) - ret = driver->ops->pin_pages (container->iommu_data, user_pfn, + ret = driver->ops->pin_pages (container->iommu_data, + group->iommu_group, user_pfn, npage, prot, phys_pfn);else ret = -ENOTTY ;
@@ -2050,8 +2057,8 @@ int vfio_group_pin_pages(struct vfio_group *group, driver = container->iommu_driver;if (likely (driver && driver->ops->pin_pages)) ret = driver->ops->pin_pages (container->iommu_data, - user_iova_pfn, npage, - prot, phys_pfn); + group->iommu_group, user_iova_pfn, + npage, prot, phys_pfn);else ret = -ENOTTY ;
drivers/vfio/vfio_iommu_type1.c +96 -8
201 lines not on this page · raw message
include/linux/vfio.h +3 -1
@@ -76,7 +76,9 @@ struct vfio_iommu_driver_ops { struct iommu_group *group);void (*detach_group)(void *iommu_data,struct iommu_group *group); -int (*pin_pages)(void *iommu_data,unsigned long *user_pfn, +int (*pin_pages)(void *iommu_data, +struct iommu_group *group, +unsigned long *user_pfn,int npage,int prot,unsigned long *phys_pfn); int (*unpin_pages)(void *iommu_data,
Signature · 1 line
2.7.0
Hi Yan & Intel folks,
I'm starting to run out of comments on this series, where are you with porting GVT-g migration to this API? Are there remaining blocking issues? Are we satisfied that the API is sufficient to support vIOMMU now? Thanks,
Alex
Quoted 193 lines · On Fri, 15 May 2020 02:07:39 +0530 Kirti Wankhede <[email protected]> wrote:
Hi,
This patch set adds:
* IOCTLVFIO_IOMMU_DIRTY_PAGESto get dirty pages bitmap with respect to IOMMU container rather than per device. All pages pinned by vendor driver throughvfio_pin_pagesexternal API has to be marked as dirty during migration. When IOMMU capable device is present in the container and all pages are pinned and mapped, then all pages are marked dirty.
When there are CPU writes, CPU dirty page tracking can identify dirtied pages, but any page pinned by vendor driver can also be written by device. As of now there is no device which has hardware support for dirty page tracking. So all pages which are pinned should be considered as dirty.
This ioctl is also used to start/stop dirty pages tracking for pinned and unpinned pages while migration is active.* Updated IOCTL
VFIO_IOMMU_UNMAP_DMAto get dirty pages bitmap before unmapping IO virtual address range.
With vIOMMU, during pre-copy phase of migration, while CPUs are still running, IO virtual address unmap can happen while device still keeping reference of guest pfns. Those pages should be reported as dirty before unmap, so that VFIO user space application can copy content of those pages from source to destination.* Patch 8 detect if IOMMU capable device driver is smart to report pages to be marked dirty by pinning pages using
vfio_pin_pages()API.Yet TODO:
Since there is no device which has hardware support for system memmory dirty bitmap tracking, right now there is no other API from vendor driver to VFIO IOMMU module to report dirty pages. In future, when such hardware support will be implemented, an API will be required such that vendor driver could report dirty pages to VFIO module during migration phases.Adding revision history from previous QEMU patch set to understand KABI changes done till now
v19 -> v20
- Fixed ioctl to get dirty bitmap to get bitmap of multiplevfio_dmas
- Fixed unmap ioctl to get dirty bitmap of multiplevfio_dmas.
- Removed flag definition from migration capability.v18 -> v19
- Updated migration capability with supported page sizes bitmap for dirty page tracking and maximum bitmap size supported by kernel module.
- Added patch to calculate and cachepgsize_bitmapwheniommu->domain_listis updated.
- Removed extra buffers added in previous version for bitmap manipulation and optimised the code.v17 -> v18
- Add migration capability to the capability chain forVFIO_IOMMU_GET_INFOioctl
- UpdatedUMAP_DMAioctl to return bitmap of multiplevfio_dmav16 -> v17
- Fixed errors reported by kbuild test robot <[email protected]> on i386v15 -> v16
- Minor edits and nit picks (Auger Eric)
- On copying bitmap to user, re-populated bitmap only for pinned pages, excluding unmapped pages and CPU dirtied pages.
- Patches are on tag: next-20200318 and 1-3 patches from Yan's series
https://lkml.org/lkml/2020/3/12/1255v14 -> v15
- Minor edits and nit picks.
- In the verification of user allocated bitmap memory, added check of maximum size.
- Patches are on tag: next-20200318 and 1-3 patches from Yan's series
https://lkml.org/lkml/2020/3/12/1255v13 -> v14
- Added structvfio_bitmapto kabi. updated structurevfio_iommu_type1_dirty_bitmap_getandvfio_iommu_type1_dma_unmap.
- All small changes suggested by Alex.
- Patches are on tag: next-20200318 and 1-3 patches from Yan's series
https://lkml.org/lkml/2020/3/12/1255v12 -> v13
- Changed bitmap allocation invfio_iommu_type1to pervfio_dma
- ChangedVFIO_IOMMU_DIRTY_PAGESioctl behaviour to be pervfio_dmarange.
- Changedvfio_iommu_type1_dirty_bitmapstructure to have separate data field.v11 -> v12
- Changed bitmap allocation invfio_iommu_type1.
- Remove atomicity ofref_count.
- Updated comments for migration device state structure about error reporting.
- Nit picks from v11 reviewsv10 -> v11
- Fix pin pages API to free vpfn if it is marked as unpinned tracking page.
- Added proposal to detect if IOMMU capable device calls external pin pages API to mark pages dirty.
- Nit picks from v10 reviewsv9 -> v10:
- Updated existingVFIO_IOMMU_UNMAP_DMAioctl to get dirty pages bitmap during unmap while migration is active
- Added flag inVFIO_IOMMU_GET_INFOto indicate driver support dirty page tracking.
- Ifiommu_mapped, mark all pages dirty.
- Added unpinned pages tracking while migration is active.
- Updated comments for migration device state structure with bit combination table and state transition details.v8 -> v9:
- Split patch set in 2 sets, Kernel and QEMU.
- Dirty pages bitmap is queried from IOMMU container rather than from vendor driver for per device. Added 2 ioctls to achieve this.v7 -> v8:
- Updated comments for KABI
- Added BAR address validation check during PCI device's config space load as suggested by Dr. David Alan Gilbert.
- Changedvfio_migration_set_state()to set or clear device state flags.
- Some nit fixes.v6 -> v7:
- Fix build failures.v5 -> v6:
- Fix build failure.v4 -> v5:
- Added decriptive comment about the sequence of access of members of structurevfio_device_migration_infoto be followed based on Alex's suggestion
- Updated get dirty pages sequence.
- As per Cornelia Huck's suggestion, added callbacks to VFIODeviceOps toget_object,save_configandload_config.
- Fixed multiple nit picks.
- Tested live migration with multiple vfio device assigned to a VM.v3 -> v4:
- Added one more bit for_RESUMINGflag to be set explicitly.
-data_offsetfield is read-only for user space application.
-data_sizeis read for every iteration before reading data from migration, that is removed assumption that data will be till end of migration region.
- If vendor driver supports mappable sparsed region, map those region during setup state of save/load, similarly unmap those from cleanup routines.
- Handles race condition that causes data corruption in migration region during save device state by adding mutex and serialiaingsave_bufferandget_dirty_pagesroutines.
- Skip calledget_dirty_pagesroutine for mapped MMIO region of device.
- Added trace events.
- Split into multiple functional patches.v2 -> v3:
- Removed enum of VFIO device states. Defined VFIO device state with 2 bits.
- Re-structuredvfio_device_migration_infoto keep it minimal and defined action on read and write access on its members.v1 -> v2:
- Defined MIGRATION region type and sub-type which should be used with region type capability.
- Re-structuredvfio_device_migration_info. This structure will be placed at 0th offset of migration region.
- Replaced ioctl with read/write for trapped part of migration region.
- Added both type of access support, trapped or mmapped, for data section of the region.
- Moved PCI device functions to pci file.
- Added iteration to get dirty page bitmap until bitmap for all requested pages are copied.Thanks,
KirtiKirti Wankhede (8):
vfio: UAPI for migration interface for device state
vfio iommu: Remove atomicity ofref_countof pinned pages vfio iommu: Cachepgsize_bitmapin structvfio_iommu
vfio iommu: Add ioctl definition for dirty pages tracking vfio iommu: Implementation of ioctl for dirty pages tracking vfio iommu: UpdateUNMAP_DMAioctl to get dirty bitmap before unmap vfio iommu: Add migration capability to report supported features
vfio: Selective dirty page tracking if IOMMU backed device pins pagesdrivers/vfio/vfio.c | 13 +- drivers/vfio/vfio_iommu_type1.c | 565 ++++++++++++++++++++++++++++++++++++---- include/linux/vfio.h | 4 +- include/uapi/linux/vfio.h | 315 ++++++++++++++++++++++ 4 files changed, 838 insertions(+), 59 deletions(-)
Quoted 7 lines · Alex Williamson
Hi Yan & Intel folks,
I'm starting to run out of comments on this series, where are you with porting GVT-g migration to this API? Are there remaining blocking issues? Are we satisfied that the API is sufficient to support vIOMMU now? Thanks,
hi Alex
currently, we have ported to v17 kernel + v16 qemu with some fixes. gvt is working but we didn't try viommu yet.
for this v20 kernel series, is there any qemu patches matching to it?
Thanks
Yan
Quoted 197 lines
Quoted 193 lines · On Fri, 15 May 2020 02:07:39 +0530 Kirti Wankhede <[email protected]> wrote:
Hi,
This patch set adds:
* IOCTLVFIO_IOMMU_DIRTY_PAGESto get dirty pages bitmap with respect to IOMMU container rather than per device. All pages pinned by vendor driver throughvfio_pin_pagesexternal API has to be marked as dirty during migration. When IOMMU capable device is present in the container and all pages are pinned and mapped, then all pages are marked dirty.
When there are CPU writes, CPU dirty page tracking can identify dirtied pages, but any page pinned by vendor driver can also be written by device. As of now there is no device which has hardware support for dirty page tracking. So all pages which are pinned should be considered as dirty.
This ioctl is also used to start/stop dirty pages tracking for pinned and unpinned pages while migration is active.* Updated IOCTL
VFIO_IOMMU_UNMAP_DMAto get dirty pages bitmap before unmapping IO virtual address range.
With vIOMMU, during pre-copy phase of migration, while CPUs are still running, IO virtual address unmap can happen while device still keeping reference of guest pfns. Those pages should be reported as dirty before unmap, so that VFIO user space application can copy content of those pages from source to destination.* Patch 8 detect if IOMMU capable device driver is smart to report pages to be marked dirty by pinning pages using
vfio_pin_pages()API.Yet TODO:
Since there is no device which has hardware support for system memmory dirty bitmap tracking, right now there is no other API from vendor driver to VFIO IOMMU module to report dirty pages. In future, when such hardware support will be implemented, an API will be required such that vendor driver could report dirty pages to VFIO module during migration phases.Adding revision history from previous QEMU patch set to understand KABI changes done till now
v19 -> v20
- Fixed ioctl to get dirty bitmap to get bitmap of multiplevfio_dmas
- Fixed unmap ioctl to get dirty bitmap of multiplevfio_dmas.
- Removed flag definition from migration capability.v18 -> v19
- Updated migration capability with supported page sizes bitmap for dirty page tracking and maximum bitmap size supported by kernel module.
- Added patch to calculate and cachepgsize_bitmapwheniommu->domain_listis updated.
- Removed extra buffers added in previous version for bitmap manipulation and optimised the code.v17 -> v18
- Add migration capability to the capability chain forVFIO_IOMMU_GET_INFOioctl
- UpdatedUMAP_DMAioctl to return bitmap of multiplevfio_dmav16 -> v17
- Fixed errors reported by kbuild test robot <[email protected]> on i386v15 -> v16
- Minor edits and nit picks (Auger Eric)
- On copying bitmap to user, re-populated bitmap only for pinned pages, excluding unmapped pages and CPU dirtied pages.
- Patches are on tag: next-20200318 and 1-3 patches from Yan's series
https://lkml.org/lkml/2020/3/12/1255v14 -> v15
- Minor edits and nit picks.
- In the verification of user allocated bitmap memory, added check of maximum size.
- Patches are on tag: next-20200318 and 1-3 patches from Yan's series
https://lkml.org/lkml/2020/3/12/1255v13 -> v14
- Added structvfio_bitmapto kabi. updated structurevfio_iommu_type1_dirty_bitmap_getandvfio_iommu_type1_dma_unmap.
- All small changes suggested by Alex.
- Patches are on tag: next-20200318 and 1-3 patches from Yan's series
https://lkml.org/lkml/2020/3/12/1255v12 -> v13
- Changed bitmap allocation invfio_iommu_type1to pervfio_dma
- ChangedVFIO_IOMMU_DIRTY_PAGESioctl behaviour to be pervfio_dmarange.
- Changedvfio_iommu_type1_dirty_bitmapstructure to have separate data field.v11 -> v12
- Changed bitmap allocation invfio_iommu_type1.
- Remove atomicity ofref_count.
- Updated comments for migration device state structure about error reporting.
- Nit picks from v11 reviewsv10 -> v11
- Fix pin pages API to free vpfn if it is marked as unpinned tracking page.
- Added proposal to detect if IOMMU capable device calls external pin pages API to mark pages dirty.
- Nit picks from v10 reviewsv9 -> v10:
- Updated existingVFIO_IOMMU_UNMAP_DMAioctl to get dirty pages bitmap during unmap while migration is active
- Added flag inVFIO_IOMMU_GET_INFOto indicate driver support dirty page tracking.
- Ifiommu_mapped, mark all pages dirty.
- Added unpinned pages tracking while migration is active.
- Updated comments for migration device state structure with bit combination table and state transition details.v8 -> v9:
- Split patch set in 2 sets, Kernel and QEMU.
- Dirty pages bitmap is queried from IOMMU container rather than from vendor driver for per device. Added 2 ioctls to achieve this.v7 -> v8:
- Updated comments for KABI
- Added BAR address validation check during PCI device's config space load as suggested by Dr. David Alan Gilbert.
- Changedvfio_migration_set_state()to set or clear device state flags.
- Some nit fixes.v6 -> v7:
- Fix build failures.v5 -> v6:
- Fix build failure.v4 -> v5:
- Added decriptive comment about the sequence of access of members of structurevfio_device_migration_infoto be followed based on Alex's suggestion
- Updated get dirty pages sequence.
- As per Cornelia Huck's suggestion, added callbacks to VFIODeviceOps toget_object,save_configandload_config.
- Fixed multiple nit picks.
- Tested live migration with multiple vfio device assigned to a VM.v3 -> v4:
- Added one more bit for_RESUMINGflag to be set explicitly.
-data_offsetfield is read-only for user space application.
-data_sizeis read for every iteration before reading data from migration, that is removed assumption that data will be till end of migration region.
- If vendor driver supports mappable sparsed region, map those region during setup state of save/load, similarly unmap those from cleanup routines.
- Handles race condition that causes data corruption in migration region during save device state by adding mutex and serialiaingsave_bufferandget_dirty_pagesroutines.
- Skip calledget_dirty_pagesroutine for mapped MMIO region of device.
- Added trace events.
- Split into multiple functional patches.v2 -> v3:
- Removed enum of VFIO device states. Defined VFIO device state with 2 bits.
- Re-structuredvfio_device_migration_infoto keep it minimal and defined action on read and write access on its members.v1 -> v2:
- Defined MIGRATION region type and sub-type which should be used with region type capability.
- Re-structuredvfio_device_migration_info. This structure will be placed at 0th offset of migration region.
- Replaced ioctl with read/write for trapped part of migration region.
- Added both type of access support, trapped or mmapped, for data section of the region.
- Moved PCI device functions to pci file.
- Added iteration to get dirty page bitmap until bitmap for all requested pages are copied.Thanks,
KirtiKirti Wankhede (8):
vfio: UAPI for migration interface for device state
vfio iommu: Remove atomicity ofref_countof pinned pages vfio iommu: Cachepgsize_bitmapin structvfio_iommu
vfio iommu: Add ioctl definition for dirty pages tracking vfio iommu: Implementation of ioctl for dirty pages tracking vfio iommu: UpdateUNMAP_DMAioctl to get dirty bitmap before unmap vfio iommu: Add migration capability to report supported features
vfio: Selective dirty page tracking if IOMMU backed device pins pagesdrivers/vfio/vfio.c | 13 +- drivers/vfio/vfio_iommu_type1.c | 565 ++++++++++++++++++++++++++++++++++++---- include/linux/vfio.h | 4 +- include/uapi/linux/vfio.h | 315 ++++++++++++++++++++++ 4 files changed, 838 insertions(+), 59 deletions(-)