QEMU VFIO live migration
cover QEMU VFIO live migration
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.
Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.
Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.
System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.
Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.
So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages
(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).
kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2
#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2
#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3
struct vfio_device_state_ctl {
__u32 version; /* ro */
__u32 device_state; /* VFIO device state, wo */
__u32 caps; /* ro */
struct {
__u32 action; /* wo, GET_BUFFER or SET_BUFFER */
__u64 size; /*rw*/
} device_config;
struct {
__u32 action; /* wo, GET_BUFFER or SET_BUFFER */
__u64 size; /* rw */
__u64 pos; /*the offset in total buffer of device memory*/
} device_memory;
struct {
__u64 start_addr; /* wo */
__u64 page_nr; /* wo */
} system_memory;
};Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.
Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP
RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.
STOP: In this state, a VFIO device is deactivated to interact with
device driver.
LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.
Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.
Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.
Then it will probe VFIO device's data capability by reading caps field of control region.
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.
If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------
Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING & VFIO_DEVICE_STATE_LOGGING |MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLED orMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
Live migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETED
In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.
If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.
In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.
In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.
References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/
Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability
hw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.c
Signature · 1 line
2.7.4
PATCH 1/5 vfio/migration: define kernel interfaces
- defined 4 device states regions: one control region and 3 data regions
- defined layout of control region in struct vfio_device_state_ctl
- defined 4 device states: running, stop, running&logging, stop&logging
- define 3 device data categories: device config, device memory, system memory
- defined 2 device data capabilities: device memory and system memory
- defined device state interfaces' version and 12 device state interfaces
--- linux-headers/linux/vfio.h | 260 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 260 insertions(+)
linux-headers/linux/vfio.h +260 -0
@@ -303,6 +303,56 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* Device State region type and sub-type +* +* A VFIO device driver needs to register up to four device state regions in +* total: two mandatory and another two optional, if it plans to support device +* state management. +* +* 1. region CTL : +* Mandatory. +* This is a control region. +* Its layout is defined in struct vfio_device_state_ctl. +* Reading from this region can get version, capabilities and data +* size of device state interfaces. +* Writing to this region can set device state, data size and +* choose which interface to use. +* 2. region DEVICE_CONFIG +* Mandatory. +* This is a data region that holds device config data. +* Device config is such kind of data like MMIOs, page tables... +* Every device is supposed to possess device config data. +* Usually the size of device config data is small (no big +* than 10M), and it needs to be loaded in certain strict +* order. +* Therefore no dirty data logging is enabled for device +* config and it must be got/set as a whole. +* Size of device config data is smaller than or equal to that of +* device config region. +* It is able to be mmaped into user space. +* 3. region DEVICE_MEMORY +* Optional. +* This is a data region that holds device memory data. +* Device memory is device's internal memory, standalone and outside +* system memory. It is usually very big. +* Not all device has device memory. Like IGD only uses system +* memory and has no device memory. +* Size of devie memory is usually larger than that of device +* memory region. qemu needs to save/load it in chunks of size of +* device memory region. +* It is able to be mmaped into user space. +* 4. region DIRTY_BITMAP +* Optional. +* This is a data region that holds bitmap of dirty pages in system +* memory that a VFIO devices produces. +* It is able to be mmaped into user space. +*/ +#define VFIO_REGION_TYPE_DEVICE_STATE (1 << 1) +#define VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL (1) +#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG (2) +#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY (3) +#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP (4) +/* * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped * which allows direct access to non-MSIX registers which happened to be within
@@ -816,6 +866,216 @@ struct vfio_iommu_spapr_tce_remove { };#define VFIO_IOMMU_SPAPR_TCE_REMOVE _IO(VFIO_TYPE, VFIO_BASE + 20) +/* version number of the device state interface */ +#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1 + +/* +* For devices that have devcie memory, it is required to expose +* DEVICE_MEMORY capability. +* +* For devices producing dirty pages in system memory, it is required to +* expose cap SYSTEM_MEMORY in order to get dirty bitmap in certain range +* of system memory. +*/ +#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 +#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2 + +/* +* DEVICE STATES +* +* Four states are defined for a VFIO device: +* RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP. +* They can be set by writing to device_state field of +* vfio_device_state_ctl region. +* +* RUNNING: In this state, a VFIO device is in active state ready to +* receive commands from device driver. +* It is the default state that a VFIO device enters initially. +* +* STOP: In this state, a VFIO device is deactivated to interact with +* device driver. +* +* LOGGING state is a special state that it CANNOT exist +* independently. +* It must be set alongside with state RUNNING or STOP, i.e, +* RUNNING & LOGGING, STOP & LOGGING. +* It is used for dirty data logging both for device memory +* and system memory. +* +* LOGGING only impacts device/system memory. In LOGGING state, get buffer +* of device memory returns dirty pages since last call; outside LOGGING +* state, get buffer of device memory returns whole snapshot of device +* memory. system memory's dirty page is only available in LOGGING state. +* +* Device config should be always accessible and return whole config snapshot +* regardless of LOGGING state. +* */ +#define VFIO_DEVICE_STATE_RUNNING 0 +#define VFIO_DEVICE_STATE_STOP 1 +#define VFIO_DEVICE_STATE_LOGGING 2 + +/* action to get data from device memory or device config +* the action is write to device state's control region, and data is read +* from device memory region or device config region. +* Each time before read device memory region or device config region, +* action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action +* field in control region. That is because device memory and devie config +* region is mmaped into user space. vendor driver has to be notified of +* the the GET_BUFFER action in advance. +*/ +#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1 + +/* action to set data to device memory or device config +* the action is write to device state's control region, and data is +* written to device memory region or device config region. +* Each time after write to device memory region or device config region, +* action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action +* field in control region. That is because device memory and devie config +* region is mmaped into user space. vendor driver has to be notified of +* the the SET_BUFFER action after data written. +*/ +#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2 + +/* layout of device state interfaces' control region +* By reading to control region and reading/writing data from device config +* region, device memory region, system memory regions, below interface can +* be implemented: +* +* 1. get version +* (1) user space calls read system call on "version" field of control +* region. +* (2) vendor driver writes version number of device state interfaces +* to the "version" field of control region. +* +* 2. get caps +* (1) user space calls read system call on "caps" field of control region. +* (2) if a VFIO device has huge device memory, vendor driver reports +* VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY in "caps" field of control region. +* if a VFIO device produces dirty pages in system memory, vendor driver +* reports VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY in "caps" field of +* control region. +* +* 3. set device state +* (1) user space calls write system call on "device_state" field of +* control region. +* (2) device state transitions as: +* +* RUNNING -- start dirty data logging --> RUNNING & LOGGING +* RUNNING -- deactivate --> STOP +* RUNNING -- deactivate & start dirty data longging --> STOP & LOGGING +* RUNNING & LOGGING -- stop dirty data logging --> RUNNING +* RUNNING & LOGGING -- deactivate --> STOP & LOGGING +* RUNNING & LOGGING -- deactivate & stop dirty data logging --> STOP +* STOP -- activate --> RUNNING +* STOP -- start dirty data logging --> STOP & LOGGING +* STOP -- activate & start dirty data logging --> RUNNING & LOGGING +* STOP & LOGGING -- stop dirty data logging --> STOP +* STOP & LOGGING -- activate --> RUNNING & LOGGING +* STOP & LOGGING -- activate & stop dirty data logging --> RUNNING +* +* 4. get device config size +* (1) user space calls read system call on "device_config.size" field of +* control region for the total size of device config snapshot. +* (2) vendor driver writes device config data's total size in +* "device_config.size" field of control region. +* +* 5. set device config size +* (1) user space calls write system call. +* total size of device config snapshot --> "device_config.size" field +* of control region. +* (2) vendor driver reads device config data's total size from +* "device_config.size" field of control region. +* +* 6 get device config buffer +* (1) user space calls write system call. +* "GET_BUFFER" --> "device_config.action" field of control region. +* (2) vendor driver +* a. gets whole snapshot for device config +* b. writes whole device config snapshot to region +* DEVICE_CONFIG. +* (3) user space reads the whole of device config snapshot from region +* DEVICE_CONFIG. +* +* 7. set device config buffer +* (1) user space writes whole of device config data to region +* DEVICE_CONFIG. +* (2) user space calls write system call. +* "SET_BUFFER" --> "device_config.action" field of control region. +* (3) vendor driver loads whole of device config from region DEVICE_CONFIG. +* +* 8. get device memory size +* (1) user space calls read system call on "device_memory.size" field of +* control region for device memory size. +* (2) vendor driver +* a. gets device memory snapshot (in state RUNNING or STOP), or +* gets device memory dirty data (in state RUNNING & LOGGING or +* state STOP & LOGGING) +* b. writes size in "device_memory.size" field of control region +* +* 9. set device memory size +* (1) user space calls write system call on "device_memory.size" field of +* control region to set total size of device memory snapshot. +* (2) vendor driver reads device memory's size from "device_memory.size" +* field of control region. +* +* +* 10. get device memory buffer +* (1) user space calls write system. +* pos --> "device_memory.pos" field of control region, +* "GET_BUFFER" --> "device_memory.action" field of control region. +* (pos must be 0 or multiples of length of region DEVICE_MEMORY). +* (2) vendor driver writes N'th chunk of device memory snapshot/dirty data +* to region DEVICE_MEMORY. +* (N equals to pos/(region length of DEVICE_MEMORY)) +* (3) user space reads the N'th chunk of device memory snapshot/dirty data +* from region DEVICE_MEMORY. +* +* 11. set device memory buffer +* (1) user space writes N'th chunk of device memory snapshot/dirty data to +* region DEVICE_MEMORY. +* (N equals to pos/(region length of DEVICE_MEMORY)) +* (2) user space writes pos to "device_memory.pos" field and writes +* "SET_BUFFER" to "device_memory.action" field of control region. +* (3) vendor driver loads N'th chunk of device memory snapshot/dirty data +* from region DEVICE_MEMORY. +* +* 12. get system memory dirty bitmap +* (1) user space calls write system call to specify a range of system +* memory that querying dirty pages. +* system memory's start address --> "system_memory.start_addr" field +* of control region, +* system memory's page count --> "system_memory.page_nr" field of +* control region. +* (2) if device state is not in RUNNING or STOP & LOGGING, +* vendor driver returns empty bitmap; otherwise, +* vendor driver checks the page_nr, +* if it's larger than the size that region DIRTY_BITMAP can support, +* error returns; if not, +* vendor driver returns as bitmap to specify dirty pages that +* device produces since last query in this range of system memory . +* (3) usespace reads back the dirty bitmap from region DIRTY_BITMAP. +* +*/ + +struct vfio_device_state_ctl { +__u32 version;/* ro versio of devcie state interfaces*/ +__u32 device_state;/* VFIO device state, wo */ +__u32 caps;/* ro */ +struct { +__u32 action;/* wo, GET_BUFFER or SET_BUFFER */ +__u64 size;/*rw, total size of device config*/ + } device_config; +struct { +__u32 action;/* wo, GET_BUFFER or SET_BUFFER */ +__u64 size;/* rw, total size of device memory*/ +__u64 pos;/*chunk offset in total buffer of device memory*/ + } device_memory; +struct { +__u64 start_addr;/* wo */ +__u64 page_nr;/* wo */ + } system_memory; +}__attribute__((packed)); +/* ***************************************************************** */ #endif /* VFIO_H */
Signature · 1 line
2.7.4
Re: [PATCH 1/5] vfio/migration: define kernel interfaces
Quoted 8 lines · On Tue, 19 Feb 2019 16:52:14 +0800 Yan Zhao <[email protected]> wrote:
- defined 4 device states regions: one control region and 3 data regions
- defined layout of control region in structvfio_device_state_ctl
- defined 4 device states: running, stop, running&logging, stop&logging
- define 3 device data categories: device config, device memory, system memory
- defined 2 device data capabilities: device memory and system memory
- defined device state interfaces' version and 12 device state interfaces
Quoted 6 lines
Signed-off-by Yan Zhao <[email protected]>Signed-off-by Kevin Tian <[email protected]>Signed-off-by Yulei Zhang <[email protected]>--- linux-headers/linux/vfio.h | 260 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 260 insertions(+)
[commenting here for convenience; changes obviously need to be done in the Linux patch]
Quoted 6 lines
linux-headers/linux/vfio.h +0
-0
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 310
Quoted 8 lines
linux-headers/linux/vfio.h +5
-0@@ -303,6 +303,56 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* Device State region type and sub-type +* +* A VFIO device driver needs to register up to four device state regions in +* total: two mandatory and another two optional, if it plans to support device +* state management.
Suggest to rephrase:
"A VFIO device driver that plans to support device state management needs to register..."
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 323
Quoted 13 lines
linux-headers/linux/vfio.h +13
-0+* +* 1. region CTL : +* Mandatory. +* This is a control region. +* Its layout is defined in struct vfio_device_state_ctl. +* Reading from this region can get version, capabilities and data +* size of device state interfaces. +* Writing to this region can set device state, data size and +* choose which interface to use. +* 2. region DEVICE_CONFIG +* Mandatory. +* This is a data region that holds device config data. +* Device config is such kind of data like MMIOs, page tables...
"Device config is data such as..."
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 325
Quoted 2 lines
linux-headers/linux/vfio.h +2
-0+* Every device is supposed to possess device config data. +* Usually the size of device config data is small (no big
s/no big/no bigger/
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 331
Quoted 6 lines
linux-headers/linux/vfio.h +6
-0+* than 10M), and it needs to be loaded in certain strict +* order. +* Therefore no dirty data logging is enabled for device +* config and it must be got/set as a whole. +* Size of device config data is smaller than or equal to that of +* device config region.
Not sure if I understand that sentence correctly... but what if a device has more config state than fits into this region? Is that supposed to be covered by the device memory region? Or is this assumed to be something so exotic that we don't need to plan for it?
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 336
Quoted 5 lines
linux-headers/linux/vfio.h +5
-0+* It is able to be mmaped into user space. +* 3. region DEVICE_MEMORY +* Optional. +* This is a data region that holds device memory data. +* Device memory is device's internal memory, standalone and outside
s/outside/distinct from/ ?
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 338
Quoted 2 lines
linux-headers/linux/vfio.h +2
-0+* system memory. It is usually very big. +* Not all device has device memory. Like IGD only uses system
s/all devices has/all devices have/
s/Like/E.g./
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 340
Quoted 2 lines
linux-headers/linux/vfio.h +2
-0+* memory and has no device memory. +* Size of devie memory is usually larger than that of device
s/devie/device/
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 342
Quoted 2 lines
linux-headers/linux/vfio.h +2
-0+* memory region. qemu needs to save/load it in chunks of size of +* device memory region.
I'd rather not explicitly mention QEMU in this header. Maybe "Userspace"?
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 350
Quoted 8 lines
linux-headers/linux/vfio.h +8
-0+* It is able to be mmaped into user space. +* 4. region DIRTY_BITMAP +* Optional. +* This is a data region that holds bitmap of dirty pages in system +* memory that a VFIO devices produces. +* It is able to be mmaped into user space. +*/ +#define VFIO_REGION_TYPE_DEVICE_STATE (1 << 1)
Can you make this an explicit number instead?
(FWIW, I plan to add a CCW region as type 2, whatever comes first.)
Quoted 9 lines
linux-headers/linux/vfio.h +5
-0+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL (1) +#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG (2) +#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY (3) +#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP (4) +/* * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped * which allows direct access to non-MSIX registers which happened to be within
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 870
Quoted 5 lines
linux-headers/linux/vfio.h +2
-0@@ -816,6 +866,216 @@ struct vfio_iommu_spapr_tce_remove { };#define VFIO_IOMMU_SPAPR_TCE_REMOVE _IO(VFIO_TYPE, VFIO_BASE + 20) +/* version number of the device state interface */ +#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
Hm. Is this supposed to be backwards-compatible, should we need to bump this?
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 873
Quoted 3 lines
linux-headers/linux/vfio.h +3
-0+ +/* +* For devices that have devcie memory, it is required to expose
s/devcie/device/
Quoted 6 lines
linux-headers/linux/vfio.h +6
-0+* DEVICE_MEMORY capability. +* +* For devices producing dirty pages in system memory, it is required to +* expose cap SYSTEM_MEMORY in order to get dirty bitmap in certain range +* of system memory. +*/
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 889
Quoted 10 lines
linux-headers/linux/vfio.h +10
-0+#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 +#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2 + +/* +* DEVICE STATES +* +* Four states are defined for a VFIO device: +* RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP. +* They can be set by writing to device_state field of +* vfio_device_state_ctl region.
Who controls this? Userspace?
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 896
Quoted 7 lines
linux-headers/linux/vfio.h +7
-0+* +* RUNNING: In this state, a VFIO device is in active state ready to +* receive commands from device driver. +* It is the default state that a VFIO device enters initially. +* +* STOP: In this state, a VFIO device is deactivated to interact with +* device driver.
I think 'STOPPED' would read nicer.
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 899
Quoted 3 lines
linux-headers/linux/vfio.h +3
-0+* +* LOGGING state is a special state that it CANNOT exist +* independently.
So it's not a state, but rather a modifier?
Quoted 28 lines
linux-headers/linux/vfio.h +28
-0+* It must be set alongside with state RUNNING or STOP, i.e, +* RUNNING & LOGGING, STOP & LOGGING. +* It is used for dirty data logging both for device memory +* and system memory. +* +* LOGGING only impacts device/system memory. In LOGGING state, get buffer +* of device memory returns dirty pages since last call; outside LOGGING +* state, get buffer of device memory returns whole snapshot of device +* memory. system memory's dirty page is only available in LOGGING state. +* +* Device config should be always accessible and return whole config snapshot +* regardless of LOGGING state. +* */ +#define VFIO_DEVICE_STATE_RUNNING 0 +#define VFIO_DEVICE_STATE_STOP 1 +#define VFIO_DEVICE_STATE_LOGGING 2 + +/* action to get data from device memory or device config +* the action is write to device state's control region, and data is read +* from device memory region or device config region. +* Each time before read device memory region or device config region, +* action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action +* field in control region. That is because device memory and devie config +* region is mmaped into user space. vendor driver has to be notified of +* the the GET_BUFFER action in advance. +*/ +#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1 +
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 937
Quoted 10 lines
linux-headers/linux/vfio.h +10
-0+/* action to set data to device memory or device config +* the action is write to device state's control region, and data is +* written to device memory region or device config region. +* Each time after write to device memory region or device config region, +* action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action +* field in control region. That is because device memory and devie config +* region is mmaped into user space. vendor driver has to be notified of +* the the SET_BUFFER action after data written. +*/ +#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
Let me describe this in my own words to make sure that I understand this correctly.
- The actions are set by userspace to notify the kernel that it is going to get data or that it has just written data.
- This is needed as a notification that the mmapped data should not be changed resp. just has changed.
So, how does the kernel know whether the read action has finished resp. whether the write action has started? Even if userspace reads/writes it as a whole.
Quoted 111 lines
linux-headers/linux/vfio.h +111
-0+ +/* layout of device state interfaces' control region +* By reading to control region and reading/writing data from device config +* region, device memory region, system memory regions, below interface can +* be implemented: +* +* 1. get version +* (1) user space calls read system call on "version" field of control +* region. +* (2) vendor driver writes version number of device state interfaces +* to the "version" field of control region. +* +* 2. get caps +* (1) user space calls read system call on "caps" field of control region. +* (2) if a VFIO device has huge device memory, vendor driver reports +* VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY in "caps" field of control region. +* if a VFIO device produces dirty pages in system memory, vendor driver +* reports VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY in "caps" field of +* control region. +* +* 3. set device state +* (1) user space calls write system call on "device_state" field of +* control region. +* (2) device state transitions as: +* +* RUNNING -- start dirty data logging --> RUNNING & LOGGING +* RUNNING -- deactivate --> STOP +* RUNNING -- deactivate & start dirty data longging --> STOP & LOGGING +* RUNNING & LOGGING -- stop dirty data logging --> RUNNING +* RUNNING & LOGGING -- deactivate --> STOP & LOGGING +* RUNNING & LOGGING -- deactivate & stop dirty data logging --> STOP +* STOP -- activate --> RUNNING +* STOP -- start dirty data logging --> STOP & LOGGING +* STOP -- activate & start dirty data logging --> RUNNING & LOGGING +* STOP & LOGGING -- stop dirty data logging --> STOP +* STOP & LOGGING -- activate --> RUNNING & LOGGING +* STOP & LOGGING -- activate & stop dirty data logging --> RUNNING +* +* 4. get device config size +* (1) user space calls read system call on "device_config.size" field of +* control region for the total size of device config snapshot. +* (2) vendor driver writes device config data's total size in +* "device_config.size" field of control region. +* +* 5. set device config size +* (1) user space calls write system call. +* total size of device config snapshot --> "device_config.size" field +* of control region. +* (2) vendor driver reads device config data's total size from +* "device_config.size" field of control region. +* +* 6 get device config buffer +* (1) user space calls write system call. +* "GET_BUFFER" --> "device_config.action" field of control region. +* (2) vendor driver +* a. gets whole snapshot for device config +* b. writes whole device config snapshot to region +* DEVICE_CONFIG. +* (3) user space reads the whole of device config snapshot from region +* DEVICE_CONFIG. +* +* 7. set device config buffer +* (1) user space writes whole of device config data to region +* DEVICE_CONFIG. +* (2) user space calls write system call. +* "SET_BUFFER" --> "device_config.action" field of control region. +* (3) vendor driver loads whole of device config from region DEVICE_CONFIG. +* +* 8. get device memory size +* (1) user space calls read system call on "device_memory.size" field of +* control region for device memory size. +* (2) vendor driver +* a. gets device memory snapshot (in state RUNNING or STOP), or +* gets device memory dirty data (in state RUNNING & LOGGING or +* state STOP & LOGGING) +* b. writes size in "device_memory.size" field of control region +* +* 9. set device memory size +* (1) user space calls write system call on "device_memory.size" field of +* control region to set total size of device memory snapshot. +* (2) vendor driver reads device memory's size from "device_memory.size" +* field of control region. +* +* +* 10. get device memory buffer +* (1) user space calls write system. +* pos --> "device_memory.pos" field of control region, +* "GET_BUFFER" --> "device_memory.action" field of control region. +* (pos must be 0 or multiples of length of region DEVICE_MEMORY). +* (2) vendor driver writes N'th chunk of device memory snapshot/dirty data +* to region DEVICE_MEMORY. +* (N equals to pos/(region length of DEVICE_MEMORY)) +* (3) user space reads the N'th chunk of device memory snapshot/dirty data +* from region DEVICE_MEMORY. +* +* 11. set device memory buffer +* (1) user space writes N'th chunk of device memory snapshot/dirty data to +* region DEVICE_MEMORY. +* (N equals to pos/(region length of DEVICE_MEMORY)) +* (2) user space writes pos to "device_memory.pos" field and writes +* "SET_BUFFER" to "device_memory.action" field of control region. +* (3) vendor driver loads N'th chunk of device memory snapshot/dirty data +* from region DEVICE_MEMORY. +* +* 12. get system memory dirty bitmap +* (1) user space calls write system call to specify a range of system +* memory that querying dirty pages. +* system memory's start address --> "system_memory.start_addr" field +* of control region, +* system memory's page count --> "system_memory.page_nr" field of +* control region.
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1058
Quoted 10 lines
linux-headers/linux/vfio.h +10
-0+* (2) if device state is not in RUNNING or STOP & LOGGING, +* vendor driver returns empty bitmap; otherwise, +* vendor driver checks the page_nr, +* if it's larger than the size that region DIRTY_BITMAP can support, +* error returns; if not, +* vendor driver returns as bitmap to specify dirty pages that +* device produces since last query in this range of system memory . +* (3) usespace reads back the dirty bitmap from region DIRTY_BITMAP. +* +*/
It might make sense to extract the explanations above into a separate design document in the kernel Documentation/ directory. You could also add ASCII art there :)
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1061
Quoted 3 lines
linux-headers/linux/vfio.h +3
-0+ +struct vfio_device_state_ctl { +__u32 version;/* ro versio of devcie state interfaces*/
s/versio/version/
s/devcie/device/
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1064
Quoted 3 lines
linux-headers/linux/vfio.h +3
-0+__u32 device_state;/* VFIO device state, wo */ +__u32 caps;/* ro */ +struct {
Indentation looks a bit off.
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1071
Quoted 7 lines
linux-headers/linux/vfio.h +7
-0+__u32 action;/* wo, GET_BUFFER or SET_BUFFER */ +__u64 size;/*rw, total size of device config*/ + } device_config; +struct { +__u32 action;/* wo, GET_BUFFER or SET_BUFFER */ +__u64 size;/* rw, total size of device memory*/ +__u64 pos;/*chunk offset in total buffer of device memory*/
Here as well.
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1077
Quoted 6 lines
linux-headers/linux/vfio.h +6
-0+ } device_memory; +struct { +__u64 start_addr;/* wo */ +__u64 page_nr;/* wo */ + } system_memory; +}__attribute__((packed));
For an interface definition, it's probably better to avoid packed and instead add padding if needed.
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1081
Quoted 4 lines
linux-headers/linux/vfio.h +1
-0+/* ***************************************************************** */ #endif /* VFIO_H */
On the whole, I think this is moving into the right direction.
Re: [PATCH 1/5] vfio/migration: define kernel interfaces
Quoted 18 lines · Cornelia Huck
Quoted 8 lines · On Tue, 19 Feb 2019 16:52:14 +0800 Yan Zhao <[email protected]> wrote:
- defined 4 device states regions: one control region and 3 data regions
- defined layout of control region in structvfio_device_state_ctl
- defined 4 device states: running, stop, running&logging, stop&logging
- define 3 device data categories: device config, device memory, system memory
- defined 2 device data capabilities: device memory and system memory
- defined device state interfaces' version and 12 device state interfacesQuoted 6 lines
Signed-off-by Yan Zhao <[email protected]>Signed-off-by Kevin Tian <[email protected]>Signed-off-by Yulei Zhang <[email protected]>--- linux-headers/linux/vfio.h | 260 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 260 insertions(+)
Quoted 3 lines
[commenting here for convenience; changes obviously need to be done in the Linux patch]
yes, you can find the corresponding kernel part code at
https://patchwork.freedesktop.org/series/56876/
Quoted 17 lines
Quoted 6 lines
linux-headers/linux/vfio.h +0
-0linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 310
Quoted 8 lines
linux-headers/linux/vfio.h +5
-0@@ -303,6 +303,56 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* Device State region type and sub-type +* +* A VFIO device driver needs to register up to four device state regions in +* total: two mandatory and another two optional, if it plans to support device +* state management. Suggest to rephrase:
Quoted 3 lines
"A VFIO device driver that plans to support device state management needs to register..."
ok :)
Quoted 14 lines
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 323
Quoted 13 lines
linux-headers/linux/vfio.h +13
-0+* +* 1. region CTL : +* Mandatory. +* This is a control region. +* Its layout is defined in struct vfio_device_state_ctl. +* Reading from this region can get version, capabilities and data +* size of device state interfaces. +* Writing to this region can set device state, data size and +* choose which interface to use. +* 2. region DEVICE_CONFIG +* Mandatory. +* This is a data region that holds device config data. +* Device config is such kind of data like MMIOs, page tables...
Quoted 1 line
"Device config is data such as..."
ok :)
Quoted 5 lines
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 325
Quoted 3 lines
linux-headers/linux/vfio.h +2
-0+* Every device is supposed to possess device config data. +* Usually the size of device config data is small (no big s/no big/no bigger/
right :)
Quoted 8 lines
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 331
Quoted 7 lines
linux-headers/linux/vfio.h +6
-0+* than 10M), and it needs to be loaded in certain strict +* order. +* Therefore no dirty data logging is enabled for device +* config and it must be got/set as a whole. +* Size of device config data is smaller than or equal to that of +* device config region.
Quoted 5 lines
Not sure if I understand that sentence correctly... but what if a device has more config state than fits into this region? Is that supposed to be covered by the device memory region? Or is this assumed to be something so exotic that we don't need to plan for it?
Device config data and device config region are all provided by vendor driver, so vendor driver is always able to create a large enough device config region to hold device config data.
So, if a device has data that are better to be saved after device stop and saved/loaded in strict order, the data needs to be in device config region. This kind of data is supposed to be small.
If the device data can be saved/loaded several times, it can also be put into device memory region.
Quoted 7 lines
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 336
Quoted 5 lines
linux-headers/linux/vfio.h +5
-0+* It is able to be mmaped into user space. +* 3. region DEVICE_MEMORY +* Optional. +* This is a data region that holds device memory data. +* Device memory is device's internal memory, standalone and outside s/outside/distinct from/ ?
ok.
Quoted 8 lines
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 338
Quoted 3 lines
linux-headers/linux/vfio.h +2
-0+* system memory. It is usually very big. +* Not all device has device memory. Like IGD only uses system s/all devices has/all devices have/
s/Like/E.g./
ok :)
Quoted 5 lines
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 340
Quoted 2 lines
linux-headers/linux/vfio.h +2
-0+* memory and has no device memory. +* Size of devie memory is usually larger than that of device s/devie/device/
thanks:)
Quoted 6 lines
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 342
Quoted 2 lines
linux-headers/linux/vfio.h +2
-0+* memory region. qemu needs to save/load it in chunks of size of +* device memory region. I'd rather not explicitly mention QEMU in this header. Maybe "Userspace"?
ok.
Quoted 9 lines
linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 350
Quoted 8 lines
linux-headers/linux/vfio.h +8
-0+* It is able to be mmaped into user space. +* 4. region DIRTY_BITMAP +* Optional. +* This is a data region that holds bitmap of dirty pages in system +* memory that a VFIO devices produces. +* It is able to be mmaped into user space. +*/ +#define VFIO_REGION_TYPE_DEVICE_STATE (1 << 1)
Quoted 3 lines
Can you make this an explicit number instead?
(FWIW, I plan to add a CCW region as type 2, whatever comes first.)
ok :)
Quoted 13 lines
Quoted 13 lines
linux-headers/linux/vfio.h +5
-0+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL (1) +#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG (2) +#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY (3) +#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP (4) +/* * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped * which allows direct access to non-MSIX registers which happened to be within @@ -816,6 +866,216 @@ struct vfio_iommu_spapr_tce_remove { };#define VFIO_IOMMU_SPAPR_TCE_REMOVE _IO(VFIO_TYPE, VFIO_BASE + 20)
Quoted 6 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 870
Quoted 2 lines
linux-headers/linux/vfio.h +2
-0+/* version number of the device state interface */ +#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1Hm. Is this supposed to be backwards-compatible, should we need to bump this?
currently no backwords-compatible. we can discuss on that.
Quoted 16 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 873
Quoted 3 lines
linux-headers/linux/vfio.h +3
-0+ +/* +* For devices that have devcie memory, it is required to expose s/devcie/device/
Quoted 10 lines
linux-headers/linux/vfio.h +10
-0+* DEVICE_MEMORY capability. +* +* For devices producing dirty pages in system memory, it is required to +* expose cap SYSTEM_MEMORY in order to get dirty bitmap in certain range +* of system memory. +*/ +#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 +#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2 + +/*
Quoted 8 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 889
Quoted 6 lines
linux-headers/linux/vfio.h +6
-0+* DEVICE STATES +* +* Four states are defined for a VFIO device: +* RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP. +* They can be set by writing to device_state field of +* vfio_device_state_ctl region. Who controls this? Userspace?
Yes. Userspace notifies vendor driver to do the state switching.
Quoted 10 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 896
Quoted 7 lines
linux-headers/linux/vfio.h +7
-0+* +* RUNNING: In this state, a VFIO device is in active state ready to +* receive commands from device driver. +* It is the default state that a VFIO device enters initially. +* +* STOP: In this state, a VFIO device is deactivated to interact with +* device driver. I think 'STOPPED' would read nicer.
sounds better :)
Quoted 6 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 899
Quoted 3 lines
linux-headers/linux/vfio.h +3
-0+* +* LOGGING state is a special state that it CANNOT exist +* independently. So it's not a state, but rather a modifier?
yes. or thinking LOGGING/not LOGGING as bit 1 of a device state, whereas RUNNING/STOPPED is bit 0 of a device state. They have to be got as a whole.
Quoted 42 lines
Quoted 28 lines
linux-headers/linux/vfio.h +28
-0+* It must be set alongside with state RUNNING or STOP, i.e, +* RUNNING & LOGGING, STOP & LOGGING. +* It is used for dirty data logging both for device memory +* and system memory. +* +* LOGGING only impacts device/system memory. In LOGGING state, get buffer +* of device memory returns dirty pages since last call; outside LOGGING +* state, get buffer of device memory returns whole snapshot of device +* memory. system memory's dirty page is only available in LOGGING state. +* +* Device config should be always accessible and return whole config snapshot +* regardless of LOGGING state. +* */ +#define VFIO_DEVICE_STATE_RUNNING 0 +#define VFIO_DEVICE_STATE_STOP 1 +#define VFIO_DEVICE_STATE_LOGGING 2 + +/* action to get data from device memory or device config +* the action is write to device state's control region, and data is read +* from device memory region or device config region. +* Each time before read device memory region or device config region, +* action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action +* field in control region. That is because device memory and devie config +* region is mmaped into user space. vendor driver has to be notified of +* the the GET_BUFFER action in advance. +*/ +#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1 +linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 937
Quoted 10 lines
linux-headers/linux/vfio.h +10
-0+/* action to set data to device memory or device config +* the action is write to device state's control region, and data is +* written to device memory region or device config region. +* Each time after write to device memory region or device config region, +* action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action +* field in control region. That is because device memory and devie config +* region is mmaped into user space. vendor driver has to be notified of +* the the SET_BUFFER action after data written. +*/ +#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2Let me describe this in my own words to make sure that I understand this correctly.
Quoted 4 lines
- The actions are set by userspace to notify the kernel that it is going to get data or that it has just written data.
- This is needed as a notification that the mmapped data should not be changed resp. just has changed.
we need this notification is because when userspace read the mmapped data, it's from the ptr returned from mmap(). So, when userspace reads that ptr, there will be no page fault or read/write system calls, so vendor driver does not know whether read/write opertation happens or not. Therefore, before userspace reads the ptr from mmap, it first writes action field in control region (through write system call), and vendor driver will not return the write system call until data prepared.
When userspace writes to that ptr from mmap, it writes data to the data region first, then writes the action field in control region (through write system call) to notify vendor driver. vendor driver will return the system call after it copies the buffer completely.
Quoted 5 lines
So, how does the kernel know whether the read action has finished resp. whether the write action has started? Even if userspace reads/writes it as a whole.
kernel does not touch the data region except when in response to the "action" write system call.
Quoted 117 lines
Quoted 117 lines
linux-headers/linux/vfio.h +117
-0+ +/* layout of device state interfaces' control region +* By reading to control region and reading/writing data from device config +* region, device memory region, system memory regions, below interface can +* be implemented: +* +* 1. get version +* (1) user space calls read system call on "version" field of control +* region. +* (2) vendor driver writes version number of device state interfaces +* to the "version" field of control region. +* +* 2. get caps +* (1) user space calls read system call on "caps" field of control region. +* (2) if a VFIO device has huge device memory, vendor driver reports +* VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY in "caps" field of control region. +* if a VFIO device produces dirty pages in system memory, vendor driver +* reports VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY in "caps" field of +* control region. +* +* 3. set device state +* (1) user space calls write system call on "device_state" field of +* control region. +* (2) device state transitions as: +* +* RUNNING -- start dirty data logging --> RUNNING & LOGGING +* RUNNING -- deactivate --> STOP +* RUNNING -- deactivate & start dirty data longging --> STOP & LOGGING +* RUNNING & LOGGING -- stop dirty data logging --> RUNNING +* RUNNING & LOGGING -- deactivate --> STOP & LOGGING +* RUNNING & LOGGING -- deactivate & stop dirty data logging --> STOP +* STOP -- activate --> RUNNING +* STOP -- start dirty data logging --> STOP & LOGGING +* STOP -- activate & start dirty data logging --> RUNNING & LOGGING +* STOP & LOGGING -- stop dirty data logging --> STOP +* STOP & LOGGING -- activate --> RUNNING & LOGGING +* STOP & LOGGING -- activate & stop dirty data logging --> RUNNING +* +* 4. get device config size +* (1) user space calls read system call on "device_config.size" field of +* control region for the total size of device config snapshot. +* (2) vendor driver writes device config data's total size in +* "device_config.size" field of control region. +* +* 5. set device config size +* (1) user space calls write system call. +* total size of device config snapshot --> "device_config.size" field +* of control region. +* (2) vendor driver reads device config data's total size from +* "device_config.size" field of control region. +* +* 6 get device config buffer +* (1) user space calls write system call. +* "GET_BUFFER" --> "device_config.action" field of control region. +* (2) vendor driver +* a. gets whole snapshot for device config +* b. writes whole device config snapshot to region +* DEVICE_CONFIG. +* (3) user space reads the whole of device config snapshot from region +* DEVICE_CONFIG. +* +* 7. set device config buffer +* (1) user space writes whole of device config data to region +* DEVICE_CONFIG. +* (2) user space calls write system call. +* "SET_BUFFER" --> "device_config.action" field of control region. +* (3) vendor driver loads whole of device config from region DEVICE_CONFIG. +* +* 8. get device memory size +* (1) user space calls read system call on "device_memory.size" field of +* control region for device memory size. +* (2) vendor driver +* a. gets device memory snapshot (in state RUNNING or STOP), or +* gets device memory dirty data (in state RUNNING & LOGGING or +* state STOP & LOGGING) +* b. writes size in "device_memory.size" field of control region +* +* 9. set device memory size +* (1) user space calls write system call on "device_memory.size" field of +* control region to set total size of device memory snapshot. +* (2) vendor driver reads device memory's size from "device_memory.size" +* field of control region. +* +* +* 10. get device memory buffer +* (1) user space calls write system. +* pos --> "device_memory.pos" field of control region, +* "GET_BUFFER" --> "device_memory.action" field of control region. +* (pos must be 0 or multiples of length of region DEVICE_MEMORY). +* (2) vendor driver writes N'th chunk of device memory snapshot/dirty data +* to region DEVICE_MEMORY. +* (N equals to pos/(region length of DEVICE_MEMORY)) +* (3) user space reads the N'th chunk of device memory snapshot/dirty data +* from region DEVICE_MEMORY. +* +* 11. set device memory buffer +* (1) user space writes N'th chunk of device memory snapshot/dirty data to +* region DEVICE_MEMORY. +* (N equals to pos/(region length of DEVICE_MEMORY)) +* (2) user space writes pos to "device_memory.pos" field and writes +* "SET_BUFFER" to "device_memory.action" field of control region. +* (3) vendor driver loads N'th chunk of device memory snapshot/dirty data +* from region DEVICE_MEMORY. +* +* 12. get system memory dirty bitmap +* (1) user space calls write system call to specify a range of system +* memory that querying dirty pages. +* system memory's start address --> "system_memory.start_addr" field +* of control region, +* system memory's page count --> "system_memory.page_nr" field of +* control region. +* (2) if device state is not in RUNNING or STOP & LOGGING, +* vendor driver returns empty bitmap; otherwise, +* vendor driver checks the page_nr, +* if it's larger than the size that region DIRTY_BITMAP can support, +* error returns; if not, +* vendor driver returns as bitmap to specify dirty pages that
Quoted 9 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1058
Quoted 4 lines
linux-headers/linux/vfio.h +4
-0+* device produces since last query in this range of system memory . +* (3) usespace reads back the dirty bitmap from region DIRTY_BITMAP. +* +*/ It might make sense to extract the explanations above into a separate design document in the kernel
Documentation/directory. You could also add ASCII art there :)
yes, a diagram is better:)
Quoted 7 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1061
Quoted 3 lines
linux-headers/linux/vfio.h +3
-0+ +struct vfio_device_state_ctl { +__u32 version;/* ro versio of devcie state interfaces*/ s/versio/version/
s/devcie/device/
thanks~
Quoted 6 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1064
Quoted 3 lines
linux-headers/linux/vfio.h +3
-0+__u32 device_state;/* VFIO device state, wo */ +__u32 caps;/* ro */ +struct {Indentation looks a bit off.
Quoted 10 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1071
Quoted 7 lines
linux-headers/linux/vfio.h +7
-0+__u32 action;/* wo, GET_BUFFER or SET_BUFFER */ +__u64 size;/*rw, total size of device config*/ + } device_config; +struct { +__u32 action;/* wo, GET_BUFFER or SET_BUFFER */ +__u64 size;/* rw, total size of device memory*/ +__u64 pos;/*chunk offset in total buffer of device memory*/ Here as well.
thanks~
Quoted 10 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1077
Quoted 6 lines
linux-headers/linux/vfio.h +6
-0+ } device_memory; +struct { +__u64 start_addr;/* wo */ +__u64 page_nr;/* wo */ + } system_memory; +}__attribute__((packed));For an interface definition, it's probably better to avoid packed and instead add padding if needed.
ok. so just remove the __attribute__((packed)); is enough for this interface.
Quoted 6 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1081
Quoted 4 lines
linux-headers/linux/vfio.h +1
-0+/* ***************************************************************** */ #endif /* VFIO_H */ On the whole, I think this is moving into the right direction.
Re: [PATCH 1/5] vfio/migration: define kernel interfaces
Quoted 3 lines · On Wed, 20 Feb 2019 02:36:36 -0500 Zhao Yan <[email protected]> wrote:
Quoted 2 lines · Cornelia Huck
On Tue, 19 Feb 2019 16:52:14 +0800
Yan Zhao <[email protected]> wrote:
(...)
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
+ * Size of device config data is smaller than or equal to that of
+ * device config region.Not sure if I understand that sentence correctly... but what if a device has more config state than fits into this region? Is that supposed to be covered by the device memory region? Or is this assumed to be something so exotic that we don't need to plan for it?
Quoted 8 lines
Device config data and device config region are all provided by vendor driver, so vendor driver is always able to create a large enough device config region to hold device config data.
So, if a device has data that are better to be saved after device stop and saved/loaded in strict order, the data needs to be in device config region. This kind of data is supposed to be small.
If the device data can be saved/loaded several times, it can also be put into device memory region.
So, it is the vendor driver's decision which device information should go via which region? With the device config data supposed to be saved/loaded in one go?
(...)
Quoted 7 lines
Quoted 6 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 870
Quoted 2 lines
linux-headers/linux/vfio.h +2
-0+/* version number of the device state interface */ +#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1Hm. Is this supposed to be backwards-compatible, should we need to bump this?
currently no backwords-compatible. we can discuss on that.
It might be useful if we discover that we need some extensions. But I'm not sure how much work it would be.
(...)
Quoted 11 lines
Quoted 9 lines
Quoted 7 lines
+/*
+ * DEVICE STATES
+ *
+ * Four states are defined for a VFIO device:
+ * RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP.
+ * They can be set by writing todevice_statefield of
+ *vfio_device_state_ctlregion.Who controls this? Userspace?
Yes. Userspace notifies vendor driver to do the state switching.
Might be good to mention this (just to make it obvious).
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
+ * LOGGING state is a special state that it CANNOT exist
+ * independently.So it's not a state, but rather a modifier?
yes. or thinking LOGGING/not LOGGING as bit 1 of a device state, whereas RUNNING/STOPPED is bit 0 of a device state. They have to be got as a whole.
So it is (on a bit level):
RUNNING -> 00
STOPPED -> 01
LOGGING/RUNNING -> 10
LOGGING/STOPPED -> 11
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+ * It must be set alongside with state RUNNING or STOP, i.e,
+ * RUNNING & LOGGING, STOP & LOGGING.
+ * It is used for dirty data logging both for device memory
+ * and system memory.
+ *
+ * LOGGING only impacts device/system memory. In LOGGING state, get buffer
+ * of device memory returns dirty pages since last call; outside LOGGING
+ * state, get buffer of device memory returns whole snapshot of device
Quoted 8 lines
Quoted 8 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 915
Quoted 8 lines
+ * memory. system memory's dirty page is only available in LOGGING state.
+ *
+ * Device config should be always accessible and return whole config snapshot
+ * regardless of LOGGING state.
+ * */linux-headers/linux/vfio.h +3
-0+#define VFIO_DEVICE_STATE_RUNNING 0 +#define VFIO_DEVICE_STATE_STOP 1 +#define VFIO_DEVICE_STATE_LOGGING 2
This makes it look a bit like LOGGING were an individual state, while 2 is in reality LOGGING/RUNNING... not sure how to make that more obvious. Maybe (as we are dealing with a u32):
#define VFIO_DEVICE_STATE_RUNNING 0x00000000 #define VFIO_DEVICE_STATE_STOPPED 0x00000001 #define VFIO_DEVICE_STATE_LOGGING_RUNNING 0x00000002 #define VFIO_DEVICE_STATE_LOGGING_STOPPED 0x00000003 #define VFIO_DEVICE_STATE_LOGGING_MASK 0x00000002
Quoted 43 lines
Quoted 26 lines
Quoted 12 lines
linux-headers/linux/vfio.h +12
-0+ +/* action to get data from device memory or device config +* the action is write to device state's control region, and data is read +* from device memory region or device config region. +* Each time before read device memory region or device config region, +* action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action +* field in control region. That is because device memory and devie config +* region is mmaped into user space. vendor driver has to be notified of +* the the GET_BUFFER action in advance. +*/ +#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1 +linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 937
Quoted 10 lines
linux-headers/linux/vfio.h +10
-0+/* action to set data to device memory or device config +* the action is write to device state's control region, and data is +* written to device memory region or device config region. +* Each time after write to device memory region or device config region, +* action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action +* field in control region. That is because device memory and devie config +* region is mmaped into user space. vendor driver has to be notified of +* the the SET_BUFFER action after data written. +*/ +#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2Let me describe this in my own words to make sure that I understand this correctly.
Quoted 4 lines
- The actions are set by userspace to notify the kernel that it is going to get data or that it has just written data.
- This is needed as a notification that the mmapped data should not be changed resp. just has changed.we need this notification is because when userspace read the mmapped data, it's from the ptr returned from
mmap(). So, when userspace reads that ptr, there will be no page fault or read/write system calls, so vendor driver does not know whether read/write opertation happens or not. Therefore, before userspace reads the ptr from mmap, it first writes action field in control region (through write system call), and vendor driver will not return the write system call until data prepared.When userspace writes to that ptr from mmap, it writes data to the data region first, then writes the action field in control region (through write system call) to notify vendor driver. vendor driver will return the system call after it copies the buffer completely.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
So, how does the kernel know whether the read action has finished resp. whether the write action has started? Even if userspace reads/writes it as a whole.
kernel does not touch the data region except when in response to the "action" write system call.
Thanks for the explanation, that makes sense.
(...)
Re: [PATCH 1/5] vfio/migration: define kernel interfaces
Quoted 24 lines · Cornelia Huck
Quoted 3 lines · On Wed, 20 Feb 2019 02:36:36 -0500 Zhao Yan <[email protected]> wrote:
Quoted 2 lines · Cornelia Huck
On Tue, 19 Feb 2019 16:52:14 +0800
Yan Zhao <[email protected]> wrote:(...)
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
+ * Size of device config data is smaller than or equal to that of
+ * device config region.Not sure if I understand that sentence correctly... but what if a device has more config state than fits into this region? Is that supposed to be covered by the device memory region? Or is this assumed to be something so exotic that we don't need to plan for it?
Quoted 8 lines
Device config data and device config region are all provided by vendor driver, so vendor driver is always able to create a large enough device config region to hold device config data.
So, if a device has data that are better to be saved after device stop and saved/loaded in strict order, the data needs to be in device config region. This kind of data is supposed to be small.
If the device data can be saved/loaded several times, it can also be put into device memory region.
Quoted 3 lines
So, it is the vendor driver's decision which device information should go via which region? With the device config data supposed to be saved/loaded in one go?
Right, exactly.
Quoted 21 lines
(...)
Quoted 7 lines
Quoted 6 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 870
Quoted 2 lines
linux-headers/linux/vfio.h +2
-0+/* version number of the device state interface */ +#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1Hm. Is this supposed to be backwards-compatible, should we need to bump this?
currently no backwords-compatible. we can discuss on that.
It might be useful if we discover that we need some extensions. But I'm not sure how much work it would be.
(...)
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
+/*
+ * DEVICE STATES
+ *
+ * Four states are defined for a VFIO device:
+ * RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP.
+ * They can be set by writing todevice_statefield of
+ *vfio_device_state_ctlregion.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Who controls this? Userspace?
Yes. Userspace notifies vendor driver to do the state switching.
Might be good to mention this (just to make it obvious).
Got it. thanks
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
+ * LOGGING state is a special state that it CANNOT exist
+ * independently.So it's not a state, but rather a modifier?
yes. or thinking LOGGING/not LOGGING as bit 1 of a device state, whereas RUNNING/STOPPED is bit 0 of a device state. They have to be got as a whole.
Quoted 6 lines
So it is (on a bit level):
RUNNING -> 00
STOPPED -> 01
LOGGING/RUNNING -> 10
LOGGING/STOPPED -> 11
Yes.
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+ * It must be set alongside with state RUNNING or STOP, i.e,
+ * RUNNING & LOGGING, STOP & LOGGING.
+ * It is used for dirty data logging both for device memory
+ * and system memory.
+ *
+ * LOGGING only impacts device/system memory. In LOGGING state, get buffer
+ * of device memory returns dirty pages since last call; outside LOGGING
+ * state, get buffer of device memory returns whole snapshot of deviceQuoted 8 lines
Quoted 8 lines
linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 915
Quoted 8 lines
+ * memory. system memory's dirty page is only available in LOGGING state.
+ *
+ * Device config should be always accessible and return whole config snapshot
+ * regardless of LOGGING state.
+ * */linux-headers/linux/vfio.h +3
-0+#define VFIO_DEVICE_STATE_RUNNING 0 +#define VFIO_DEVICE_STATE_STOP 1 +#define VFIO_DEVICE_STATE_LOGGING 2This makes it look a bit like LOGGING were an individual state, while 2 is in reality LOGGING/RUNNING... not sure how to make that more obvious. Maybe (as we are dealing with a u32):
Quoted 6 lines
#define
VFIO_DEVICE_STATE_RUNNING0x00000000#defineVFIO_DEVICE_STATE_STOPPED0x00000001#defineVFIO_DEVICE_STATE_LOGGING_RUNNING0x00000002#defineVFIO_DEVICE_STATE_LOGGING_STOPPED0x00000003#defineVFIO_DEVICE_STATE_LOGGING_MASK0x00000002
Yes, yours are better, thanks:)
Quoted 47 lines
Quoted 47 lines
Quoted 26 lines
Quoted 12 lines
linux-headers/linux/vfio.h +12
-0+ +/* action to get data from device memory or device config +* the action is write to device state's control region, and data is read +* from device memory region or device config region. +* Each time before read device memory region or device config region, +* action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action +* field in control region. That is because device memory and devie config +* region is mmaped into user space. vendor driver has to be notified of +* the the GET_BUFFER action in advance. +*/ +#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1 +linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 937
Quoted 10 lines
linux-headers/linux/vfio.h +10
-0+/* action to set data to device memory or device config +* the action is write to device state's control region, and data is +* written to device memory region or device config region. +* Each time after write to device memory region or device config region, +* action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action +* field in control region. That is because device memory and devie config +* region is mmaped into user space. vendor driver has to be notified of +* the the SET_BUFFER action after data written. +*/ +#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2Let me describe this in my own words to make sure that I understand this correctly.
Quoted 4 lines
- The actions are set by userspace to notify the kernel that it is going to get data or that it has just written data.
- This is needed as a notification that the mmapped data should not be changed resp. just has changed.we need this notification is because when userspace read the mmapped data, it's from the ptr returned from
mmap(). So, when userspace reads that ptr, there will be no page fault or read/write system calls, so vendor driver does not know whether read/write opertation happens or not. Therefore, before userspace reads the ptr from mmap, it first writes action field in control region (through write system call), and vendor driver will not return the write system call until data prepared.When userspace writes to that ptr from mmap, it writes data to the data region first, then writes the action field in control region (through write system call) to notify vendor driver. vendor driver will return the system call after it copies the buffer completely.
Quoted 5 lines
So, how does the kernel know whether the read action has finished resp. whether the write action has started? Even if userspace reads/writes it as a whole.
Quoted 5 lines
Quoted 2 lines
kernel does not touch the data region except when in response to the "action" write system call.
Thanks for the explanation, that makes sense.
(...)
welcome:)
Quoted 4 lines
_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
PATCH 2/5 vfio/migration: support device of device config capability
In reply to Yan ZhaoDevice config is the default data that every device should have. so device config capability is by default on, no need to set.
- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.
Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state to VFIO_DEVICE_STATE_LOGGING
--- hw/vfio/Makefile.objs | 2 +- hw/vfio/migration.c | 633 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 1 - hw/vfio/pci.h | 25 +- include/hw/vfio/vfio-common.h | 1 + 5 files changed, 659 insertions(+), 3 deletions(-) create mode 100644 hw/vfio/migration.c
hw/vfio/Makefile.objs +1 -1
@@ -1,6 +1,6 @@ ifeq ($(CONFIG_LINUX), y) obj-$(CONFIG_SOFTMMU) += common.o -obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o +obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.o obj-$(CONFIG_VFIO_CCW) += ccw.o obj-$(CONFIG_SOFTMMU) += platform.o obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o
hw/vfio/migration.c new file +633 -0
633 lines not on this page · raw message
hw/vfio/pci.c +0 -1
@@ -37,7 +37,6 @@#define MSIX_CAP_LENGTH 12 -#define TYPE_VFIO_PCI "vfio-pci"#define PCI_VFIO (obj) OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)static void vfio_disable_interrupts (VFIOPCIDevice *vdev);
hw/vfio/pci.h +24 -1
@@ -19,6 +19,7 @@#include "qemu/event_notifier.h" #include "qemu/queue.h" #include "qemu/timer.h" +#include "sysemu/sysemu.h" #define PCI_ANY_ID (~0)
@@ -56,6 +57,21 @@ typedef struct VFIOBAR { QLIST_HEAD(, VFIOQuirk) quirks; }VFIOBAR ; +enum { +VFIO_DEVSTATE_REGION_CTL =0 , +VFIO_DEVSTATE_REGION_DATA_CONFIG , +VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY , +VFIO_DEVSTATE_REGION_DATA_BITMAP , +VFIO_DEVSTATE_REGION_NUM , +}; +typedef struct VFIOMigration { +VFIORegion region[VFIO_DEVSTATE_REGION_NUM ]; +uint32_t data_caps; +uint32_t device_state; +uint64_t devconfig_size; +VMChangeStateEntry *vm_state; +}VFIOMigration ; + typedef struct VFIOVGARegion { MemoryRegion mem; off_t offset;
@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {VFIOBAR bars[PCI_NUM_REGIONS -1 ];/* No ROM */ VFIOVGA *vga;/* 0xa0000, 0x3b0, 0x3c0 */ void *igd_opregion; +VFIOMigration *migration; +Error *migration_blocker;PCIHostDeviceAddress host;EventNotifier err_notifier;EventNotifier req_notifier;
@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,void vfio_display_reset (VFIOPCIDevice *vdev);int vfio_display_probe (VFIOPCIDevice *vdev,Error **errp);void vfio_display_finalize (VFIOPCIDevice *vdev); - +bool vfio_device_data_cap_system_memory (VFIOPCIDevice *vdev); +bool vfio_device_data_cap_device_memory (VFIOPCIDevice *vdev); +int vfio_set_dirty_page_bitmap (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr); +int vfio_migration_init (VFIOPCIDevice *vdev,Error **errp); +void vfio_migration_finalize (VFIOPCIDevice *vdev);#endif /* HW_VFIO_VFIO_PCI_H */
include/hw/vfio/vfio-common.h +1 -0
@@ -32,6 +32,7 @@#endif #define VFIO_MSG_PREFIX "vfio %s: " +#define TYPE_VFIO_PCI "vfio-pci"enum {VFIO_DEVICE_TYPE_PCI =0 ,
Signature · 1 line
2.7.4
Re: [PATCH 2/5] vfio/migration: support device of device config capability
Quoted 184 lines · * Yan Zhao ([email protected]) wrote:
Device config is the default data that every device should have. so device config capability is by default on, no need to set.
- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state toVFIO_DEVICE_STATE_LOGGINGSigned-off-by Yan Zhao <[email protected]>Signed-off-by Yulei Zhang <[email protected]>--- hw/vfio/Makefile.objs | 2 +- hw/vfio/migration.c | 633 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 1 - hw/vfio/pci.h | 25 +- include/hw/vfio/vfio-common.h | 1 + 5 files changed, 659 insertions(+), 3 deletions(-) create mode 100644 hw/vfio/migration.chw/vfio/Makefile.objs +1
-1@@ -1,6 +1,6 @@ ifeq ($(CONFIG_LINUX), y) obj-$(CONFIG_SOFTMMU) += common.o -obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o +obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.o obj-$(CONFIG_VFIO_CCW) += ccw.o obj-$(CONFIG_SOFTMMU) += platform.o obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.ohw/vfio/migration.c new file +137
-0@@ -0,0 +1,633 @@ +#include "qemu/osdep.h" + +#include "hw/vfio/vfio-common.h" +#include "migration/blocker.h" +#include "migration/register.h" +#include "qapi/error.h" +#include "pci.h" +#include "sysemu/kvm.h" +#include "exec/ram_addr.h" + +#define VFIO_SAVE_FLAG_SETUP 0 +#define VFIO_SAVE_FLAG_PCI 1 +#define VFIO_SAVE_FLAG_DEVCONFIG 2 +#define VFIO_SAVE_FLAG_DEVMEMORY 4 +#define VFIO_SAVE_FLAG_CONTINUE 8 + +static int vfio_device_state_region_setup (VFIOPCIDevice *vdev, +VFIORegion *region,uint32_t subtype,const char *name) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +struct vfio_region_info *info; +int ret; + + ret =vfio_get_dev_region_info (vbasedev,VFIO_REGION_TYPE_DEVICE_STATE , + subtype, &info); +if (ret) { +error_report ("Failed to get info of region %s" , name); +return ret; + } + +if (vfio_region_setup (OBJECT (vdev), vbasedev, + region, info->index, name)) { +error_report ("Failed to setup migrtion region %s" , name); +return ret; + } + +if (vfio_region_mmap (region)) { +error_report ("Failed to mmap migrtion region %s" , name); + } + +return 0 ; +} + +bool vfio_device_data_cap_system_memory (VFIOPCIDevice *vdev) +{ +return !!(vdev->migration->data_caps &VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY ); +} + +bool vfio_device_data_cap_device_memory (VFIOPCIDevice *vdev) +{ +return !!(vdev->migration->data_caps &VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY ); +} + +static bool vfio_device_state_region_mmaped (VFIORegion *region) +{ +bool mmaped = true; +if (region->nr_mmaps !=1 || region->mmaps[0 ].offset || + (region->size != region->mmaps[0 ].size) || + (region->mmaps[0 ].mmap ==NULL )) { + mmaped = false; + } + +return mmaped; +} + +static int vfio_get_device_config_size (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +uint64_t len; +int sz; + + sz =sizeof (len); +if (pread (vbasedev->fd, &len, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.size)) + != sz) { +error_report ("vfio: Failed to get length of device config" ); +return -1 ; + } +if (len > region_config->size) { +error_report ("vfio: Error device config length" ); +return -1 ; + } + vdev->migration->devconfig_size = len; + +return 0 ; +} + +static int vfio_set_device_config_size (VFIOPCIDevice *vdev,uint64_t size) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +int sz; + +if (size > region_config->size) { +return -1 ; + } + + sz =sizeof (size); +if (pwrite (vbasedev->fd, &size, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.size)) + != sz) { +error_report ("vfio: Failed to set length of device config" ); +return -1 ; + } + vdev->migration->devconfig_size = size; +return 0 ; +} + +static int vfio_save_data_device_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_GET_BUFFER ; +uint64_t len = vdev->migration->devconfig_size; + +qemu_put_be64 (f, len); + + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.action)) + != sz) {
hw/vfio/migration.c line 147
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+error_report ("vfio: action failure for device config get buffer" ); +return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_config)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + }
g_malloc never returns NULL; it aborts on failure to allocate. So you can either drop the check, or my preference is to use g_try_malloc for large/unknown areas, and it can return NULL.
Quoted 276 lines
hw/vfio/migration.c new file +276
-0+if (pread (vbasedev->fd, buf, len, region_config->fd_offset) != len) { +error_report ("vfio: Failed read device config buffer" ); +return -1 ; + } +qemu_put_buffer (f, buf, len); +g_free (buf); + }else { + dest = region_config->mmaps[0 ].mmap; +qemu_put_buffer (f, dest, len); + } +return 0 ; +} + +static int vfio_load_data_device_config (VFIOPCIDevice *vdev, +QEMUFile *f,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_SET_BUFFER ; + +vfio_set_device_config_size (vdev, len); + +if (!vfio_device_state_region_mmaped (region_config)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +qemu_get_buffer (f, buf, len); +if (pwrite (vbasedev->fd, buf, len, + region_config->fd_offset) != len) { +error_report ("vfio: Failed to write devie config buffer" ); +return -1 ; + } +g_free (buf); + }else { + dest = region_config->mmaps[0 ].mmap; +qemu_get_buffer (f, dest, len); + } + + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.action)) + != sz) { +error_report ("vfio: action failure for device config set buffer" ); +return -1 ; + } + +return 0 ; +} + +static int vfio_set_dirty_page_bitmap_chunk (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr) +{ + +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_bitmap = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ]; +unsigned long bitmap_size = +BITS_TO_LONGS (page_nr) *sizeof (unsigned long ); +uint32_t sz; + +struct { +__u64 start_addr; +__u64 page_nr; + } system_memory; + system_memory.start_addr = start_addr; + system_memory.page_nr = page_nr; + sz =sizeof (system_memory); +if (pwrite (vbasedev->fd, &system_memory, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl , system_memory)) + != sz) { +error_report ("vfio: Failed to set system memory range for dirty pages" ); +return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_bitmap)) { +void *bitmap =g_malloc0 (bitmap_size); + +if (pread (vbasedev->fd, bitmap, bitmap_size, + region_bitmap->fd_offset) != bitmap_size) { +error_report ("vfio: Failed to read dirty bitmap data" ); +return -1 ; + } + +cpu_physical_memory_set_dirty_lebitmap (bitmap, start_addr, page_nr); + +g_free (bitmap); + }else { +cpu_physical_memory_set_dirty_lebitmap ( + region_bitmap->mmaps[0 ].mmap, + start_addr, page_nr); + } +return 0 ; +} + +int vfio_set_dirty_page_bitmap (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr) +{ +VFIORegion *region_bitmap = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ]; +unsigned long chunk_size = region_bitmap->size; +uint64_t chunk_pg_nr = (chunk_size /sizeof (unsigned long )) * +BITS_PER_LONG ; + +uint64_t cnt_left; +int rc =0 ; + + cnt_left = page_nr; + +while (cnt_left >= chunk_pg_nr) { + rc =vfio_set_dirty_page_bitmap_chunk (vdev, start_addr, chunk_pg_nr); +if (rc) { + goto exit; + } + cnt_left -= chunk_pg_nr; + start_addr += start_addr; + } + rc =vfio_set_dirty_page_bitmap_chunk (vdev, start_addr, cnt_left); + +exit: +return rc; +} + +static int vfio_set_device_state (VFIOPCIDevice *vdev, +uint32_t dev_state) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +uint32_t sz =sizeof (dev_state); + +if (!vdev->migration) { +return -1 ; + } + +if (pwrite (vbasedev->fd, &dev_state, sz, + region->fd_offset + + offsetof(struct vfio_device_state_ctl , device_state)) + != sz) { +error_report ("vfio: Failed to set device state %d" , dev_state); +return -1 ; + } + vdev->migration->device_state = dev_state; +return 0 ; +} + +static int vfio_get_device_data_caps (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; + +uint32_t caps; +uint32_t size =sizeof (caps); + +if (pread (vbasedev->fd, &caps, size, + region->fd_offset + + offsetof(struct vfio_device_state_ctl , caps)) + != size) { +error_report ("%s Failed to read data caps of device states" , + vbasedev->name); +return -1 ; + } + vdev->migration->data_caps = caps; +return 0 ; +} + + +static int vfio_check_devstate_version (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; + +uint32_t version; +uint32_t size =sizeof (version); + +if (pread (vbasedev->fd, &version, size, + region->fd_offset + + offsetof(struct vfio_device_state_ctl , version)) + != size) { +error_report ("%s Failed to read version of device state interfaces" , + vbasedev->name); +return -1 ; + } + +if (version !=VFIO_DEVICE_STATE_INTERFACE_VERSION ) { +error_report ("%s migration version mismatch, right version is %d" , + vbasedev->name,VFIO_DEVICE_STATE_INTERFACE_VERSION ); +return -1 ; + } + +return 0 ; +} + +static void vfio_vm_change_state_handler (void *pv,int running,RunState state) +{ +VFIOPCIDevice *vdev = pv; +uint32_t dev_state = vdev->migration->device_state; + +if (!running) { + dev_state |=VFIO_DEVICE_STATE_STOP ; + }else { + dev_state &= ~VFIO_DEVICE_STATE_STOP ; + } + +vfio_set_device_state (vdev, dev_state); +} + +static void vfio_save_live_pending (QEMUFile *f,void *opaque, +uint64_t max_size, +uint64_t *res_precopy_only, +uint64_t *res_compatible, +uint64_t *res_post_copy_only) +{ +VFIOPCIDevice *vdev = opaque; + +if (!vfio_device_data_cap_device_memory (vdev)) { +return ; + } + +return ; +} + +static int vfio_save_iterate (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; + +if (!vfio_device_data_cap_device_memory (vdev)) { +return 0 ; + } + +return 0 ; +} + +static void vfio_pci_load_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +PCIDevice *pdev = &vdev->pdev; +uint32_t ctl, msi_lo, msi_hi, msi_data, bar_cfg, i; +bool msi_64bit; + +/* retore pci bar configuration */ + ctl =pci_default_read_config (pdev,PCI_COMMAND ,2 ); +vfio_pci_write_config (pdev,PCI_COMMAND , + ctl & (!(PCI_COMMAND_IO |PCI_COMMAND_MEMORY )),2 ); +for (i =0 ; i <PCI_ROM_SLOT ; i++) { + bar_cfg =qemu_get_be32 (f); +vfio_pci_write_config (pdev,PCI_BASE_ADDRESS_0 + i *4 , bar_cfg,4 ); + } +vfio_pci_write_config (pdev,PCI_COMMAND , + ctl |PCI_COMMAND_IO |PCI_COMMAND_MEMORY ,2 ); + +/* restore msi configuration */ + ctl =pci_default_read_config (pdev, pdev->msi_cap +PCI_MSI_FLAGS ,2 ); + msi_64bit = !!(ctl &PCI_MSI_FLAGS_64BIT ); + +vfio_pci_write_config (&vdev->pdev, + pdev->msi_cap +PCI_MSI_FLAGS , + ctl & (!PCI_MSI_FLAGS_ENABLE ),2 ); + + msi_lo =qemu_get_be32 (f); +vfio_pci_write_config (pdev, pdev->msi_cap +PCI_MSI_ADDRESS_LO , msi_lo,4 ); + +if (msi_64bit) { + msi_hi =qemu_get_be32 (f);
hw/vfio/migration.c line 433
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+vfio_pci_write_config (pdev, pdev->msi_cap +PCI_MSI_ADDRESS_HI , + msi_hi,4 ); + } + msi_data =qemu_get_be32 (f); +vfio_pci_write_config (pdev, + pdev->msi_cap + (msi_64bit ?PCI_MSI_DATA_64 :PCI_MSI_DATA_32 ), + msi_data,2 ); + +vfio_pci_write_config (&vdev->pdev, pdev->msi_cap +PCI_MSI_FLAGS , + ctl |PCI_MSI_FLAGS_ENABLE ,2 );
It would probably be best to use a VMStateDescription and the macros for this if possible; I bet you'll want to add more fields in the future for example.
Also what happens if the data read from the migration stream is bad or doesn't agree with this devices hardware? How does this fail?
Quoted 281 lines
hw/vfio/migration.c new file +199
-0+} + +static int vfio_load_state (QEMUFile *f,void *opaque,int version_id) +{ +VFIOPCIDevice *vdev = opaque; +int flag; +uint64_t len; +int ret =0 ; + +if (version_id !=VFIO_DEVICE_STATE_INTERFACE_VERSION ) { +return -EINVAL ; + } + +do { + flag =qemu_get_byte (f); + +switch (flag & ~VFIO_SAVE_FLAG_CONTINUE ) { +case VFIO_SAVE_FLAG_SETUP : +break ; +case VFIO_SAVE_FLAG_PCI : +vfio_pci_load_config (vdev, f); +break ; +case VFIO_SAVE_FLAG_DEVCONFIG : + len =qemu_get_be64 (f); +vfio_load_data_device_config (vdev, f, len); +break ; +default : + ret = -EINVAL ; + } + }while (flag &VFIO_SAVE_FLAG_CONTINUE ); + +return ret; +} + +static void vfio_pci_save_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +PCIDevice *pdev = &vdev->pdev; +uint32_t msi_cfg, msi_lo, msi_hi, msi_data, bar_cfg, i; +bool msi_64bit; + +for (i =0 ; i <PCI_ROM_SLOT ; i++) { + bar_cfg =pci_default_read_config (pdev,PCI_BASE_ADDRESS_0 + i *4 ,4 ); +qemu_put_be32 (f, bar_cfg); + } + + msi_cfg =pci_default_read_config (pdev, pdev->msi_cap +PCI_MSI_FLAGS ,2 ); + msi_64bit = !!(msi_cfg &PCI_MSI_FLAGS_64BIT ); + + msi_lo =pci_default_read_config (pdev, + pdev->msi_cap +PCI_MSI_ADDRESS_LO ,4 ); +qemu_put_be32 (f, msi_lo); + +if (msi_64bit) { + msi_hi =pci_default_read_config (pdev, + pdev->msi_cap +PCI_MSI_ADDRESS_HI , +4 ); +qemu_put_be32 (f, msi_hi); + } + + msi_data =pci_default_read_config (pdev, + pdev->msi_cap + (msi_64bit ?PCI_MSI_DATA_64 :PCI_MSI_DATA_32 ), +2 ); +qemu_put_be32 (f, msi_data); + +} + +static int vfio_save_complete_precopy (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +int rc =0 ; + +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE ); +vfio_pci_save_config (vdev, f); + +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVCONFIG ); + rc +=vfio_get_device_config_size (vdev); + rc +=vfio_save_data_device_config (vdev, f); + +return rc; +} + +static int vfio_save_setup (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING | +VFIO_DEVICE_STATE_LOGGING ); +return 0 ; +} + +static int vfio_load_setup (QEMUFile *f,void *opaque) +{ +return 0 ; +} + +static void vfio_save_cleanup (void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +uint32_t dev_state = vdev->migration->device_state; + + dev_state &= ~VFIO_DEVICE_STATE_LOGGING ; + +vfio_set_device_state (vdev, dev_state); +} + +static SaveVMHandlers savevm_vfio_handlers = { + .save_setup = vfio_save_setup, + .save_live_pending = vfio_save_live_pending, + .save_live_iterate = vfio_save_iterate, + .save_live_complete_precopy = vfio_save_complete_precopy, + .save_cleanup = vfio_save_cleanup, + .load_setup = vfio_load_setup, + .load_state = vfio_load_state, +}; + +int vfio_migration_init (VFIOPCIDevice *vdev,Error **errp) +{ +int ret; +Error *local_err =NULL ; + vdev->migration =g_new0 (VFIOMigration,1 ); + +if (vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL , +"device-state-ctl" )) { + goto error; + } + +if (vfio_check_devstate_version (vdev)) { + goto error; + } + +if (vfio_get_device_data_caps (vdev)) { + goto error; + } + +if (vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG , +"device-state-data-device-config" )) { + goto error; + } + +if (vfio_device_data_cap_device_memory (vdev)) { +error_report ("No suppport of data cap device memory Yet" ); + goto error; + } + +if (vfio_device_data_cap_system_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP , +"device-state-data-dirtybitmap" )) { + goto error; + } + + vdev->migration->device_state =VFIO_DEVICE_STATE_RUNNING ; + +register_savevm_live (NULL ,TYPE_VFIO_PCI ,-1 , +VFIO_DEVICE_STATE_INTERFACE_VERSION , + &savevm_vfio_handlers, + vdev); + + vdev->migration->vm_state = +qemu_add_vm_change_state_handler (vfio_vm_change_state_handler, vdev); + +return 0 ; +error: +error_setg (&vdev->migration_blocker, +"VFIO device doesn't support migration" ); + ret =migrate_add_blocker (vdev->migration_blocker, &local_err); +if (local_err) { +error_propagate (errp, local_err); +error_free (vdev->migration_blocker); + } + +g_free (vdev->migration); + vdev->migration =NULL ; + +return ret; +} + +void vfio_migration_finalize (VFIOPCIDevice *vdev) +{ +if (vdev->migration) { +int i; +qemu_del_vm_change_state_handler (vdev->migration->vm_state); +unregister_savevm (NULL ,TYPE_VFIO_PCI , vdev); +for (i =0 ; i <VFIO_DEVSTATE_REGION_NUM ; i++) { +vfio_region_finalize (&vdev->migration->region[i]); + } +g_free (vdev->migration); + vdev->migration =NULL ; + }else if (vdev->migration_blocker) { +migrate_del_blocker (vdev->migration_blocker); +error_free (vdev->migration_blocker); + } +}hw/vfio/pci.c +0
-1@@ -37,7 +37,6 @@#define MSIX_CAP_LENGTH 12 -#define TYPE_VFIO_PCI "vfio-pci"#define PCI_VFIO (obj) OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)static void vfio_disable_interrupts (VFIOPCIDevice *vdev);hw/vfio/pci.h +24
-1@@ -19,6 +19,7 @@#include "qemu/event_notifier.h" #include "qemu/queue.h" #include "qemu/timer.h" +#include "sysemu/sysemu.h" #define PCI_ANY_ID (~0)@@ -56,6 +57,21 @@ typedef struct VFIOBAR { QLIST_HEAD(, VFIOQuirk) quirks; }VFIOBAR ; +enum { +VFIO_DEVSTATE_REGION_CTL =0 , +VFIO_DEVSTATE_REGION_DATA_CONFIG , +VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY , +VFIO_DEVSTATE_REGION_DATA_BITMAP , +VFIO_DEVSTATE_REGION_NUM , +}; +typedef struct VFIOMigration { +VFIORegion region[VFIO_DEVSTATE_REGION_NUM ]; +uint32_t data_caps; +uint32_t device_state; +uint64_t devconfig_size; +VMChangeStateEntry *vm_state; +}VFIOMigration ; + typedef struct VFIOVGARegion { MemoryRegion mem; off_t offset;@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {VFIOBAR bars[PCI_NUM_REGIONS -1 ];/* No ROM */ VFIOVGA *vga;/* 0xa0000, 0x3b0, 0x3c0 */ void *igd_opregion; +VFIOMigration *migration; +Error *migration_blocker;PCIHostDeviceAddress host;EventNotifier err_notifier;EventNotifier req_notifier;@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,void vfio_display_reset (VFIOPCIDevice *vdev);int vfio_display_probe (VFIOPCIDevice *vdev,Error **errp);void vfio_display_finalize (VFIOPCIDevice *vdev); - +bool vfio_device_data_cap_system_memory (VFIOPCIDevice *vdev); +bool vfio_device_data_cap_device_memory (VFIOPCIDevice *vdev); +int vfio_set_dirty_page_bitmap (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr); +int vfio_migration_init (VFIOPCIDevice *vdev,Error **errp); +void vfio_migration_finalize (VFIOPCIDevice *vdev);#endif /* HW_VFIO_VFIO_PCI_H */ include/hw/vfio/vfio-common.h +1
-0@@ -32,6 +32,7 @@#endif #define VFIO_MSG_PREFIX "vfio %s: " +#define TYPE_VFIO_PCI "vfio-pci"enum {VFIO_DEVICE_TYPE_PCI =0 ,Signature · 1 line
2.7.4
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/5] vfio/migration: support device of device config capability
Quoted 191 lines · Dr. David Alan Gilbert
Quoted 190 lines · * Yan Zhao ([email protected]) wrote:
Device config is the default data that every device should have. so device config capability is by default on, no need to set.
- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state toVFIO_DEVICE_STATE_LOGGINGSigned-off-by Yan Zhao <[email protected]>Signed-off-by Yulei Zhang <[email protected]>--- hw/vfio/Makefile.objs | 2 +- hw/vfio/migration.c | 633 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 1 - hw/vfio/pci.h | 25 +- include/hw/vfio/vfio-common.h | 1 + 5 files changed, 659 insertions(+), 3 deletions(-) create mode 100644 hw/vfio/migration.chw/vfio/Makefile.objs +1
-1@@ -1,6 +1,6 @@ ifeq ($(CONFIG_LINUX), y) obj-$(CONFIG_SOFTMMU) += common.o -obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o +obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.o obj-$(CONFIG_VFIO_CCW) += ccw.o obj-$(CONFIG_SOFTMMU) += platform.o obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.ohw/vfio/migration.c new file +143
-0@@ -0,0 +1,633 @@ +#include "qemu/osdep.h" + +#include "hw/vfio/vfio-common.h" +#include "migration/blocker.h" +#include "migration/register.h" +#include "qapi/error.h" +#include "pci.h" +#include "sysemu/kvm.h" +#include "exec/ram_addr.h" + +#define VFIO_SAVE_FLAG_SETUP 0 +#define VFIO_SAVE_FLAG_PCI 1 +#define VFIO_SAVE_FLAG_DEVCONFIG 2 +#define VFIO_SAVE_FLAG_DEVMEMORY 4 +#define VFIO_SAVE_FLAG_CONTINUE 8 + +static int vfio_device_state_region_setup (VFIOPCIDevice *vdev, +VFIORegion *region,uint32_t subtype,const char *name) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +struct vfio_region_info *info; +int ret; + + ret =vfio_get_dev_region_info (vbasedev,VFIO_REGION_TYPE_DEVICE_STATE , + subtype, &info); +if (ret) { +error_report ("Failed to get info of region %s" , name); +return ret; + } + +if (vfio_region_setup (OBJECT (vdev), vbasedev, + region, info->index, name)) { +error_report ("Failed to setup migrtion region %s" , name); +return ret; + } + +if (vfio_region_mmap (region)) { +error_report ("Failed to mmap migrtion region %s" , name); + } + +return 0 ; +} + +bool vfio_device_data_cap_system_memory (VFIOPCIDevice *vdev) +{ +return !!(vdev->migration->data_caps &VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY ); +} + +bool vfio_device_data_cap_device_memory (VFIOPCIDevice *vdev) +{ +return !!(vdev->migration->data_caps &VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY ); +} + +static bool vfio_device_state_region_mmaped (VFIORegion *region) +{ +bool mmaped = true; +if (region->nr_mmaps !=1 || region->mmaps[0 ].offset || + (region->size != region->mmaps[0 ].size) || + (region->mmaps[0 ].mmap ==NULL )) { + mmaped = false; + } + +return mmaped; +} + +static int vfio_get_device_config_size (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +uint64_t len; +int sz; + + sz =sizeof (len); +if (pread (vbasedev->fd, &len, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.size)) + != sz) { +error_report ("vfio: Failed to get length of device config" ); +return -1 ; + } +if (len > region_config->size) { +error_report ("vfio: Error device config length" ); +return -1 ; + } + vdev->migration->devconfig_size = len; + +return 0 ; +} + +static int vfio_set_device_config_size (VFIOPCIDevice *vdev,uint64_t size) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +int sz; + +if (size > region_config->size) { +return -1 ; + } + + sz =sizeof (size); +if (pwrite (vbasedev->fd, &size, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.size)) + != sz) { +error_report ("vfio: Failed to set length of device config" ); +return -1 ; + } + vdev->migration->devconfig_size = size; +return 0 ; +} + +static int vfio_save_data_device_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_GET_BUFFER ; +uint64_t len = vdev->migration->devconfig_size; + +qemu_put_be64 (f, len); + + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.action)) + != sz) { +error_report ("vfio: action failure for device config get buffer" ); +return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_config)) { + buf =g_malloc (len);
Quoted 9 lines
hw/vfio/migration.c line 147
Quoted 4 lines
hw/vfio/migration.c new file +4
-0+if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + }
g_mallocnever returns NULL; it aborts on failure to allocate. So you can either drop the check, or my preference is to useg_try_mallocfor large/unknown areas, and it can return NULL.
ok. got that. I'll use g_try_malloc next time :)
Quoted 282 lines
Quoted 282 lines
hw/vfio/migration.c new file +282
-0+if (pread (vbasedev->fd, buf, len, region_config->fd_offset) != len) { +error_report ("vfio: Failed read device config buffer" ); +return -1 ; + } +qemu_put_buffer (f, buf, len); +g_free (buf); + }else { + dest = region_config->mmaps[0 ].mmap; +qemu_put_buffer (f, dest, len); + } +return 0 ; +} + +static int vfio_load_data_device_config (VFIOPCIDevice *vdev, +QEMUFile *f,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_SET_BUFFER ; + +vfio_set_device_config_size (vdev, len); + +if (!vfio_device_state_region_mmaped (region_config)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +qemu_get_buffer (f, buf, len); +if (pwrite (vbasedev->fd, buf, len, + region_config->fd_offset) != len) { +error_report ("vfio: Failed to write devie config buffer" ); +return -1 ; + } +g_free (buf); + }else { + dest = region_config->mmaps[0 ].mmap; +qemu_get_buffer (f, dest, len); + } + + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.action)) + != sz) { +error_report ("vfio: action failure for device config set buffer" ); +return -1 ; + } + +return 0 ; +} + +static int vfio_set_dirty_page_bitmap_chunk (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr) +{ + +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_bitmap = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ]; +unsigned long bitmap_size = +BITS_TO_LONGS (page_nr) *sizeof (unsigned long ); +uint32_t sz; + +struct { +__u64 start_addr; +__u64 page_nr; + } system_memory; + system_memory.start_addr = start_addr; + system_memory.page_nr = page_nr; + sz =sizeof (system_memory); +if (pwrite (vbasedev->fd, &system_memory, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl , system_memory)) + != sz) { +error_report ("vfio: Failed to set system memory range for dirty pages" ); +return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_bitmap)) { +void *bitmap =g_malloc0 (bitmap_size); + +if (pread (vbasedev->fd, bitmap, bitmap_size, + region_bitmap->fd_offset) != bitmap_size) { +error_report ("vfio: Failed to read dirty bitmap data" ); +return -1 ; + } + +cpu_physical_memory_set_dirty_lebitmap (bitmap, start_addr, page_nr); + +g_free (bitmap); + }else { +cpu_physical_memory_set_dirty_lebitmap ( + region_bitmap->mmaps[0 ].mmap, + start_addr, page_nr); + } +return 0 ; +} + +int vfio_set_dirty_page_bitmap (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr) +{ +VFIORegion *region_bitmap = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ]; +unsigned long chunk_size = region_bitmap->size; +uint64_t chunk_pg_nr = (chunk_size /sizeof (unsigned long )) * +BITS_PER_LONG ; + +uint64_t cnt_left; +int rc =0 ; + + cnt_left = page_nr; + +while (cnt_left >= chunk_pg_nr) { + rc =vfio_set_dirty_page_bitmap_chunk (vdev, start_addr, chunk_pg_nr); +if (rc) { + goto exit; + } + cnt_left -= chunk_pg_nr; + start_addr += start_addr; + } + rc =vfio_set_dirty_page_bitmap_chunk (vdev, start_addr, cnt_left); + +exit: +return rc; +} + +static int vfio_set_device_state (VFIOPCIDevice *vdev, +uint32_t dev_state) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +uint32_t sz =sizeof (dev_state); + +if (!vdev->migration) { +return -1 ; + } + +if (pwrite (vbasedev->fd, &dev_state, sz, + region->fd_offset + + offsetof(struct vfio_device_state_ctl , device_state)) + != sz) { +error_report ("vfio: Failed to set device state %d" , dev_state); +return -1 ; + } + vdev->migration->device_state = dev_state; +return 0 ; +} + +static int vfio_get_device_data_caps (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; + +uint32_t caps; +uint32_t size =sizeof (caps); + +if (pread (vbasedev->fd, &caps, size, + region->fd_offset + + offsetof(struct vfio_device_state_ctl , caps)) + != size) { +error_report ("%s Failed to read data caps of device states" , + vbasedev->name); +return -1 ; + } + vdev->migration->data_caps = caps; +return 0 ; +} + + +static int vfio_check_devstate_version (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; + +uint32_t version; +uint32_t size =sizeof (version); + +if (pread (vbasedev->fd, &version, size, + region->fd_offset + + offsetof(struct vfio_device_state_ctl , version)) + != size) { +error_report ("%s Failed to read version of device state interfaces" , + vbasedev->name); +return -1 ; + } + +if (version !=VFIO_DEVICE_STATE_INTERFACE_VERSION ) { +error_report ("%s migration version mismatch, right version is %d" , + vbasedev->name,VFIO_DEVICE_STATE_INTERFACE_VERSION ); +return -1 ; + } + +return 0 ; +} + +static void vfio_vm_change_state_handler (void *pv,int running,RunState state) +{ +VFIOPCIDevice *vdev = pv; +uint32_t dev_state = vdev->migration->device_state; + +if (!running) { + dev_state |=VFIO_DEVICE_STATE_STOP ; + }else { + dev_state &= ~VFIO_DEVICE_STATE_STOP ; + } + +vfio_set_device_state (vdev, dev_state); +} + +static void vfio_save_live_pending (QEMUFile *f,void *opaque, +uint64_t max_size, +uint64_t *res_precopy_only, +uint64_t *res_compatible, +uint64_t *res_post_copy_only) +{ +VFIOPCIDevice *vdev = opaque; + +if (!vfio_device_data_cap_device_memory (vdev)) { +return ; + } + +return ; +} + +static int vfio_save_iterate (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; + +if (!vfio_device_data_cap_device_memory (vdev)) { +return 0 ; + } + +return 0 ; +} + +static void vfio_pci_load_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +PCIDevice *pdev = &vdev->pdev; +uint32_t ctl, msi_lo, msi_hi, msi_data, bar_cfg, i; +bool msi_64bit; + +/* retore pci bar configuration */ + ctl =pci_default_read_config (pdev,PCI_COMMAND ,2 ); +vfio_pci_write_config (pdev,PCI_COMMAND , + ctl & (!(PCI_COMMAND_IO |PCI_COMMAND_MEMORY )),2 ); +for (i =0 ; i <PCI_ROM_SLOT ; i++) { + bar_cfg =qemu_get_be32 (f); +vfio_pci_write_config (pdev,PCI_BASE_ADDRESS_0 + i *4 , bar_cfg,4 ); + } +vfio_pci_write_config (pdev,PCI_COMMAND , + ctl |PCI_COMMAND_IO |PCI_COMMAND_MEMORY ,2 ); + +/* restore msi configuration */ + ctl =pci_default_read_config (pdev, pdev->msi_cap +PCI_MSI_FLAGS ,2 ); + msi_64bit = !!(ctl &PCI_MSI_FLAGS_64BIT ); + +vfio_pci_write_config (&vdev->pdev, + pdev->msi_cap +PCI_MSI_FLAGS , + ctl & (!PCI_MSI_FLAGS_ENABLE ),2 ); + + msi_lo =qemu_get_be32 (f); +vfio_pci_write_config (pdev, pdev->msi_cap +PCI_MSI_ADDRESS_LO , msi_lo,4 ); + +if (msi_64bit) { + msi_hi =qemu_get_be32 (f); +vfio_pci_write_config (pdev, pdev->msi_cap +PCI_MSI_ADDRESS_HI , + msi_hi,4 ); + } + msi_data =qemu_get_be32 (f); + vfio_pci_write_config(pdev, + pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
Quoted 8 lines
hw/vfio/migration.c line 433
Quoted 4 lines
hw/vfio/migration.c new file +4
-0+ msi_data, 2); + +vfio_pci_write_config (&vdev->pdev, pdev->msi_cap +PCI_MSI_FLAGS , + ctl |PCI_MSI_FLAGS_ENABLE ,2 );It would probably be best to use a VMStateDescription and the macros for this if possible; I bet you'll want to add more fields in the future for example.
yes, it is also a good idea to use VMStateDescription for pci general config data, which are saved one time in stop-and-copy phase. But it's a little hard to maintain two type of interfaces. Maybe use the way as what VMStateDescription did, i.e. using field list and macros?
Quoted 3 lines
Also what happens if the data read from the migration stream is bad or doesn't agree with this devices hardware? How does this fail?
right, errors in migration stream needs to be checked. I'll add the error check.
For the hardware incompatiable issue, seems vfio_pci_write_config does not return error if device driver returns error for this write. Seems it needs to be addressed in somewhere else like checking compitibility before lauching migration.
Quoted 287 lines
Quoted 281 lines
hw/vfio/migration.c new file +199
-0+} + +static int vfio_load_state (QEMUFile *f,void *opaque,int version_id) +{ +VFIOPCIDevice *vdev = opaque; +int flag; +uint64_t len; +int ret =0 ; + +if (version_id !=VFIO_DEVICE_STATE_INTERFACE_VERSION ) { +return -EINVAL ; + } + +do { + flag =qemu_get_byte (f); + +switch (flag & ~VFIO_SAVE_FLAG_CONTINUE ) { +case VFIO_SAVE_FLAG_SETUP : +break ; +case VFIO_SAVE_FLAG_PCI : +vfio_pci_load_config (vdev, f); +break ; +case VFIO_SAVE_FLAG_DEVCONFIG : + len =qemu_get_be64 (f); +vfio_load_data_device_config (vdev, f, len); +break ; +default : + ret = -EINVAL ; + } + }while (flag &VFIO_SAVE_FLAG_CONTINUE ); + +return ret; +} + +static void vfio_pci_save_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +PCIDevice *pdev = &vdev->pdev; +uint32_t msi_cfg, msi_lo, msi_hi, msi_data, bar_cfg, i; +bool msi_64bit; + +for (i =0 ; i <PCI_ROM_SLOT ; i++) { + bar_cfg =pci_default_read_config (pdev,PCI_BASE_ADDRESS_0 + i *4 ,4 ); +qemu_put_be32 (f, bar_cfg); + } + + msi_cfg =pci_default_read_config (pdev, pdev->msi_cap +PCI_MSI_FLAGS ,2 ); + msi_64bit = !!(msi_cfg &PCI_MSI_FLAGS_64BIT ); + + msi_lo =pci_default_read_config (pdev, + pdev->msi_cap +PCI_MSI_ADDRESS_LO ,4 ); +qemu_put_be32 (f, msi_lo); + +if (msi_64bit) { + msi_hi =pci_default_read_config (pdev, + pdev->msi_cap +PCI_MSI_ADDRESS_HI , +4 ); +qemu_put_be32 (f, msi_hi); + } + + msi_data =pci_default_read_config (pdev, + pdev->msi_cap + (msi_64bit ?PCI_MSI_DATA_64 :PCI_MSI_DATA_32 ), +2 ); +qemu_put_be32 (f, msi_data); + +} + +static int vfio_save_complete_precopy (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +int rc =0 ; + +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE ); +vfio_pci_save_config (vdev, f); + +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVCONFIG ); + rc +=vfio_get_device_config_size (vdev); + rc +=vfio_save_data_device_config (vdev, f); + +return rc; +} + +static int vfio_save_setup (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING | +VFIO_DEVICE_STATE_LOGGING ); +return 0 ; +} + +static int vfio_load_setup (QEMUFile *f,void *opaque) +{ +return 0 ; +} + +static void vfio_save_cleanup (void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +uint32_t dev_state = vdev->migration->device_state; + + dev_state &= ~VFIO_DEVICE_STATE_LOGGING ; + +vfio_set_device_state (vdev, dev_state); +} + +static SaveVMHandlers savevm_vfio_handlers = { + .save_setup = vfio_save_setup, + .save_live_pending = vfio_save_live_pending, + .save_live_iterate = vfio_save_iterate, + .save_live_complete_precopy = vfio_save_complete_precopy, + .save_cleanup = vfio_save_cleanup, + .load_setup = vfio_load_setup, + .load_state = vfio_load_state, +}; + +int vfio_migration_init (VFIOPCIDevice *vdev,Error **errp) +{ +int ret; +Error *local_err =NULL ; + vdev->migration =g_new0 (VFIOMigration,1 ); + +if (vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL , +"device-state-ctl" )) { + goto error; + } + +if (vfio_check_devstate_version (vdev)) { + goto error; + } + +if (vfio_get_device_data_caps (vdev)) { + goto error; + } + +if (vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG , +"device-state-data-device-config" )) { + goto error; + } + +if (vfio_device_data_cap_device_memory (vdev)) { +error_report ("No suppport of data cap device memory Yet" ); + goto error; + } + +if (vfio_device_data_cap_system_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP , +"device-state-data-dirtybitmap" )) { + goto error; + } + + vdev->migration->device_state =VFIO_DEVICE_STATE_RUNNING ; + +register_savevm_live (NULL ,TYPE_VFIO_PCI ,-1 , +VFIO_DEVICE_STATE_INTERFACE_VERSION , + &savevm_vfio_handlers, + vdev); + + vdev->migration->vm_state = +qemu_add_vm_change_state_handler (vfio_vm_change_state_handler, vdev); + +return 0 ; +error: +error_setg (&vdev->migration_blocker, +"VFIO device doesn't support migration" ); + ret =migrate_add_blocker (vdev->migration_blocker, &local_err); +if (local_err) { +error_propagate (errp, local_err); +error_free (vdev->migration_blocker); + } + +g_free (vdev->migration); + vdev->migration =NULL ; + +return ret; +} + +void vfio_migration_finalize (VFIOPCIDevice *vdev) +{ +if (vdev->migration) { +int i; +qemu_del_vm_change_state_handler (vdev->migration->vm_state); +unregister_savevm (NULL ,TYPE_VFIO_PCI , vdev); +for (i =0 ; i <VFIO_DEVSTATE_REGION_NUM ; i++) { +vfio_region_finalize (&vdev->migration->region[i]); + } +g_free (vdev->migration); + vdev->migration =NULL ; + }else if (vdev->migration_blocker) { +migrate_del_blocker (vdev->migration_blocker); +error_free (vdev->migration_blocker); + } +}hw/vfio/pci.c +0
-1@@ -37,7 +37,6 @@#define MSIX_CAP_LENGTH 12 -#define TYPE_VFIO_PCI "vfio-pci"#define PCI_VFIO (obj) OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)static void vfio_disable_interrupts (VFIOPCIDevice *vdev);hw/vfio/pci.h +24
-1@@ -19,6 +19,7 @@#include "qemu/event_notifier.h" #include "qemu/queue.h" #include "qemu/timer.h" +#include "sysemu/sysemu.h" #define PCI_ANY_ID (~0)@@ -56,6 +57,21 @@ typedef struct VFIOBAR { QLIST_HEAD(, VFIOQuirk) quirks; }VFIOBAR ; +enum { +VFIO_DEVSTATE_REGION_CTL =0 , +VFIO_DEVSTATE_REGION_DATA_CONFIG , +VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY , +VFIO_DEVSTATE_REGION_DATA_BITMAP , +VFIO_DEVSTATE_REGION_NUM , +}; +typedef struct VFIOMigration { +VFIORegion region[VFIO_DEVSTATE_REGION_NUM ]; +uint32_t data_caps; +uint32_t device_state; +uint64_t devconfig_size; +VMChangeStateEntry *vm_state; +}VFIOMigration ; + typedef struct VFIOVGARegion { MemoryRegion mem; off_t offset;@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {VFIOBAR bars[PCI_NUM_REGIONS -1 ];/* No ROM */ VFIOVGA *vga;/* 0xa0000, 0x3b0, 0x3c0 */ void *igd_opregion; +VFIOMigration *migration; +Error *migration_blocker;PCIHostDeviceAddress host;EventNotifier err_notifier;EventNotifier req_notifier;@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,void vfio_display_reset (VFIOPCIDevice *vdev);int vfio_display_probe (VFIOPCIDevice *vdev,Error **errp);void vfio_display_finalize (VFIOPCIDevice *vdev); - +bool vfio_device_data_cap_system_memory (VFIOPCIDevice *vdev); +bool vfio_device_data_cap_device_memory (VFIOPCIDevice *vdev); +int vfio_set_dirty_page_bitmap (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr); +int vfio_migration_init (VFIOPCIDevice *vdev,Error **errp); +void vfio_migration_finalize (VFIOPCIDevice *vdev);#endif /* HW_VFIO_VFIO_PCI_H */ include/hw/vfio/vfio-common.h +1
-0@@ -32,6 +32,7 @@#endif #define VFIO_MSG_PREFIX "vfio %s: " +#define TYPE_VFIO_PCI "vfio-pci"enum {VFIO_DEVICE_TYPE_PCI =0 ,Signature · 1 line
2.7.4--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Re: [PATCH 2/5] vfio/migration: support device of device config capability
Quoted 2 lines · * Zhao Yan ([email protected]) wrote:
Quoted 1 line · Dr. David Alan Gilbert
* Yan Zhao ([email protected]) wrote:
<snip>
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
hw/vfio/migration.c new file +7
-0+ msi_data =qemu_get_be32 (f); +vfio_pci_write_config (pdev, + pdev->msi_cap + (msi_64bit ?PCI_MSI_DATA_64 :PCI_MSI_DATA_32 ), + msi_data,2 ); + +vfio_pci_write_config (&vdev->pdev, pdev->msi_cap +PCI_MSI_FLAGS , + ctl |PCI_MSI_FLAGS_ENABLE ,2 );
Quoted 8 lines
Quoted 3 lines
It would probably be best to use a VMStateDescription and the macros for this if possible; I bet you'll want to add more fields in the future for example.
yes, it is also a good idea to use VMStateDescription for pci general config data, which are saved one time in stop-and-copy phase. But it's a little hard to maintain two type of interfaces. Maybe use the way as what VMStateDescription did, i.e. using field list and macros?
Maybe, but if you can actually use the VMStateDescription it would give you the versioning and conditional fields and things like that.
Quoted 7 lines
Quoted 3 lines
Also what happens if the data read from the migration stream is bad or doesn't agree with this devices hardware? How does this fail?
right, errors in migration stream needs to be checked. I'll add the error check.
For the hardware incompatiable issue, seemsvfio_pci_write_configdoes not return error if device driver returns error for this write.
It would be great if we could make that fail properly.
Quoted 2 lines
Seems it needs to be addressed in somewhere else like checking compitibility before lauching migration.
That would be good; but we should still guard against it being wrong if possible - these type of checks for compatibility are probably quite difficult.
Dave
Quoted 276 lines
Quoted 276 lines
Quoted 276 lines
hw/vfio/migration.c new file +199
-0+} + +static int vfio_load_state (QEMUFile *f,void *opaque,int version_id) +{ +VFIOPCIDevice *vdev = opaque; +int flag; +uint64_t len; +int ret =0 ; + +if (version_id !=VFIO_DEVICE_STATE_INTERFACE_VERSION ) { +return -EINVAL ; + } + +do { + flag =qemu_get_byte (f); + +switch (flag & ~VFIO_SAVE_FLAG_CONTINUE ) { +case VFIO_SAVE_FLAG_SETUP : +break ; +case VFIO_SAVE_FLAG_PCI : +vfio_pci_load_config (vdev, f); +break ; +case VFIO_SAVE_FLAG_DEVCONFIG : + len =qemu_get_be64 (f); +vfio_load_data_device_config (vdev, f, len); +break ; +default : + ret = -EINVAL ; + } + }while (flag &VFIO_SAVE_FLAG_CONTINUE ); + +return ret; +} + +static void vfio_pci_save_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +PCIDevice *pdev = &vdev->pdev; +uint32_t msi_cfg, msi_lo, msi_hi, msi_data, bar_cfg, i; +bool msi_64bit; + +for (i =0 ; i <PCI_ROM_SLOT ; i++) { + bar_cfg =pci_default_read_config (pdev,PCI_BASE_ADDRESS_0 + i *4 ,4 ); +qemu_put_be32 (f, bar_cfg); + } + + msi_cfg =pci_default_read_config (pdev, pdev->msi_cap +PCI_MSI_FLAGS ,2 ); + msi_64bit = !!(msi_cfg &PCI_MSI_FLAGS_64BIT ); + + msi_lo =pci_default_read_config (pdev, + pdev->msi_cap +PCI_MSI_ADDRESS_LO ,4 ); +qemu_put_be32 (f, msi_lo); + +if (msi_64bit) { + msi_hi =pci_default_read_config (pdev, + pdev->msi_cap +PCI_MSI_ADDRESS_HI , +4 ); +qemu_put_be32 (f, msi_hi); + } + + msi_data =pci_default_read_config (pdev, + pdev->msi_cap + (msi_64bit ?PCI_MSI_DATA_64 :PCI_MSI_DATA_32 ), +2 ); +qemu_put_be32 (f, msi_data); + +} + +static int vfio_save_complete_precopy (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +int rc =0 ; + +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE ); +vfio_pci_save_config (vdev, f); + +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVCONFIG ); + rc +=vfio_get_device_config_size (vdev); + rc +=vfio_save_data_device_config (vdev, f); + +return rc; +} + +static int vfio_save_setup (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING | +VFIO_DEVICE_STATE_LOGGING ); +return 0 ; +} + +static int vfio_load_setup (QEMUFile *f,void *opaque) +{ +return 0 ; +} + +static void vfio_save_cleanup (void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +uint32_t dev_state = vdev->migration->device_state; + + dev_state &= ~VFIO_DEVICE_STATE_LOGGING ; + +vfio_set_device_state (vdev, dev_state); +} + +static SaveVMHandlers savevm_vfio_handlers = { + .save_setup = vfio_save_setup, + .save_live_pending = vfio_save_live_pending, + .save_live_iterate = vfio_save_iterate, + .save_live_complete_precopy = vfio_save_complete_precopy, + .save_cleanup = vfio_save_cleanup, + .load_setup = vfio_load_setup, + .load_state = vfio_load_state, +}; + +int vfio_migration_init (VFIOPCIDevice *vdev,Error **errp) +{ +int ret; +Error *local_err =NULL ; + vdev->migration =g_new0 (VFIOMigration,1 ); + +if (vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL , +"device-state-ctl" )) { + goto error; + } + +if (vfio_check_devstate_version (vdev)) { + goto error; + } + +if (vfio_get_device_data_caps (vdev)) { + goto error; + } + +if (vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG , +"device-state-data-device-config" )) { + goto error; + } + +if (vfio_device_data_cap_device_memory (vdev)) { +error_report ("No suppport of data cap device memory Yet" ); + goto error; + } + +if (vfio_device_data_cap_system_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP , +"device-state-data-dirtybitmap" )) { + goto error; + } + + vdev->migration->device_state =VFIO_DEVICE_STATE_RUNNING ; + +register_savevm_live (NULL ,TYPE_VFIO_PCI ,-1 , +VFIO_DEVICE_STATE_INTERFACE_VERSION , + &savevm_vfio_handlers, + vdev); + + vdev->migration->vm_state = +qemu_add_vm_change_state_handler (vfio_vm_change_state_handler, vdev); + +return 0 ; +error: +error_setg (&vdev->migration_blocker, +"VFIO device doesn't support migration" ); + ret =migrate_add_blocker (vdev->migration_blocker, &local_err); +if (local_err) { +error_propagate (errp, local_err); +error_free (vdev->migration_blocker); + } + +g_free (vdev->migration); + vdev->migration =NULL ; + +return ret; +} + +void vfio_migration_finalize (VFIOPCIDevice *vdev) +{ +if (vdev->migration) { +int i; +qemu_del_vm_change_state_handler (vdev->migration->vm_state); +unregister_savevm (NULL ,TYPE_VFIO_PCI , vdev); +for (i =0 ; i <VFIO_DEVSTATE_REGION_NUM ; i++) { +vfio_region_finalize (&vdev->migration->region[i]); + } +g_free (vdev->migration); + vdev->migration =NULL ; + }else if (vdev->migration_blocker) { +migrate_del_blocker (vdev->migration_blocker); +error_free (vdev->migration_blocker); + } +}hw/vfio/pci.c +0
-1@@ -37,7 +37,6 @@#define MSIX_CAP_LENGTH 12 -#define TYPE_VFIO_PCI "vfio-pci"#define PCI_VFIO (obj) OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)static void vfio_disable_interrupts (VFIOPCIDevice *vdev);hw/vfio/pci.h +24
-1@@ -19,6 +19,7 @@#include "qemu/event_notifier.h" #include "qemu/queue.h" #include "qemu/timer.h" +#include "sysemu/sysemu.h" #define PCI_ANY_ID (~0)@@ -56,6 +57,21 @@ typedef struct VFIOBAR { QLIST_HEAD(, VFIOQuirk) quirks; }VFIOBAR ; +enum { +VFIO_DEVSTATE_REGION_CTL =0 , +VFIO_DEVSTATE_REGION_DATA_CONFIG , +VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY , +VFIO_DEVSTATE_REGION_DATA_BITMAP , +VFIO_DEVSTATE_REGION_NUM , +}; +typedef struct VFIOMigration { +VFIORegion region[VFIO_DEVSTATE_REGION_NUM ]; +uint32_t data_caps; +uint32_t device_state; +uint64_t devconfig_size; +VMChangeStateEntry *vm_state; +}VFIOMigration ; + typedef struct VFIOVGARegion { MemoryRegion mem; off_t offset;@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {VFIOBAR bars[PCI_NUM_REGIONS -1 ];/* No ROM */ VFIOVGA *vga;/* 0xa0000, 0x3b0, 0x3c0 */ void *igd_opregion; +VFIOMigration *migration; +Error *migration_blocker;PCIHostDeviceAddress host;EventNotifier err_notifier;EventNotifier req_notifier;@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,void vfio_display_reset (VFIOPCIDevice *vdev);int vfio_display_probe (VFIOPCIDevice *vdev,Error **errp);void vfio_display_finalize (VFIOPCIDevice *vdev); - +bool vfio_device_data_cap_system_memory (VFIOPCIDevice *vdev); +bool vfio_device_data_cap_device_memory (VFIOPCIDevice *vdev); +int vfio_set_dirty_page_bitmap (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr); +int vfio_migration_init (VFIOPCIDevice *vdev,Error **errp); +void vfio_migration_finalize (VFIOPCIDevice *vdev);#endif /* HW_VFIO_VFIO_PCI_H */ include/hw/vfio/vfio-common.h +1
-0@@ -32,6 +32,7 @@#endif #define VFIO_MSG_PREFIX "vfio %s: " +#define TYPE_VFIO_PCI "vfio-pci"
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
include/hw/vfio/vfio-common.h +0
-0enum {VFIO_DEVICE_TYPE_PCI =0 ,Signature · 1 line
2.7.4--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/5] vfio/migration: support device of device config capability
In reply to Yan ZhaoQuoted 34 lines · On Tue, 19 Feb 2019 16:52:27 +0800 Yan Zhao <[email protected]> wrote:
Device config is the default data that every device should have. so device config capability is by default on, no need to set.
- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state toVFIO_DEVICE_STATE_LOGGINGSigned-off-by Yan Zhao <[email protected]>Signed-off-by Yulei Zhang <[email protected]>--- hw/vfio/Makefile.objs | 2 +- hw/vfio/migration.c | 633 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 1 - hw/vfio/pci.h | 25 +- include/hw/vfio/vfio-common.h | 1 + 5 files changed, 659 insertions(+), 3 deletions(-) create mode 100644 hw/vfio/migration.chw/vfio/Makefile.objs +0
-0
hw/vfio/Makefile.objs line 3
Quoted 4 lines
hw/vfio/Makefile.objs +1
-1@@ -1,6 +1,6 @@ ifeq ($(CONFIG_LINUX), y) obj-$(CONFIG_SOFTMMU) += common.o -obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o +obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.o
I think you want to split the migration code: The type-independent code, and the pci-specific code.
Quoted 17 lines
hw/vfio/Makefile.objs +0
-0obj-$(CONFIG_VFIO_CCW) += ccw.o obj-$(CONFIG_SOFTMMU) += platform.o obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.ohw/vfio/migration.c new file +8
-0@@ -0,0 +1,633 @@ +#include "qemu/osdep.h" + +#include "hw/vfio/vfio-common.h" +#include "migration/blocker.h" +#include "migration/register.h" +#include "qapi/error.h" +#include "pci.h" +#include "sysemu/kvm.h"
hw/vfio/migration.c line 18
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+#include "exec/ram_addr.h" + +#define VFIO_SAVE_FLAG_SETUP 0 +#define VFIO_SAVE_FLAG_PCI 1 +#define VFIO_SAVE_FLAG_DEVCONFIG 2 +#define VFIO_SAVE_FLAG_DEVMEMORY 4 +#define VFIO_SAVE_FLAG_CONTINUE 8 + +static int vfio_device_state_region_setup (VFIOPCIDevice *vdev, +VFIORegion *region,uint32_t subtype,const char *name)
This function looks like it should be more generic and e.g. take a VFIODevice instead of a VFIOPCIDevice as argument.
Quoted 24 lines
hw/vfio/migration.c new file +24
-0+{ +VFIODevice *vbasedev = &vdev->vbasedev; +struct vfio_region_info *info; +int ret; + + ret =vfio_get_dev_region_info (vbasedev,VFIO_REGION_TYPE_DEVICE_STATE , + subtype, &info); +if (ret) { +error_report ("Failed to get info of region %s" , name); +return ret; + } + +if (vfio_region_setup (OBJECT (vdev), vbasedev, + region, info->index, name)) { +error_report ("Failed to setup migrtion region %s" , name); +return ret; + } + +if (vfio_region_mmap (region)) { +error_report ("Failed to mmap migrtion region %s" , name); + } + +return 0 ; +}
hw/vfio/migration.c line 52
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+ +bool vfio_device_data_cap_system_memory (VFIOPCIDevice *vdev) +{ +return !!(vdev->migration->data_caps &VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY ); +} + +bool vfio_device_data_cap_device_memory (VFIOPCIDevice *vdev) +{ +return !!(vdev->migration->data_caps &VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY ); +}
These two as well. The migration structure should probably hang off the VFIODevice instead.
hw/vfio/migration.c line 64
Quoted 12 lines
hw/vfio/migration.c new file +12
-0+ +static bool vfio_device_state_region_mmaped (VFIORegion *region) +{ +bool mmaped = true; +if (region->nr_mmaps !=1 || region->mmaps[0 ].offset || + (region->size != region->mmaps[0 ].size) || + (region->mmaps[0 ].mmap ==NULL )) { + mmaped = false; + } + +return mmaped; +}
s/mmaped/mmapped/ ?
hw/vfio/migration.c line 72
Quoted 8 lines
hw/vfio/migration.c new file +8
-0+ +static int vfio_get_device_config_size (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ];
This looks like it should not depend on pci, either.
Quoted 17 lines
hw/vfio/migration.c new file +17
-0+uint64_t len; +int sz; + + sz =sizeof (len); +if (pread (vbasedev->fd, &len, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.size)) + != sz) { +error_report ("vfio: Failed to get length of device config" ); +return -1 ; + } +if (len > region_config->size) { +error_report ("vfio: Error device config length" ); +return -1 ; + } + vdev->migration->devconfig_size = len; +
hw/vfio/migration.c line 99
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+return 0 ; +} + +static int vfio_set_device_config_size (VFIOPCIDevice *vdev,uint64_t size) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ];
Ditto. Also for the functions below.
Quoted 22 lines
hw/vfio/migration.c new file +22
-0+int sz; + +if (size > region_config->size) { +return -1 ; + } + + sz =sizeof (size); +if (pwrite (vbasedev->fd, &size, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.size)) + != sz) { +error_report ("vfio: Failed to set length of device config" ); +return -1 ; + } + vdev->migration->devconfig_size = size; +return 0 ; +} + +static int vfio_save_data_device_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +VFIODevice *vbasedev = &vdev->vbasedev; + VFIORegion *region_ctl =
hw/vfio/migration.c line 131
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+ &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_GET_BUFFER ; +uint64_t len = vdev->migration->devconfig_size; + +qemu_put_be64 (f, len);
Why big endian? (Generally, do we need any endianness considerations?)
hw/vfio/migration.c line 140
Quoted 9 lines
hw/vfio/migration.c new file +9
-0+ + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.action)) + != sz) { +error_report ("vfio: action failure for device config get buffer" ); +return -1 ; + }
Might make sense to wrap this into a set_action() helper that takes a SET_BUFFER/GET_BUFFER argument.
Quoted 148 lines
hw/vfio/migration.c new file +148
-0+ +if (!vfio_device_state_region_mmaped (region_config)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +if (pread (vbasedev->fd, buf, len, region_config->fd_offset) != len) { +error_report ("vfio: Failed read device config buffer" ); +return -1 ; + } +qemu_put_buffer (f, buf, len); +g_free (buf); + }else { + dest = region_config->mmaps[0 ].mmap; +qemu_put_buffer (f, dest, len); + } +return 0 ; +} + +static int vfio_load_data_device_config (VFIOPCIDevice *vdev, +QEMUFile *f,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_SET_BUFFER ; + +vfio_set_device_config_size (vdev, len); + +if (!vfio_device_state_region_mmaped (region_config)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +qemu_get_buffer (f, buf, len); +if (pwrite (vbasedev->fd, buf, len, + region_config->fd_offset) != len) { +error_report ("vfio: Failed to write devie config buffer" ); +return -1 ; + } +g_free (buf); + }else { + dest = region_config->mmaps[0 ].mmap; +qemu_get_buffer (f, dest, len); + } + + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.action)) + != sz) { +error_report ("vfio: action failure for device config set buffer" ); +return -1 ; + } + +return 0 ; +} + +static int vfio_set_dirty_page_bitmap_chunk (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr) +{ + +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_bitmap = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ]; +unsigned long bitmap_size = +BITS_TO_LONGS (page_nr) *sizeof (unsigned long ); +uint32_t sz; + +struct { +__u64 start_addr; +__u64 page_nr; + } system_memory; + system_memory.start_addr = start_addr; + system_memory.page_nr = page_nr; + sz =sizeof (system_memory); +if (pwrite (vbasedev->fd, &system_memory, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl , system_memory)) + != sz) { +error_report ("vfio: Failed to set system memory range for dirty pages" ); +return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_bitmap)) { +void *bitmap =g_malloc0 (bitmap_size); + +if (pread (vbasedev->fd, bitmap, bitmap_size, + region_bitmap->fd_offset) != bitmap_size) { +error_report ("vfio: Failed to read dirty bitmap data" ); +return -1 ; + } + +cpu_physical_memory_set_dirty_lebitmap (bitmap, start_addr, page_nr); + +g_free (bitmap); + }else { +cpu_physical_memory_set_dirty_lebitmap ( + region_bitmap->mmaps[0 ].mmap, + start_addr, page_nr); + } +return 0 ; +} + +int vfio_set_dirty_page_bitmap (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr) +{ +VFIORegion *region_bitmap = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ]; +unsigned long chunk_size = region_bitmap->size; +uint64_t chunk_pg_nr = (chunk_size /sizeof (unsigned long )) * +BITS_PER_LONG ; + +uint64_t cnt_left; +int rc =0 ; + + cnt_left = page_nr; + +while (cnt_left >= chunk_pg_nr) { + rc =vfio_set_dirty_page_bitmap_chunk (vdev, start_addr, chunk_pg_nr); +if (rc) { + goto exit; + } + cnt_left -= chunk_pg_nr; + start_addr += start_addr; + } + rc =vfio_set_dirty_page_bitmap_chunk (vdev, start_addr, cnt_left); + +exit: +return rc; +} + +static int vfio_set_device_state (VFIOPCIDevice *vdev, +uint32_t dev_state) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +uint32_t sz =sizeof (dev_state);
hw/vfio/migration.c line 298
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+ +if (!vdev->migration) { +return -1 ; + } + +if (pwrite (vbasedev->fd, &dev_state, sz, + region->fd_offset + + offsetof(struct vfio_device_state_ctl , device_state)) + != sz) { +error_report ("vfio: Failed to set device state %d" , dev_state);
Can the kernel reject this if a state transition is not allowed (or are all transitions allowed?)
Quoted 39 lines
hw/vfio/migration.c new file +39
-0+return -1 ; + } + vdev->migration->device_state = dev_state; +return 0 ; +} + +static int vfio_get_device_data_caps (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; + +uint32_t caps; +uint32_t size =sizeof (caps); + +if (pread (vbasedev->fd, &caps, size, + region->fd_offset + + offsetof(struct vfio_device_state_ctl , caps)) + != size) { +error_report ("%s Failed to read data caps of device states" , + vbasedev->name); +return -1 ; + } + vdev->migration->data_caps = caps; +return 0 ; +} + + +static int vfio_check_devstate_version (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; + +uint32_t version; +uint32_t size =sizeof (version); + + if (pread(vbasedev->fd, &version, size, + region->fd_offset +
hw/vfio/migration.c line 347
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+ offsetof(struct vfio_device_state_ctl, version)) + != size) { +error_report ("%s Failed to read version of device state interfaces" , + vbasedev->name); +return -1 ; + } + +if (version !=VFIO_DEVICE_STATE_INTERFACE_VERSION ) { +error_report ("%s migration version mismatch, right version is %d" , + vbasedev->name,VFIO_DEVICE_STATE_INTERFACE_VERSION );
So, we require an exact match... or should we allow to extend the interface in an backwards-compatible way, in which case we'd require (QEMU interface version) <= (kernel interface version)?
Quoted 35 lines
hw/vfio/migration.c new file +35
-0+return -1 ; + } + +return 0 ; +} + +static void vfio_vm_change_state_handler (void *pv,int running,RunState state) +{ +VFIOPCIDevice *vdev = pv; +uint32_t dev_state = vdev->migration->device_state; + +if (!running) { + dev_state |=VFIO_DEVICE_STATE_STOP ; + }else { + dev_state &= ~VFIO_DEVICE_STATE_STOP ; + } + +vfio_set_device_state (vdev, dev_state); +} + +static void vfio_save_live_pending (QEMUFile *f,void *opaque, +uint64_t max_size, +uint64_t *res_precopy_only, +uint64_t *res_compatible, +uint64_t *res_post_copy_only) +{ +VFIOPCIDevice *vdev = opaque; + +if (!vfio_device_data_cap_device_memory (vdev)) { +return ; + } + +return ; +} +
hw/vfio/migration.c line 392
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+static int vfio_save_iterate (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; + +if (!vfio_device_data_cap_device_memory (vdev)) { +return 0 ; + } + +return 0 ; +}
These look a bit weird...
Quoted 32 lines
hw/vfio/migration.c new file +32
-0+ +static void vfio_pci_load_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +PCIDevice *pdev = &vdev->pdev; +uint32_t ctl, msi_lo, msi_hi, msi_data, bar_cfg, i; +bool msi_64bit; + +/* retore pci bar configuration */ + ctl =pci_default_read_config (pdev,PCI_COMMAND ,2 ); +vfio_pci_write_config (pdev,PCI_COMMAND , + ctl & (!(PCI_COMMAND_IO |PCI_COMMAND_MEMORY )),2 ); +for (i =0 ; i <PCI_ROM_SLOT ; i++) { + bar_cfg =qemu_get_be32 (f); +vfio_pci_write_config (pdev,PCI_BASE_ADDRESS_0 + i *4 , bar_cfg,4 ); + } +vfio_pci_write_config (pdev,PCI_COMMAND , + ctl |PCI_COMMAND_IO |PCI_COMMAND_MEMORY ,2 ); + +/* restore msi configuration */ + ctl =pci_default_read_config (pdev, pdev->msi_cap +PCI_MSI_FLAGS ,2 ); + msi_64bit = !!(ctl &PCI_MSI_FLAGS_64BIT ); + +vfio_pci_write_config (&vdev->pdev, + pdev->msi_cap +PCI_MSI_FLAGS , + ctl & (!PCI_MSI_FLAGS_ENABLE ),2 ); + + msi_lo =qemu_get_be32 (f); +vfio_pci_write_config (pdev, pdev->msi_cap +PCI_MSI_ADDRESS_LO , msi_lo,4 ); + +if (msi_64bit) { + msi_hi =qemu_get_be32 (f); + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_HI,
hw/vfio/migration.c line 434
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+ msi_hi, 4); + } + msi_data =qemu_get_be32 (f); +vfio_pci_write_config (pdev, + pdev->msi_cap + (msi_64bit ?PCI_MSI_DATA_64 :PCI_MSI_DATA_32 ), + msi_data,2 ); + +vfio_pci_write_config (&vdev->pdev, pdev->msi_cap +PCI_MSI_FLAGS , + ctl |PCI_MSI_FLAGS_ENABLE ,2 ); +
Ok, this function is indeed pci-specific and probably should be moved to the vfio-pci code (other types could hook themselves up in the same place, then).
Quoted 95 lines
hw/vfio/migration.c new file +95
-0+} + +static int vfio_load_state (QEMUFile *f,void *opaque,int version_id) +{ +VFIOPCIDevice *vdev = opaque; +int flag; +uint64_t len; +int ret =0 ; + +if (version_id !=VFIO_DEVICE_STATE_INTERFACE_VERSION ) { +return -EINVAL ; + } + +do { + flag =qemu_get_byte (f); + +switch (flag & ~VFIO_SAVE_FLAG_CONTINUE ) { +case VFIO_SAVE_FLAG_SETUP : +break ; +case VFIO_SAVE_FLAG_PCI : +vfio_pci_load_config (vdev, f); +break ; +case VFIO_SAVE_FLAG_DEVCONFIG : + len =qemu_get_be64 (f); +vfio_load_data_device_config (vdev, f, len); +break ; +default : + ret = -EINVAL ; + } + }while (flag &VFIO_SAVE_FLAG_CONTINUE ); + +return ret; +} + +static void vfio_pci_save_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +PCIDevice *pdev = &vdev->pdev; +uint32_t msi_cfg, msi_lo, msi_hi, msi_data, bar_cfg, i; +bool msi_64bit; + +for (i =0 ; i <PCI_ROM_SLOT ; i++) { + bar_cfg =pci_default_read_config (pdev,PCI_BASE_ADDRESS_0 + i *4 ,4 ); +qemu_put_be32 (f, bar_cfg); + } + + msi_cfg =pci_default_read_config (pdev, pdev->msi_cap +PCI_MSI_FLAGS ,2 ); + msi_64bit = !!(msi_cfg &PCI_MSI_FLAGS_64BIT ); + + msi_lo =pci_default_read_config (pdev, + pdev->msi_cap +PCI_MSI_ADDRESS_LO ,4 ); +qemu_put_be32 (f, msi_lo); + +if (msi_64bit) { + msi_hi =pci_default_read_config (pdev, + pdev->msi_cap +PCI_MSI_ADDRESS_HI , +4 ); +qemu_put_be32 (f, msi_hi); + } + + msi_data =pci_default_read_config (pdev, + pdev->msi_cap + (msi_64bit ?PCI_MSI_DATA_64 :PCI_MSI_DATA_32 ), +2 ); +qemu_put_be32 (f, msi_data); + +} + +static int vfio_save_complete_precopy (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +int rc =0 ; + +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE ); +vfio_pci_save_config (vdev, f); + +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVCONFIG ); + rc +=vfio_get_device_config_size (vdev); + rc +=vfio_save_data_device_config (vdev, f); + +return rc; +} + +static int vfio_save_setup (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING | +VFIO_DEVICE_STATE_LOGGING ); +return 0 ; +} + +static int vfio_load_setup (QEMUFile *f,void *opaque) +{ +return 0 ; +}
hw/vfio/migration.c line 539
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+ +static void vfio_save_cleanup (void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +uint32_t dev_state = vdev->migration->device_state; + + dev_state &= ~VFIO_DEVICE_STATE_LOGGING ; + +vfio_set_device_state (vdev, dev_state); +}
These look like they should be type-independent, again.
Quoted 99 lines
hw/vfio/migration.c new file +94
-0+ +static SaveVMHandlers savevm_vfio_handlers = { + .save_setup = vfio_save_setup, + .save_live_pending = vfio_save_live_pending, + .save_live_iterate = vfio_save_iterate, + .save_live_complete_precopy = vfio_save_complete_precopy, + .save_cleanup = vfio_save_cleanup, + .load_setup = vfio_load_setup, + .load_state = vfio_load_state, +}; + +int vfio_migration_init (VFIOPCIDevice *vdev,Error **errp) +{ +int ret; +Error *local_err =NULL ; + vdev->migration =g_new0 (VFIOMigration,1 ); + +if (vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL , +"device-state-ctl" )) { + goto error; + } + +if (vfio_check_devstate_version (vdev)) { + goto error; + } + +if (vfio_get_device_data_caps (vdev)) { + goto error; + } + +if (vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG , +"device-state-data-device-config" )) { + goto error; + } + +if (vfio_device_data_cap_device_memory (vdev)) { +error_report ("No suppport of data cap device memory Yet" ); + goto error; + } + +if (vfio_device_data_cap_system_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP , +"device-state-data-dirtybitmap" )) { + goto error; + } + + vdev->migration->device_state =VFIO_DEVICE_STATE_RUNNING ; + +register_savevm_live (NULL ,TYPE_VFIO_PCI ,-1 , +VFIO_DEVICE_STATE_INTERFACE_VERSION , + &savevm_vfio_handlers, + vdev); + + vdev->migration->vm_state = +qemu_add_vm_change_state_handler (vfio_vm_change_state_handler, vdev); + +return 0 ; +error: +error_setg (&vdev->migration_blocker, +"VFIO device doesn't support migration" ); + ret =migrate_add_blocker (vdev->migration_blocker, &local_err); +if (local_err) { +error_propagate (errp, local_err); +error_free (vdev->migration_blocker); + } + +g_free (vdev->migration); + vdev->migration =NULL ; + +return ret; +} + +void vfio_migration_finalize (VFIOPCIDevice *vdev) +{ +if (vdev->migration) { +int i; +qemu_del_vm_change_state_handler (vdev->migration->vm_state); +unregister_savevm (NULL ,TYPE_VFIO_PCI , vdev); +for (i =0 ; i <VFIO_DEVSTATE_REGION_NUM ; i++) { +vfio_region_finalize (&vdev->migration->region[i]); + } +g_free (vdev->migration); + vdev->migration =NULL ; + }else if (vdev->migration_blocker) { +migrate_del_blocker (vdev->migration_blocker); +error_free (vdev->migration_blocker); + } +}hw/vfio/pci.c +0
-0
hw/vfio/pci.c line 40
Quoted 4 lines
hw/vfio/pci.c +0
-1@@ -37,7 +37,6 @@#define MSIX_CAP_LENGTH 12 -#define TYPE_VFIO_PCI "vfio-pci"
Why do you need to move this? That looks like a sign that the layering needs work.
Quoted 38 lines
hw/vfio/pci.c +0
-0#define PCI_VFIO (obj) OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)static void vfio_disable_interrupts (VFIOPCIDevice *vdev);hw/vfio/pci.h +16
-0@@ -19,6 +19,7 @@#include "qemu/event_notifier.h" #include "qemu/queue.h" #include "qemu/timer.h" +#include "sysemu/sysemu.h" #define PCI_ANY_ID (~0)@@ -56,6 +57,21 @@ typedef struct VFIOBAR { QLIST_HEAD(, VFIOQuirk) quirks; }VFIOBAR ; +enum { +VFIO_DEVSTATE_REGION_CTL =0 , +VFIO_DEVSTATE_REGION_DATA_CONFIG , +VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY , +VFIO_DEVSTATE_REGION_DATA_BITMAP , +VFIO_DEVSTATE_REGION_NUM , +}; +typedef struct VFIOMigration { +VFIORegion region[VFIO_DEVSTATE_REGION_NUM ]; +uint32_t data_caps; +uint32_t device_state; +uint64_t devconfig_size; +VMChangeStateEntry *vm_state; +}VFIOMigration ; + typedef struct VFIOVGARegion { MemoryRegion mem; off_t offset;
hw/vfio/pci.h typedef struct VFIOPCIDevice { · line 151
Quoted 4 lines
hw/vfio/pci.h +1
-0@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {VFIOBAR bars[PCI_NUM_REGIONS -1 ];/* No ROM */ VFIOVGA *vga;/* 0xa0000, 0x3b0, 0x3c0 */ void *igd_opregion; +VFIOMigration *migration;
As said, it would probably be better to hang this off VFIODevice.
Quoted 5 lines
hw/vfio/pci.h +1
-0+Error *migration_blocker;PCIHostDeviceAddress host;EventNotifier err_notifier;EventNotifier req_notifier;
hw/vfio/pci.h int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev, · line 224
Quoted 10 lines
hw/vfio/pci.h +6
-1@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,void vfio_display_reset (VFIOPCIDevice *vdev);int vfio_display_probe (VFIOPCIDevice *vdev,Error **errp);void vfio_display_finalize (VFIOPCIDevice *vdev); - +bool vfio_device_data_cap_system_memory (VFIOPCIDevice *vdev); +bool vfio_device_data_cap_device_memory (VFIOPCIDevice *vdev); +int vfio_set_dirty_page_bitmap (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr); +int vfio_migration_init (VFIOPCIDevice *vdev,Error **errp); +void vfio_migration_finalize (VFIOPCIDevice *vdev);
And the interfaces should be in vfio-common.
Quoted 13 lines
hw/vfio/pci.h +0
-0#endif /* HW_VFIO_VFIO_PCI_H */ include/hw/vfio/vfio-common.h +1
-0@@ -32,6 +32,7 @@#endif #define VFIO_MSG_PREFIX "vfio %s: " +#define TYPE_VFIO_PCI "vfio-pci"enum {VFIO_DEVICE_TYPE_PCI =0 ,
Re: [PATCH 2/5] vfio/migration: support device of device config capability
Quoted 32 lines · Cornelia Huck
Quoted 29 lines · On Tue, 19 Feb 2019 16:52:27 +0800 Yan Zhao <[email protected]> wrote:
Device config is the default data that every device should have. so device config capability is by default on, no need to set.
- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state toVFIO_DEVICE_STATE_LOGGINGSigned-off-by Yan Zhao <[email protected]>Signed-off-by Yulei Zhang <[email protected]>--- hw/vfio/Makefile.objs | 2 +- hw/vfio/migration.c | 633 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 1 - hw/vfio/pci.h | 25 +- include/hw/vfio/vfio-common.h | 1 + 5 files changed, 659 insertions(+), 3 deletions(-) create mode 100644 hw/vfio/migration.c
Quoted 13 lines
hw/vfio/Makefile.objs line 3
Quoted 9 lines
hw/vfio/Makefile.objs +1
-1@@ -1,6 +1,6 @@ ifeq ($(CONFIG_LINUX), y) obj-$(CONFIG_SOFTMMU) += common.o -obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o +obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.oI think you want to split the migration code: The type-independent code, and the pci-specific code.
ok. actually, now only saving/loading of pci generic config data is pci-specific. the data getting/setting through device state interfaces are type-independent.
Quoted 60 lines
Quoted 17 lines
hw/vfio/Makefile.objs +0
-0obj-$(CONFIG_VFIO_CCW) += ccw.o obj-$(CONFIG_SOFTMMU) += platform.o obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.ohw/vfio/migration.c new file +8
-0@@ -0,0 +1,633 @@ +#include "qemu/osdep.h" + +#include "hw/vfio/vfio-common.h" +#include "migration/blocker.h" +#include "migration/register.h" +#include "qapi/error.h" +#include "pci.h" +#include "sysemu/kvm.h" hw/vfio/migration.c line 18
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+#include "exec/ram_addr.h" + +#define VFIO_SAVE_FLAG_SETUP 0 +#define VFIO_SAVE_FLAG_PCI 1 +#define VFIO_SAVE_FLAG_DEVCONFIG 2 +#define VFIO_SAVE_FLAG_DEVMEMORY 4 +#define VFIO_SAVE_FLAG_CONTINUE 8 + +static int vfio_device_state_region_setup (VFIOPCIDevice *vdev, +VFIORegion *region,uint32_t subtype,const char *name)This function looks like it should be more generic and e.g. take a VFIODevice instead of a VFIOPCIDevice as argument.
Quoted 29 lines
hw/vfio/migration.c new file +29
-0+{ +VFIODevice *vbasedev = &vdev->vbasedev; +struct vfio_region_info *info; +int ret; + + ret =vfio_get_dev_region_info (vbasedev,VFIO_REGION_TYPE_DEVICE_STATE , + subtype, &info); +if (ret) { +error_report ("Failed to get info of region %s" , name); +return ret; + } + +if (vfio_region_setup (OBJECT (vdev), vbasedev, + region, info->index, name)) { +error_report ("Failed to setup migrtion region %s" , name); +return ret; + } + +if (vfio_region_mmap (region)) { +error_report ("Failed to mmap migrtion region %s" , name); + } + +return 0 ; +} + +bool vfio_device_data_cap_system_memory (VFIOPCIDevice *vdev) +{ +return !!(vdev->migration->data_caps &VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY ); +}
Quoted 9 lines
hw/vfio/migration.c line 52
Quoted 5 lines
hw/vfio/migration.c new file +5
-0+ +bool vfio_device_data_cap_device_memory (VFIOPCIDevice *vdev) +{ +return !!(vdev->migration->data_caps &VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY ); +}These two as well. The migration structure should probably hang off the VFIODevice instead.
ok.
Quoted 6 lines
Quoted 6 lines
hw/vfio/migration.c new file +6
-0+ +static bool vfio_device_state_region_mmaped (VFIORegion *region) +{ +bool mmaped = true; + if (region->nr_mmaps != 1 || region->mmaps[0].offset || + (region->size != region->mmaps[0].size) ||
Quoted 8 lines
hw/vfio/migration.c line 64
Quoted 6 lines
hw/vfio/migration.c new file +6
-0+ (region->mmaps[0].mmap == NULL)) { + mmaped = false; + } + +return mmaped; +}s/mmaped/mmapped/ ?
yes :)
Quoted 12 lines
hw/vfio/migration.c line 72
Quoted 9 lines
hw/vfio/migration.c new file +8
-0+ +static int vfio_get_device_config_size (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ];This looks like it should not depend on pci, either.
right.
Quoted 56 lines
Quoted 17 lines
hw/vfio/migration.c new file +17
-0+uint64_t len; +int sz; + + sz =sizeof (len); +if (pread (vbasedev->fd, &len, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.size)) + != sz) { +error_report ("vfio: Failed to get length of device config" ); +return -1 ; + } +if (len > region_config->size) { +error_report ("vfio: Error device config length" ); +return -1 ; + } + vdev->migration->devconfig_size = len; +hw/vfio/migration.c line 99
Quoted 10 lines
hw/vfio/migration.c new file +10
-0+return 0 ; +} + +static int vfio_set_device_config_size (VFIOPCIDevice *vdev,uint64_t size) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ];Ditto. Also for the functions below.
Quoted 26 lines
hw/vfio/migration.c new file +26
-0+int sz; + +if (size > region_config->size) { +return -1 ; + } + + sz =sizeof (size); +if (pwrite (vbasedev->fd, &size, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.size)) + != sz) { +error_report ("vfio: Failed to set length of device config" ); +return -1 ; + } + vdev->migration->devconfig_size = size; +return 0 ; +} + +static int vfio_save_data_device_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +void *dest;
Quoted 9 lines
hw/vfio/migration.c line 131
Quoted 6 lines
hw/vfio/migration.c new file +6
-0+uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_GET_BUFFER ; +uint64_t len = vdev->migration->devconfig_size; + +qemu_put_be64 (f, len);Why big endian? (Generally, do we need any endianness considerations?)
I think big endian is the endian for qemu to save file. as long as qemu_put and qemu_get use the same endian, it will be no problem.
do you think so?
Quoted 10 lines
hw/vfio/migration.c line 140
Quoted 9 lines
hw/vfio/migration.c new file +9
-0+ + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.action)) + != sz) { +error_report ("vfio: action failure for device config get buffer" ); +return -1 ; + }
Quoted 3 lines
Might make sense to wrap this into a
set_action()helper that takes a SET_BUFFER/GET_BUFFER argument.
right. it makes sense :)
Quoted 153 lines
Quoted 153 lines
hw/vfio/migration.c new file +153
-0+ +if (!vfio_device_state_region_mmaped (region_config)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +if (pread (vbasedev->fd, buf, len, region_config->fd_offset) != len) { +error_report ("vfio: Failed read device config buffer" ); +return -1 ; + } +qemu_put_buffer (f, buf, len); +g_free (buf); + }else { + dest = region_config->mmaps[0 ].mmap; +qemu_put_buffer (f, dest, len); + } +return 0 ; +} + +static int vfio_load_data_device_config (VFIOPCIDevice *vdev, +QEMUFile *f,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ]; +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_SET_BUFFER ; + +vfio_set_device_config_size (vdev, len); + +if (!vfio_device_state_region_mmaped (region_config)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +qemu_get_buffer (f, buf, len); +if (pwrite (vbasedev->fd, buf, len, + region_config->fd_offset) != len) { +error_report ("vfio: Failed to write devie config buffer" ); +return -1 ; + } +g_free (buf); + }else { + dest = region_config->mmaps[0 ].mmap; +qemu_get_buffer (f, dest, len); + } + + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_config.action)) + != sz) { +error_report ("vfio: action failure for device config set buffer" ); +return -1 ; + } + +return 0 ; +} + +static int vfio_set_dirty_page_bitmap_chunk (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr) +{ + +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_bitmap = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ]; +unsigned long bitmap_size = +BITS_TO_LONGS (page_nr) *sizeof (unsigned long ); +uint32_t sz; + +struct { +__u64 start_addr; +__u64 page_nr; + } system_memory; + system_memory.start_addr = start_addr; + system_memory.page_nr = page_nr; + sz =sizeof (system_memory); +if (pwrite (vbasedev->fd, &system_memory, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl , system_memory)) + != sz) { +error_report ("vfio: Failed to set system memory range for dirty pages" ); +return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_bitmap)) { +void *bitmap =g_malloc0 (bitmap_size); + +if (pread (vbasedev->fd, bitmap, bitmap_size, + region_bitmap->fd_offset) != bitmap_size) { +error_report ("vfio: Failed to read dirty bitmap data" ); +return -1 ; + } + +cpu_physical_memory_set_dirty_lebitmap (bitmap, start_addr, page_nr); + +g_free (bitmap); + }else { +cpu_physical_memory_set_dirty_lebitmap ( + region_bitmap->mmaps[0 ].mmap, + start_addr, page_nr); + } +return 0 ; +} + +int vfio_set_dirty_page_bitmap (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr) +{ +VFIORegion *region_bitmap = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ]; +unsigned long chunk_size = region_bitmap->size; +uint64_t chunk_pg_nr = (chunk_size /sizeof (unsigned long )) * +BITS_PER_LONG ; + +uint64_t cnt_left; +int rc =0 ; + + cnt_left = page_nr; + +while (cnt_left >= chunk_pg_nr) { + rc =vfio_set_dirty_page_bitmap_chunk (vdev, start_addr, chunk_pg_nr); +if (rc) { + goto exit; + } + cnt_left -= chunk_pg_nr; + start_addr += start_addr; + } + rc =vfio_set_dirty_page_bitmap_chunk (vdev, start_addr, cnt_left); + +exit: +return rc; +} + +static int vfio_set_device_state (VFIOPCIDevice *vdev, +uint32_t dev_state) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +uint32_t sz =sizeof (dev_state); + +if (!vdev->migration) { +return -1 ; + } +
Quoted 9 lines
hw/vfio/migration.c line 298
Quoted 5 lines
hw/vfio/migration.c new file +5
-0+if (pwrite (vbasedev->fd, &dev_state, sz, + region->fd_offset + + offsetof(struct vfio_device_state_ctl , device_state)) + != sz) { +error_report ("vfio: Failed to set device state %d" , dev_state);Can the kernel reject this if a state transition is not allowed (or are all transitions allowed?)
yes, kernel can reject some state transition if it's not allowed. But currently all transitions are allowed.
Maybe a check of self-to-self transition is needed in kernel.
Quoted 45 lines
Quoted 45 lines
hw/vfio/migration.c new file +45
-0+return -1 ; + } + vdev->migration->device_state = dev_state; +return 0 ; +} + +static int vfio_get_device_data_caps (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; + +uint32_t caps; +uint32_t size =sizeof (caps); + +if (pread (vbasedev->fd, &caps, size, + region->fd_offset + + offsetof(struct vfio_device_state_ctl , caps)) + != size) { +error_report ("%s Failed to read data caps of device states" , + vbasedev->name); +return -1 ; + } + vdev->migration->data_caps = caps; +return 0 ; +} + + +static int vfio_check_devstate_version (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; + +uint32_t version; +uint32_t size =sizeof (version); + +if (pread (vbasedev->fd, &version, size, + region->fd_offset + + offsetof(struct vfio_device_state_ctl , version)) + != size) { +error_report ("%s Failed to read version of device state interfaces" , + vbasedev->name); +return -1 ; + }
Quoted 9 lines
hw/vfio/migration.c line 347
Quoted 4 lines
hw/vfio/migration.c new file +4
-0+ +if (version !=VFIO_DEVICE_STATE_INTERFACE_VERSION ) { +error_report ("%s migration version mismatch, right version is %d" , + vbasedev->name,VFIO_DEVICE_STATE_INTERFACE_VERSION );So, we require an exact match... or should we allow to extend the interface in an backwards-compatible way, in which case we'd require (QEMU interface version) <= (kernel interface version)?
currently yes. we can discuss on that.
Quoted 39 lines
Quoted 39 lines
hw/vfio/migration.c new file +39
-0+return -1 ; + } + +return 0 ; +} + +static void vfio_vm_change_state_handler (void *pv,int running,RunState state) +{ +VFIOPCIDevice *vdev = pv; +uint32_t dev_state = vdev->migration->device_state; + +if (!running) { + dev_state |=VFIO_DEVICE_STATE_STOP ; + }else { + dev_state &= ~VFIO_DEVICE_STATE_STOP ; + } + +vfio_set_device_state (vdev, dev_state); +} + +static void vfio_save_live_pending (QEMUFile *f,void *opaque, +uint64_t max_size, +uint64_t *res_precopy_only, +uint64_t *res_compatible, +uint64_t *res_post_copy_only) +{ +VFIOPCIDevice *vdev = opaque; + +if (!vfio_device_data_cap_device_memory (vdev)) { +return ; + } + +return ; +} + +static int vfio_save_iterate (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +
Quoted 9 lines
hw/vfio/migration.c line 392
Quoted 6 lines
hw/vfio/migration.c new file +6
-0+if (!vfio_device_data_cap_device_memory (vdev)) { +return 0 ; + } + +return 0 ; +}These look a bit weird...
patch 5 (adding device memory cap support ) will make it look better :)
Quoted 38 lines
Quoted 38 lines
hw/vfio/migration.c new file +38
-0+ +static void vfio_pci_load_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +PCIDevice *pdev = &vdev->pdev; +uint32_t ctl, msi_lo, msi_hi, msi_data, bar_cfg, i; +bool msi_64bit; + +/* retore pci bar configuration */ + ctl =pci_default_read_config (pdev,PCI_COMMAND ,2 ); +vfio_pci_write_config (pdev,PCI_COMMAND , + ctl & (!(PCI_COMMAND_IO |PCI_COMMAND_MEMORY )),2 ); +for (i =0 ; i <PCI_ROM_SLOT ; i++) { + bar_cfg =qemu_get_be32 (f); +vfio_pci_write_config (pdev,PCI_BASE_ADDRESS_0 + i *4 , bar_cfg,4 ); + } +vfio_pci_write_config (pdev,PCI_COMMAND , + ctl |PCI_COMMAND_IO |PCI_COMMAND_MEMORY ,2 ); + +/* restore msi configuration */ + ctl =pci_default_read_config (pdev, pdev->msi_cap +PCI_MSI_FLAGS ,2 ); + msi_64bit = !!(ctl &PCI_MSI_FLAGS_64BIT ); + +vfio_pci_write_config (&vdev->pdev, + pdev->msi_cap +PCI_MSI_FLAGS , + ctl & (!PCI_MSI_FLAGS_ENABLE ),2 ); + + msi_lo =qemu_get_be32 (f); +vfio_pci_write_config (pdev, pdev->msi_cap +PCI_MSI_ADDRESS_LO , msi_lo,4 ); + +if (msi_64bit) { + msi_hi =qemu_get_be32 (f); +vfio_pci_write_config (pdev, pdev->msi_cap +PCI_MSI_ADDRESS_HI , + msi_hi,4 ); + } + msi_data =qemu_get_be32 (f); +vfio_pci_write_config (pdev, + pdev->msi_cap + (msi_64bit ?PCI_MSI_DATA_64 :PCI_MSI_DATA_32 ), + msi_data,2 );
Quoted 9 lines
hw/vfio/migration.c line 434
Quoted 4 lines
hw/vfio/migration.c new file +4
-0+ +vfio_pci_write_config (&vdev->pdev, pdev->msi_cap +PCI_MSI_FLAGS , + ctl |PCI_MSI_FLAGS_ENABLE ,2 ); +Ok, this function is indeed pci-specific and probably should be moved to the vfio-pci code (other types could hook themselves up in the same place, then).
yes, this is the only pci-specific code.
maybe using VFIO_DEVICE_TYPE_PCI as a sign to decide whether to save/load pci config data?
or as Dave said, put saving/loading of pci config data into VMStateDescription interface?
Quoted 99 lines
Quoted 99 lines
hw/vfio/migration.c new file +99
-0+} + +static int vfio_load_state (QEMUFile *f,void *opaque,int version_id) +{ +VFIOPCIDevice *vdev = opaque; +int flag; +uint64_t len; +int ret =0 ; + +if (version_id !=VFIO_DEVICE_STATE_INTERFACE_VERSION ) { +return -EINVAL ; + } + +do { + flag =qemu_get_byte (f); + +switch (flag & ~VFIO_SAVE_FLAG_CONTINUE ) { +case VFIO_SAVE_FLAG_SETUP : +break ; +case VFIO_SAVE_FLAG_PCI : +vfio_pci_load_config (vdev, f); +break ; +case VFIO_SAVE_FLAG_DEVCONFIG : + len =qemu_get_be64 (f); +vfio_load_data_device_config (vdev, f, len); +break ; +default : + ret = -EINVAL ; + } + }while (flag &VFIO_SAVE_FLAG_CONTINUE ); + +return ret; +} + +static void vfio_pci_save_config (VFIOPCIDevice *vdev,QEMUFile *f) +{ +PCIDevice *pdev = &vdev->pdev; +uint32_t msi_cfg, msi_lo, msi_hi, msi_data, bar_cfg, i; +bool msi_64bit; + +for (i =0 ; i <PCI_ROM_SLOT ; i++) { + bar_cfg =pci_default_read_config (pdev,PCI_BASE_ADDRESS_0 + i *4 ,4 ); +qemu_put_be32 (f, bar_cfg); + } + + msi_cfg =pci_default_read_config (pdev, pdev->msi_cap +PCI_MSI_FLAGS ,2 ); + msi_64bit = !!(msi_cfg &PCI_MSI_FLAGS_64BIT ); + + msi_lo =pci_default_read_config (pdev, + pdev->msi_cap +PCI_MSI_ADDRESS_LO ,4 ); +qemu_put_be32 (f, msi_lo); + +if (msi_64bit) { + msi_hi =pci_default_read_config (pdev, + pdev->msi_cap +PCI_MSI_ADDRESS_HI , +4 ); +qemu_put_be32 (f, msi_hi); + } + + msi_data =pci_default_read_config (pdev, + pdev->msi_cap + (msi_64bit ?PCI_MSI_DATA_64 :PCI_MSI_DATA_32 ), +2 ); +qemu_put_be32 (f, msi_data); + +} + +static int vfio_save_complete_precopy (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +int rc =0 ; + +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE ); +vfio_pci_save_config (vdev, f); + +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVCONFIG ); + rc +=vfio_get_device_config_size (vdev); + rc +=vfio_save_data_device_config (vdev, f); + +return rc; +} + +static int vfio_save_setup (QEMUFile *f,void *opaque) +{ +VFIOPCIDevice *vdev = opaque; +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING | +VFIO_DEVICE_STATE_LOGGING ); +return 0 ; +} + +static int vfio_load_setup (QEMUFile *f,void *opaque) +{ +return 0 ; +} + +static void vfio_save_cleanup (void *opaque) +{ +VFIOPCIDevice *vdev = opaque;
Quoted 9 lines
hw/vfio/migration.c line 539
Quoted 6 lines
hw/vfio/migration.c new file +6
-0+uint32_t dev_state = vdev->migration->device_state; + + dev_state &= ~VFIO_DEVICE_STATE_LOGGING ; + +vfio_set_device_state (vdev, dev_state); +}These look like they should be type-independent, again.
right:)
Quoted 100 lines
Quoted 100 lines
hw/vfio/migration.c new file +94
-0+ +static SaveVMHandlers savevm_vfio_handlers = { + .save_setup = vfio_save_setup, + .save_live_pending = vfio_save_live_pending, + .save_live_iterate = vfio_save_iterate, + .save_live_complete_precopy = vfio_save_complete_precopy, + .save_cleanup = vfio_save_cleanup, + .load_setup = vfio_load_setup, + .load_state = vfio_load_state, +}; + +int vfio_migration_init (VFIOPCIDevice *vdev,Error **errp) +{ +int ret; +Error *local_err =NULL ; + vdev->migration =g_new0 (VFIOMigration,1 ); + +if (vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL , +"device-state-ctl" )) { + goto error; + } + +if (vfio_check_devstate_version (vdev)) { + goto error; + } + +if (vfio_get_device_data_caps (vdev)) { + goto error; + } + +if (vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG , +"device-state-data-device-config" )) { + goto error; + } + +if (vfio_device_data_cap_device_memory (vdev)) { +error_report ("No suppport of data cap device memory Yet" ); + goto error; + } + +if (vfio_device_data_cap_system_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP , +"device-state-data-dirtybitmap" )) { + goto error; + } + + vdev->migration->device_state =VFIO_DEVICE_STATE_RUNNING ; + +register_savevm_live (NULL ,TYPE_VFIO_PCI ,-1 , +VFIO_DEVICE_STATE_INTERFACE_VERSION , + &savevm_vfio_handlers, + vdev); + + vdev->migration->vm_state = +qemu_add_vm_change_state_handler (vfio_vm_change_state_handler, vdev); + +return 0 ; +error: +error_setg (&vdev->migration_blocker, +"VFIO device doesn't support migration" ); + ret =migrate_add_blocker (vdev->migration_blocker, &local_err); +if (local_err) { +error_propagate (errp, local_err); +error_free (vdev->migration_blocker); + } + +g_free (vdev->migration); + vdev->migration =NULL ; + +return ret; +} + +void vfio_migration_finalize (VFIOPCIDevice *vdev) +{ +if (vdev->migration) { +int i; +qemu_del_vm_change_state_handler (vdev->migration->vm_state); +unregister_savevm (NULL ,TYPE_VFIO_PCI , vdev); +for (i =0 ; i <VFIO_DEVSTATE_REGION_NUM ; i++) { +vfio_region_finalize (&vdev->migration->region[i]); + } +g_free (vdev->migration); + vdev->migration =NULL ; + }else if (vdev->migration_blocker) { +migrate_del_blocker (vdev->migration_blocker); +error_free (vdev->migration_blocker); + } +}hw/vfio/pci.c +0
-0@@ -37,7 +37,6 @@
Quoted 7 lines
hw/vfio/pci.c line 40
Quoted 3 lines
hw/vfio/pci.c +0
-1#define MSIX_CAP_LENGTH 12 -#define TYPE_VFIO_PCI "vfio-pci"Why do you need to move this? That looks like a sign that the layering needs work.
I moved it for passing it as an idstr for register_savevm_live. Seems it's not necessary if we make it pci independent. Thanks:)
Quoted 36 lines
Quoted 36 lines
hw/vfio/pci.c +0
-0#define PCI_VFIO (obj) OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)static void vfio_disable_interrupts (VFIOPCIDevice *vdev);hw/vfio/pci.h +16
-0@@ -19,6 +19,7 @@#include "qemu/event_notifier.h" #include "qemu/queue.h" #include "qemu/timer.h" +#include "sysemu/sysemu.h" #define PCI_ANY_ID (~0)@@ -56,6 +57,21 @@ typedef struct VFIOBAR { QLIST_HEAD(, VFIOQuirk) quirks; }VFIOBAR ; +enum { +VFIO_DEVSTATE_REGION_CTL =0 , +VFIO_DEVSTATE_REGION_DATA_CONFIG , +VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY , +VFIO_DEVSTATE_REGION_DATA_BITMAP , +VFIO_DEVSTATE_REGION_NUM , +}; +typedef struct VFIOMigration { +VFIORegion region[VFIO_DEVSTATE_REGION_NUM ]; +uint32_t data_caps; +uint32_t device_state; +uint64_t devconfig_size; +VMChangeStateEntry *vm_state; +}VFIOMigration ; + typedef struct VFIOVGARegion { MemoryRegion mem;
Quoted 9 lines
hw/vfio/pci.h typedef struct VFIOPCIDevice { · line 151
Quoted 6 lines
hw/vfio/pci.h +1
-0off_t offset;@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {VFIOBAR bars[PCI_NUM_REGIONS -1 ];/* No ROM */ VFIOVGA *vga;/* 0xa0000, 0x3b0, 0x3c0 */ void *igd_opregion; +VFIOMigration *migration;As said, it would probably be better to hang this off VFIODevice.
ok.
Quoted 9 lines
Quoted 9 lines
hw/vfio/pci.h +1
-1+Error *migration_blocker;PCIHostDeviceAddress host;EventNotifier err_notifier;EventNotifier req_notifier;@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,void vfio_display_reset (VFIOPCIDevice *vdev);int vfio_display_probe (VFIOPCIDevice *vdev,Error **errp);void vfio_display_finalize (VFIOPCIDevice *vdev); -
Quoted 9 lines
hw/vfio/pci.h int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev, · line 224
Quoted 6 lines
hw/vfio/pci.h +6
-0+bool vfio_device_data_cap_system_memory (VFIOPCIDevice *vdev); +bool vfio_device_data_cap_device_memory (VFIOPCIDevice *vdev); +int vfio_set_dirty_page_bitmap (VFIOPCIDevice *vdev, +uint64_t start_addr,uint64_t page_nr); +int vfio_migration_init (VFIOPCIDevice *vdev,Error **errp); +void vfio_migration_finalize (VFIOPCIDevice *vdev);And the interfaces should be in vfio-common.
ok.
Quoted 14 lines
Quoted 13 lines
hw/vfio/pci.h +0
-0#endif /* HW_VFIO_VFIO_PCI_H */ include/hw/vfio/vfio-common.h +1
-0@@ -32,6 +32,7 @@#endif #define VFIO_MSG_PREFIX "vfio %s: " +#define TYPE_VFIO_PCI "vfio-pci"enum {VFIO_DEVICE_TYPE_PCI =0 ,
Re: [PATCH 2/5] vfio/migration: support device of device config capability
Quoted 46 lines · On Wed, 20 Feb 2019 17:54:00 -0500 Zhao Yan <[email protected]> wrote:
Quoted 32 lines · Cornelia Huck
Quoted 29 lines · On Tue, 19 Feb 2019 16:52:27 +0800 Yan Zhao <[email protected]> wrote:
Device config is the default data that every device should have. so device config capability is by default on, no need to set.
- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state toVFIO_DEVICE_STATE_LOGGINGSigned-off-by Yan Zhao <[email protected]>Signed-off-by Yulei Zhang <[email protected]>--- hw/vfio/Makefile.objs | 2 +- hw/vfio/migration.c | 633 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 1 - hw/vfio/pci.h | 25 +- include/hw/vfio/vfio-common.h | 1 + 5 files changed, 659 insertions(+), 3 deletions(-) create mode 100644 hw/vfio/migration.cQuoted 13 lines
hw/vfio/Makefile.objs line 3
Quoted 9 lines
hw/vfio/Makefile.objs +1
-1@@ -1,6 +1,6 @@ ifeq ($(CONFIG_LINUX), y) obj-$(CONFIG_SOFTMMU) += common.o -obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o +obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.oI think you want to split the migration code: The type-independent code, and the pci-specific code.
Quoted 3 lines
ok. actually, now only saving/loading of pci generic config data is pci-specific. the data getting/setting through device state interfaces are type-independent.
Yes. If it has capability chains, it can probably be made to work.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
hw/vfio/Makefile.objs +0
-0obj-$(CONFIG_VFIO_CCW) += ccw.o obj-$(CONFIG_SOFTMMU) += platform.o obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o
(...)
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
hw/vfio/Makefile.objs +8
-0+static int vfio_save_data_device_config(VFIOPCIDevice *vdev, QEMUFile *f) +{ + VFIODevice *vbasedev = &vdev->vbasedev; + VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL]; + VFIORegion *region_config = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG]; + void *dest;Quoted 9 lines
hw/vfio/Makefile.objs
Quoted 6 lines
hw/vfio/Makefile.objs +6
-0+ uint32_t sz; + uint8_t *buf = NULL; + uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER; + uint64_t len = vdev->migration->devconfig_size; + + qemu_put_be64(f, len);Why big endian? (Generally, do we need any endianness considerations?)
Quoted 4 lines
I think big endian is the endian for qemu to save file. as long as
qemu_putandqemu_getuse the same endian, it will be no problem.
do you think so?
Yes, as long we are explicit about the endianness. I'm not sure whether e.g. power even has the ability to mix endianness.
(...)
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
hw/vfio/Makefile.objs +12
-0+static int vfio_set_device_state(VFIOPCIDevice *vdev, + uint32_t dev_state) +{ + VFIODevice *vbasedev = &vdev->vbasedev; + VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL]; + uint32_t sz = sizeof(dev_state); + + if (!vdev->migration) { + return -1; + } +Quoted 9 lines
hw/vfio/Makefile.objs
Quoted 5 lines
hw/vfio/Makefile.objs +5
-0+ if (pwrite(vbasedev->fd, &dev_state, sz, + region->fd_offset + + offsetof(struct vfio_device_state_ctl, device_state)) + != sz) { + error_report("vfio: Failed to set device state %d", dev_state);Can the kernel reject this if a state transition is not allowed (or are all transitions allowed?)
Quoted 3 lines
yes, kernel can reject some state transition if it's not allowed. But currently all transitions are allowed.
Maybe a check of self-to-self transition is needed in kernel.
Self-to-self looks benign enough to simply return success early.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
hw/vfio/Makefile.objs +5
-0+ return -1; + } + vdev->migration->device_state = dev_state; + return 0; +}
(...)
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
hw/vfio/Makefile.objs +17
-0+static int vfio_check_devstate_version(VFIOPCIDevice *vdev) +{ + VFIODevice *vbasedev = &vdev->vbasedev; + VFIORegion *region = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL]; + + uint32_t version; + uint32_t size = sizeof(version); + + if (pread(vbasedev->fd, &version, size, + region->fd_offset + + offsetof(struct vfio_device_state_ctl, version)) + != size) { + error_report("%s Failed to read version of device state interfaces", + vbasedev->name); + return -1; + }Quoted 9 lines
hw/vfio/Makefile.objs
Quoted 4 lines
hw/vfio/Makefile.objs +4
-0+ + if (version != VFIO_DEVICE_STATE_INTERFACE_VERSION) { + error_report("%s migration version mismatch, right version is %d", + vbasedev->name, VFIO_DEVICE_STATE_INTERFACE_VERSION);So, we require an exact match... or should we allow to extend the interface in an backwards-compatible way, in which case we'd require (QEMU interface version) <= (kernel interface version)?
Quoted 1 line
currently yes. we can discuss on that.
If we want to allow that, we need to have a strictly monotonous progression of versions here (which means dragging along old compatibility code for basically forever). Maintaining a table about which version is compatible with which other version would get insane pretty quickly.
Can we somehow accommodate more optional regions via capabilities? Maybe via optional vmstates?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
hw/vfio/Makefile.objs +5
-0+ return -1; + } + + return 0; +}
(...)
Quoted 46 lines
Quoted 37 lines
Quoted 37 lines
hw/vfio/Makefile.objs +37
-0+static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f) +{ + PCIDevice *pdev = &vdev->pdev; + uint32_t ctl, msi_lo, msi_hi, msi_data, bar_cfg, i; + bool msi_64bit; + + /* retore pci bar configuration */ + ctl = pci_default_read_config(pdev, PCI_COMMAND, 2); + vfio_pci_write_config(pdev, PCI_COMMAND, + ctl & (!(PCI_COMMAND_IO | PCI_COMMAND_MEMORY)), 2); + for (i = 0; i < PCI_ROM_SLOT; i++) { + bar_cfg = qemu_get_be32(f); + vfio_pci_write_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, bar_cfg, 4); + } + vfio_pci_write_config(pdev, PCI_COMMAND, + ctl | PCI_COMMAND_IO | PCI_COMMAND_MEMORY, 2); + + /* restore msi configuration */ + ctl = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, 2); + msi_64bit = !!(ctl & PCI_MSI_FLAGS_64BIT); + + vfio_pci_write_config(&vdev->pdev, + pdev->msi_cap + PCI_MSI_FLAGS, + ctl & (!PCI_MSI_FLAGS_ENABLE), 2); + + msi_lo = qemu_get_be32(f); + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_LO, msi_lo, 4); + + if (msi_64bit) { + msi_hi = qemu_get_be32(f); + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_HI, + msi_hi, 4); + } + msi_data = qemu_get_be32(f); + vfio_pci_write_config(pdev, + pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32), + msi_data, 2);Quoted 9 lines
hw/vfio/Makefile.objs
Quoted 4 lines
hw/vfio/Makefile.objs +4
-0+ + vfio_pci_write_config(&vdev->pdev, pdev->msi_cap + PCI_MSI_FLAGS, + ctl | PCI_MSI_FLAGS_ENABLE, 2); +Ok, this function is indeed pci-specific and probably should be moved to the vfio-pci code (other types could hook themselves up in the same place, then).
Quoted 5 lines
yes, this is the only pci-specific code.
maybe usingVFIO_DEVICE_TYPE_PCIas a sign to decide whether to save/load pci config data?
or as Dave said, put saving/loading of pci config data into VMStateDescription interface?
I like having an interface like vmstate where other types can register themselves better than introducing conditional handling based on hard-coded type values.
PATCH 3/5 vfio/migration: tracking of dirty page in system memory
In reply to Yan Zhaoregister the log_sync interface to hook into ram's live migration callbacks.
ram_save_pending
|->migration_bitmap_sync
|->memory_global_dirty_log_sync
|->memory_region_sync_dirty_bitmap
|->listener->log_sync(listener, &mrs);So, the dirty page produced by vfio device in system memory will be save/load by ram's live migration code iteratively.
Bitmap of device's dirty page in system memory is retrieved from Dirty Bitmap Region
--- hw/vfio/common.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
hw/vfio/common.c +26 -0
@@ -27,6 +27,7 @@#include "hw/vfio/vfio-common.h" #include "hw/vfio/vfio.h" +#include "hw/vfio/pci.h" #include "exec/address-spaces.h" #include "exec/memory.h" #include "hw/hw.h"
@@ -698,9 +699,34 @@ static void vfio_listener_region_del(MemoryListener *listener, } } +static void vfio_log_sync (MemoryListener *listener, +MemoryRegionSection *section) +{ +VFIOContainer *container =container_of (listener, VFIOContainer, listener); +VFIOGroup *group =QLIST_FIRST (&container->group_list); +VFIODevice *vbasedev; +VFIOPCIDevice *vdev; + +ram_addr_t size =int128_get64 (section->size); +uint64_t page_nr = size >>TARGET_PAGE_BITS ; +uint64_t start_addr = section->offset_within_address_space; + + QLIST_FOREACH(vbasedev, &group->device_list, next) { + vdev =container_of (vbasedev, VFIOPCIDevice, vbasedev); +if (!vdev->migration || + !vfio_device_data_cap_system_memory (vdev) || + !(vdev->migration->device_state &VFIO_DEVICE_STATE_LOGGING )) { +continue ; + } + +vfio_set_dirty_page_bitmap (vdev, start_addr, page_nr); + } +} +static const MemoryListener vfio_memory_listener = { .region_add = vfio_listener_region_add, .region_del = vfio_listener_region_del, + .log_sync = vfio_log_sync, };static void vfio_listener_release (VFIOContainer *container)
Signature · 1 line
2.7.4
PATCH 4/5 vfio/migration: turn on migration
In reply to Yan Zhaoinit vfio migration in vfio_realize() and register migraton blocker if failure met.
finalize all migration resources when vfio_instance_finalize().
--- hw/vfio/pci.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)
hw/vfio/pci.c +3 -6
@@ -3068,6 +3068,8 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) goto out_teardown; } +vfio_migration_init (vdev, errp); +vfio_register_err_notifier (vdev);vfio_register_req_notifier (vdev);vfio_setup_resetfn_quirk (vdev);
@@ -3089,6 +3091,7 @@ static void vfio_instance_finalize(Object *obj)vfio_display_finalize (vdev);vfio_bars_finalize (vdev); +vfio_migration_finalize (vdev);g_free (vdev->emulated_config_bits);g_free (vdev->rom);/*
@@ -3221,11 +3224,6 @@ static Property vfio_pci_dev_properties[] = { DEFINE_PROP_END_OF_LIST(), }; -static const VMStateDescription vfio_pci_vmstate = { - .name ="vfio-pci" , - .unmigratable =1 , -}; -static void vfio_pci_dev_class_init (ObjectClass *klass,void *data) {DeviceClass *dc =DEVICE_CLASS (klass);
@@ -3233,7 +3231,6 @@ static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) dc->reset = vfio_pci_reset; dc->props = vfio_pci_dev_properties; - dc->vmsd = &vfio_pci_vmstate; dc->desc ="VFIO-based PCI device assignment" ;set_bit (DEVICE_CATEGORY_MISC , dc->categories); pdc->realize = vfio_realize;
Signature · 1 line
2.7.4
PATCH 5/5 vfio/migration: support device memory capability
In reply to Yan ZhaoIf a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.
LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.
Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.
--- hw/vfio/migration.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++-- hw/vfio/pci.h | 1 + 2 files changed, 231 insertions(+), 5 deletions(-)
hw/vfio/migration.c +230 -5
@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,return 0 ; } +static int vfio_get_device_memory_size (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +uint64_t len; +int sz; + + sz =sizeof (len); +if (pread (vbasedev->fd, &len, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { +error_report ("vfio: Failed to get length of device memory" ); +return -1 ; + } + vdev->migration->devmem_size = len; +return 0 ; +} + +static int vfio_set_device_memory_size (VFIOPCIDevice *vdev,uint64_t size) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +int sz; + + sz =sizeof (size); +if (pwrite (vbasedev->fd, &size, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { +error_report ("vfio: Failed to set length of device comemory" ); +return -1 ; + } + vdev->migration->devmem_size = size; +return 0 ; +} + +static +int vfio_save_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_GET_BUFFER ; + +if (len > region_devmem->size) { +return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set save buffer pos" ); +return -1 ; + } + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set save buffer action" ); +return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +if (pread (vbasedev->fd, buf, len, region_devmem->fd_offset) != len) { +error_report ("vfio: error load device memory buffer" ); +return -1 ; + } +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, buf, len); +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, dest, len); + } +return 0 ; +} + +static int vfio_save_data_device_memory (VFIOPCIDevice *vdev,QEMUFile *f) +{ +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +uint64_t total_len = vdev->migration->devmem_size; +uint64_t pos =0 ; + +qemu_put_be64 (f, total_len); +while (pos < total_len) { +uint64_t len = region_devmem->size; + +if (pos + len >= total_len) { + len = total_len - pos; + } +if (vfio_save_data_device_memory_chunk (vdev, f, pos, len)) { +return -1 ; + } + } + +return 0 ; +} + +static +int vfio_load_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; + +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_SET_BUFFER ; + +if (len > region_devmem->size) { +return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set device memory buffer pos" ); +return -1 ; + } +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +qemu_get_buffer (f, buf, len); +if (pwrite (vbasedev->fd, buf, len, + region_devmem->fd_offset) != len) { +error_report ("vfio: Failed to load devie memory buffer" ); +return -1 ; + } +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_get_buffer (f, dest, len); + } + + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set load device memory buffer action" ); +return -1 ; + } + +return 0 ; + +} + +static int vfio_load_data_device_memory (VFIOPCIDevice *vdev, +QEMUFile *f,uint64_t total_len) +{ +uint64_t pos =0 , len =0 ; + +vfio_set_device_memory_size (vdev, total_len); + +while (pos + len < total_len) { + len =qemu_get_be64 (f); + pos =qemu_get_be64 (f); + +vfio_load_data_device_memory_chunk (vdev, f, pos, len); + } + +return 0 ; +} + +static int vfio_set_dirty_page_bitmap_chunk (VFIOPCIDevice *vdev,uint64_t start_addr,uint64_t page_nr) {
@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,return ; } +/* get dirty data size of device memory */ +vfio_get_device_memory_size (vdev); + + *res_precopy_only += vdev->migration->devmem_size;return ; }
@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)return 0 ; } -return 0 ; +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get dirty data of device memory */ +return vfio_save_data_device_memory (vdev, f); }static void vfio_pci_load_config (VFIOPCIDevice *vdev,QEMUFile *f)
@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) len =qemu_get_be64 (f);vfio_load_data_device_config (vdev, f, len);break ; +case VFIO_SAVE_FLAG_DEVMEMORY : + len =qemu_get_be64 (f); +vfio_load_data_device_memory (vdev, f, len); +break ;default : ret = -EINVAL ; }
@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)VFIOPCIDevice *vdev = opaque;int rc =0 ; +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY |VFIO_SAVE_FLAG_CONTINUE ); +/* get dirty data of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + } +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE );vfio_pci_save_config (vdev, f);
@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)static int vfio_save_setup (QEMUFile *f,void *opaque) { +int rc =0 ;VFIOPCIDevice *vdev = opaque; -qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP |VFIO_SAVE_FLAG_CONTINUE ); +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get whole snapshot of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + }else { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + }vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING |VFIO_DEVICE_STATE_LOGGING ); -return 0 ; +return rc; }static int vfio_load_setup (QEMUFile *f,void *opaque)
@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp) goto error; } -if (vfio_device_data_cap_device_memory (vdev)) { -error_report ("No suppport of data cap device memory Yet" ); +if (vfio_device_data_cap_device_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY , +"device-state-data-device-memory" )) { goto error; }
hw/vfio/pci.h +1 -0
@@ -69,6 +69,7 @@ typedef struct VFIOMigration {uint32_t data_caps;uint32_t device_state;uint64_t devconfig_size; +uint64_t devmem_size;VMChangeStateEntry *vm_state; } VFIOMigration;
Signature · 1 line
2.7.4
Re: [PATCH 5/5] vfio/migration: support device memory capability
Quoted 98 lines · * Yan Zhao ([email protected]) wrote:
If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.
LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.
Signed-off-by Yan Zhao <[email protected]>Signed-off-by Kirti Wankhede <[email protected]>--- hw/vfio/migration.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++-- hw/vfio/pci.h | 1 + 2 files changed, 231 insertions(+), 5 deletions(-)hw/vfio/migration.c +72
-0@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,return 0 ; } +static int vfio_get_device_memory_size (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +uint64_t len; +int sz; + + sz =sizeof (len); +if (pread (vbasedev->fd, &len, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { +error_report ("vfio: Failed to get length of device memory" ); +return -1 ; + } + vdev->migration->devmem_size = len; +return 0 ; +} + +static int vfio_set_device_memory_size (VFIOPCIDevice *vdev,uint64_t size) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +int sz; + + sz =sizeof (size); +if (pwrite (vbasedev->fd, &size, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { +error_report ("vfio: Failed to set length of device comemory" ); +return -1 ; + } + vdev->migration->devmem_size = size; +return 0 ; +} + +static +int vfio_save_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_GET_BUFFER ; + +if (len > region_devmem->size) { +return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set save buffer pos" ); +return -1 ; + } + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set save buffer action" ); +return -1 ;
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 287
Quoted 10 lines
hw/vfio/migration.c +10
-0+ } + +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +if (pread (vbasedev->fd, buf, len, region_devmem->fd_offset) != len) { +error_report ("vfio: error load device memory buffer" );
That's forgotten to g_free(buf)
Quoted 64 lines
hw/vfio/migration.c +64
-0+return -1 ; + } +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, buf, len); +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, dest, len); + } +return 0 ; +} + +static int vfio_save_data_device_memory (VFIOPCIDevice *vdev,QEMUFile *f) +{ +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +uint64_t total_len = vdev->migration->devmem_size; +uint64_t pos =0 ; + +qemu_put_be64 (f, total_len); +while (pos < total_len) { +uint64_t len = region_devmem->size; + +if (pos + len >= total_len) { + len = total_len - pos; + } +if (vfio_save_data_device_memory_chunk (vdev, f, pos, len)) { +return -1 ; + } + } + +return 0 ; +} + +static +int vfio_load_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; + +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_SET_BUFFER ; + +if (len > region_devmem->size) { +return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set device memory buffer pos" ); +return -1 ; + }
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 361
Quoted 10 lines
hw/vfio/migration.c +10
-0+if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +qemu_get_buffer (f, buf, len); +if (pwrite (vbasedev->fd, buf, len, + region_devmem->fd_offset) != len) { +error_report ("vfio: Failed to load devie memory buffer" );
Again, failed to free buf
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 368
Quoted 7 lines
hw/vfio/migration.c +7
-0+return -1 ; + } +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_get_buffer (f, dest, len); + }
You might want to use qemu_file_get_error(f) before writing the data to the device, to check for the case of a read error on the migration stream that happened somewhere in the pevious qemu_get's
Quoted 13 lines
hw/vfio/migration.c +13
-0+ sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set load device memory buffer action" ); +return -1 ; + } + +return 0 ; + +} +
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392
Quoted 10 lines
hw/vfio/migration.c +10
-0+static int vfio_load_data_device_memory (VFIOPCIDevice *vdev, +QEMUFile *f,uint64_t total_len) +{ +uint64_t pos =0 , len =0 ; + +vfio_set_device_memory_size (vdev, total_len); + +while (pos + len < total_len) { + len =qemu_get_be64 (f); + pos =qemu_get_be64 (f);
Please check len/pos - always assume that the migration stream could be (maliciously or accidentally) corrupt.
Quoted 111 lines
hw/vfio/migration.c +42
-5+vfio_load_data_device_memory_chunk (vdev, f, pos, len); + } + +return 0 ; +} + +static int vfio_set_dirty_page_bitmap_chunk (VFIOPCIDevice *vdev,uint64_t start_addr,uint64_t page_nr) {@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,return ; } +/* get dirty data size of device memory */ +vfio_get_device_memory_size (vdev); + + *res_precopy_only += vdev->migration->devmem_size;return ; }@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)return 0 ; } -return 0 ; +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get dirty data of device memory */ +return vfio_save_data_device_memory (vdev, f); }static void vfio_pci_load_config (VFIOPCIDevice *vdev,QEMUFile *f)@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) len =qemu_get_be64 (f);vfio_load_data_device_config (vdev, f, len);break ; +case VFIO_SAVE_FLAG_DEVMEMORY : + len =qemu_get_be64 (f); +vfio_load_data_device_memory (vdev, f, len); +break ;default : ret = -EINVAL ; }@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)VFIOPCIDevice *vdev = opaque;int rc =0 ; +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY |VFIO_SAVE_FLAG_CONTINUE ); +/* get dirty data of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + } +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE );vfio_pci_save_config (vdev, f);@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)static int vfio_save_setup (QEMUFile *f,void *opaque) { +int rc =0 ;VFIOPCIDevice *vdev = opaque; -qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP |VFIO_SAVE_FLAG_CONTINUE ); +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get whole snapshot of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + }else { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + }vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING |VFIO_DEVICE_STATE_LOGGING ); -return 0 ; +return rc; }static int vfio_load_setup (QEMUFile *f,void *opaque)@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp) goto error; } -if (vfio_device_data_cap_device_memory (vdev)) { -error_report ("No suppport of data cap device memory Yet" ); +if (vfio_device_data_cap_device_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY , +"device-state-data-device-memory" )) { goto error; }hw/vfio/pci.h +1
-0@@ -69,6 +69,7 @@ typedef struct VFIOMigration {uint32_t data_caps;uint32_t device_state;uint64_t devconfig_size; +uint64_t devmem_size;VMChangeStateEntry *vm_state; } VFIOMigration;Signature · 1 line
2.7.4
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 5/5] vfio/migration: support device memory capability
Quoted 103 lines · Dr. David Alan Gilbert
Quoted 102 lines · * Yan Zhao ([email protected]) wrote:
If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.
LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.
Signed-off-by Yan Zhao <[email protected]>Signed-off-by Kirti Wankhede <[email protected]>--- hw/vfio/migration.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++-- hw/vfio/pci.h | 1 + 2 files changed, 231 insertions(+), 5 deletions(-)hw/vfio/migration.c +76
-0@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,return 0 ; } +static int vfio_get_device_memory_size (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +uint64_t len; +int sz; + + sz =sizeof (len); +if (pread (vbasedev->fd, &len, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { +error_report ("vfio: Failed to get length of device memory" ); +return -1 ; + } + vdev->migration->devmem_size = len; +return 0 ; +} + +static int vfio_set_device_memory_size (VFIOPCIDevice *vdev,uint64_t size) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +int sz; + + sz =sizeof (size); +if (pwrite (vbasedev->fd, &size, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { +error_report ("vfio: Failed to set length of device comemory" ); +return -1 ; + } + vdev->migration->devmem_size = size; +return 0 ; +} + +static +int vfio_save_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_GET_BUFFER ; + +if (len > region_devmem->size) { +return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set save buffer pos" ); +return -1 ; + } + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set save buffer action" ); +return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len);
Quoted 9 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 287
Quoted 6 lines
hw/vfio/migration.c +6
-0+if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +if (pread (vbasedev->fd, buf, len, region_devmem->fd_offset) != len) { +error_report ("vfio: error load device memory buffer" );That's forgotten to
g_free(buf)
Right, I'll correct that.
Quoted 68 lines
Quoted 68 lines
hw/vfio/migration.c +68
-0+return -1 ; + } +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, buf, len); +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, dest, len); + } +return 0 ; +} + +static int vfio_save_data_device_memory (VFIOPCIDevice *vdev,QEMUFile *f) +{ +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +uint64_t total_len = vdev->migration->devmem_size; +uint64_t pos =0 ; + +qemu_put_be64 (f, total_len); +while (pos < total_len) { +uint64_t len = region_devmem->size; + +if (pos + len >= total_len) { + len = total_len - pos; + } +if (vfio_save_data_device_memory_chunk (vdev, f, pos, len)) { +return -1 ; + } + } + +return 0 ; +} + +static +int vfio_load_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; + +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_SET_BUFFER ; + +if (len > region_devmem->size) { +return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set device memory buffer pos" ); +return -1 ; + } +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" );
Quoted 9 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 361
Quoted 6 lines
hw/vfio/migration.c +6
-0+return -1 ; + } +qemu_get_buffer (f, buf, len); +if (pwrite (vbasedev->fd, buf, len, + region_devmem->fd_offset) != len) { +error_report ("vfio: Failed to load devie memory buffer" );Again, failed to free buf
Right, I'll correct that.
Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 368
Quoted 7 lines
hw/vfio/migration.c +7
-0+return -1 ; + } +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_get_buffer (f, dest, len); + }
Quoted 4 lines
You might want to use
qemu_file_get_error(f)before writing the data to the device, to check for the case of a read error on the migration stream that happened somewhere in the pevious qemu_get's
ok.
Quoted 18 lines
Quoted 18 lines
hw/vfio/migration.c +18
-0+ sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set load device memory buffer action" ); +return -1 ; + } + +return 0 ; + +} + +static int vfio_load_data_device_memory (VFIOPCIDevice *vdev, +QEMUFile *f,uint64_t total_len) +{ +uint64_t pos =0 , len =0 ; +
Quoted 9 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392
Quoted 5 lines
hw/vfio/migration.c +5
-0+vfio_set_device_memory_size (vdev, total_len); + +while (pos + len < total_len) { + len =qemu_get_be64 (f); + pos =qemu_get_be64 (f);Please check len/pos - always assume that the migration stream could be (maliciously or accidentally) corrupt.
ok.
Quoted 117 lines
Quoted 111 lines
hw/vfio/migration.c +42
-5+vfio_load_data_device_memory_chunk (vdev, f, pos, len); + } + +return 0 ; +} + +static int vfio_set_dirty_page_bitmap_chunk (VFIOPCIDevice *vdev,uint64_t start_addr,uint64_t page_nr) {@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,return ; } +/* get dirty data size of device memory */ +vfio_get_device_memory_size (vdev); + + *res_precopy_only += vdev->migration->devmem_size;return ; }@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)return 0 ; } -return 0 ; +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get dirty data of device memory */ +return vfio_save_data_device_memory (vdev, f); }static void vfio_pci_load_config (VFIOPCIDevice *vdev,QEMUFile *f)@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) len =qemu_get_be64 (f);vfio_load_data_device_config (vdev, f, len);break ; +case VFIO_SAVE_FLAG_DEVMEMORY : + len =qemu_get_be64 (f); +vfio_load_data_device_memory (vdev, f, len); +break ;default : ret = -EINVAL ; }@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)VFIOPCIDevice *vdev = opaque;int rc =0 ; +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY |VFIO_SAVE_FLAG_CONTINUE ); +/* get dirty data of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + } +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE );vfio_pci_save_config (vdev, f);@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)static int vfio_save_setup (QEMUFile *f,void *opaque) { +int rc =0 ;VFIOPCIDevice *vdev = opaque; -qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP |VFIO_SAVE_FLAG_CONTINUE ); +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get whole snapshot of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + }else { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + }vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING |VFIO_DEVICE_STATE_LOGGING ); -return 0 ; +return rc; }static int vfio_load_setup (QEMUFile *f,void *opaque)@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp) goto error; } -if (vfio_device_data_cap_device_memory (vdev)) { -error_report ("No suppport of data cap device memory Yet" ); +if (vfio_device_data_cap_device_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY , +"device-state-data-device-memory" )) { goto error; }hw/vfio/pci.h +1
-0@@ -69,6 +69,7 @@ typedef struct VFIOMigration {uint32_t data_caps;uint32_t device_state;uint64_t devconfig_size; +uint64_t devmem_size;VMChangeStateEntry *vm_state; } VFIOMigration;Signature · 1 line
2.7.4--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Re: [PATCH 5/5] vfio/migration: support device memory capability
In reply to Yan ZhaoQuoted 34 lines · Yan Zhao
If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.
LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.
Signed-off-by Yan Zhao <[email protected]>Signed-off-by Kirti Wankhede <[email protected]>---
hw/vfio/migration.c| 235 ++++++++++++++++++++++++++++++++++++++++++++++++++--hw/vfio/pci.h| 1 +2 files changed, 231 insertions(+), 5 deletions(-)hw/vfio/migration.c +4
-0@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,return 0 ; } +static int vfio_get_device_memory_size (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; + VFIORegion *region_ctl =
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 219
Quoted 10 lines
hw/vfio/migration.c +10
-0+ &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +uint64_t len; +int sz; + + sz =sizeof (len); +if (pread (vbasedev->fd, &len, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { + error_report("vfio: Failed to get length of device memory”);
s/length/size/ ? (to be consistent with function name)
Quoted 9 lines
hw/vfio/migration.c +9
-0+return -1 ; + } + vdev->migration->devmem_size = len; +return 0 ; +} + +static int vfio_set_device_memory_size (VFIOPCIDevice *vdev,uint64_t size) +{ +VFIODevice *vbasedev = &vdev->vbasedev;
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 10 lines
hw/vfio/migration.c +10
-0+VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +int sz; + + sz =sizeof (size); +if (pwrite (vbasedev->fd, &size, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { + error_report("vfio: Failed to set length of device comemory”);
What is comemory? Typo?
Same comment about length vs size
Quoted 11 lines
hw/vfio/migration.c +11
-0+return -1 ; + } + vdev->migration->devmem_size = size; +return 0 ; +} + +static +int vfio_save_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev;
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 259
Quoted 10 lines
hw/vfio/migration.c +10
-0+VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_GET_BUFFER ; + +if (len > region_devmem->size) {
Is it intentional that there is no error_report here?
Quoted 14 lines
hw/vfio/migration.c +14
-0+return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set save buffer pos" ); +return -1 ; + } + sz =sizeof (action); + if (pwrite(vbasedev->fd, &action, sz, + region_ctl->fd_offset +
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 10 lines
hw/vfio/migration.c +10
-0+ offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set save buffer action" ); +return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { + error_report("vfio: Failed to allocate memory for migrate”);
s/migrate/migration/ ?
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 4 lines
hw/vfio/migration.c +4
-0+return -1 ; + } +if (pread (vbasedev->fd, buf, len, region_devmem->fd_offset) != len) { + error_report("vfio: error load device memory buffer”);
s/load/loading/ ?
Quoted 22 lines
hw/vfio/migration.c +22
-0+return -1 ; + } +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, buf, len); +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, dest, len); + } +return 0 ; +} + +static int vfio_save_data_device_memory (VFIOPCIDevice *vdev,QEMUFile *f) +{ +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +uint64_t total_len = vdev->migration->devmem_size; +uint64_t pos =0 ; +
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 319
Quoted 10 lines
hw/vfio/migration.c +10
-0+qemu_put_be64 (f, total_len); +while (pos < total_len) { +uint64_t len = region_devmem->size; + +if (pos + len >= total_len) { + len = total_len - pos; + } +if (vfio_save_data_device_memory_chunk (vdev, f, pos, len)) { +return -1 ; + }
I don’t see where pos is incremented in this loop
Quoted 11 lines
hw/vfio/migration.c +11
-0+ } + +return 0 ; +} + +static +int vfio_load_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; + VFIORegion *region_ctl =
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 340
Quoted 10 lines
hw/vfio/migration.c +10
-0+ &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; + +void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_SET_BUFFER ; + +if (len > region_devmem->size) {
error_report?
Quoted 42 lines
hw/vfio/migration.c +42
-0+return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set device memory buffer pos" ); +return -1 ; + } +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +qemu_get_buffer (f, buf, len); +if (pwrite (vbasedev->fd, buf, len, + region_devmem->fd_offset) != len) { +error_report ("vfio: Failed to load devie memory buffer" ); +return -1 ; + } +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_get_buffer (f, dest, len); + } + + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set load device memory buffer action" ); +return -1 ; + } + +return 0 ; + +} +
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392
Quoted 10 lines
hw/vfio/migration.c +10
-0+static int vfio_load_data_device_memory (VFIOPCIDevice *vdev, +QEMUFile *f,uint64_t total_len) +{ +uint64_t pos =0 , len =0 ; + +vfio_set_device_memory_size (vdev, total_len); + +while (pos + len < total_len) { + len =qemu_get_be64 (f); + pos =qemu_get_be64 (f);
Nit: load reads len/pos in the loop, whereas save does it in the inner function (vfio_save_data_device_memory_chunk)
Quoted 116 lines
hw/vfio/migration.c +43
-5+ +vfio_load_data_device_memory_chunk (vdev, f, pos, len); + } + +return 0 ; +} + +static int vfio_set_dirty_page_bitmap_chunk (VFIOPCIDevice *vdev,uint64_t start_addr,uint64_t page_nr) {@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,return ; } +/* get dirty data size of device memory */ +vfio_get_device_memory_size (vdev); + + *res_precopy_only += vdev->migration->devmem_size;return ; }@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)return 0 ; } -return 0 ; +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get dirty data of device memory */ +return vfio_save_data_device_memory (vdev, f); }static void vfio_pci_load_config (VFIOPCIDevice *vdev,QEMUFile *f)@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) len =qemu_get_be64 (f);vfio_load_data_device_config (vdev, f, len);break ; +case VFIO_SAVE_FLAG_DEVMEMORY : + len =qemu_get_be64 (f); +vfio_load_data_device_memory (vdev, f, len); +break ;default : ret = -EINVAL ; }@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)VFIOPCIDevice *vdev = opaque;int rc =0 ; +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY |VFIO_SAVE_FLAG_CONTINUE ); +/* get dirty data of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + } +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE );vfio_pci_save_config (vdev, f);@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)static int vfio_save_setup (QEMUFile *f,void *opaque) { +int rc =0 ;VFIOPCIDevice *vdev = opaque; -qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP |VFIO_SAVE_FLAG_CONTINUE ); +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get whole snapshot of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + }else { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + }vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING |VFIO_DEVICE_STATE_LOGGING ); -return 0 ; +return rc; }static int vfio_load_setup (QEMUFile *f,void *opaque)@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp) goto error; } -if (vfio_device_data_cap_device_memory (vdev)) { -error_report ("No suppport of data cap device memory Yet" ); +if (vfio_device_data_cap_device_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY , +"device-state-data-device-memory" )) { goto error; }hw/vfio/pci.h +1
-0@@ -69,6 +69,7 @@ typedef struct VFIOMigration {uint32_t data_caps;uint32_t device_state;uint64_t devconfig_size; +uint64_t devmem_size;VMChangeStateEntry *vm_state; } VFIOMigration;Signature · 6 lines
2.7.4 _______________________________________________ intel-gvt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Re: [PATCH 5/5] vfio/migration: support device memory capability
Quoted 30 lines · Christophe de Dinechin
Quoted 30 lines · Yan Zhao
If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.
LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.
Signed-off-by Yan Zhao <[email protected]>Signed-off-by Kirti Wankhede <[email protected]>---
hw/vfio/migration.c| 235 ++++++++++++++++++++++++++++++++++++++++++++++++++--hw/vfio/pci.h| 1 +2 files changed, 231 insertions(+), 5 deletions(-)hw/vfio/migration.c +0
-0@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,return 0 ; }
Quoted 16 lines
Quoted 4 lines
hw/vfio/migration.c +4
-0+static int vfio_get_device_memory_size (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; + VFIORegion *region_ctl =hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 10 lines
hw/vfio/migration.c +10
-0+ &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +uint64_t len; +int sz; + + sz =sizeof (len); +if (pread (vbasedev->fd, &len, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { + error_report("vfio: Failed to get length of device memory”);s/length/size/ ? (to be consistent with function name)
ok. thanks
Quoted 13 lines
Quoted 13 lines
hw/vfio/migration.c +13
-0+return -1 ; + } + vdev->migration->devmem_size = len; +return 0 ; +} + +static int vfio_set_device_memory_size (VFIOPCIDevice *vdev,uint64_t size) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +int sz; +
Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 6 lines
hw/vfio/migration.c +6
-0+ sz =sizeof (size); +if (pwrite (vbasedev->fd, &size, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { + error_report("vfio: Failed to set length of device comemory”);What is comemory? Typo?
Right, typo. should be "memory" :)
Quoted 3 lines
Same comment about length vs size
got it. thanks
Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15
-0+return -1 ; + } + vdev->migration->devmem_size = size; +return 0 ; +} + +static +int vfio_save_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ];
Quoted 9 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 259
Quoted 6 lines
hw/vfio/migration.c +6
-0+void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_GET_BUFFER ; + +if (len > region_devmem->size) {Is it intentional that there is no
error_reporthere?
an error_report here may be better.
Quoted 17 lines
Quoted 17 lines
hw/vfio/migration.c +17
-0+return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set save buffer pos" ); +return -1 ; + } + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set save buffer action" );
Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 7 lines
hw/vfio/migration.c +7
-0+return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { + error_report("vfio: Failed to allocate memory for migrate”);s/migrate/migration/ ?
yes, thanks
Quoted 5 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 4 lines
hw/vfio/migration.c +4
-0+return -1 ; + } +if (pread (vbasedev->fd, buf, len, region_devmem->fd_offset) != len) { + error_report("vfio: error load device memory buffer”);s/load/loading/ ?
error to load? :)
Quoted 26 lines
Quoted 26 lines
hw/vfio/migration.c +26
-0+return -1 ; + } +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, buf, len); +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, dest, len); + } +return 0 ; +} + +static int vfio_save_data_device_memory (VFIOPCIDevice *vdev,QEMUFile *f) +{ +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +uint64_t total_len = vdev->migration->devmem_size; +uint64_t pos =0 ; + +qemu_put_be64 (f, total_len); +while (pos < total_len) { +uint64_t len = region_devmem->size; +
Quoted 9 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 319
Quoted 6 lines
hw/vfio/migration.c +6
-0+if (pos + len >= total_len) { + len = total_len - pos; + } +if (vfio_save_data_device_memory_chunk (vdev, f, pos, len)) { +return -1 ; + }I don’t see where pos is incremented in this loop
yes, missing one line "pos += len;"
Currently, code is not verified in hardware with device memory cap on. Thanks:)
Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15
-0+ } + +return 0 ; +} + +static +int vfio_load_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +
Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 340
Quoted 6 lines
hw/vfio/migration.c +6
-0+void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_SET_BUFFER ; + +if (len > region_devmem->size) {
error_report?
seems better to add error_report.
Quoted 47 lines
Quoted 47 lines
hw/vfio/migration.c +47
-0+return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set device memory buffer pos" ); +return -1 ; + } +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +qemu_get_buffer (f, buf, len); +if (pwrite (vbasedev->fd, buf, len, + region_devmem->fd_offset) != len) { +error_report ("vfio: Failed to load devie memory buffer" ); +return -1 ; + } +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_get_buffer (f, dest, len); + } + + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set load device memory buffer action" ); +return -1 ; + } + +return 0 ; + +} + +static int vfio_load_data_device_memory (VFIOPCIDevice *vdev, +QEMUFile *f,uint64_t total_len) +{ +uint64_t pos =0 , len =0 ; +
Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392
Quoted 5 lines
hw/vfio/migration.c +5
-0+vfio_set_device_memory_size (vdev, total_len); + +while (pos + len < total_len) { + len =qemu_get_be64 (f); + pos =qemu_get_be64 (f);Nit: load reads len/pos in the loop, whereas save does it in the inner function (
vfio_save_data_device_memory_chunk)
right, load has to read len/pos in the loop.
Quoted 122 lines
Quoted 117 lines
hw/vfio/migration.c +43
-5+ +vfio_load_data_device_memory_chunk (vdev, f, pos, len); + } + +return 0 ; +} + +static int vfio_set_dirty_page_bitmap_chunk (VFIOPCIDevice *vdev,uint64_t start_addr,uint64_t page_nr) {@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,return ; } +/* get dirty data size of device memory */ +vfio_get_device_memory_size (vdev); + + *res_precopy_only += vdev->migration->devmem_size;return ; }@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)return 0 ; } -return 0 ; +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get dirty data of device memory */ +return vfio_save_data_device_memory (vdev, f); }static void vfio_pci_load_config (VFIOPCIDevice *vdev,QEMUFile *f)@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) len =qemu_get_be64 (f);vfio_load_data_device_config (vdev, f, len);break ; +case VFIO_SAVE_FLAG_DEVMEMORY : + len =qemu_get_be64 (f); +vfio_load_data_device_memory (vdev, f, len); +break ;default : ret = -EINVAL ; }@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)VFIOPCIDevice *vdev = opaque;int rc =0 ; +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY |VFIO_SAVE_FLAG_CONTINUE ); +/* get dirty data of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + } +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE );vfio_pci_save_config (vdev, f);@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)static int vfio_save_setup (QEMUFile *f,void *opaque) { +int rc =0 ;VFIOPCIDevice *vdev = opaque; -qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP |VFIO_SAVE_FLAG_CONTINUE ); +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get whole snapshot of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + }else { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + }vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING |VFIO_DEVICE_STATE_LOGGING ); -return 0 ; +return rc; }static int vfio_load_setup (QEMUFile *f,void *opaque)@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp) goto error; } -if (vfio_device_data_cap_device_memory (vdev)) { -error_report ("No suppport of data cap device memory Yet" ); +if (vfio_device_data_cap_device_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY , +"device-state-data-device-memory" )) { goto error; }hw/vfio/pci.h +1
-0@@ -69,6 +69,7 @@ typedef struct VFIOMigration {uint32_t data_caps;uint32_t device_state;uint64_t devconfig_size; +uint64_t devmem_size;VMChangeStateEntry *vm_state; } VFIOMigration;Signature · 6 lines
2.7.4 _______________________________________________ intel-gvt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Re: [PATCH 5/5] vfio/migration: support device memory capability
Quoted 132 lines · Zhao Yan
Quoted 30 lines · Christophe de Dinechin
Quoted 30 lines · Yan Zhao
If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.
LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.
Signed-off-by Yan Zhao <[email protected]>Signed-off-by Kirti Wankhede <[email protected]>---
hw/vfio/migration.c| 235 ++++++++++++++++++++++++++++++++++++++++++++++++++--hw/vfio/pci.h| 1 +2 files changed, 231 insertions(+), 5 deletions(-)hw/vfio/migration.c +0
-0@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,return 0 ; }Quoted 16 lines
Quoted 4 lines
hw/vfio/migration.c +4
-0+static int vfio_get_device_memory_size (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; + VFIORegion *region_ctl =hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 10 lines
hw/vfio/migration.c +10
-0+ &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +uint64_t len; +int sz; + + sz =sizeof (len); +if (pread (vbasedev->fd, &len, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { + error_report("vfio: Failed to get length of device memory”);s/length/size/ ? (to be consistent with function name)
ok. thanks
Quoted 13 lines
Quoted 13 lines
hw/vfio/migration.c +13
-0+return -1 ; + } + vdev->migration->devmem_size = len; +return 0 ; +} + +static int vfio_set_device_memory_size (VFIOPCIDevice *vdev,uint64_t size) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +int sz; +Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 6 lines
hw/vfio/migration.c +6
-0+ sz =sizeof (size); +if (pwrite (vbasedev->fd, &size, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { + error_report("vfio: Failed to set length of device comemory”);What is comemory? Typo?
Right, typo. should be "memory" :)
Quoted 3 lines
Same comment about length vs size
got it. thanks
Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15
-0+return -1 ; + } + vdev->migration->devmem_size = size; +return 0 ; +} + +static +int vfio_save_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ];Quoted 9 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 259
Quoted 6 lines
hw/vfio/migration.c +6
-0+void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_GET_BUFFER ; + +if (len > region_devmem->size) {Is it intentional that there is no
error_reporthere?an
error_reporthere may be better.Quoted 17 lines
Quoted 17 lines
hw/vfio/migration.c +17
-0+return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set save buffer pos" ); +return -1 ; + } + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set save buffer action" );Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 7 lines
hw/vfio/migration.c +7
-0+return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { + error_report("vfio: Failed to allocate memory for migrate”);s/migrate/migration/ ?
Quoted 7 lines
yes, thanks
Quoted 5 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 4 lines
hw/vfio/migration.c +4
-0+return -1 ; + } +if (pread (vbasedev->fd, buf, len, region_devmem->fd_offset) != len) { + error_report("vfio: error load device memory buffer”);s/load/loading/ ?
error to load? :)
I’d check with a native speaker, but I believe it’s “error loading”.
To me (to be checked), the two sentences don’t have the same meaning:
“It is an error to load device memory buffer” -> “You are not allowed to do that” “I had an error loading device memory buffer” -> “I tried, but it failed"
Quoted 242 lines
Quoted 27 lines
Quoted 27 lines
hw/vfio/migration.c +26
-0+return -1 ; + } +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, buf, len); +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, dest, len); + } +return 0 ; +} + +static int vfio_save_data_device_memory (VFIOPCIDevice *vdev,QEMUFile *f) +{ +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +uint64_t total_len = vdev->migration->devmem_size; +uint64_t pos =0 ; + +qemu_put_be64 (f, total_len); +while (pos < total_len) { +uint64_t len = region_devmem->size; +Quoted 9 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 319
Quoted 6 lines
hw/vfio/migration.c +6
-0+if (pos + len >= total_len) { + len = total_len - pos; + } +if (vfio_save_data_device_memory_chunk (vdev, f, pos, len)) { +return -1 ; + }I don’t see where pos is incremented in this loop
yes, missing one line "pos += len;"
Currently, code is not verified in hardware with device memory cap on. Thanks:)Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15
-0+ } + +return 0 ; +} + +static +int vfio_load_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 340
Quoted 6 lines
hw/vfio/migration.c +6
-0+void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_SET_BUFFER ; + +if (len > region_devmem->size) {
error_report?seems better to add
error_report.Quoted 47 lines
Quoted 47 lines
hw/vfio/migration.c +47
-0+return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set device memory buffer pos" ); +return -1 ; + } +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +qemu_get_buffer (f, buf, len); +if (pwrite (vbasedev->fd, buf, len, + region_devmem->fd_offset) != len) { +error_report ("vfio: Failed to load devie memory buffer" ); +return -1 ; + } +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_get_buffer (f, dest, len); + } + + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set load device memory buffer action" ); +return -1 ; + } + +return 0 ; + +} + +static int vfio_load_data_device_memory (VFIOPCIDevice *vdev, +QEMUFile *f,uint64_t total_len) +{ +uint64_t pos =0 , len =0 ; +Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392
Quoted 5 lines
hw/vfio/migration.c +5
-0+vfio_set_device_memory_size (vdev, total_len); + +while (pos + len < total_len) { + len =qemu_get_be64 (f); + pos =qemu_get_be64 (f);Nit: load reads len/pos in the loop, whereas save does it in the inner function (
vfio_save_data_device_memory_chunk)right, load has to read len/pos in the loop.
Quoted 122 lines
Quoted 117 lines
hw/vfio/migration.c +43
-5+ +vfio_load_data_device_memory_chunk (vdev, f, pos, len); + } + +return 0 ; +} + +static int vfio_set_dirty_page_bitmap_chunk (VFIOPCIDevice *vdev,uint64_t start_addr,uint64_t page_nr) {@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,return ; } +/* get dirty data size of device memory */ +vfio_get_device_memory_size (vdev); + + *res_precopy_only += vdev->migration->devmem_size;return ; }@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)return 0 ; } -return 0 ; +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get dirty data of device memory */ +return vfio_save_data_device_memory (vdev, f); }static void vfio_pci_load_config (VFIOPCIDevice *vdev,QEMUFile *f)@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) len =qemu_get_be64 (f);vfio_load_data_device_config (vdev, f, len);break ; +case VFIO_SAVE_FLAG_DEVMEMORY : + len =qemu_get_be64 (f); +vfio_load_data_device_memory (vdev, f, len); +break ;default : ret = -EINVAL ; }@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)VFIOPCIDevice *vdev = opaque;int rc =0 ; +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY |VFIO_SAVE_FLAG_CONTINUE ); +/* get dirty data of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + } +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE );vfio_pci_save_config (vdev, f);@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)static int vfio_save_setup (QEMUFile *f,void *opaque) { +int rc =0 ;VFIOPCIDevice *vdev = opaque; -qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP |VFIO_SAVE_FLAG_CONTINUE ); +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get whole snapshot of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + }else { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + }vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING |VFIO_DEVICE_STATE_LOGGING ); -return 0 ; +return rc; }static int vfio_load_setup (QEMUFile *f,void *opaque)@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp) goto error; } -if (vfio_device_data_cap_device_memory (vdev)) { -error_report ("No suppport of data cap device memory Yet" ); +if (vfio_device_data_cap_device_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY , +"device-state-data-device-memory" )) { goto error; }hw/vfio/pci.h +1
-0@@ -69,6 +69,7 @@ typedef struct VFIOMigration {uint32_t data_caps;uint32_t device_state;uint64_t devconfig_size; +uint64_t devmem_size;VMChangeStateEntry *vm_state; } VFIOMigration;Signature · 6 lines
2.7.4 _______________________________________________ intel-gvt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Re: [PATCH 5/5] vfio/migration: support device memory capability
Quoted 138 lines · Christophe de Dinechin
Quoted 138 lines · Zhao Yan
Quoted 30 lines · Christophe de Dinechin
Quoted 30 lines · Yan Zhao
If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.
LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.
Signed-off-by Yan Zhao <[email protected]>Signed-off-by Kirti Wankhede <[email protected]>---
hw/vfio/migration.c| 235 ++++++++++++++++++++++++++++++++++++++++++++++++++--hw/vfio/pci.h| 1 +2 files changed, 231 insertions(+), 5 deletions(-)hw/vfio/migration.c +0
-0@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,return 0 ; }Quoted 16 lines
Quoted 4 lines
hw/vfio/migration.c +4
-0+static int vfio_get_device_memory_size (VFIOPCIDevice *vdev) +{ +VFIODevice *vbasedev = &vdev->vbasedev; + VFIORegion *region_ctl =hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 10 lines
hw/vfio/migration.c +10
-0+ &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +uint64_t len; +int sz; + + sz =sizeof (len); +if (pread (vbasedev->fd, &len, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { + error_report("vfio: Failed to get length of device memory”);s/length/size/ ? (to be consistent with function name)
ok. thanks
Quoted 13 lines
Quoted 13 lines
hw/vfio/migration.c +13
-0+return -1 ; + } + vdev->migration->devmem_size = len; +return 0 ; +} + +static int vfio_set_device_memory_size (VFIOPCIDevice *vdev,uint64_t size) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +int sz; +Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 6 lines
hw/vfio/migration.c +6
-0+ sz =sizeof (size); +if (pwrite (vbasedev->fd, &size, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.size)) + != sz) { + error_report("vfio: Failed to set length of device comemory”);What is comemory? Typo?
Right, typo. should be "memory" :)
Quoted 3 lines
Same comment about length vs size
got it. thanks
Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15
-0+return -1 ; + } + vdev->migration->devmem_size = size; +return 0 ; +} + +static +int vfio_save_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ];Quoted 9 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 259
Quoted 6 lines
hw/vfio/migration.c +6
-0+void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_GET_BUFFER ; + +if (len > region_devmem->size) {Is it intentional that there is no
error_reporthere?an
error_reporthere may be better.Quoted 17 lines
Quoted 17 lines
hw/vfio/migration.c +17
-0+return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set save buffer pos" ); +return -1 ; + } + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set save buffer action" );Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 7 lines
hw/vfio/migration.c +7
-0+return -1 ; + } + +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { + error_report("vfio: Failed to allocate memory for migrate”);s/migrate/migration/ ?
yes, thanks
Quoted 5 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
Quoted 4 lines
hw/vfio/migration.c +4
-0+return -1 ; + } +if (pread (vbasedev->fd, buf, len, region_devmem->fd_offset) != len) { + error_report("vfio: error load device memory buffer”);s/load/loading/ ?
Quoted 9 lines
Quoted 1 line
error to load? :)
I’d check with a native speaker, but I believe it’s “error loading”.
To me (to be checked), the two sentences don’t have the same meaning:
“It is an error to load device memory buffer” -> “You are not allowed to do that” “I had an error loading device memory buffer” -> “I tried, but it failed"
haha, ok, I'll change it to loading, thanks :)
Quoted 243 lines
Quoted 242 lines
Quoted 27 lines
Quoted 27 lines
hw/vfio/migration.c +26
-0+return -1 ; + } +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, buf, len); +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_put_be64 (f, len); +qemu_put_be64 (f, pos); +qemu_put_buffer (f, dest, len); + } +return 0 ; +} + +static int vfio_save_data_device_memory (VFIOPCIDevice *vdev,QEMUFile *f) +{ +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +uint64_t total_len = vdev->migration->devmem_size; +uint64_t pos =0 ; + +qemu_put_be64 (f, total_len); +while (pos < total_len) { +uint64_t len = region_devmem->size; +Quoted 9 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 319
Quoted 6 lines
hw/vfio/migration.c +6
-0+if (pos + len >= total_len) { + len = total_len - pos; + } +if (vfio_save_data_device_memory_chunk (vdev, f, pos, len)) { +return -1 ; + }I don’t see where pos is incremented in this loop
yes, missing one line "pos += len;"
Currently, code is not verified in hardware with device memory cap on. Thanks:)Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15
-0+ } + +return 0 ; +} + +static +int vfio_load_data_device_memory_chunk (VFIOPCIDevice *vdev,QEMUFile *f, +uint64_t pos,uint64_t len) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIORegion *region_ctl = + &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL ]; +VFIORegion *region_devmem = + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ]; +Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 340
Quoted 6 lines
hw/vfio/migration.c +6
-0+void *dest; +uint32_t sz; +uint8_t *buf =NULL ; +uint32_t action =VFIO_DEVICE_DATA_ACTION_SET_BUFFER ; + +if (len > region_devmem->size) {
error_report?seems better to add
error_report.Quoted 47 lines
Quoted 47 lines
hw/vfio/migration.c +47
-0+return -1 ; + } + + sz =sizeof (pos); +if (pwrite (vbasedev->fd, &pos, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.pos)) + != sz) { +error_report ("vfio: Failed to set device memory buffer pos" ); +return -1 ; + } +if (!vfio_device_state_region_mmaped (region_devmem)) { + buf =g_malloc (len); +if (buf ==NULL ) { +error_report ("vfio: Failed to allocate memory for migrate" ); +return -1 ; + } +qemu_get_buffer (f, buf, len); +if (pwrite (vbasedev->fd, buf, len, + region_devmem->fd_offset) != len) { +error_report ("vfio: Failed to load devie memory buffer" ); +return -1 ; + } +g_free (buf); + }else { + dest = region_devmem->mmaps[0 ].mmap; +qemu_get_buffer (f, dest, len); + } + + sz =sizeof (action); +if (pwrite (vbasedev->fd, &action, sz, + region_ctl->fd_offset + + offsetof(struct vfio_device_state_ctl, device_memory.action)) + != sz) { +error_report ("vfio: Failed to set load device memory buffer action" ); +return -1 ; + } + +return 0 ; + +} + +static int vfio_load_data_device_memory (VFIOPCIDevice *vdev, +QEMUFile *f,uint64_t total_len) +{ +uint64_t pos =0 , len =0 ; +Quoted 8 lines
hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392
Quoted 5 lines
hw/vfio/migration.c +5
-0+vfio_set_device_memory_size (vdev, total_len); + +while (pos + len < total_len) { + len =qemu_get_be64 (f); + pos =qemu_get_be64 (f);Nit: load reads len/pos in the loop, whereas save does it in the inner function (
vfio_save_data_device_memory_chunk)right, load has to read len/pos in the loop.
Quoted 122 lines
Quoted 117 lines
hw/vfio/migration.c +43
-5+ +vfio_load_data_device_memory_chunk (vdev, f, pos, len); + } + +return 0 ; +} + +static int vfio_set_dirty_page_bitmap_chunk (VFIOPCIDevice *vdev,uint64_t start_addr,uint64_t page_nr) {@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,return ; } +/* get dirty data size of device memory */ +vfio_get_device_memory_size (vdev); + + *res_precopy_only += vdev->migration->devmem_size;return ; }@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)return 0 ; } -return 0 ; +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get dirty data of device memory */ +return vfio_save_data_device_memory (vdev, f); }static void vfio_pci_load_config (VFIOPCIDevice *vdev,QEMUFile *f)@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) len =qemu_get_be64 (f);vfio_load_data_device_config (vdev, f, len);break ; +case VFIO_SAVE_FLAG_DEVMEMORY : + len =qemu_get_be64 (f); +vfio_load_data_device_memory (vdev, f, len); +break ;default : ret = -EINVAL ; }@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)VFIOPCIDevice *vdev = opaque;int rc =0 ; +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY |VFIO_SAVE_FLAG_CONTINUE ); +/* get dirty data of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + } +qemu_put_byte (f,VFIO_SAVE_FLAG_PCI |VFIO_SAVE_FLAG_CONTINUE );vfio_pci_save_config (vdev, f);@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)static int vfio_save_setup (QEMUFile *f,void *opaque) { +int rc =0 ;VFIOPCIDevice *vdev = opaque; -qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + +if (vfio_device_data_cap_device_memory (vdev)) { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP |VFIO_SAVE_FLAG_CONTINUE ); +qemu_put_byte (f,VFIO_SAVE_FLAG_DEVMEMORY ); +/* get whole snapshot of device memory */ +vfio_get_device_memory_size (vdev); + rc =vfio_save_data_device_memory (vdev, f); + }else { +qemu_put_byte (f,VFIO_SAVE_FLAG_SETUP ); + }vfio_set_device_state (vdev,VFIO_DEVICE_STATE_RUNNING |VFIO_DEVICE_STATE_LOGGING ); -return 0 ; +return rc; }static int vfio_load_setup (QEMUFile *f,void *opaque)@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp) goto error; } -if (vfio_device_data_cap_device_memory (vdev)) { -error_report ("No suppport of data cap device memory Yet" ); +if (vfio_device_data_cap_device_memory (vdev) && +vfio_device_state_region_setup (vdev, + &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY ], +VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY , +"device-state-data-device-memory" )) { goto error; }hw/vfio/pci.h +1
-0@@ -69,6 +69,7 @@ typedef struct VFIOMigration {uint32_t data_caps;uint32_t device_state;uint64_t devconfig_size; +uint64_t devmem_size;VMChangeStateEntry *vm_state; } VFIOMigration;Signature · 6 lines
2.7.4 _______________________________________________ intel-gvt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Hi,
I've sent minor comments to later patches; but some minor general
comments:
a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?
c) Consider using the trace_ mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.
Dave
(P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'
Quoted 264 lines
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages(The reason why four seperate regions are defined is that the unit of mmap system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3struct vfio_device_state_ctl { __u32 version; /* ro */ __u32 device_state; /* VFIO device state, wo */ __u32 caps; /* ro */ struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /*rw*/ } device_config; struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /* rw */ __u64 pos; /*the offset in total buffer of device memory*/ } device_memory; struct { __u64 start_addr; /* wo */ __u64 page_nr; /* wo */ } system_memory; };Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Then it will probe VFIO device's data capability by reading caps field of control region.
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state -->VFIO_DEVICE_STATE_RUNNING&VFIO_DEVICE_STATE_LOGGING|MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->VFIO_DEVICE_STATE_STOPMIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETEDIn source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.In precopy phase, if a device has
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYon, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis off, .log_sync callback just returns without call to vendor driver.In stop-and-copy phase, device state will be set to STOP & LOGGING first. in
save_live_complete_precopycallback,
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability
Quoted 13 lines
hw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.cSignature · 1 line
2.7.4
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Quoted 6 lines
Hi,
I've sent minor comments to later patches; but some minor general
comments:a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.
hi Dave
Thanks for this suggestion. I'll add more checks for migration streams.
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?
Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
This patchset is intended to settle down the main device state interfaces for VFIO migration. So that we can work on that and improve it.
Quoted 5 lines
c) Consider using the
trace_mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.Dave
Got it. many thanks~~
Quoted 2 lines
(
P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'
sorry :)
Quoted 284 lines
Quoted 265 lines
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages(The reason why four seperate regions are defined is that the unit of mmap system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3struct vfio_device_state_ctl { __u32 version; /* ro */ __u32 device_state; /* VFIO device state, wo */ __u32 caps; /* ro */ struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /*rw*/ } device_config; struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /* rw */ __u64 pos; /*the offset in total buffer of device memory*/ } device_memory; struct { __u64 start_addr; /* wo */ __u64 page_nr; /* wo */ } system_memory; };Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Then it will probe VFIO device's data capability by reading caps field of control region.
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state -->VFIO_DEVICE_STATE_RUNNING&VFIO_DEVICE_STATE_LOGGING|MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->VFIO_DEVICE_STATE_STOPMIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETEDIn source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.In precopy phase, if a device has
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYon, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis off, .log_sync callback just returns without call to vendor driver.In stop-and-copy phase, device state will be set to STOP & LOGGING first. in
save_live_complete_precopycallback,
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capabilityQuoted 13 lines
hw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.cSignature · 1 line
2.7.4--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Quoted 34 lines · * Zhao Yan ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Quoted 6 lines
Hi,
I've sent minor comments to later patches; but some minor general
comments:a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.
hi Dave
Thanks for this suggestion. I'll add more checks for migration streams.Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
This patchset is intended to settle down the main device state interfaces for VFIO migration. So that we can work on that and improve it.Quoted 5 lines
c) Consider using the
trace_mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.Dave
Quoted 7 lines
Got it. many thanks~~
Quoted 2 lines
(
P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'sorry :)
No problem.
Given the mails, I'm guessing you've mostly tested this on graphics devices? Have you also checked with VFIO network cards?
Also see the mail I sent in reply to Kirti's series; we need to boil these down to one solution.
Dave
Quoted 278 lines
Quoted 278 lines
Quoted 278 lines
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages(The reason why four seperate regions are defined is that the unit of mmap system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3struct vfio_device_state_ctl { __u32 version; /* ro */ __u32 device_state; /* VFIO device state, wo */ __u32 caps; /* ro */ struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /*rw*/ } device_config; struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /* rw */ __u64 pos; /*the offset in total buffer of device memory*/ } device_memory; struct { __u64 start_addr; /* wo */ __u64 page_nr; /* wo */ } system_memory; };Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Then it will probe VFIO device's data capability by reading caps field of control region.
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state -->VFIO_DEVICE_STATE_RUNNING&VFIO_DEVICE_STATE_LOGGING|MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->VFIO_DEVICE_STATE_STOPMIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETEDIn source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.In precopy phase, if a device has
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYon, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis off, .log_sync callback just returns without call to vendor driver.In stop-and-copy phase, device state will be set to STOP & LOGGING first. in
save_live_complete_precopycallback,
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capabilityhw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.cSignature · 1 line
2.7.4
Quoted 6 lines
Quoted 6 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 16 lines
-----Original Message-----
From: Dr. David Alan Gilbert [mailto:[email protected]]
Sent: Wednesday, February 20, 2019 7:02 PM
To: Zhao Yan <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration
Quoted 26 lines
Quoted 25 lines · * Zhao Yan ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Quoted 6 lines
Hi,
I've sent minor comments to later patches; but some minor general
comments:a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.
hi Dave
Thanks for this suggestion. I'll add more checks for migration streams.Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Quoted 348 lines
Quoted 9 lines
This patchset is intended to settle down the main device state interfaces for VFIO migration. So that we can work on that and improve it.
Quoted 5 lines
c) Consider using the
trace_mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.Dave
Quoted 7 lines
Got it. many thanks~~
Quoted 2 lines
(
P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'sorry :)
No problem.
Given the mails, I'm guessing you've mostly tested this on graphics devices? Have you also checked with VFIO network cards?
Also see the mail I sent in reply to Kirti's series; we need to boil these down to one solution.
Dave
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.Quoted 1 line
Quoted 1 line
Quoted 1 line
Device Memory: device's internal memory, standalone and outside
system
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses systemmemory.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range ofsystem
Quoted 1 line
Quoted 1 line
Quoted 1 line
memory. This dirty bitmap is queried in pre-copy and
stop-and-copy
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by
ram's
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
live migration code.
The dirty bitmap of system memory is held in dirty bitmapregion.
Quoted 1 line
Quoted 1 line
Quoted 1 line
If system memory range is larger than that dirty bitmap region
can
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
hold, qemu will cut it into several chunks and get dirty bitmap in succession.
Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and anothertwo
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
optional regions if it plans to support device state management.
So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of systemmemory
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
dirty pages
(The reason why four seperate regions are defined is that the unit of
mmap
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one bigregion
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
padded and sparse mmaped).
kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u32 device_state; /* VFIO device state, wo */ __u32 caps; /* ro */Quoted 8 lines
Quoted 8 lines
struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /*rw*/ } device_config; struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /* rw */ __u64 pos; /*the offset in total buffer of devicememory*/
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
} device_memory; struct { __u64 start_addr; /* wo */ __u64 page_nr; /* wo */ } system_memory; };Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING &LOGGING,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, thenvendor
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They returnwhole
Quoted 1 line
Quoted 1 line
Quoted 1 line
snapshot outside LOGGING and dirty data since last get
operation
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
inside LOGGING.
Device config should be always accessible and return wholeconfig
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
snapshot regardless of LOGGING state.
Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Then it will probe VFIO device's data capability by reading caps field of control region.
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/loaddata of
Quoted 1 line
Quoted 1 line
Quoted 1 line
device memory in pre-copy and stop-and-copy phase. The data
of
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirtypages
Quoted 1 line
Quoted 1 line
Quoted 1 line
produced by VFIO device during pre-copy and stop-and-copy
phase.
Quoted 1 line
Quoted 1 line
Quoted 1 line
The dirty bitmap of system memory is held in dirty bitmap
region.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE
STATE)
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size) get device memory buffer (whole snapshot data) set device state -->VFIO_DEVICE_STATE_RUNNING&
VFIO_DEVICE_STATE_LOGGINGQuoted 9 lines
Quoted 9 lines
Quoted 9 lines
|
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGINGQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->
VFIO_DEVICE_STATE_STOPQuoted 11 lines
Quoted 11 lines
Quoted 11 lines
MIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE
STATE)
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device configsize,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETEDQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
In source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field ofcontrol
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.actionfields of
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopyphase
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.
In precopy phase, if a device has
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYon,Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
.log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (
GET_BITMAP) to "system_memory.start_addr","system_memory.page_nr", and
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
"system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis off, .log_sync callbackjust
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
returns without call to vendor driver.
In stop-and-copy phase, device state will be set to STOP & LOGGING first. in
save_live_complete_precopycallback,
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state willbe
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capabilityQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
hw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858++++++++++++++++++++++++++++++++++++++++++
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.c--
2.7.4Quoted 6 lines
Quoted 6 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 28 lines · On Wed, 20 Feb 2019 11:28:46 +0000 "Gonglei (Arei)" <[email protected]>…
Quoted 28 lines
-----Original Message-----
From: Dr. David Alan Gilbert [mailto:[email protected]]
Sent: Wednesday, February 20, 2019 7:02 PM
To: Zhao Yan <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 12 lines · * Zhao Yan ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed, lower->higher is the one direction that might make sense to support.
But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
Quoted 63 lines
-----Original Message-----
From: Cornelia Huck [mailto:[email protected]]
Sent: Wednesday, February 20, 2019 7:43 PM
To: Gonglei (Arei) <[email protected]>
Cc: Dr. David Alan Gilbert <[email protected]>; Zhao Yan <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 5 lines · On Wed, 20 Feb 2019 11:28:46 +0000 "Gonglei (Arei)" <[email protected]>…
Quoted 5 lines
-----Original Message-----
From: Dr. David Alan Gilbert [mailto:[email protected]]
Sent: Wednesday, February 20, 2019 7:02 PM
To: Zhao Yan <[email protected]>
Cc: [email protected]; [email protected]; [email protected];Quoted 8 lines
Quoted 8 lines
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];
Quoted 9 lines
Quoted 9 lines
Subject: Re: [PATCH 0/5] QEMU VFIO live migration
Quoted 6 lines · * Zhao Yan ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Yan Zhao ([email protected]) wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration
patches,
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
including device states, VFIO device state region layout, dirty bitmap's query.
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public
cloud
Quoted 1 line
providers. But we still think about the live migration between from the the
lower
Quoted 1 line
generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.
Quoted 8 lines
But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
We can provide an interface to query if the VM support live migration or not in prepare phase of Libvirt.
Can we get the revision_id from the vendor driver ? before invoking
register_savevm_live(NULL, TYPE_VFIO_PCI, -1,
revision_id,
&savevm_vfio_handlers,
vdev);then limit the live migration form higher gens to lower gens?
Regards,
-Gonglei
Quoted 14 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.
Quoted 8 lines
But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?
version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".
struct vfio_device_state_ctl {
__u32 version; /* ro */
__u8 device_version[MAX_DEVICE_VERSION_LEN]; /* ro */
struct {
__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
...
}data;
...};
Then, an action IS_COMPATIBLE is added to check device compatibility.
The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.
The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.
Thanks
Yan
Quoted 57 lines · * Zhao Yan ([email protected]) wrote:
Quoted 14 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.Quoted 8 lines
But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".struct vfio_device_state_ctl { __u32 version; /* ro */ __u8 device_version[MAX_DEVICE_VERSION_LEN]; /* ro */ struct { __u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/ ... }data; ...};
Then, an action
IS_COMPATIBLEis added to check device compatibility.The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device'sdevice_versionstring
2. in target side's .load_state, load source device's device version string and write it to data region, and callIS_COMPATIBLEaction to ask vendor driver to check whether the source device is compatible to it.
Quoted 6 lines
The advantage of adding an
IS_COMPATIBLEaction is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
Indevice_versionstring, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.
It would also be good if the 'IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.
Dave
Quoted 5 lines
Thanks
Yan
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
Quoted 7 lines
Quoted 7 lines
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.
Quoted 3 lines
Quoted 3 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.
Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)
For vendor specific regions.
Quoted 5 lines
Quoted 5 lines
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".
Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u8 device_version[MAX_DEVICE_VERSION_LEN]; /* ro */ struct { __u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/ ... }data; ...};
We have a buffer area where we can read and write data from the vendor driver, why would we have this IS_COMPATIBLE protocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's use GET_VERSION, CHECK_VERSION actions that make use of the buffer area and allow vendor specific version information length.
Quoted 10 lines
Quoted 10 lines
Then, an action
IS_COMPATIBLEis added to check device compatibility.The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device'sdevice_versionstring
2. in target side's .load_state, load source device's device version string and write it to data region, and callIS_COMPATIBLEaction to ask vendor driver to check whether the source device is compatible to it.
Quoted 6 lines
Quoted 6 lines
The advantage of adding an
IS_COMPATIBLEaction is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
Indevice_versionstring, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.
I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.
Quoted 4 lines
It would also be good if the '
IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.
I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex
hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.
Quoted 31 lines · Alex Williamson
Quoted 28 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 27 lines · * Zhao Yan ([email protected]) wrote:
Quoted 14 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.Quoted 8 lines
But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct
vfio_info_cap_headerincludes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.
yes, good idea!
so, what about declaring below new cap?
#define VFIO_REGION_INFO_CAP_MIGRATION 4
struct vfio_region_info_cap_migration {
struct vfio_info_cap_header header;
__u32 device_version_len;
__u8 device_version[];
};Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.
Quoted 8 lines
Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
#define
VFIO_REGION_TYPE_PCI_VENDOR_TYPE(1 << 31)
#defineVFIO_REGION_TYPE_PCI_VENDOR_MASK(0xffff)For vendor specific regions.
Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what is VFIO_REGION_TYPE_PCI_VENDOR_MASK used for? why it's 0xffff? I searched QEMU and kernel code and did not find anywhere uses it.
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".
Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u8 device_version[MAX_DEVICE_VERSION_LEN]; /* ro */ struct { __u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/ ... }data; ...};
Quoted 6 lines
We have a buffer area where we can read and write data from the vendor driver, why would we have this
IS_COMPATIBLEprotocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's useGET_VERSION,CHECK_VERSIONactions that make use of the buffer area and allow vendor specific version information length.
you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. use GET_VERSION action, and read from region buffer
2. get it when querying cap VFIO_REGION_INFO_CAP_MIGRATION
seems approach 1 is better, and cap VFIO_REGION_INFO_CAP_MIGRATION is only for checking migration interface's version?
Quoted 17 lines
Quoted 10 lines
Quoted 10 lines
Then, an action
IS_COMPATIBLEis added to check device compatibility.The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device'sdevice_versionstring
2. in target side's .load_state, load source device's device version string and write it to data region, and callIS_COMPATIBLEaction to ask vendor driver to check whether the source device is compatible to it.Quoted 6 lines
Quoted 6 lines
The advantage of adding an
IS_COMPATIBLEaction is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
Indevice_versionstring, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.
Quoted 3 lines
I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.
:)
Quoted 4 lines
Quoted 4 lines
It would also be good if the '
IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.
so qemu needs to expose two qmp/sysfs interfaces: GET_VERSION and CHECK_VERSION. GET_VERSION returns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?
Quoted 8 lines
I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex
do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version
Thanks
Yan
Quoted 145 lines · Zhao Yan
hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.Quoted 31 lines · Alex Williamson
Quoted 28 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 27 lines · * Zhao Yan ([email protected]) wrote:
Quoted 14 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.Quoted 8 lines
But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct
vfio_info_cap_headerincludes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.yes, good idea!
so, what about declaring below new cap?#define VFIO_REGION_INFO_CAP_MIGRATION 4 struct vfio_region_info_cap_migration { struct vfio_info_cap_header header; __u32 device_version_len; __u8 device_version[]; };Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.Quoted 8 lines
Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
#define
VFIO_REGION_TYPE_PCI_VENDOR_TYPE(1 << 31)
#defineVFIO_REGION_TYPE_PCI_VENDOR_MASK(0xffff)For vendor specific regions.
Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what isVFIO_REGION_TYPE_PCI_VENDOR_MASKused for? why it's0xffff? I searched QEMU and kernel code and did not find anywhere uses it.Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".
Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u8 device_version[MAX_DEVICE_VERSION_LEN]; /* ro */ struct { __u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/ ... }data; ...};
Quoted 6 lines
We have a buffer area where we can read and write data from the vendor driver, why would we have this
IS_COMPATIBLEprotocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's useGET_VERSION,CHECK_VERSIONactions that make use of the buffer area and allow vendor specific version information length.you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. useGET_VERSIONaction, and read from region buffer
2. get it when querying capVFIO_REGION_INFO_CAP_MIGRATIONseems approach 1 is better, and cap
VFIO_REGION_INFO_CAP_MIGRATIONis only for checking migration interface's version?Quoted 17 lines
Quoted 10 lines
Quoted 10 lines
Then, an action
IS_COMPATIBLEis added to check device compatibility.The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device'sdevice_versionstring
2. in target side's .load_state, load source device's device version string and write it to data region, and callIS_COMPATIBLEaction to ask vendor driver to check whether the source device is compatible to it.Quoted 6 lines
Quoted 6 lines
The advantage of adding an
IS_COMPATIBLEaction is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
Indevice_versionstring, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.Quoted 3 lines
I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.
:)
Quoted 4 lines
Quoted 4 lines
It would also be good if the '
IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.so qemu needs to expose two qmp/sysfs interfaces:
GET_VERSIONandCHECK_VERSION.GET_VERSIONreturns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?Quoted 8 lines
I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex
Quoted 4 lines
do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version
Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?
Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.
On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.
So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.
Regards,
Erik
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan
hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
Quoted 7 lines
Quoted 7 lines
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct
vfio_info_cap_headerincludes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.
Quoted 8 lines
Quoted 8 lines
yes, good idea!
so, what about declaring below new cap?#define VFIO_REGION_INFO_CAP_MIGRATION 4 struct vfio_region_info_cap_migration { struct vfio_info_cap_header header; __u32 device_version_len; __u8 device_version[]; };
I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
#define
VFIO_REGION_TYPE_PCI_VENDOR_TYPE(1 << 31)
#defineVFIO_REGION_TYPE_PCI_VENDOR_MASK(0xffff)
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
For vendor specific regions.
Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what isVFIO_REGION_TYPE_PCI_VENDOR_MASKused for? why it's0xffff? I searched QEMU and kernel code and did not find anywhere uses it.
PCI vendor IDs are 16bits, it's just indicating that when the PCI_VENDOR_TYPE bit is set the valid data is the lower 16bits.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 5 lines
Quoted 5 lines
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".
Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u8 device_version[MAX_DEVICE_VERSION_LEN]; /* ro */ struct { __u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/ ... }data; ...};
We have a buffer area where we can read and write data from the vendor driver, why would we have this
IS_COMPATIBLEprotocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's useGET_VERSION,CHECK_VERSIONactions that make use of the buffer area and allow vendor specific version information length.
Quoted 7 lines
Quoted 7 lines
you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. useGET_VERSIONaction, and read from region buffer
2. get it when querying capVFIO_REGION_INFO_CAP_MIGRATIONseems approach 1 is better, and cap
VFIO_REGION_INFO_CAP_MIGRATIONis only for checking migration interface's version?
I think 1 provides the most flexibility to the vendor driver.
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Then, an action
IS_COMPATIBLEis added to check device compatibility.The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device'sdevice_versionstring
2. in target side's .load_state, load source device's device version string and write it to data region, and callIS_COMPATIBLEaction to ask vendor driver to check whether the source device is compatible to it.Quoted 6 lines
Quoted 6 lines
The advantage of adding an
IS_COMPATIBLEaction is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
Indevice_versionstring, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.Quoted 3 lines
I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.
:)
Quoted 4 lines
Quoted 4 lines
It would also be good if the '
IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.
Quoted 5 lines
Quoted 5 lines
so qemu needs to expose two qmp/sysfs interfaces:
GET_VERSIONandCHECK_VERSION.GET_VERSIONreturns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?
That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex
Quoted 4 lines
Quoted 4 lines
do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version
I'd think it might live more in the mdev_support_types area, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:
/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version
Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.
Quoted 16 lines
Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?
Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.
Quoted 8 lines
On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.
So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.
Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,
Alex
Quoted 57 lines · Alex Williamson
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan
hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
Quoted 7 lines
Quoted 7 lines
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct
vfio_info_cap_headerincludes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.Quoted 8 lines
Quoted 8 lines
yes, good idea!
so, what about declaring below new cap?#define VFIO_REGION_INFO_CAP_MIGRATION 4 struct vfio_region_info_cap_migration { struct vfio_info_cap_header header; __u32 device_version_len; __u8 device_version[]; };
Quoted 4 lines
I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.
so, to get version of VFIO live migration device state interface (simply call it migration interface?),
a new cap looks like this:
#define VFIO_REGION_INFO_CAP_MIGRATION 4
it contains struct vfio_info_cap_header only.
when get region info of the migration region, we query this cap and get migration interface's version. right?
or just directly use VFIO_REGION_INFO_CAP_TYPE is fine?
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
#define
VFIO_REGION_TYPE_PCI_VENDOR_TYPE(1 << 31)
#defineVFIO_REGION_TYPE_PCI_VENDOR_MASK(0xffff)Quoted 8 lines
Quoted 8 lines
Quoted 1 line
For vendor specific regions.
Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what isVFIO_REGION_TYPE_PCI_VENDOR_MASKused for? why it's0xffff? I searched QEMU and kernel code and did not find anywhere uses it.
Quoted 2 lines
PCI vendor IDs are 16bits, it's just indicating that when the
PCI_VENDOR_TYPEbit is set the valid data is the lower 16bits.
thanks:)
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".
Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u8 device_version[MAX_DEVICE_VERSION_LEN]; /* ro */ struct { __u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/ ... }data; ...};
Quoted 6 lines
We have a buffer area where we can read and write data from the vendor driver, why would we have this
IS_COMPATIBLEprotocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's useGET_VERSION,CHECK_VERSIONactions that make use of the buffer area and allow vendor specific version information length.you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. useGET_VERSIONaction, and read from region buffer
2. get it when querying capVFIO_REGION_INFO_CAP_MIGRATION
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
seems approach 1 is better, and cap
VFIO_REGION_INFO_CAP_MIGRATIONis only for checking migration interface's version?I think 1 provides the most flexibility to the vendor driver.
Got it.
For VFIO live migration, compared to reuse device state region (which takes GET_BUFFER/SET_BUFFER actions),
could we create a new region for GET_VERSION & CHECK_VERSION ?
Quoted 49 lines
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Then, an action
IS_COMPATIBLEis added to check device compatibility.The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device'sdevice_versionstring
2. in target side's .load_state, load source device's device version string and write it to data region, and callIS_COMPATIBLEaction to ask vendor driver to check whether the source device is compatible to it.Quoted 6 lines
Quoted 6 lines
The advantage of adding an
IS_COMPATIBLEaction is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
Indevice_versionstring, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.Quoted 3 lines
I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.
:)
Quoted 4 lines
Quoted 4 lines
It would also be good if the '
IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.Quoted 5 lines
Quoted 5 lines
so qemu needs to expose two qmp/sysfs interfaces:
GET_VERSIONandCHECK_VERSION.GET_VERSIONreturns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
AlexQuoted 4 lines
Quoted 4 lines
do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version
Quoted 8 lines
I'd think it might live more in the
mdev_support_typesarea, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version
Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.
yes, knowing if a device is compatible before it's created is good. but do you think check whether a device is compatible after it's created is also required? For live migration, user usually only queries this information when it's really required, i.e. when a device has been created. maybe we can add this version get/check at both places?
Quoted 26 lines
Quoted 17 lines
Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?
Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.Quoted 8 lines
On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.
So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.
Quoted 11 lines
Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,
So QEMU vfio uses sysfs to check device compatiblity in migration's save_setup phase?
Thanks
Yan
Quoted 70 lines · On Thu, 28 Mar 2019 22:47:04 -0400 Zhao Yan <[email protected]> wrote:
Quoted 57 lines · Alex Williamson
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan
hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
Quoted 7 lines
Quoted 7 lines
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct
vfio_info_cap_headerincludes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.Quoted 8 lines
Quoted 8 lines
yes, good idea!
so, what about declaring below new cap?#define VFIO_REGION_INFO_CAP_MIGRATION 4 struct vfio_region_info_cap_migration { struct vfio_info_cap_header header; __u32 device_version_len; __u8 device_version[]; };Quoted 4 lines
I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.
so, to get version of VFIO live migration device state interface (simply call it migration interface?),
a new cap looks like this:
#defineVFIO_REGION_INFO_CAP_MIGRATION4
it contains structvfio_info_cap_headeronly.
when get region info of the migration region, we query this cap and get migration interface's version. right?
Quoted 1 line
or just directly use
VFIO_REGION_INFO_CAP_TYPEis fine?
Again, why a new region. I'm imagining we have one region and this is just asking for a slightly different thing from it. But TBH, I'm not sure we need it at all vs the sysfs interface.
Quoted 53 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
#define
VFIO_REGION_TYPE_PCI_VENDOR_TYPE(1 << 31)
#defineVFIO_REGION_TYPE_PCI_VENDOR_MASK(0xffff)Quoted 8 lines
Quoted 8 lines
Quoted 1 line
For vendor specific regions.
Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what isVFIO_REGION_TYPE_PCI_VENDOR_MASKused for? why it's0xffff? I searched QEMU and kernel code and did not find anywhere uses it.Quoted 2 lines
PCI vendor IDs are 16bits, it's just indicating that when the
PCI_VENDOR_TYPEbit is set the valid data is the lower 16bits.thanks:)
Quoted 29 lines
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".
Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u8 device_version[MAX_DEVICE_VERSION_LEN]; /* ro */ struct { __u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/ ... }data; ...};
Quoted 6 lines
We have a buffer area where we can read and write data from the vendor driver, why would we have this
IS_COMPATIBLEprotocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's useGET_VERSION,CHECK_VERSIONactions that make use of the buffer area and allow vendor specific version information length.you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. useGET_VERSIONaction, and read from region buffer
2. get it when querying capVFIO_REGION_INFO_CAP_MIGRATIONseems approach 1 is better, and cap
VFIO_REGION_INFO_CAP_MIGRATIONis only for checking migration interface's version?
Quoted 6 lines
Quoted 1 line
I think 1 provides the most flexibility to the vendor driver.
Got it.
For VFIO live migration, compared to reuse device state region (which takes GET_BUFFER/SET_BUFFER actions),
could we create a new region forGET_VERSION&CHECK_VERSION?
Why?
Quoted 55 lines
Quoted 55 lines
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Then, an action
IS_COMPATIBLEis added to check device compatibility.The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device'sdevice_versionstring
2. in target side's .load_state, load source device's device version string and write it to data region, and callIS_COMPATIBLEaction to ask vendor driver to check whether the source device is compatible to it.Quoted 6 lines
Quoted 6 lines
The advantage of adding an
IS_COMPATIBLEaction is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
Indevice_versionstring, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.Quoted 3 lines
I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.
:)
Quoted 4 lines
Quoted 4 lines
It would also be good if the '
IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.Quoted 5 lines
Quoted 5 lines
so qemu needs to expose two qmp/sysfs interfaces:
GET_VERSIONandCHECK_VERSION.GET_VERSIONreturns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
AlexQuoted 4 lines
Quoted 4 lines
do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_versionI'd think it might live more in the
mdev_support_typesarea, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version
Quoted 7 lines
Quoted 2 lines
Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.
yes, knowing if a device is compatible before it's created is good. but do you think check whether a device is compatible after it's created is also required? For live migration, user usually only queries this information when it's really required, i.e. when a device has been created. maybe we can add this version get/check at both places?
Why does an instantiated device suddenly not follow the version and compatibility rules of an uninstantiated device? IOW, if the version and compatibility check are on the mdev type, why can't we trace back from the device to the mdev type and make use of that same interface? Seems the only question is whether we require an interface through the vfio API directly or if sysfs is sufficient.
Quoted 36 lines
Quoted 25 lines
Quoted 16 lines
Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?
Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.Quoted 8 lines
On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.
So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.
Quoted 11 lines
Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,
Quoted 2 lines
So QEMU vfio uses sysfs to check device compatiblity in migration's
save_setupphase?
The migration stream between source and target device are the ultimate test of compatibility, the vendor driver should never rely on userspace validating compatibility of the migration. At the point it could do so, the migration has already begun, so we're only testing how quickly we can fail the migration. The management layer setting up the migration can test via sysfs for compatibility and the migration stream itself needs to be self validating, so what value is added for QEMU to perform a version compatibility test? Thanks,
Alex
Quoted 204 lines · Alex Williamson
Quoted 70 lines · On Thu, 28 Mar 2019 22:47:04 -0400 Zhao Yan <[email protected]> wrote:
Quoted 57 lines · Alex Williamson
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan
hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps
a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
Quoted 7 lines
Quoted 7 lines
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct
vfio_info_cap_headerincludes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.Quoted 8 lines
Quoted 8 lines
yes, good idea!
so, what about declaring below new cap?#define VFIO_REGION_INFO_CAP_MIGRATION 4 struct vfio_region_info_cap_migration { struct vfio_info_cap_header header; __u32 device_version_len; __u8 device_version[]; };Quoted 4 lines
I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.
so, to get version of VFIO live migration device state interface (simply call it migration interface?),
a new cap looks like this:
#defineVFIO_REGION_INFO_CAP_MIGRATION4
it contains structvfio_info_cap_headeronly.
when get region info of the migration region, we query this cap and get migration interface's version. right?Quoted 1 line
or just directly use
VFIO_REGION_INFO_CAP_TYPEis fine?Again, why a new region. I'm imagining we have one region and this is just asking for a slightly different thing from it. But TBH, I'm not sure we need it at all vs the sysfs interface.
Quoted 53 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
#define
VFIO_REGION_TYPE_PCI_VENDOR_TYPE(1 << 31)
#defineVFIO_REGION_TYPE_PCI_VENDOR_MASK(0xffff)Quoted 8 lines
Quoted 8 lines
Quoted 1 line
For vendor specific regions.
Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what isVFIO_REGION_TYPE_PCI_VENDOR_MASKused for? why it's0xffff? I searched QEMU and kernel code and did not find anywhere uses it.Quoted 2 lines
PCI vendor IDs are 16bits, it's just indicating that when the
PCI_VENDOR_TYPEbit is set the valid data is the lower 16bits.thanks:)
Quoted 29 lines
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".
Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u8 device_version[MAX_DEVICE_VERSION_LEN]; /* ro */ struct { __u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/ ... }data; ...};
Quoted 6 lines
We have a buffer area where we can read and write data from the vendor driver, why would we have this
IS_COMPATIBLEprotocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's useGET_VERSION,CHECK_VERSIONactions that make use of the buffer area and allow vendor specific version information length.you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. useGET_VERSIONaction, and read from region buffer
2. get it when querying capVFIO_REGION_INFO_CAP_MIGRATIONseems approach 1 is better, and cap
VFIO_REGION_INFO_CAP_MIGRATIONis only for checking migration interface's version?Quoted 6 lines
Quoted 1 line
I think 1 provides the most flexibility to the vendor driver.
Got it.
For VFIO live migration, compared to reuse device state region (which takes GET_BUFFER/SET_BUFFER actions),
could we create a new region forGET_VERSION&CHECK_VERSION?Why?
Quoted 55 lines
Quoted 55 lines
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Then, an action
IS_COMPATIBLEis added to check device compatibility.The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device'sdevice_versionstring
2. in target side's .load_state, load source device's device version string and write it to data region, and callIS_COMPATIBLEaction to ask vendor driver to check whether the source device is compatible to it.Quoted 6 lines
Quoted 6 lines
The advantage of adding an
IS_COMPATIBLEaction is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
Indevice_versionstring, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.Quoted 3 lines
I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.
:)
Quoted 4 lines
Quoted 4 lines
It would also be good if the '
IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.Quoted 5 lines
Quoted 5 lines
so qemu needs to expose two qmp/sysfs interfaces:
GET_VERSIONandCHECK_VERSION.GET_VERSIONreturns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
AlexQuoted 4 lines
Quoted 4 lines
do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_versionI'd think it might live more in the
mdev_support_typesarea, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version
Quoted 7 lines
Quoted 2 lines
Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.
yes, knowing if a device is compatible before it's created is good. but do you think check whether a device is compatible after it's created is also required? For live migration, user usually only queries this information when it's really required, i.e. when a device has been created. maybe we can add this version get/check at both places?
Quoted 6 lines
Why does an instantiated device suddenly not follow the version and compatibility rules of an uninstantiated device? IOW, if the version and compatibility check are on the mdev type, why can't we trace back from the device to the mdev type and make use of that same interface? Seems the only question is whether we require an interface through the vfio API directly or if sysfs is sufficient.
ok. got it.
Quoted 39 lines
Quoted 36 lines
Quoted 25 lines
Quoted 16 lines
Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?
Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.Quoted 8 lines
On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.
So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.
Quoted 11 lines
Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,
Quoted 2 lines
So QEMU vfio uses sysfs to check device compatiblity in migration's
save_setupphase?
Quoted 8 lines
The migration stream between source and target device are the ultimate test of compatibility, the vendor driver should never rely on userspace validating compatibility of the migration. At the point it could do so, the migration has already begun, so we're only testing how quickly we can fail the migration. The management layer setting up the migration can test via sysfs for compatibility and the migration stream itself needs to be self validating, so what value is added for QEMU to perform a version compatibility test? Thanks,
oh, do you mean vendor driver should embed source device's version in migration stream, which is opaque to qemu?
otherwise, I can't think of a quick way for vendor driver to determine whether source device is an incompatible device.
Thanks
Yan
Quoted 260 lines · On Fri, 29 Mar 2019 19:10:50 -0400 Zhao Yan <[email protected]> wrote:
Quoted 204 lines · Alex Williamson
Quoted 70 lines · On Thu, 28 Mar 2019 22:47:04 -0400 Zhao Yan <[email protected]> wrote:
Quoted 57 lines · Alex Williamson
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan
hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps
a device that has less device memory ?
Actually it's still an open for VFIO migration. Need to think about
whether it's better to check that in libvirt or qemu (like a device magic
along with verion ?).We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
Quoted 7 lines
Quoted 7 lines
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct
vfio_info_cap_headerincludes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.Quoted 8 lines
Quoted 8 lines
yes, good idea!
so, what about declaring below new cap?#define VFIO_REGION_INFO_CAP_MIGRATION 4 struct vfio_region_info_cap_migration { struct vfio_info_cap_header header; __u32 device_version_len; __u8 device_version[]; };Quoted 4 lines
I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.
so, to get version of VFIO live migration device state interface (simply call it migration interface?),
a new cap looks like this:
#defineVFIO_REGION_INFO_CAP_MIGRATION4
it contains structvfio_info_cap_headeronly.
when get region info of the migration region, we query this cap and get migration interface's version. right?Quoted 1 line
or just directly use
VFIO_REGION_INFO_CAP_TYPEis fine?Again, why a new region. I'm imagining we have one region and this is just asking for a slightly different thing from it. But TBH, I'm not sure we need it at all vs the sysfs interface.
Quoted 53 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
#define
VFIO_REGION_TYPE_PCI_VENDOR_TYPE(1 << 31)
#defineVFIO_REGION_TYPE_PCI_VENDOR_MASK(0xffff)Quoted 8 lines
Quoted 8 lines
Quoted 1 line
For vendor specific regions.
Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what isVFIO_REGION_TYPE_PCI_VENDOR_MASKused for? why it's0xffff? I searched QEMU and kernel code and did not find anywhere uses it.Quoted 2 lines
PCI vendor IDs are 16bits, it's just indicating that when the
PCI_VENDOR_TYPEbit is set the valid data is the lower 16bits.thanks:)
Quoted 29 lines
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".
Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u8 device_version[MAX_DEVICE_VERSION_LEN]; /* ro */ struct { __u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/ ... }data; ...};
Quoted 6 lines
We have a buffer area where we can read and write data from the vendor driver, why would we have this
IS_COMPATIBLEprotocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's useGET_VERSION,CHECK_VERSIONactions that make use of the buffer area and allow vendor specific version information length.you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. useGET_VERSIONaction, and read from region buffer
2. get it when querying capVFIO_REGION_INFO_CAP_MIGRATIONseems approach 1 is better, and cap
VFIO_REGION_INFO_CAP_MIGRATIONis only for checking migration interface's version?Quoted 6 lines
Quoted 1 line
I think 1 provides the most flexibility to the vendor driver.
Got it.
For VFIO live migration, compared to reuse device state region (which takes GET_BUFFER/SET_BUFFER actions),
could we create a new region forGET_VERSION&CHECK_VERSION?Why?
Quoted 55 lines
Quoted 55 lines
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Then, an action
IS_COMPATIBLEis added to check device compatibility.The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device'sdevice_versionstring
2. in target side's .load_state, load source device's device version string and write it to data region, and callIS_COMPATIBLEaction to ask vendor driver to check whether the source device is compatible to it.Quoted 6 lines
Quoted 6 lines
The advantage of adding an
IS_COMPATIBLEaction is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
Indevice_versionstring, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.Quoted 3 lines
I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.
:)
Quoted 4 lines
Quoted 4 lines
It would also be good if the '
IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.Quoted 5 lines
Quoted 5 lines
so qemu needs to expose two qmp/sysfs interfaces:
GET_VERSIONandCHECK_VERSION.GET_VERSIONreturns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
AlexQuoted 4 lines
Quoted 4 lines
do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_versionI'd think it might live more in the
mdev_support_typesarea, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version
Quoted 7 lines
Quoted 2 lines
Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.
yes, knowing if a device is compatible before it's created is good. but do you think check whether a device is compatible after it's created is also required? For live migration, user usually only queries this information when it's really required, i.e. when a device has been created. maybe we can add this version get/check at both places?
Quoted 6 lines
Why does an instantiated device suddenly not follow the version and compatibility rules of an uninstantiated device? IOW, if the version and compatibility check are on the mdev type, why can't we trace back from the device to the mdev type and make use of that same interface? Seems the only question is whether we require an interface through the vfio API directly or if sysfs is sufficient.
ok. got it.
Quoted 39 lines
Quoted 36 lines
Quoted 25 lines
Quoted 16 lines
Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?
Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.Quoted 8 lines
On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.
So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.
Quoted 11 lines
Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,
Quoted 2 lines
So QEMU vfio uses sysfs to check device compatiblity in migration's
save_setupphase?Quoted 8 lines
The migration stream between source and target device are the ultimate test of compatibility, the vendor driver should never rely on userspace validating compatibility of the migration. At the point it could do so, the migration has already begun, so we're only testing how quickly we can fail the migration. The management layer setting up the migration can test via sysfs for compatibility and the migration stream itself needs to be self validating, so what value is added for QEMU to perform a version compatibility test? Thanks,
Quoted 4 lines
oh, do you mean vendor driver should embed source device's version in migration stream, which is opaque to qemu?
otherwise, I can't think of a quick way for vendor driver to determine whether source device is an incompatible device.
Yes, the vendor driver cannot rely on the user to make sure the incoming migration stream is compatible, the vendor driver must take responsibility for this. Therefore, regardless of what other interfaces we have for the user to test the compatibility between devices, the vendor driver must make no assumptions about the validity or integrity of the data stream. Plan for and protect against a malicious or incompetent user. Thanks,
Alex
Quoted 268 lines · Alex Williamson
Quoted 260 lines · On Fri, 29 Mar 2019 19:10:50 -0400 Zhao Yan <[email protected]> wrote:
Quoted 204 lines · Alex Williamson
Quoted 70 lines · On Thu, 28 Mar 2019 22:47:04 -0400 Zhao Yan <[email protected]> wrote:
Quoted 57 lines · Alex Williamson
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan
hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps
a device that has less device memory ?
Actually it's still an open for VFIO migration. Need to think about
whether it's better to check that in libvirt or qemu (like a device magic
along with verion ?).We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
Quoted 7 lines
Quoted 7 lines
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct
vfio_info_cap_headerincludes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.Quoted 8 lines
Quoted 8 lines
yes, good idea!
so, what about declaring below new cap?#define VFIO_REGION_INFO_CAP_MIGRATION 4 struct vfio_region_info_cap_migration { struct vfio_info_cap_header header; __u32 device_version_len; __u8 device_version[]; };Quoted 4 lines
I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.
so, to get version of VFIO live migration device state interface (simply call it migration interface?),
a new cap looks like this:
#defineVFIO_REGION_INFO_CAP_MIGRATION4
it contains structvfio_info_cap_headeronly.
when get region info of the migration region, we query this cap and get migration interface's version. right?Quoted 1 line
or just directly use
VFIO_REGION_INFO_CAP_TYPEis fine?Again, why a new region. I'm imagining we have one region and this is just asking for a slightly different thing from it. But TBH, I'm not sure we need it at all vs the sysfs interface.
Quoted 53 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
#define
VFIO_REGION_TYPE_PCI_VENDOR_TYPE(1 << 31)
#defineVFIO_REGION_TYPE_PCI_VENDOR_MASK(0xffff)Quoted 8 lines
Quoted 8 lines
Quoted 1 line
For vendor specific regions.
Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what isVFIO_REGION_TYPE_PCI_VENDOR_MASKused for? why it's0xffff? I searched QEMU and kernel code and did not find anywhere uses it.Quoted 2 lines
PCI vendor IDs are 16bits, it's just indicating that when the
PCI_VENDOR_TYPEbit is set the valid data is the lower 16bits.thanks:)
Quoted 29 lines
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".
Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u8 device_version[MAX_DEVICE_VERSION_LEN]; /* ro */ struct { __u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/ ... }data; ...};
Quoted 6 lines
We have a buffer area where we can read and write data from the vendor driver, why would we have this
IS_COMPATIBLEprotocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's useGET_VERSION,CHECK_VERSIONactions that make use of the buffer area and allow vendor specific version information length.you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. useGET_VERSIONaction, and read from region buffer
2. get it when querying capVFIO_REGION_INFO_CAP_MIGRATIONseems approach 1 is better, and cap
VFIO_REGION_INFO_CAP_MIGRATIONis only for checking migration interface's version?Quoted 6 lines
Quoted 1 line
I think 1 provides the most flexibility to the vendor driver.
Got it.
For VFIO live migration, compared to reuse device state region (which takes GET_BUFFER/SET_BUFFER actions),
could we create a new region forGET_VERSION&CHECK_VERSION?Why?
Quoted 55 lines
Quoted 55 lines
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Then, an action
IS_COMPATIBLEis added to check device compatibility.The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device'sdevice_versionstring
2. in target side's .load_state, load source device's device version string and write it to data region, and callIS_COMPATIBLEaction to ask vendor driver to check whether the source device is compatible to it.Quoted 6 lines
Quoted 6 lines
The advantage of adding an
IS_COMPATIBLEaction is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
Indevice_versionstring, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.Quoted 3 lines
I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.
:)
Quoted 4 lines
Quoted 4 lines
It would also be good if the '
IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.Quoted 5 lines
Quoted 5 lines
so qemu needs to expose two qmp/sysfs interfaces:
GET_VERSIONandCHECK_VERSION.GET_VERSIONreturns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
AlexQuoted 4 lines
Quoted 4 lines
do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_versionI'd think it might live more in the
mdev_support_typesarea, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version
Quoted 7 lines
Quoted 2 lines
Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.
yes, knowing if a device is compatible before it's created is good. but do you think check whether a device is compatible after it's created is also required? For live migration, user usually only queries this information when it's really required, i.e. when a device has been created. maybe we can add this version get/check at both places?
Quoted 6 lines
Why does an instantiated device suddenly not follow the version and compatibility rules of an uninstantiated device? IOW, if the version and compatibility check are on the mdev type, why can't we trace back from the device to the mdev type and make use of that same interface? Seems the only question is whether we require an interface through the vfio API directly or if sysfs is sufficient.
ok. got it.
Quoted 39 lines
Quoted 36 lines
Quoted 25 lines
Quoted 16 lines
Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?
Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.Quoted 8 lines
On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.
So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.
Quoted 11 lines
Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,
Quoted 2 lines
So QEMU vfio uses sysfs to check device compatiblity in migration's
save_setupphase?Quoted 8 lines
The migration stream between source and target device are the ultimate test of compatibility, the vendor driver should never rely on userspace validating compatibility of the migration. At the point it could do so, the migration has already begun, so we're only testing how quickly we can fail the migration. The management layer setting up the migration can test via sysfs for compatibility and the migration stream itself needs to be self validating, so what value is added for QEMU to perform a version compatibility test? Thanks,
Quoted 4 lines
oh, do you mean vendor driver should embed source device's version in migration stream, which is opaque to qemu?
otherwise, I can't think of a quick way for vendor driver to determine whether source device is an incompatible device.
Quoted 8 lines
Yes, the vendor driver cannot rely on the user to make sure the incoming migration stream is compatible, the vendor driver must take responsibility for this. Therefore, regardless of what other interfaces we have for the user to test the compatibility between devices, the vendor driver must make no assumptions about the validity or integrity of the data stream. Plan for and protect against a malicious or incompetent user. Thanks,
ok. got it.
Thank you :)
Yan
Quoted 44 lines · On Wed, 27 Mar 2019 16:10:20 -0600 Alex Williamson…
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
Quoted 7 lines
Quoted 7 lines
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct
vfio_info_cap_headerincludes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.Quoted 3 lines
Quoted 3 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.
Quoted 8 lines
Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
#define
VFIO_REGION_TYPE_PCI_VENDOR_TYPE(1 << 31)
#defineVFIO_REGION_TYPE_PCI_VENDOR_MASK(0xffff)For vendor specific regions.
Just browsing through the thread... if I don't misunderstand, we could use a vfio-ccw region type id here for ccw, couldn't we? Just to make sure that this is not pci-specific.
Quoted 51 lines · Cornelia Huck
Quoted 48 lines · On Wed, 27 Mar 2019 16:10:20 -0600 Alex Williamson…
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
Quoted 7 lines
Quoted 7 lines
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct
vfio_info_cap_headerincludes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.Quoted 3 lines
Quoted 3 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
Quoted 8 lines
Quoted 4 lines
#define
VFIO_REGION_TYPE_PCI_VENDOR_TYPE(1 << 31)
#defineVFIO_REGION_TYPE_PCI_VENDOR_MASK(0xffff)For vendor specific regions.
Just browsing through the thread... if I don't misunderstand, we could use a vfio-ccw region type id here for ccw, couldn't we? Just to make sure that this is not pci-specific.
CCW could use another bit other than bit 31?
e.g.
#define VFIO_REGION_TYPE_CCW_VENDOR_TYPE (1 << 30)
then ccw device use (VFIO_REGION_TYPE_CCW_VENDOR_TYPE | vendor id) as its first 32 bit for device version string.
But as Alex said we'll not provide an extra region to get device version, and device version is only exported in sysfs, probably we should define them as below:
#define VFIO_DEVICE_VERSION_TYPE_PCI (1<<31)
#define VFIO_DEVICE_VERSION_TYPE_CCW (1<<30)
Do you think it's ok?
Thanks
Yan
Quoted 66 lines · On Mon, 1 Apr 2019 04:40:03 -0400 Yan Zhao <[email protected]> wrote:
Quoted 51 lines · Cornelia Huck
Quoted 48 lines · On Wed, 27 Mar 2019 16:10:20 -0600 Alex Williamson…
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.
Agreed,
lower->higheris the one direction that might make sense to support.But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.
How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?
Quoted 7 lines
Quoted 7 lines
hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.
Don't we essentially already have this via the device specific region? The struct
vfio_info_cap_headerincludes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.Quoted 3 lines
Quoted 3 lines
device_versionfield consists two parts:
1. vendor id : it takes 32 bits. e.g.0x8086.Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:
Quoted 8 lines
Quoted 4 lines
#define
VFIO_REGION_TYPE_PCI_VENDOR_TYPE(1 << 31)
#defineVFIO_REGION_TYPE_PCI_VENDOR_MASK(0xffff)For vendor specific regions.
Just browsing through the thread... if I don't misunderstand, we could use a vfio-ccw region type id here for ccw, couldn't we? Just to make sure that this is not pci-specific.
CCW could use another bit other than bit 31?
e.g.
#defineVFIO_REGION_TYPE_CCW_VENDOR_TYPE(1 << 30)
then ccw device use (VFIO_REGION_TYPE_CCW_VENDOR_TYPE| vendor id) as its first 32 bit for device version string.
Quoted 7 lines
But as Alex said we'll not provide an extra region to get device version, and device version is only exported in sysfs, probably we should define them as below:
#defineVFIO_DEVICE_VERSION_TYPE_PCI(1<<31)
#defineVFIO_DEVICE_VERSION_TYPE_CCW(1<<30)Do you think it's ok?
We already had this discussion for device specific regions and decided that CCW doesn't have enough vendors to justify a full subset of the available address space. Also, this doesn't need to imply the device interface, we're simply specifying a vendor registrar such that we can give each vendor their own namespace, so I don't think it would be a problem for a CCW to specify a namespace using a PCI vendor ID. Finally, if we have such need for this in the future, because I'm not sure where we stand with this in the current proposals, maybe we should make use of an IEEE OUI rather than a PCI database to avoid this sort of confusion and mis-association if we have further need. Thanks,
Alex
Quoted 41 lines · Dr. David Alan Gilbert
Quoted 40 lines · * Zhao Yan ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Quoted 6 lines
Hi,
I've sent minor comments to later patches; but some minor general
comments:a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.
hi Dave
Thanks for this suggestion. I'll add more checks for migration streams.Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
This patchset is intended to settle down the main device state interfaces for VFIO migration. So that we can work on that and improve it.Quoted 5 lines
c) Consider using the
trace_mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.Dave
Got it. many thanks~~
Quoted 2 lines
(
P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'
Quoted 7 lines
Quoted 1 line
sorry :)
No problem.
Given the mails, I'm guessing you've mostly tested this on graphics devices? Have you also checked with VFIO network cards?
yes, I tested it on Intel's graphics devices which do not have device memory. so the cap of device-memory is off.
I believe this patchset can work well on VFIO network cards as well, because Gonglei once said their NIC can work well on our previous code (i.e. device-memory cap off).
Quoted 3 lines
Also see the mail I sent in reply to Kirti's series; we need to boil these down to one solution.
Maybe Kirti can merge their implementaion into the code for device-memory cap (like in my patch 5 for device-memory).
Quoted 292 lines
Dave
Quoted 278 lines
Quoted 278 lines
Quoted 278 lines
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages(The reason why four seperate regions are defined is that the unit of mmap system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3struct vfio_device_state_ctl { __u32 version; /* ro */ __u32 device_state; /* VFIO device state, wo */ __u32 caps; /* ro */ struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /*rw*/ } device_config; struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /* rw */ __u64 pos; /*the offset in total buffer of device memory*/ } device_memory; struct { __u64 start_addr; /* wo */ __u64 page_nr; /* wo */ } system_memory; };Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Then it will probe VFIO device's data capability by reading caps field of control region.
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state -->VFIO_DEVICE_STATE_RUNNING&VFIO_DEVICE_STATE_LOGGING|MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->VFIO_DEVICE_STATE_STOPMIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETEDIn source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.In precopy phase, if a device has
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYon, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis off, .log_sync callback just returns without call to vendor driver.In stop-and-copy phase, device state will be set to STOP & LOGGING first. in
save_live_complete_precopycallback,
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capabilityhw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.cSignature · 1 line
2.7.4Quoted 6 lines
Quoted 6 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Quoted 46 lines · * Zhao Yan ([email protected]) wrote:
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 34 lines · * Zhao Yan ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Quoted 6 lines
Hi,
I've sent minor comments to later patches; but some minor general
comments:a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.
hi Dave
Thanks for this suggestion. I'll add more checks for migration streams.Quoted 3 lines
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
This patchset is intended to settle down the main device state interfaces for VFIO migration. So that we can work on that and improve it.Quoted 5 lines
c) Consider using the
trace_mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.Dave
Quoted 7 lines
Got it. many thanks~~
Quoted 2 lines
(
P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'sorry :)
No problem.
Quoted 8 lines
Quoted 3 lines
Given the mails, I'm guessing you've mostly tested this on graphics devices? Have you also checked with VFIO network cards?
yes, I tested it on Intel's graphics devices which do not have device memory. so the cap of device-memory is off.
I believe this patchset can work well on VFIO network cards as well, because Gonglei once said their NIC can work well on our previous code (i.e. device-memory cap off).
It would be great if you could find some Intel NIC people to help test it out.
Quoted 6 lines
Quoted 4 lines
Also see the mail I sent in reply to Kirti's series; we need to boil these down to one solution.
Maybe Kirti can merge their implementaion into the code for device-memory cap (like in my patch 5 for device-memory).
It would be great to come up with one patchset between yourself and Kirti that was tested for Intel and Nvidia GPUs and Intel NICs (and anyone else who wants to jump on board!).
Dave
Quoted 286 lines
Quoted 286 lines
Dave
Quoted 284 lines
Quoted 284 lines
Quoted 265 lines
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages(The reason why four seperate regions are defined is that the unit of mmap system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3struct vfio_device_state_ctl { __u32 version; /* ro */ __u32 device_state; /* VFIO device state, wo */ __u32 caps; /* ro */ struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /*rw*/ } device_config; struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /* rw */ __u64 pos; /*the offset in total buffer of device memory*/ } device_memory; struct { __u64 start_addr; /* wo */ __u64 page_nr; /* wo */ } system_memory; };Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Then it will probe VFIO device's data capability by reading caps field of control region.
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state -->VFIO_DEVICE_STATE_RUNNING&VFIO_DEVICE_STATE_LOGGING|MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->VFIO_DEVICE_STATE_STOPMIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETEDIn source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.In precopy phase, if a device has
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYon, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis off, .log_sync callback just returns without call to vendor driver.In stop-and-copy phase, device state will be set to STOP & LOGGING first. in
save_live_complete_precopycallback,
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capabilityQuoted 13 lines
hw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.cSignature · 1 line
2.7.4--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Quoted 6 lines
Quoted 6 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Hi yan,
Thanks for your work.
I have some suggestions or questions:
1) Would you add msix mode support,? if not, pls add a check in vfio_pci_save_config(), likes Nvidia's solution.
2) We should start vfio devices before vcpu resumes, so we can't rely on vm start change handler completely.
3) We'd better support live migration rollback since have many failure scenarios,
register a migration notifier is a good choice.
4) Four memory region for live migration is too complicated IMHO.
5) About log sync, why not register log_global_start/stop in vfio_memory_listener?
Regards,
-Gonglei
Quoted 305 lines
-----Original Message-----
From: Yan Zhao [mailto:[email protected]]
Sent: Tuesday, February 19, 2019 4:51 PM
To: [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; [email protected]; Yan Zhao <[email protected]>
Subject: [PATCH 0/5] QEMU VFIO live migrationThis patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages(The reason why four seperate regions are defined is that the unit of mmap system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3struct vfio_device_state_ctl { __u32 version; /* ro */ __u32 device_state; /* VFIO device state, wo */ __u32 caps; /* ro */ struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /*rw*/ } device_config; struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /* rw */ __u64 pos; /*the offset in total buffer of device memory*/ } device_memory; struct { __u64 start_addr; /* wo */ __u64 page_nr; /* wo */ } system_memory; };Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING &
LOGGING,
STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Then it will probe VFIO device's data capability by reading caps field of control region.
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state -->VFIO_DEVICE_STATE_RUNNING&VFIO_DEVICE_STATE_LOGGING
|MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->VFIO_DEVICE_STATE_STOPMIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETEDIn source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopy phase
like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.In precopy phase, if a device has
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYon,
.log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and
"system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis off, .log_sync callback just returns without call to vendor driver.In stop-and-copy phase, device state will be set to STOP & LOGGING first. in
save_live_complete_precopycallback,
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capabilityhw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.c--
2.7.4
Quoted 7 lines · Gonglei (Arei)
Hi yan,
Thanks for your work.
I have some suggestions or questions:
1) Would you add msix mode support,? if not, pls add a check in
vfio_pci_save_config(), likes Nvidia's solution.
ok.
Quoted 1 line
2) We should start vfio devices before vcpu resumes, so we can't rely on vm start change handler completely.
vfio devices is by default set to running state.
In the target machine, its state transition flow is running->stop->running. so, maybe you can ignore the stop notification in kernel?
Quoted 2 lines
3) We'd better support live migration rollback since have many failure scenarios, register a migration notifier is a good choice.
I think this patchset can also handle the failure case well. if migration failure or cancelling happens,
in cleanup handler, LOGGING state is cleared. device state(running or stopped) keeps as it is).
then,
if vm switches back to running, device state will be set to running; if vm stayes at stopped state, device state is also stopped (it has no meaning to let it in running state).
Do you think so ?
Quoted 1 line
4) Four memory region for live migration is too complicated IMHO.
one big region requires the sub-regions well padded.
like for the first control fields, they have to be padded to 4K. the same for other data fields.
Otherwise, mmap simply fails, because the start-offset and size for mmap both need to be PAGE aligned.
Also, 4 regions is clearer in my view :)
Quoted 3 lines
5) About log sync, why not register log_global_start/stop in
vfio_memory_listener?
seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?
Quoted 310 lines
Regards,
-GongleiQuoted 305 lines
-----Original Message-----
From: Yan Zhao [mailto:[email protected]]
Sent: Tuesday, February 19, 2019 4:51 PM
To: [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; [email protected]; Yan Zhao <[email protected]>
Subject: [PATCH 0/5] QEMU VFIO live migrationThis patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages(The reason why four seperate regions are defined is that the unit of mmap system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3struct vfio_device_state_ctl { __u32 version; /* ro */ __u32 device_state; /* VFIO device state, wo */ __u32 caps; /* ro */ struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /*rw*/ } device_config; struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /* rw */ __u64 pos; /*the offset in total buffer of device memory*/ } device_memory; struct { __u64 start_addr; /* wo */ __u64 page_nr; /* wo */ } system_memory; };Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING &
LOGGING,
STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Then it will probe VFIO device's data capability by reading caps field of control region.
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state -->VFIO_DEVICE_STATE_RUNNING&VFIO_DEVICE_STATE_LOGGING
|MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->VFIO_DEVICE_STATE_STOPMIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETEDIn source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopy phase
like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.In precopy phase, if a device has
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYon,
.log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and
"system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis off, .log_sync callback just returns without call to vendor driver.In stop-and-copy phase, device state will be set to STOP & LOGGING first. in
save_live_complete_precopycallback,
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capabilityhw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.c--
2.7.4
Quoted 24 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 8:25 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 6 lines · Gonglei (Arei)
Hi yan,
Thanks for your work.
I have some suggestions or questions:
Quoted 8 lines
Quoted 1 line
1) Would you add msix mode support,? if not, pls add a check in
vfio_pci_save_config(), likes Nvidia's solution.
ok.Quoted 1 line
2) We should start vfio devices before vcpu resumes, so we can't rely on vm
start change handler completely.
vfio devices is by default set to running state.
In the target machine, its state transition flow isrunning->stop->running.
That's confusing. We should start vfio devices after vfio_load_state, otherwise how can you keep the devices' information are the same between source side and destination side?
Quoted 8 lines
so, maybe you can ignore the stop notification in kernel?
Quoted 1 line
3) We'd better support live migration rollback since have many failure
scenarios,
Quoted 1 line
register a migration notifier is a good choice.
I think this patchset can also handle the failure case well. if migration failure or cancelling happens,
in cleanup handler, LOGGING state is cleared. device state(running or stopped) keeps as it is).
IIRC there're many failure paths don't calling cleanup handler.
Quoted 6 lines
then,
if vm switches back to running, device state will be set to running; if vm stayes at stopped state, device state is also stopped (it has no meaning to let it in running state).
Do you think so ?
IF the underlying state machine is complicated,
We should tell the canceling state to vendor driver proactively.
Quoted 7 lines
Quoted 1 line
4) Four memory region for live migration is too complicated IMHO.
one big region requires the sub-regions well padded.
like for the first control fields, they have to be padded to 4K. the same for other data fields.
Otherwise, mmap simply fails, because the start-offset and size for mmap both need to be PAGE aligned.
But if we don't need use mmap for control filed and device state, they are small basically. The performance is enough using pread/pwrite.
Quoted 10 lines
Also, 4 regions is clearer in my view :)
Quoted 1 line
5) About log sync, why not register log_global_start/stop in
vfio_memory_listener?Quoted 2 lines
seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?
We just need invoking only once for start and stop logging. Why we need to call them literately? See memory_listener of vhost.
Regards,
-Gonglei
Quoted 28 lines · Gonglei (Arei)
Quoted 28 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 8:25 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 7 lines · Gonglei (Arei)
Hi yan,
Thanks for your work.
I have some suggestions or questions:
1) Would you add msix mode support,? if not, pls add a check in
vfio_pci_save_config(), likes Nvidia's solution.
ok.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
2) We should start vfio devices before vcpu resumes, so we can't rely on vm
start change handler completely.
vfio devices is by default set to running state.
In the target machine, its state transition flow isrunning->stop->running.That's confusing. We should start vfio devices after
vfio_load_state, otherwise how can you keep the devices' information are the same between source side and destination side?
so, your meaning is to set device state to running in the first call to vfio_load_state?
Quoted 11 lines
Quoted 8 lines
so, maybe you can ignore the stop notification in kernel?
Quoted 1 line
3) We'd better support live migration rollback since have many failure
scenarios,
Quoted 1 line
register a migration notifier is a good choice.
I think this patchset can also handle the failure case well. if migration failure or cancelling happens,
in cleanup handler, LOGGING state is cleared. device state(running or stopped) keeps as it is).IIRC there're many failure paths don't calling cleanup handler.
could you take an example?
Quoted 9 lines
Quoted 6 lines
then,
if vm switches back to running, device state will be set to running; if vm stayes at stopped state, device state is also stopped (it has no meaning to let it in running state).
Do you think so ?IF the underlying state machine is complicated,
We should tell the canceling state to vendor driver proactively.
That makes sense.
Quoted 10 lines
Quoted 7 lines
Quoted 1 line
4) Four memory region for live migration is too complicated IMHO.
one big region requires the sub-regions well padded.
like for the first control fields, they have to be padded to 4K. the same for other data fields.
Otherwise, mmap simply fails, because the start-offset and size for mmap both need to be PAGE aligned.But if we don't need use mmap for control filed and device state, they are small basically. The performance is enough using pread/pwrite.
we don't mmap control fields. but if data fields going immedately after control fields (e.g. just 64 bytes), we can't mmap data fields successfully because its start offset is 64. Therefore control fields have to be padded to 4k to let data fields start from 4k.
That's the drawback of one big region holding both control and data fields.
Quoted 19 lines
Quoted 10 lines
Also, 4 regions is clearer in my view :)
Quoted 1 line
5) About log sync, why not register log_global_start/stop in
vfio_memory_listener?Quoted 2 lines
seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?
We just need invoking only once for start and stop logging. Why we need to call them literately? See
memory_listenerof vhost.Regards,
-Gonglei
Quoted 53 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 9:59 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 10 lines · Gonglei (Arei)
Quoted 10 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 8:25 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];Quoted 3 lines
Quoted 9 lines
Quoted 9 lines
[email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 4 lines · Gonglei (Arei)
Hi yan,
Thanks for your work.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
I have some suggestions or questions:
1) Would you add msix mode support,? if not, pls add a check in
vfio_pci_save_config(), likes Nvidia's solution.
ok.Quoted 1 line
2) We should start vfio devices before vcpu resumes, so we can't rely on
vm
Quoted 4 lines
Quoted 3 lines
start change handler completely.
vfio devices is by default set to running state.
In the target machine, its state transition flow isrunning->stop->running.
Quoted 8 lines
Quoted 1 line
That's confusing. We should start vfio devices after
vfio_load_state,otherwise
Quoted 3 lines
how can you keep the devices' information are the same between source side and destination side?
so, your meaning is to set device state to running in the first call to
vfio_load_state?
No, it should start devices after vfio_load_state and before vcpu resuming.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
so, maybe you can ignore the stop notification in kernel?
Quoted 1 line
3) We'd better support live migration rollback since have many failure
scenarios,
Quoted 1 line
register a migration notifier is a good choice.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
I think this patchset can also handle the failure case well. if migration failure or cancelling happens,
in cleanup handler, LOGGING state is cleared. device state(running or stopped) keeps as it is).IIRC there're many failure paths don't calling cleanup handler.
could you take an example?
Never mind, that's another bug I think.
Quoted 48 lines
Quoted 9 lines
Quoted 6 lines
then,
if vm switches back to running, device state will be set to running; if vm stayes at stopped state, device state is also stopped (it has no meaning to let it in running state).
Do you think so ?IF the underlying state machine is complicated,
We should tell the canceling state to vendor driver proactively.That makes sense.
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
4) Four memory region for live migration is too complicated IMHO.
one big region requires the sub-regions well padded.
like for the first control fields, they have to be padded to 4K. the same for other data fields.
Otherwise, mmap simply fails, because the start-offset and size for mmap both need to be PAGE aligned.But if we don't need use mmap for control filed and device state, they are
small basically.
Quoted 2 lines
The performance is enough using pread/pwrite.
we don't mmap control fields. but if data fields going immedately after control fields (e.g. just 64 bytes), we can't mmap data fields successfully because its start offset is 64. Therefore control fields have to be padded to 4k to let data fields start from 4k.
That's the drawback of one big region holding both control and data fields.Quoted 6 lines
Quoted 6 lines
Also, 4 regions is clearer in my view :)
Quoted 1 line
5) About log sync, why not register log_global_start/stop in
vfio_memory_listener?Quoted 2 lines
Quoted 6 lines
Quoted 4 lines
seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?
We just need invoking only once for start and stop logging. Why we need to
call
Quoted 7 lines
them literately? See
memory_listenerof vhost.Regards,
-Gonglei
Quoted 54 lines · Gonglei (Arei)
Quoted 54 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 9:59 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 10 lines · Gonglei (Arei)
Quoted 10 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 8:25 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];Quoted 3 lines
Quoted 9 lines
Quoted 9 lines
[email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 4 lines · Gonglei (Arei)
Hi yan,
Thanks for your work.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
I have some suggestions or questions:
1) Would you add msix mode support,? if not, pls add a check in
vfio_pci_save_config(), likes Nvidia's solution.
ok.Quoted 1 line
2) We should start vfio devices before vcpu resumes, so we can't rely on
vm
Quoted 5 lines
Quoted 3 lines
start change handler completely.
vfio devices is by default set to running state.
In the target machine, its state transition flow isrunning->stop->running.That's confusing. We should start vfio devices after
vfio_load_state,
Quoted 9 lines
Quoted 7 lines
otherwise
Quoted 3 lines
how can you keep the devices' information are the same between source side and destination side?
so, your meaning is to set device state to running in the first call to
vfio_load_state?No, it should start devices after
vfio_load_stateand before vcpu resuming.
What about set device state to running in load_cleanup handler ?
Quoted 67 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
so, maybe you can ignore the stop notification in kernel?
Quoted 1 line
3) We'd better support live migration rollback since have many failure
scenarios,
Quoted 1 line
register a migration notifier is a good choice.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
I think this patchset can also handle the failure case well. if migration failure or cancelling happens,
in cleanup handler, LOGGING state is cleared. device state(running or stopped) keeps as it is).IIRC there're many failure paths don't calling cleanup handler.
could you take an example?
Never mind, that's another bug I think.
Quoted 34 lines
Quoted 9 lines
Quoted 6 lines
then,
if vm switches back to running, device state will be set to running; if vm stayes at stopped state, device state is also stopped (it has no meaning to let it in running state).
Do you think so ?IF the underlying state machine is complicated,
We should tell the canceling state to vendor driver proactively.That makes sense.
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
4) Four memory region for live migration is too complicated IMHO.
one big region requires the sub-regions well padded.
like for the first control fields, they have to be padded to 4K. the same for other data fields.
Otherwise, mmap simply fails, because the start-offset and size for mmap both need to be PAGE aligned.But if we don't need use mmap for control filed and device state, they are
small basically.
Quoted 2 lines
The performance is enough using pread/pwrite.
we don't mmap control fields. but if data fields going immedately after control fields (e.g. just 64 bytes), we can't mmap data fields successfully because its start offset is 64. Therefore control fields have to be padded to 4k to let data fields start from 4k.
That's the drawback of one big region holding both control and data fields.Quoted 6 lines
Quoted 6 lines
Also, 4 regions is clearer in my view :)
Quoted 1 line
5) About log sync, why not register log_global_start/stop in
vfio_memory_listener?Quoted 2 lines
Quoted 14 lines
Quoted 6 lines
Quoted 4 lines
seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?
We just need invoking only once for start and stop logging. Why we need to
call
Quoted 7 lines
them literately? See
memory_listenerof vhost.Regards,
-Gonglei_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Quoted 85 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 12:08 PM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 9 lines · Gonglei (Arei)
Quoted 9 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 9:59 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];Quoted 3 lines
Quoted 29 lines
Quoted 29 lines
[email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 10 lines · Gonglei (Arei)
Quoted 10 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 8:25 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];Quoted 3 lines
Quoted 9 lines
Quoted 9 lines
[email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 4 lines · Gonglei (Arei)
Hi yan,
Thanks for your work.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
I have some suggestions or questions:
1) Would you add msix mode support,? if not, pls add a check in
vfio_pci_save_config(), likes Nvidia's solution. ok.Quoted 1 line
2) We should start vfio devices before vcpu resumes, so we can't rely
on
Quoted 4 lines
Quoted 4 lines
vm
Quoted 3 lines
Quoted 3 lines
start change handler completely.
vfio devices is by default set to running state. In the target machine, its state transition flow is
running->stop->running.Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
That's confusing. We should start vfio devices after
vfio_load_state,otherwise
Quoted 1 line
how can you keep the devices' information are the same between source
side
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
and destination side?
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
so, your meaning is to set device state to running in the first call to
vfio_load_state?No, it should start devices after
vfio_load_stateand before vcpu resuming.What about set device state to running in
load_cleanuphandler ?
The timing is fine, but you should also think about if should set device state to running in failure branches when calling load_cleanup handler.
Regards,
-Gonglei
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
5) About log sync, why not register log_global_start/stop in
vfio_memory_listener?Quoted 2 lines
Quoted 8 lines
Quoted 4 lines
seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?
We just need invoking only once for start and stop logging. Why we need to call them literately? See
memory_listenerof vhost.
the dirty pages in system memory produces by device is incremental. if it can be got iteratively, the dirty pages in stop-and-copy phase can be minimal.
:)
Quoted 2 lines
Regards,
-Gonglei
Quoted 28 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 10:05 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 10 lines
Quoted 9 lines
Quoted 2 lines
5) About log sync, why not register log_global_start/stop in
vfio_memory_listener?Quoted 2 lines
seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?
Quoted 9 lines
Quoted 1 line
We just need invoking only once for start and stop logging. Why we need to
call
Quoted 2 lines
them literately? See
memory_listenerof vhost.the dirty pages in system memory produces by device is incremental. if it can be got iteratively, the dirty pages in stop-and-copy phase can be minimal.
:)
I mean starting or stopping the capability of logging, not log sync.
We register the below callbacks:
.log_sync = vfio_log_sync,
.log_global_start = vfio_log_global_start,
.log_global_stop = vfio_log_global_stop,
Regards,
-Gonglei
Quoted 37 lines · Gonglei (Arei)
Quoted 28 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 10:05 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 10 lines
Quoted 9 lines
Quoted 2 lines
5) About log sync, why not register log_global_start/stop in
vfio_memory_listener?Quoted 2 lines
seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?
Quoted 9 lines
Quoted 1 line
We just need invoking only once for start and stop logging. Why we need to
call
Quoted 2 lines
them literately? See
memory_listenerof vhost.the dirty pages in system memory produces by device is incremental. if it can be got iteratively, the dirty pages in stop-and-copy phase can be minimal.
:)
Quoted 8 lines
I mean starting or stopping the capability of logging, not log sync.
We register the below callbacks:
.log_sync =
vfio_log_sync,
.log_global_start =vfio_log_global_start,
.log_global_stop =vfio_log_global_stop,
.log_global_start is also a good point to notify logging state. But if notifying in .save_setup handler, we can do fine-grained control of when to notify of logging starting together with get_buffer operation.
Is there any special benifit by registering to .log_global_start/stop?
Quoted 2 lines
Regards,
-Gonglei
Quoted 47 lines
Quoted 9 lines
Quoted 9 lines
-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 10:05 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];Quoted 3 lines
Quoted 10 lines
Quoted 10 lines
[email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migrationQuoted 6 lines
Quoted 6 lines
Quoted 2 lines
5) About log sync, why not register log_global_start/stop in
vfio_memory_listener?Quoted 2 lines
seems log_global_start/stop cannot be iterately called in pre-copy
phase?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?
We just need invoking only once for start and stop logging. Why we need
to
Quoted 8 lines
Quoted 8 lines
call
Quoted 2 lines
them literately? See
memory_listenerof vhost.the dirty pages in system memory produces by device is incremental. if it can be got iteratively, the dirty pages in stop-and-copy phase can be minimal.
:)Quoted 8 lines
I mean starting or stopping the capability of logging, not log sync.
We register the below callbacks:
.log_sync =
vfio_log_sync,
.log_global_start =vfio_log_global_start,
.log_global_stop =vfio_log_global_stop,
Quoted 6 lines
.log_global_start is also a good point to notify logging state. But if notifying in .save_setup handler, we can do fine-grained control of when to notify of logging starting together with
get_bufferoperation.
Is there any special benifit by registering to .log_global_start/stop?
Performance benefit when one VM has multiple same vfio devices.
Regards,
-Gonglei
Hi Yan,
Thanks for working on this!
Quoted 7 lines · On Tue, 19 Feb 2019 16:50:54 +0800 Yan Zhao <[email protected]> wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Quoted 8 lines
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it
I'm not sure how we can really impose a limit here, it is what it is for a device. A smaller state is obviously desirable to reduce downtime, but some devices could have very large states.
Quoted 6 lines
needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.
So the intention here is that this is the last data read from the device and it's done in one pass, so the region needs to be large enough to expose all config data at once. On restore it's the last data written before switching the device to the run state.
Quoted 2 lines
Device Memory: device's internal memory, standalone and outside system
s/system/VM/
Quoted 1 line
memory. It is usually very big.
Or it doesn't exist. Not sure we should be setting expectations since it will vary per device.
Quoted 7 lines
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.
It seems a little gratuitous to me that this is a separate region or that this data is handled separately. All of this data is opaque to QEMU, so why do we need to separate it?
Quoted 17 lines
System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.
Quoted 5 lines
So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.
Is mmap mandatory? I would think this would be defined by the mdev device what access they want to support per region. We don't want to impose a more complicated interface if the device doesn't require it.
Quoted 9 lines
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages(The reason why four seperate regions are defined is that the unit of mmap system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).
It's not obvious to me how this is better, a big region isn't padded, there's simply a gap in the file descriptor. Is having a sub-PAGE_SIZE gap in a file really of any consequence? Each region beyond the header is more than likely larger than PAGE_SIZE, therefore they can be nicely aligned together. We still need fields to tell us how much data is available in each area, so another to tell us the start of each area is a minor detail. And I think we still want to allow drivers to specify which parts of which areas support mmap, so I don't think we're getting away from sparse mmap support.
Quoted 5 lines
kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2
If we were to go with this multi-region solution, isn't it evident from the regions exposed that device memory and a dirty bitmap are provided? Alternatively, I believe Kirti's proposal doesn't require this distinction between device memory and device config, a device not requiring runtime migration data would simply report no data until the device moved to the stopped state, making it consistent path for userspace. Likewise, the dirty bitmap could report a zero page count in the bitmap rather than branching based on device support.
Note that it seems that cap VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY implies VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP, but there's no consistency in the naming.
Quoted 3 lines
#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2
It looks like these are being defined as bits, since patch 1 talks about RUNNING & LOGGING and STOP & LOGGING. I think Connie already posted some comments about this. I'm not sure anything prevents us from defining RUNNING a 1 and STOPPED as 0 so we don't have the polarity flip vs LOGGING though.
The state "STOP & LOGGING" also feels like a strange "device state", if the device is stopped, it's not logging any new state, so I think this is more that the device state is STOP, but the LOGGING feature is active. Maybe we should consider these independent bits. LOGGING is active as we stop a device so that we can fetch the last dirtied pages, but disabled as we load the state of the device into the target.
Quoted 4 lines
#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3
Quoted 8 lines
struct vfio_device_state_ctl {
__u32 version; /* ro */
__u32 device_state; /* VFIO device state, wo */
__u32 caps; /* ro */
struct {
__u32 action; /* wo, GET_BUFFER or SET_BUFFER */
__u64 size; /*rw*/
} device_config;Patch 1 indicates that to get the config buffer we write GET_BUFFER to action and read from the config region. The size is previously read and apparently constant. To set the config buffer, the config region is written followed by writing SET_BUFFER to action. Why is size listed as read-write?
Doesn't this protocol also require that the mdev driver consume each full region's worth of host kernel memory for backing pages in anticipation of a rare event like migration? This might be a strike against separate regions if the driver needs to provide backing pages for 3 separate regions vs 1. To avoid this runtime overhead, would it be expected that the user only mmap the regions during migration and the mdev driver allocate backing pages on mmap? Should the mmap be restricted to the LOGGING feature being enabled? Maybe I'm mistaken in how the mdev driver would back these mmap'd pages.
Quoted 4 lines
struct {
__u32 action; /* wo, GET_BUFFER or SET_BUFFER */
__u64 size; /* rw */
__u64 pos; /*the offset in total buffer of device memory*/Patch 1 outlines the protocol here that getting device memory begins with writing the position field, followed by reading from the device memory region. Setting device memory begins with writing the data to the device memory region, followed by writing the position field. Why does the user need to have visibility of data position? This is opaque data to the user, the device should manage how the chunks fit together.
How does the user know when they reach the end?
Bullets 8 and 9 in patch 1 also discuss setting and getting the device memory size, but these aren't well integrated into the protocol for getting and setting the memory buffer. Is getting the device memory really started by reading the size, which triggers the vendor driver to snapshot the state in an internal buffer which the user then iterates through using GET_BUFFER? Therefore re-reading the size field could corrupt the data stream? Wouldn't it be easier to report bytes available and countdown as the user marks them read? What does position mean when we switch from snapshot to dirty data?
Quoted 6 lines
} device_memory;
struct {
__u64 start_addr; /* wo */
__u64 page_nr; /* wo */
} system_memory;
};Why is one specified as an address and the other as pages? Note that Kirti's implementation has an optimization to know how many pages are set within a range to avoid unnecessary buffer reads.
Quoted 42 lines
Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Then it will probe VFIO device's data capability by reading caps field of control region.
Quoted 8 lines
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.
As above, these capabilities seem redundant to the existence of the device specific regions in this implementation.
Quoted 65 lines
If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state -->VFIO_DEVICE_STATE_RUNNING&VFIO_DEVICE_STATE_LOGGING|MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->VFIO_DEVICE_STATE_STOPMIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETED
Quoted 7 lines
In source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
This requires iterative reads of device memory buffer but the protocol is unclear (to me) how the user knows how to do this or interact with the position field.
Quoted 5 lines
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (
GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
What if there's not enough dirty data to fill the region? The data is always padded to fill the region?
Quoted 6 lines
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.
If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.
Therefore through the entire precopy phase we have no data from source to target to begin a compatibility check :-\ I think both proposals currently still lack any sort of device compatibility or data versioning check between source and target. Thanks,
Alex
Quoted 49 lines
In precopy phase, if a device has
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYon, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis off, .log_sync callback just returns without call to vendor driver.In stop-and-copy phase, device state will be set to STOP & LOGGING first. in
save_live_complete_precopycallback,
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capabilityhw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.c
Quoted 57 lines · Alex Williamson
Hi Yan,
Thanks for working on this!
Quoted 7 lines · On Tue, 19 Feb 2019 16:50:54 +0800 Yan Zhao <[email protected]> wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Quoted 8 lines
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and itI'm not sure how we can really impose a limit here, it is what it is for a device. A smaller state is obviously desirable to reduce downtime, but some devices could have very large states.
Quoted 6 lines
needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.So the intention here is that this is the last data read from the device and it's done in one pass, so the region needs to be large enough to expose all config data at once. On restore it's the last data written before switching the device to the run state.
Quoted 2 lines
Device Memory: device's internal memory, standalone and outside system
s/system/VM/
Quoted 1 line
memory. It is usually very big.
Or it doesn't exist. Not sure we should be setting expectations since it will vary per device.
Quoted 7 lines
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.
Quoted 3 lines
It seems a little gratuitous to me that this is a separate region or that this data is handled separately. All of this data is opaque to QEMU, so why do we need to separate it?
hi Alex,
as the device state interfaces are provided by kernel, it is expected to meet as general needs as possible. So, do you think there are such use cases from user space that user space knows well of the device, and it wants kernel to return desired data back to it.
E.g. It just wants to get whole device config data including all mmios, page tables, pci config data...
or, It just wants to get current device memory snapshot, not including any dirty data.
Or, It just needs the dirty pages in device memory or system memory. With all this accurate query, quite a lot of useful features can be developped in user space.
If all of this data is opaque to user app, seems the only use case is for live migration.
Quoted 1 line
From another aspect, if the final solution is to let the data opaque to
user space, like what NV did, kernel side's implementation will be more complicated, and actually a little challenge to vendor driver. in that case, in pre-copy phase,
1. in not LOGGING state, vendor driver first returns full data including full device memory snapshot
2. user space reads some data (you can't expect it to finish reading all data)
3. then userspace set the device state to LOGGING to start dirty data logging
4. vendor driver starts dirty data logging, and appends the dirty data to the tail of remaining unread full data and increase the pending data size?
5. user space keeps reading data.
6. vendor driver keeps appending new dirty data to the tail of remaining unread full data/dirty data and increase the pending data size?
in stop-and-copy phase
1. user space sets device state to exit LOGGING state,
2. vendor driver stops data logging. it has to append device config data at the tail of remaining dirty data unread by userspace.
during this flow, when vendor driver should get dirty data? just keeps logging and appends to tail? how to ensure dirty data is refresh new before LOGGING state exit? how does vendor driver know whether certain dirty data is copied or not?
I've no idea how NVidia handle this problem, and they don't open their kernel side code.
just feel it's a bit hard for other vendor drivers to follow:)
Quoted 23 lines
Quoted 17 lines
System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.Quoted 5 lines
So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.
Quoted 3 lines
Is mmap mandatory? I would think this would be defined by the mdev device what access they want to support per region. We don't want to impose a more complicated interface if the device doesn't require it.
I think it's "mmap is preferred, but allowed to fail".
just like a normal region with MMAP flag on (like bar regions), we also allow its mmap to fail, right?
Quoted 10 lines
Quoted 9 lines
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages(The reason why four seperate regions are defined is that the unit of mmap system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).
Quoted 9 lines
It's not obvious to me how this is better, a big region isn't padded, there's simply a gap in the file descriptor. Is having a sub-PAGE_SIZE gap in a file really of any consequence? Each region beyond the header is more than likely larger than
PAGE_SIZE, therefore they can be nicely aligned together. We still need fields to tell us how much data is available in each area, so another to tell us the start of each area is a minor detail. And I think we still want to allow drivers to specify which parts of which areas support mmap, so I don't think we're getting away from sparse mmap support.
with seperate regions and sub-region type defined, user space can explictly know which region is which region after vfio_get_dev_region_info(). along with it, user space knows region offset and size. mmap is allowed to fail and falls back to normal read/write to the region.
But with one big region and sparse mmapped subregions (1 data subregion or 3 data subregions, whatever), userspace can't tell which subregion is which one.
So, if using one big region, I think we need to explictly define subregions' sequence (like index 0 is dedicated to control subregion, index 1 is for device config data subregion ...). Vendor driver cannot freely change the sequence.
Then keep data offset the same as region->mmaps[i].offset, and data size the same as region->mmaps[i].size. (i.e. let actual data starts immediatly from first byte of its data subregion)
Also, mmaps for sparse mmaped subregions are not allowed to fail.
With one big region, we also need to consider the case when vendor driver does not want the data subregion to be mmaped.
so, what is the data layout for that case?
data subregion immedately follows control subregion, or not?
Of couse, for this condition, we can specify the data filed's start offset and size through control region. And we must not expect the data start offset in source and target are equal.
(because the big region's fd_offset
may vary in source and target. consider the case when both source and target have one opregion and one device state region, but source has opregion in the first and target has device state region in the first. If we think this case is illegal, we must be able to detect it in the first place).
Also, we must keep the start offset and size consistent with the above mmap case.
Quoted 10 lines
Quoted 5 lines
kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2If we were to go with this multi-region solution, isn't it evident from the regions exposed that device memory and a dirty bitmap are provided? Alternatively, I believe Kirti's proposal doesn't require
Quoted 5 lines
this distinction between device memory and device config, a device not requiring runtime migration data would simply report no data until the device moved to the stopped state, making it consistent path for userspace. Likewise, the dirty bitmap could report a zero page count in the bitmap rather than branching based on device support.
If the path in userspace is consistent for device config and device memory, there will be many unnecessary call of getting data size into vendor driver.
Quoted 34 lines
Note that it seems that cap
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYimpliesVFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP, but there's no consistency in the naming.Quoted 3 lines
#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2It looks like these are being defined as bits, since patch 1 talks about RUNNING & LOGGING and STOP & LOGGING. I think Connie already posted some comments about this. I'm not sure anything prevents us from defining RUNNING a 1 and STOPPED as 0 so we don't have the polarity flip vs LOGGING though.
The state "STOP & LOGGING" also feels like a strange "device state", if the device is stopped, it's not logging any new state, so I think this is more that the device state is STOP, but the LOGGING feature is active. Maybe we should consider these independent bits. LOGGING is active as we stop a device so that we can fetch the last dirtied pages, but disabled as we load the state of the device into the target.
Quoted 4 lines
#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3Quoted 8 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u32 device_state; /* VFIO device state, wo */ __u32 caps; /* ro */ struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /*rw*/ } device_config;
Quoted 5 lines
Patch 1 indicates that to get the config buffer we write
GET_BUFFERto action and read from the config region. The size is previously read and apparently constant. To set the config buffer, the config region is written followed by writingSET_BUFFERto action. Why is size listed as read-write?
this is the size of config data.
size of config data <= size of config data region.
Quoted 10 lines
Doesn't this protocol also require that the mdev driver consume each full region's worth of host kernel memory for backing pages in anticipation of a rare event like migration? This might be a strike against separate regions if the driver needs to provide backing pages for 3 separate regions vs 1. To avoid this runtime overhead, would it be expected that the user only mmap the regions during migration and the mdev driver allocate backing pages on mmap? Should the mmap be restricted to the LOGGING feature being enabled? Maybe I'm mistaken in how the mdev driver would back these mmap'd pages.
yes, 3 seperate regions consume a little more memory than 1 region. but it's just a little overhead.
As in intel's kernel implementation,
device config region's size is 9M, dirty bitmap region's size is 16k. if there is device memory region, its size can be defined as 100M? so it's 109M vs 100M ?
Quoted 5 lines
Quoted 4 lines
struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /* rw */ __u64 pos; /*the offset in total buffer of device memory*/
Quoted 8 lines
Patch 1 outlines the protocol here that getting device memory begins with writing the position field, followed by reading from the device memory region. Setting device memory begins with writing the data to the device memory region, followed by writing the position field. Why does the user need to have visibility of data position? This is opaque data to the user, the device should manage how the chunks fit together.
How does the user know when they reach the end?
sorry, maybe I didn't explain clearly here.
device ________________________________________
memory | | |////| | | | | |
data: |____|____|////|____|____|____|____|____|
:pos :
: :
device :____:
memory | |
region: |____|the whole sequence is like this:
1. user space reads device_memory.size
2. driver gets device memory's data(full snapshot or dirty data, depending on it's in LOGGING state or not), and return the total size of this data.
3. user space finishes reading device_memory.size (>= device memory region's size)
4. user space starts a loop like
while (pos < total_len) {
uint64_t len = region_devmem->size; if (pos + len >= total_len) {
len = total_len - pos;
}
if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) {
return -1;
}
pos += len;
} vfio_save_data_device_memory_chunk reads each chunk from device memory region by writing GET_BUFFER to device_memory.action, and pos to device_memory.pos.
so. each time, userspace will finish getting device memory data in one time.
specifying "pos" is just like the "lseek" before "write".
Quoted 9 lines
Bullets 8 and 9 in patch 1 also discuss setting and getting the device memory size, but these aren't well integrated into the protocol for getting and setting the memory buffer. Is getting the device memory really started by reading the size, which triggers the vendor driver to snapshot the state in an internal buffer which the user then iterates through using
GET_BUFFER? Therefore re-reading the size field could corrupt the data stream? Wouldn't it be easier to report bytes available and countdown as the user marks them read? What does position mean when we switch from snapshot to dirty data?
when switching to device memory's dirty data, pos means the pos in whole dirty data.
.save_live_pending ==> driver gets dirty data in device memory and returns total size.
.save_live_iterate ==> userspace reads all dirty data from device memory region chunk by chunk
So, in an iteration, all dirty data are saved. then in next iteration, dirty data is recalculated.
Quoted 9 lines
Quoted 7 lines
} device_memory; struct { __u64 start_addr; /* wo */ __u64 page_nr; /* wo */ } system_memory; };Why is one specified as an address and the other as pages? Note that
Yes, start_addr ==> start pfn is better
Quoted 3 lines
Kirti's implementation has an optimization to know how many pages are set within a range to avoid unnecessary buffer reads.
Let's use start_pfn_all, page_nr_all to represent the start pfn and page_nr passed in from qemu .log_sync interface.
and use start_pfn_i, page_nr_i to the value passed to driver.
start_pfn_all
| start_pfn_i
| |
\ /_______\_/_____________________________
| | |////| | | | | |
|____|____|////|____|____|____|____|____|
: :
: :
:____:
bitmap | |
region: |____|1. Each time QEMU queries dirty bitmap from driver, it passes in start_pfn_i, and page_nr_i. (page_nr_i is the biggest page number the bitmap region can hold).
2. driver queries memory range from start_pfn_i with size page_nr_i.
3. driver return a bitmap (if no dirty data, the bitmap is all 0).
4. QEMU saves the pages according to the bitmap
If there's no dirty data found in step 2, step 4 can be skipped. (I'll add this check before step 4 in future, thanks)
but if there's even 1 bit set in the bitmap, no step from 1-4 can be skipped.
Honestly, after reviewing Kirti's implementation, I don't think it's an optimization. As in below pseudo code for Kirti's code, I would think the copied_pfns corresponds to the page_nr_i in my case. so, the case of copied_pfns equaling to 0 is for the tail chunk? don't think it's working..
write start_pfn to driver
write page_size to driver
write pfn_count to driver
do {
read copied_pfns from driver.
if (copied_pfns == 0) {
break;
}
bitmap_size = (BITS_TO_LONGS(copied_pfns) + 1) * sizeof(unsigned long);
buf = get bitmap from driver
cpu_physical_memory_set_dirty_lebitmap((unsigned long *)buf,
(start_pfn + count) * page_size,
copied_pfns);
count += copied_pfns;} while (count < pfn_count);
Quoted 51 lines
Quoted 42 lines
Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Then it will probe VFIO device's data capability by reading caps field of control region.
Quoted 8 lines
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.
Quoted 3 lines
As above, these capabilities seem redundant to the existence of the device specific regions in this implementation.
seems so :)
Quoted 77 lines
Quoted 65 lines
If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state -->VFIO_DEVICE_STATE_RUNNING&VFIO_DEVICE_STATE_LOGGING|MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->VFIO_DEVICE_STATE_STOPMIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETEDQuoted 7 lines
In source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.This requires iterative reads of device memory buffer but the protocol is unclear (to me) how the user knows how to do this or interact with the position field.
Quoted 9 lines
Quoted 5 lines
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (
GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).What if there's not enough dirty data to fill the region? The data is always padded to fill the region?
I think dirty data in vendor driver is orgnaized in a format like:
(addr0, len0, data0, addr1, len1, data1, addr2, len2, data2,....addrN, lenN, dataN).
for full snapshot, it's like (addr0, len0, data0).
so, to userspace and data region, it doesn't matter whether it's full snapshot or dirty data.
Quoted 11 lines
Quoted 6 lines
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.
If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.Therefore through the entire precopy phase we have no data from source to target to begin a compatibility check :-\ I think both proposals currently still lack any sort of device compatibility or data versioning check between source and target. Thanks,
I checked the compatibility, though not good enough:)
in migration_init, vfio_check_devstate_version() checked version from kernel with VFIO_DEVICE_STATE_INTERFACE_VERSION in both source and target, and in target side, vfio_load_state() checked source side version.
int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
{
...
if (version_id != VFIO_DEVICE_STATE_INTERFACE_VERSION) {
return -EINVAL;
}
...
}Thanks
Yan
Quoted 52 lines
Alex
Quoted 49 lines
In precopy phase, if a device has
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYon, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis off, .log_sync callback just returns without call to vendor driver.In stop-and-copy phase, device state will be set to STOP & LOGGING first. in
save_live_complete_precopycallback,
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capabilityhw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.c
hi Alex
we still have some opens as below. could you kindly help review on that? :)
Thanks
Yan
Quoted 659 lines · Zhao Yan
Quoted 57 lines · Alex Williamson
Hi Yan,
Thanks for working on this!
Quoted 7 lines · On Tue, 19 Feb 2019 16:50:54 +0800 Yan Zhao <[email protected]> wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Quoted 8 lines
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and itI'm not sure how we can really impose a limit here, it is what it is for a device. A smaller state is obviously desirable to reduce downtime, but some devices could have very large states.
Quoted 6 lines
needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.So the intention here is that this is the last data read from the device and it's done in one pass, so the region needs to be large enough to expose all config data at once. On restore it's the last data written before switching the device to the run state.
Quoted 2 lines
Device Memory: device's internal memory, standalone and outside system
s/system/VM/
Quoted 1 line
memory. It is usually very big.
Or it doesn't exist. Not sure we should be setting expectations since it will vary per device.
Quoted 7 lines
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.Quoted 3 lines
It seems a little gratuitous to me that this is a separate region or that this data is handled separately. All of this data is opaque to QEMU, so why do we need to separate it?
hi Alex,
as the device state interfaces are provided by kernel, it is expected to meet as general needs as possible. So, do you think there are such use cases from user space that user space knows well of the device, and it wants kernel to return desired data back to it.
E.g. It just wants to get whole device config data including all mmios, page tables, pci config data...
or, It just wants to get current device memory snapshot, not including any dirty data.
Or, It just needs the dirty pages in device memory or system memory. With all this accurate query, quite a lot of useful features can be developped in user space.If all of this data is opaque to user app, seems the only use case is for live migration.
From another aspect, if the final solution is to let the data opaque to user space, like what NV did, kernel side's implementation will be more complicated, and actually a little challenge to vendor driver. in that case, in pre-copy phase,
1. in not LOGGING state, vendor driver first returns full data including full device memory snapshot
2. user space reads some data (you can't expect it to finish reading all data)
3. then userspace set the device state to LOGGING to start dirty data logging
4. vendor driver starts dirty data logging, and appends the dirty data to the tail of remaining unread full data and increase the pending data size?
5. user space keeps reading data.
6. vendor driver keeps appending new dirty data to the tail of remaining unread full data/dirty data and increase the pending data size?in stop-and-copy phase
1. user space sets device state to exit LOGGING state,
2. vendor driver stops data logging. it has to append device config data at the tail of remaining dirty data unread by userspace.during this flow, when vendor driver should get dirty data? just keeps logging and appends to tail? how to ensure dirty data is refresh new before LOGGING state exit? how does vendor driver know whether certain dirty data is copied or not?
I've no idea how NVidia handle this problem, and they don't open their kernel side code.
just feel it's a bit hard for other vendor drivers to follow:)Quoted 23 lines
Quoted 17 lines
System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.Quoted 5 lines
So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.Quoted 3 lines
Is mmap mandatory? I would think this would be defined by the mdev device what access they want to support per region. We don't want to impose a more complicated interface if the device doesn't require it.
I think it's "mmap is preferred, but allowed to fail".
just like a normal region with MMAP flag on (like bar regions), we also allow its mmap to fail, right?Quoted 10 lines
Quoted 9 lines
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages(The reason why four seperate regions are defined is that the unit of mmap system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).Quoted 9 lines
It's not obvious to me how this is better, a big region isn't padded, there's simply a gap in the file descriptor. Is having a sub-PAGE_SIZE gap in a file really of any consequence? Each region beyond the header is more than likely larger than
PAGE_SIZE, therefore they can be nicely aligned together. We still need fields to tell us how much data is available in each area, so another to tell us the start of each area is a minor detail. And I think we still want to allow drivers to specify which parts of which areas support mmap, so I don't think we're getting away from sparse mmap support.with seperate regions and sub-region type defined, user space can explictly know which region is which region after
vfio_get_dev_region_info(). along with it, user space knows region offset and size. mmap is allowed to fail and falls back to normal read/write to the region.But with one big region and sparse mmapped subregions (1 data subregion or 3 data subregions, whatever), userspace can't tell which subregion is which one.
So, if using one big region, I think we need to explictly define subregions' sequence (like index 0 is dedicated to control subregion, index 1 is for device config data subregion ...). Vendor driver cannot freely change the sequence.
Then keep data offset the same as region->mmaps[i].offset, and data size the same as region->mmaps[i].size. (i.e. let actual data starts immediatly from first byte of its data subregion)
Also, mmaps for sparse mmaped subregions are not allowed to fail.With one big region, we also need to consider the case when vendor driver does not want the data subregion to be mmaped.
so, what is the data layout for that case?
data subregion immedately follows control subregion, or not?
Of couse, for this condition, we can specify the data filed's start offset and size through control region. And we must not expect the data start offset in source and target are equal.
(because the big region'sfd_offset
may vary in source and target. consider the case when both source and target have one opregion and one device state region, but source has opregion in the first and target has device state region in the first. If we think this case is illegal, we must be able to detect it in the first place).
Also, we must keep the start offset and size consistent with the above mmap case.Quoted 10 lines
Quoted 5 lines
kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2If we were to go with this multi-region solution, isn't it evident from the regions exposed that device memory and a dirty bitmap are provided? Alternatively, I believe Kirti's proposal doesn't require
Quoted 5 lines
this distinction between device memory and device config, a device not requiring runtime migration data would simply report no data until the device moved to the stopped state, making it consistent path for userspace. Likewise, the dirty bitmap could report a zero page count in the bitmap rather than branching based on device support.
If the path in userspace is consistent for device config and device memory, there will be many unnecessary call of getting data size into vendor driver.
Quoted 34 lines
Note that it seems that cap
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYimpliesVFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP, but there's no consistency in the naming.Quoted 3 lines
#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2It looks like these are being defined as bits, since patch 1 talks about RUNNING & LOGGING and STOP & LOGGING. I think Connie already posted some comments about this. I'm not sure anything prevents us from defining RUNNING a 1 and STOPPED as 0 so we don't have the polarity flip vs LOGGING though.
The state "STOP & LOGGING" also feels like a strange "device state", if the device is stopped, it's not logging any new state, so I think this is more that the device state is STOP, but the LOGGING feature is active. Maybe we should consider these independent bits. LOGGING is active as we stop a device so that we can fetch the last dirtied pages, but disabled as we load the state of the device into the target.
Quoted 4 lines
#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3Quoted 8 lines
struct vfio_device_state_ctl { __u32 version; /* ro */ __u32 device_state; /* VFIO device state, wo */ __u32 caps; /* ro */ struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /*rw*/ } device_config;Quoted 5 lines
Patch 1 indicates that to get the config buffer we write
GET_BUFFERto action and read from the config region. The size is previously read and apparently constant. To set the config buffer, the config region is written followed by writingSET_BUFFERto action. Why is size listed as read-write?this is the size of config data.
size of config data <= size of config data region.Quoted 10 lines
Doesn't this protocol also require that the mdev driver consume each full region's worth of host kernel memory for backing pages in anticipation of a rare event like migration? This might be a strike against separate regions if the driver needs to provide backing pages for 3 separate regions vs 1. To avoid this runtime overhead, would it be expected that the user only mmap the regions during migration and the mdev driver allocate backing pages on mmap? Should the mmap be restricted to the LOGGING feature being enabled? Maybe I'm mistaken in how the mdev driver would back these mmap'd pages.
yes, 3 seperate regions consume a little more memory than 1 region. but it's just a little overhead.
As in intel's kernel implementation,
device config region's size is 9M, dirty bitmap region's size is 16k. if there is device memory region, its size can be defined as 100M? so it's 109M vs 100M ?Quoted 5 lines
Quoted 4 lines
struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /* rw */ __u64 pos; /*the offset in total buffer of device memory*/Quoted 8 lines
Patch 1 outlines the protocol here that getting device memory begins with writing the position field, followed by reading from the device memory region. Setting device memory begins with writing the data to the device memory region, followed by writing the position field. Why does the user need to have visibility of data position? This is opaque data to the user, the device should manage how the chunks fit together.
How does the user know when they reach the end?
sorry, maybe I didn't explain clearly here.
device ________________________________________ memory | | |////| | | | | | data: |____|____|////|____|____|____|____|____| :pos : : : device :____: memory | | region: |____|the whole sequence is like this:
1. user space reads device_memory.size
2. driver gets device memory'sdata(full snapshot or dirty data, depending on it's in LOGGING state or not), and return the total size of this data.
3. user space finishes reading device_memory.size (>= device memory region's size)4. user space starts a loop like
while (pos < total_len) { uint64_t len = region_devmem->size;if (pos + len >= total_len) { len = total_len - pos; } if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) { return -1; } pos += len; }
vfio_save_data_device_memory_chunkreads each chunk from device memory region by writingGET_BUFFERto device_memory.action, and pos to device_memory.pos.so. each time, userspace will finish getting device memory data in one time.
specifying "pos" is just like the "lseek" before "write".
Quoted 9 lines
Bullets 8 and 9 in patch 1 also discuss setting and getting the device memory size, but these aren't well integrated into the protocol for getting and setting the memory buffer. Is getting the device memory really started by reading the size, which triggers the vendor driver to snapshot the state in an internal buffer which the user then iterates through using
GET_BUFFER? Therefore re-reading the size field could corrupt the data stream? Wouldn't it be easier to report bytes available and countdown as the user marks them read? What does position mean when we switch from snapshot to dirty data?when switching to device memory's dirty data, pos means the pos in whole dirty data.
.save_live_pending ==> driver gets dirty data in device memory and returns total size.
.save_live_iterate ==> userspace reads all dirty data from device memory region chunk by chunk
So, in an iteration, all dirty data are saved. then in next iteration, dirty data is recalculated.
Quoted 9 lines
Quoted 7 lines
} device_memory; struct { __u64 start_addr; /* wo */ __u64 page_nr; /* wo */ } system_memory; };Why is one specified as an address and the other as pages? Note that
Yes,
start_addr==> start pfn is betterQuoted 3 lines
Kirti's implementation has an optimization to know how many pages are set within a range to avoid unnecessary buffer reads.
Let's use
start_pfn_all,page_nr_allto represent the start pfn andpage_nrpassed in from qemu .log_sync interface.and use
start_pfn_i,page_nr_ito the value passed to driver.
start_pfn_all| start_pfn_i | | \ /_______\_/_____________________________ | | |////| | | | | | |____|____|////|____|____|____|____|____| : : : : :____: bitmap | | region: |____|1. Each time QEMU queries dirty bitmap from driver, it passes in
start_pfn_i, andpage_nr_i. (page_nr_iis the biggest page number the bitmap region can hold).
2. driver queries memory range fromstart_pfn_iwith sizepage_nr_i.
3. driver return a bitmap (if no dirty data, the bitmap is all 0).
4. QEMU saves the pages according to the bitmapIf there's no dirty data found in step 2, step 4 can be skipped. (I'll add this check before step 4 in future, thanks)
but if there's even 1 bit set in the bitmap, no step from 1-4 can be skipped.Honestly, after reviewing Kirti's implementation, I don't think it's an optimization. As in below pseudo code for Kirti's code, I would think the
copied_pfnscorresponds to thepage_nr_iin my case. so, the case ofcopied_pfnsequaling to 0 is for the tail chunk? don't think it's working..write
start_pfnto driver
writepage_sizeto driver
writepfn_countto driverdo { read copied_pfns from driver. if (copied_pfns == 0) { break; } bitmap_size = (BITS_TO_LONGS(copied_pfns) + 1) * sizeof(unsigned long); buf = get bitmap from driver cpu_physical_memory_set_dirty_lebitmap((unsigned long *)buf, (start_pfn + count) * page_size, copied_pfns); count += copied_pfns;} while (count <
pfn_count);Quoted 51 lines
Quoted 42 lines
Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Then it will probe VFIO device's data capability by reading caps field of control region.
Quoted 8 lines
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.Quoted 3 lines
As above, these capabilities seem redundant to the existence of the device specific regions in this implementation.
seems so :)
Quoted 77 lines
Quoted 65 lines
If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state -->VFIO_DEVICE_STATE_RUNNING&VFIO_DEVICE_STATE_LOGGING|MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->VFIO_DEVICE_STATE_STOPMIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETEDQuoted 7 lines
In source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.This requires iterative reads of device memory buffer but the protocol is unclear (to me) how the user knows how to do this or interact with the position field.
Quoted 9 lines
Quoted 5 lines
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (
GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).What if there's not enough dirty data to fill the region? The data is always padded to fill the region?
I think dirty data in vendor driver is orgnaized in a format like:
(addr0, len0, data0, addr1, len1, data1, addr2, len2, data2,....addrN, lenN, dataN).
for full snapshot, it's like (addr0, len0, data0).
so, to userspace and data region, it doesn't matter whether it's full snapshot or dirty data.Quoted 11 lines
Quoted 6 lines
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.
If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.Therefore through the entire precopy phase we have no data from source to target to begin a compatibility check :-\ I think both proposals currently still lack any sort of device compatibility or data versioning check between source and target. Thanks,
I checked the compatibility, though not good enough:)
in
migration_init,vfio_check_devstate_version()checked version from kernel withVFIO_DEVICE_STATE_INTERFACE_VERSIONin both source and target, and in target side,vfio_load_state()checked source side version.int
vfio_load_state(QEMUFile *f, void *opaque, int version_id){ ... if (version_id != VFIO_DEVICE_STATE_INTERFACE_VERSION) { return -EINVAL; } ... }Thanks
YanQuoted 52 lines
Alex
Quoted 49 lines
In precopy phase, if a device has
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYon, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis off, .log_sync callback just returns without call to vendor driver.In stop-and-copy phase, device state will be set to STOP & LOGGING first. in
save_live_complete_precopycallback,
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capabilityhw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 26 ++ hw/vfio/migration.c | 858 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 10 +- hw/vfio/pci.h | 26 +- include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 260 +++++++++++++ 7 files changed, 1174 insertions(+), 9 deletions(-) create mode 100644 hw/vfio/migration.c_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Hi Yan,
Sorry for the delay, I've been on PTO...
Quoted 74 lines · On Sun, 24 Feb 2019 21:22:56 -0500 Zhao Yan <[email protected]> wrote:
Quoted 57 lines · Alex Williamson
Hi Yan,
Thanks for working on this!
Quoted 7 lines · On Tue, 19 Feb 2019 16:50:54 +0800 Yan Zhao <[email protected]> wrote:
This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.
It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.
Quoted 8 lines
Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and itI'm not sure how we can really impose a limit here, it is what it is for a device. A smaller state is obviously desirable to reduce downtime, but some devices could have very large states.
Quoted 6 lines
needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.So the intention here is that this is the last data read from the device and it's done in one pass, so the region needs to be large enough to expose all config data at once. On restore it's the last data written before switching the device to the run state.
Quoted 2 lines
Device Memory: device's internal memory, standalone and outside system
s/system/VM/
Quoted 1 line
memory. It is usually very big.
Or it doesn't exist. Not sure we should be setting expectations since it will vary per device.
Quoted 7 lines
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.Quoted 3 lines
It seems a little gratuitous to me that this is a separate region or that this data is handled separately. All of this data is opaque to QEMU, so why do we need to separate it?
hi Alex,
as the device state interfaces are provided by kernel, it is expected to meet as general needs as possible. So, do you think there are such use cases from user space that user space knows well of the device, and it wants kernel to return desired data back to it.
E.g. It just wants to get whole device config data including all mmios, page tables, pci config data...
or, It just wants to get current device memory snapshot, not including any dirty data.
Or, It just needs the dirty pages in device memory or system memory. With all this accurate query, quite a lot of useful features can be developped in user space.
Quoted 2 lines
If all of this data is opaque to user app, seems the only use case is for live migration.
I can certainly appreciate a more versatile interface, but I think we're also trying to create the most simple interface we can, with the primary target being live migration. As soon as we start defining this type of device memory and that type of device memory, we're going to have another device come along that needs yet another because they have a slightly different requirement. Even without that, we're going to have vendor drivers implement it differently, so what works for one device for a more targeted approach may not work for all devices. Can you enumerate some specific examples of the use cases you imagine your design to enable?
Quoted 6 lines
From another aspect, if the final solution is to let the data opaque to user space, like what NV did, kernel side's implementation will be more complicated, and actually a little challenge to vendor driver. in that case, in pre-copy phase,
1. in not LOGGING state, vendor driver first returns full data including full device memory snapshot
When we're not LOGGING, does the vendor driver need to return anything? It seems that LOGGING could be considered an enable switch for the interface.
Quoted 6 lines
2. user space reads some data (you can't expect it to finish reading all data)
3. then userspace set the device state to LOGGING to start dirty data logging
4. vendor driver starts dirty data logging, and appends the dirty data to the tail of remaining unread full data and increase the pending data size?
It seems a lot of overhead to expect the vendor driver to consider state read by the user prior to LOGGING being enabled. Does it log those changes forever? It seems like we should consider LOGGING enabled to be a "session".
Quoted 3 lines
5. user space keeps reading data.
6. vendor driver keeps appending new dirty data to the tail of remaining unread full data/dirty data and increase the pending data size?
Until the device is stopped it can always generate new pending date and the size of that pending data needs to be considered volatile by the user, right? What's different here? This all seems to factor into when the user decides whether the migration is converting and whether to transition to stopped phase to force that convergence.
Quoted 9 lines
in stop-and-copy phase
1. user space sets device state to exit LOGGING state,
2. vendor driver stops data logging. it has to append device config data at the tail of remaining dirty data unread by userspace.during this flow, when vendor driver should get dirty data? just keeps logging and appends to tail? how to ensure dirty data is refresh new before LOGGING state exit? how does vendor driver know whether certain dirty data is copied or not?
At stop-and-copy, I'd assume LOGGING remains enabled, only adding STOP, such that the device does not generate new data, but perhaps I've forgotten the details on vacation. As above, I'd think we'd want to bound any sort of dirty state tracking to a session bounded by the LOGGING state. The protocol defined with userspace needs to account for determining what the user has and has not read, for instance to support mmap'd data, a trapped interface needs to be used to setup the data and acknowledge a read of that data.
Quoted 3 lines
I've no idea how NVidia handle this problem, and they don't open their kernel side code.
just feel it's a bit hard for other vendor drivers to follow:)
Their interface proposal is available on the list, I don't have access to their proprietary driver either, but I expect the best ideas from each proposal to be combined into a unified solution.
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in structvfio_device_state_ctl
Three data regions: mmaped into qemu.
Quoted 6 lines
Quoted 3 lines
Is mmap mandatory? I would think this would be defined by the mdev device what access they want to support per region. We don't want to impose a more complicated interface if the device doesn't require it.
I think it's "mmap is preferred, but allowed to fail".
just like a normal region with MMAP flag on (like bar regions), we also allow its mmap to fail, right?
Currently mmap support for any region is optional both from the vendor driver and the user. The vendor driver may or may not support mmap of a region (or subset of region with sparse mmap) and the user may or may not make use of mmap if it is available. The question here was whether this interface requires the vendor driver to support mmap of these device specific regions.
Quoted 20 lines
Quoted 10 lines
Quoted 9 lines
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages(The reason why four seperate regions are defined is that the unit of mmap system call is
PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).Quoted 9 lines
It's not obvious to me how this is better, a big region isn't padded, there's simply a gap in the file descriptor. Is having a sub-PAGE_SIZE gap in a file really of any consequence? Each region beyond the header is more than likely larger than
PAGE_SIZE, therefore they can be nicely aligned together. We still need fields to tell us how much data is available in each area, so another to tell us the start of each area is a minor detail. And I think we still want to allow drivers to specify which parts of which areas support mmap, so I don't think we're getting away from sparse mmap support.
Quoted 8 lines
with seperate regions and sub-region type defined, user space can explictly know which region is which region after
vfio_get_dev_region_info(). along with it, user space knows region offset and size. mmap is allowed to fail and falls back to normal read/write to the region.But with one big region and sparse mmapped subregions (1 data subregion or 3 data subregions, whatever), userspace can't tell which subregion is which one.
Of course they can, this is part of defining the header structure. One region could define a header including config_offset, config_size, memory_offset, memory_size, dirty_offset, dirty_size. Notice how Kirti even uses the same area to support all of these (which leaves some issues with vendor driver flexibility, but at least shows this is possible).
Quoted 8 lines
So, if using one big region, I think we need to explictly define subregions' sequence (like index 0 is dedicated to control subregion, index 1 is for device config data subregion ...). Vendor driver cannot freely change the sequence.
Then keep data offset the same as region->mmaps[i].offset, and data size the same as region->mmaps[i].size. (i.e. let actual data starts immediatly from first byte of its data subregion)
Also, mmaps for sparse mmaped subregions are not allowed to fail.
This doesn't make any sense to me, the vendor driver can define the start and size of each area within the region with simple header fields. We don't need fixed sequence fields. Likewise the sparse mmap capability for the region can define which of those areas within the region support mmap. The mmap can be optional for both vendor driver and user, just as it is elsewhere. The header fields defining the sub-areas can be read-only to the user, the sparse mmap only needs to match what the vendor driver defines and supports.
Quoted 3 lines
With one big region, we also need to consider the case when vendor driver does not want the data subregion to be mmaped.
so, what is the data layout for that case?
Vendor driver defines data_offset and data_size, sparse mmap capability does not list that area as mmap capable.
Quoted 1 line
data subregion immedately follows control subregion, or not?
The header needs to begin at offset zero, the layout of the rest is defined by the vendor driver within this header. I believe this is (mostly) implemented in Kirti's version.
Quoted 11 lines
Of couse, for this condition, we can specify the data filed's start offset and size through control region. And we must not expect the data start offset in source and target are equal.
(because the big region'sfd_offset
may vary in source and target. consider the case when both source and target have one opregion and one device state region, but source has opregion in the first and target has device state region in the first. If we think this case is illegal, we must be able to detect it in the first place).
Also, we must keep the start offset and size consistent with the above mmap case.
AFAICT, these are all non-issues. Please look at Kirti's proposal. The (one) migration region can define a header at offset zero that allows the vendor driver to define where within that region the data, config, and dirty bitmap areas are and the sparse mmap capability defines which of those are mmap capable. Clearly this migration region offset within the vfio device file descriptor is independent between source and target, as are the offsets of the sub-areas within the migration region. These all need to be defined as part of the migration protocol where the source and target implement the same protocol but have no requirement to be absolutely identical.
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
kernel device state interface [1]
--------------------------------------
#defineVFIO_DEVICE_STATE_INTERFACE_VERSION1
#defineVFIO_DEVICE_DATA_CAP_DEVICE_MEMORY1
#defineVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY2If we were to go with this multi-region solution, isn't it evident from the regions exposed that device memory and a dirty bitmap are provided? Alternatively, I believe Kirti's proposal doesn't require
Quoted 8 lines
Quoted 5 lines
this distinction between device memory and device config, a device not requiring runtime migration data would simply report no data until the device moved to the stopped state, making it consistent path for userspace. Likewise, the dirty bitmap could report a zero page count in the bitmap rather than branching based on device support.
If the path in userspace is consistent for device config and device memory, there will be many unnecessary call of getting data size into vendor driver.
Consistency seems like a good thing, it makes code more simple, we don't behave differently in one case versus another. If the vendor reports no data, skip. It also provides versatility. Are the "many unnecessary call[s]" quantifiable?
Quoted 34 lines
Quoted 34 lines
Note that it seems that cap
VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYimpliesVFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP, but there's no consistency in the naming.Quoted 3 lines
#define
VFIO_DEVICE_STATE_RUNNING0
#defineVFIO_DEVICE_STATE_STOP1
#defineVFIO_DEVICE_STATE_LOGGING2It looks like these are being defined as bits, since patch 1 talks about RUNNING & LOGGING and STOP & LOGGING. I think Connie already posted some comments about this. I'm not sure anything prevents us from defining RUNNING a 1 and STOPPED as 0 so we don't have the polarity flip vs LOGGING though.
The state "STOP & LOGGING" also feels like a strange "device state", if the device is stopped, it's not logging any new state, so I think this is more that the device state is STOP, but the LOGGING feature is active. Maybe we should consider these independent bits. LOGGING is active as we stop a device so that we can fetch the last dirtied pages, but disabled as we load the state of the device into the target.
Quoted 12 lines
#define
VFIO_DEVICE_DATA_ACTION_GET_BUFFER1
#defineVFIO_DEVICE_DATA_ACTION_SET_BUFFER2
#defineVFIO_DEVICE_DATA_ACTION_GET_BITMAP3struct vfio_device_state_ctl { __u32 version; /* ro */ __u32 device_state; /* VFIO device state, wo */ __u32 caps; /* ro */ struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /*rw*/ } device_config;
Quoted 7 lines
Quoted 5 lines
Patch 1 indicates that to get the config buffer we write
GET_BUFFERto action and read from the config region. The size is previously read and apparently constant. To set the config buffer, the config region is written followed by writingSET_BUFFERto action. Why is size listed as read-write?this is the size of config data.
size of config data <= size of config data region.
Where in the usage protocol does the user WRITE the config data size?
Quoted 10 lines
Quoted 10 lines
Doesn't this protocol also require that the mdev driver consume each full region's worth of host kernel memory for backing pages in anticipation of a rare event like migration? This might be a strike against separate regions if the driver needs to provide backing pages for 3 separate regions vs 1. To avoid this runtime overhead, would it be expected that the user only mmap the regions during migration and the mdev driver allocate backing pages on mmap? Should the mmap be restricted to the LOGGING feature being enabled? Maybe I'm mistaken in how the mdev driver would back these mmap'd pages.
Quoted 6 lines
yes, 3 seperate regions consume a little more memory than 1 region. but it's just a little overhead.
As in intel's kernel implementation,
device config region's size is 9M, dirty bitmap region's size is 16k. if there is device memory region, its size can be defined as 100M? so it's 109M vs 100M ?
But what if it's 100M config with no device memory? This proposal requires 100M in-kernel backing due to the definition of the config region when it could be implemented with significantly less by allowing a small data area to be read multiple times until a bytes remaining counter becomes zero.
Quoted 52 lines
Quoted 5 lines
Quoted 4 lines
struct { __u32 action; /* wo, GET_BUFFER or SET_BUFFER */ __u64 size; /* rw */ __u64 pos; /*the offset in total buffer of device memory*/Quoted 8 lines
Patch 1 outlines the protocol here that getting device memory begins with writing the position field, followed by reading from the device memory region. Setting device memory begins with writing the data to the device memory region, followed by writing the position field. Why does the user need to have visibility of data position? This is opaque data to the user, the device should manage how the chunks fit together.
How does the user know when they reach the end?
sorry, maybe I didn't explain clearly here.
device ________________________________________ memory | | |////| | | | | | data: |____|____|////|____|____|____|____|____| :pos : : : device :____: memory | | region: |____|the whole sequence is like this:
1. user space reads device_memory.size
2. driver gets device memory'sdata(full snapshot or dirty data, depending on it's in LOGGING state or not), and return the total size of this data.
3. user space finishes reading device_memory.size (>= device memory region's size)4. user space starts a loop like
while (pos < total_len) { uint64_t len = region_devmem->size;if (pos + len >= total_len) { len = total_len - pos; } if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) { return -1; } pos += len; }
vfio_save_data_device_memory_chunkreads each chunk from device memory region by writingGET_BUFFERto device_memory.action, and pos to device_memory.pos.
Quoted 4 lines
so. each time, userspace will finish getting device memory data in one time.
specifying "pos" is just like the "lseek" before "write".
This could also be implemented as a remaining bytes counter in the interface where the vendor driver wouldn't rely on the user to manage the position. What internal consistency checking is going to protect the host kernel when the user writes data to the wrong position? If we consider the data to be opaque, the vendor driver can embed that sort of meta data into the data blob the user reads and reassemble it correctly or generate a consistency failure itself.
Quoted 88 lines
Quoted 9 lines
Bullets 8 and 9 in patch 1 also discuss setting and getting the device memory size, but these aren't well integrated into the protocol for getting and setting the memory buffer. Is getting the device memory really started by reading the size, which triggers the vendor driver to snapshot the state in an internal buffer which the user then iterates through using
GET_BUFFER? Therefore re-reading the size field could corrupt the data stream? Wouldn't it be easier to report bytes available and countdown as the user marks them read? What does position mean when we switch from snapshot to dirty data?when switching to device memory's dirty data, pos means the pos in whole dirty data.
.save_live_pending ==> driver gets dirty data in device memory and returns total size.
.save_live_iterate ==> userspace reads all dirty data from device memory region chunk by chunk
So, in an iteration, all dirty data are saved. then in next iteration, dirty data is recalculated.
Quoted 8 lines
Quoted 6 lines
} device_memory; struct { __u64 start_addr; /* wo */ __u64 page_nr; /* wo */ } system_memory; };Why is one specified as an address and the other as pages? Note that
Yes,
start_addr==> start pfn is betterQuoted 3 lines
Kirti's implementation has an optimization to know how many pages are set within a range to avoid unnecessary buffer reads.
Let's use
start_pfn_all,page_nr_allto represent the start pfn andpage_nrpassed in from qemu .log_sync interface.and use
start_pfn_i,page_nr_ito the value passed to driver.
start_pfn_all| start_pfn_i | | \ /_______\_/_____________________________ | | |////| | | | | | |____|____|////|____|____|____|____|____| : : : : :____: bitmap | | region: |____|1. Each time QEMU queries dirty bitmap from driver, it passes in
start_pfn_i, andpage_nr_i. (page_nr_iis the biggest page number the bitmap region can hold).
2. driver queries memory range fromstart_pfn_iwith sizepage_nr_i.
3. driver return a bitmap (if no dirty data, the bitmap is all 0).
4. QEMU saves the pages according to the bitmapIf there's no dirty data found in step 2, step 4 can be skipped. (I'll add this check before step 4 in future, thanks)
but if there's even 1 bit set in the bitmap, no step from 1-4 can be skipped.Honestly, after reviewing Kirti's implementation, I don't think it's an optimization. As in below pseudo code for Kirti's code, I would think the
copied_pfnscorresponds to thepage_nr_iin my case. so, the case ofcopied_pfnsequaling to 0 is for the tail chunk? don't think it's working..write
start_pfnto driver
writepage_sizeto driver
writepfn_countto driverdo { read copied_pfns from driver. if (copied_pfns == 0) { break; } bitmap_size = (BITS_TO_LONGS(copied_pfns) + 1) * sizeof(unsigned long); buf = get bitmap from driver cpu_physical_memory_set_dirty_lebitmap((unsigned long *)buf, (start_pfn + count) * page_size, copied_pfns); count += copied_pfns;
Quoted 1 line
} while (count <
pfn_count);
The intent of Kirti's copied_pfns is clearly to avoid unnecessarily reading pages from the kernel when nothing has changed. Perhaps the implementation still requires work, but I don't see from above how that's not considered an optimization.
Quoted 168 lines
Quoted 51 lines
Quoted 42 lines
Devcie States
-------------
After migration is initialzed, it will set device state via writing todevice_statefield of control region.Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOPRUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.STOP: In this state, a VFIO device is deactivated to interact with
device driver.LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that regionvfio_device_state_ctlfails to get registered. In that condition, a device needs be in active state by default.Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.Then it will probe VFIO device's data capability by reading caps field of control region.
Quoted 8 lines
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1 #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
IfVFIO_DEVICE_DATA_CAP_SYSTEM_MEMORYis on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.Quoted 3 lines
As above, these capabilities seem redundant to the existence of the device specific regions in this implementation.
seems so :)
Quoted 77 lines
Quoted 65 lines
If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.
Flows to call device state interface for VFIO live migration
------------------------------------------------------------Live migration save path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state -->VFIO_DEVICE_STATE_RUNNING&VFIO_DEVICE_STATE_LOGGING|MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP&VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->VFIO_DEVICE_STATE_STOPMIGRATION_STATUS_COMPLETED
MIGRATION_STATUS_CANCELLEDorMIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNINGLive migration load path:
(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)
MIGRATION_STATUS_NONE-->VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state -->VFIO_DEVICE_STATE_STOP
|MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state -->VFIO_DEVICE_STATE_RUNNING
|MIGRATION_STATUS_COMPLETEDQuoted 7 lines
In source VM side,
In precopy phase,
if a device hasVFIO_DEVICE_DATA_CAP_DEVICE_MEMORYon,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.This requires iterative reads of device memory buffer but the protocol is unclear (to me) how the user knows how to do this or interact with the position field.
Quoted 9 lines
Quoted 5 lines
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (
GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).What if there's not enough dirty data to fill the region? The data is always padded to fill the region?
I think dirty data in vendor driver is orgnaized in a format like:
(addr0, len0, data0, addr1, len1, data1, addr2, len2, data2,....addrN, lenN, dataN).
for full snapshot, it's like (addr0, len0, data0).
so, to userspace and data region, it doesn't matter whether it's full snapshot or dirty data.Quoted 11 lines
Quoted 6 lines
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.
If
VFIO_DEVICE_DATA_CAP_DEVICE_MEMORYis off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.Therefore through the entire precopy phase we have no data from source to target to begin a compatibility check :-\ I think both proposals currently still lack any sort of device compatibility or data versioning check between source and target. Thanks,
I checked the compatibility, though not good enough:)
in
migration_init,vfio_check_devstate_version()checked version from kernel withVFIO_DEVICE_STATE_INTERFACE_VERSIONin both source and target, and in target side,vfio_load_state()checked source side version.
Quoted 8 lines
int
vfio_load_state(QEMUFile *f, void *opaque, int version_id){ ... if (version_id != VFIO_DEVICE_STATE_INTERFACE_VERSION) { return -EINVAL; } ... }
But this only checks that both source and target are using the same migration interface, how do we know that they're compatible devices and that the vendor data stream is compatible between source and target? Whether both ends use the same migration interface is potentially not relevant if the data stream is compatible. Thanks,
Alex