Skip to contentSkip to conversation

QEMU VFIO live migration

PATCH In review Yan Zhao 8 participants 77 messages Feb 19, 2019 – Apr 1, 2019 KVM Page 1 of 2

lore
Share & tools
lore

Yan Zhao

author
#0 lore raw

cover QEMU VFIO live migration

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;    /*rw*/
	} device_config;
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;     /* rw */  
                __u64 pos; /*the offset in total buffer of device memory*/
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING & VFIO_DEVICE_STATE_LOGGING |
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH 1/5 vfio/migration: define kernel interfaces

- defined 4 device states regions: one control region and 3 data regions
- defined layout of control region in struct vfio_device_state_ctl
- defined 4 device states: running, stop, running&logging, stop&logging
- define 3 device data categories: device config, device memory, system memory
- defined 2 device data capabilities: device memory and system memory
- defined device state interfaces' version and 12 device state interfaces

---
 linux-headers/linux/vfio.h | 260 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 260 insertions(+)
linux-headers/linux/vfio.h +260 -0
index ceb6453..a124fc1 100644
@@ -303,6 +303,56 @@ struct vfio_region_info_cap_type {
 #define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG	(2)
 #define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG	(3)
 
+/* Device State region type and sub-type
+ *
+ * A VFIO device driver needs to register up to four device state regions in
+ * total: two mandatory and another two optional, if it plans to support device
+ * state management.
+ *
+ * 1. region CTL :
+ *          Mandatory.
+ *          This is a control region.
+ *          Its layout is defined in struct vfio_device_state_ctl.
+ *          Reading from this region can get version, capabilities and data
+ *          size of device state interfaces.
+ *          Writing to this region can set device state, data size and
+ *          choose which interface to use.
+ * 2. region DEVICE_CONFIG
+ *          Mandatory.
+ *          This is a data region that holds device config data.
+ *          Device config is such kind of data like MMIOs, page tables...
+ *          Every device is supposed to possess device config data.
+ *          Usually the size of device config data is small (no big
+ *          than 10M), and it needs to be loaded in certain strict
+ *          order.
+ *          Therefore no dirty data logging is enabled for device
+ *          config and it must be got/set as a whole.
+ *          Size of device config data is smaller than or equal to that of
+ *          device config region.
+ *          It is able to be mmaped into user space.
+ * 3. region DEVICE_MEMORY
+ *          Optional.
+ *          This is a data region that holds device memory data.
+ *          Device memory is device's internal memory, standalone and outside
+ *          system memory.  It is usually very big.
+ *          Not all device has device memory. Like IGD only uses system
+ *          memory and has no device memory.
+ *          Size of devie memory is usually larger than that of device
+ *          memory region. qemu needs to save/load it in chunks of size of
+ *          device memory region.
+ *          It is able to be mmaped into user space.
+ * 4. region DIRTY_BITMAP
+ *          Optional.
+ *          This is a data region that holds bitmap of dirty pages in system
+ *          memory that a VFIO devices produces.
+ *          It is able to be mmaped into user space.
+ */
+#define VFIO_REGION_TYPE_DEVICE_STATE           (1 << 1)
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL       (1)
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG      (2)
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY      (3)
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP (4)
+
 /*
  * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped
  * which allows direct access to non-MSIX registers which happened to be within
@@ -816,6 +866,216 @@ struct vfio_iommu_spapr_tce_remove {
 };
 #define VFIO_IOMMU_SPAPR_TCE_REMOVE	_IO(VFIO_TYPE, VFIO_BASE + 20)
 
+/* version number of the device state interface */
+#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
+
+/*
+ * For devices that have devcie memory, it is required to expose
+ * DEVICE_MEMORY capability.
+ *
+ * For devices producing dirty pages in system memory, it is required to
+ * expose cap SYSTEM_MEMORY in order to get dirty bitmap in certain range
+ * of system memory.
+ */
+#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
+#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2
+
+/*
+ * DEVICE STATES
+ *
+ * Four states are defined for a VFIO device:
+ * RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP.
+ * They can be set by writing to device_state field of
+ * vfio_device_state_ctl region.
+ *
+ * RUNNING: In this state, a VFIO device is in active state ready to
+ * receive commands from device driver.
+ * It is the default state that a VFIO device enters initially.
+ *
+ * STOP: In this state, a VFIO device is deactivated to interact with
+ * device driver.
+ *
+ * LOGGING state is a special state that it CANNOT exist
+ * independently.
+ * It must be set alongside with state RUNNING or STOP, i.e,
+ * RUNNING & LOGGING, STOP & LOGGING.
+ * It is used for dirty data logging both for device memory
+ * and system memory.
+ *
+ * LOGGING only impacts device/system memory. In LOGGING state, get buffer
+ * of device memory returns dirty pages since last call; outside LOGGING
+ * state, get buffer of device memory returns whole snapshot of device
+ * memory. system memory's dirty page is only available in LOGGING state.
+ *
+ * Device config should be always accessible and return whole config snapshot
+ * regardless of LOGGING state.
+ * */
+#define VFIO_DEVICE_STATE_RUNNING 0
+#define VFIO_DEVICE_STATE_STOP 1
+#define VFIO_DEVICE_STATE_LOGGING 2
+
+/* action to get data from device memory or device config
+ * the action is write to device state's control region, and data is read
+ * from device memory region or device config region.
+ * Each time before read device memory region or device config region,
+ * action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action
+ * field in control region. That is because device memory and devie config
+ * region is mmaped into user space. vendor driver has to be notified of
+ * the the GET_BUFFER action in advance.
+ */
+#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
+
+/* action to set data to device memory or device config
+ * the action is write to device state's control region, and data is
+ * written to device memory region or device config region.
+ * Each time after write to device memory region or device config region,
+ * action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action
+ * field in control region. That is because device memory and devie config
+ * region is mmaped into user space. vendor driver has to be notified of
+ * the the SET_BUFFER action after data written.
+ */
+#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
+
+/* layout of device state interfaces' control region
+ * By reading to control region and reading/writing data from device config
+ * region, device memory region, system memory regions, below interface can
+ * be implemented:
+ *
+ * 1. get version
+ *   (1) user space calls read system call on "version" field of control
+ *   region.
+ *   (2) vendor driver writes version number of device state interfaces
+ *   to the "version" field of control region.
+ *
+ * 2. get caps
+ *   (1) user space calls read system call on "caps" field of control region.
+ *   (2) if a VFIO device has huge device memory, vendor driver reports
+ *      VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY in "caps" field of control region.
+ *      if a VFIO device produces dirty pages in system memory, vendor driver
+ *      reports VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY in "caps" field of
+ *      control region.
+ *
+ * 3. set device state
+ *    (1) user space calls write system call on "device_state" field of
+ *    control region.
+ *    (2) device state transitions as:
+ *
+ *    RUNNING -- start dirty data logging --> RUNNING & LOGGING
+ *    RUNNING -- deactivate --> STOP
+ *    RUNNING -- deactivate & start dirty data longging --> STOP & LOGGING
+ *    RUNNING & LOGGING -- stop dirty data logging --> RUNNING
+ *    RUNNING & LOGGING -- deactivate --> STOP & LOGGING
+ *    RUNNING & LOGGING -- deactivate & stop dirty data logging --> STOP
+ *    STOP -- activate --> RUNNING
+ *    STOP -- start dirty data logging --> STOP & LOGGING
+ *    STOP -- activate & start dirty data logging --> RUNNING & LOGGING
+ *    STOP & LOGGING -- stop dirty data logging --> STOP
+ *    STOP & LOGGING -- activate --> RUNNING & LOGGING
+ *    STOP & LOGGING -- activate & stop dirty data logging --> RUNNING
+ *
+ * 4. get device config size
+ *   (1) user space calls read system call on "device_config.size" field of
+ *       control region for the total size of device config snapshot.
+ *   (2) vendor driver writes device config data's total size in
+ *       "device_config.size" field of control region.
+ *
+ * 5. set device config size
+ *   (1) user space calls write system call.
+ *       total size of device config snapshot --> "device_config.size" field
+ *       of control region.
+ *   (2) vendor driver reads device config data's total size from
+ *       "device_config.size" field of control region.
+ *
+ * 6 get device config buffer
+ *   (1) user space calls write system call.
+ *       "GET_BUFFER" --> "device_config.action" field of control region.
+ *   (2) vendor driver
+ *       a. gets whole snapshot for device config
+ *       b. writes whole device config snapshot to region
+ *       DEVICE_CONFIG.
+ *   (3) user space reads the whole of device config snapshot from region
+ *       DEVICE_CONFIG.
+ *
+ * 7. set device config buffer
+ *   (1) user space writes whole of device config data to region
+ *       DEVICE_CONFIG.
+ *   (2) user space calls write system call.
+ *       "SET_BUFFER" --> "device_config.action" field of control region.
+ *   (3) vendor driver loads whole of device config from region DEVICE_CONFIG.
+ *
+ * 8. get device memory size
+ *   (1) user space calls read system call on "device_memory.size" field of
+ *       control region for device memory size.
+ *   (2) vendor driver
+ *       a. gets device memory snapshot (in state RUNNING or STOP), or
+ *          gets device memory dirty data (in state RUNNING & LOGGING or
+ *          state STOP & LOGGING)
+ *       b. writes size in "device_memory.size" field of control region
+ *
+ * 9. set device memory size
+ *   (1) user space calls write system call on "device_memory.size" field of
+ *       control region to set total size of device memory snapshot.
+ *   (2) vendor driver reads device memory's size from "device_memory.size"
+ *       field of control region.
+ *
+ *
+ * 10. get device memory buffer
+ *   (1) user space calls write system.
+ *       pos --> "device_memory.pos" field of control region,
+ *       "GET_BUFFER" --> "device_memory.action" field of control region.
+ *       (pos must be 0 or multiples of length of region DEVICE_MEMORY).
+ *   (2) vendor driver writes N'th chunk of device memory snapshot/dirty data
+ *       to region DEVICE_MEMORY.
+ *       (N equals to pos/(region length of DEVICE_MEMORY))
+ *   (3) user space reads the N'th chunk of device memory snapshot/dirty data
+ *       from region DEVICE_MEMORY.
+ *
+ * 11. set device memory buffer
+ *   (1) user space writes N'th chunk of device memory snapshot/dirty data to
+ *       region DEVICE_MEMORY.
+ *       (N equals to pos/(region length of DEVICE_MEMORY))
+ *   (2) user space writes pos to "device_memory.pos" field and writes
+ *       "SET_BUFFER" to "device_memory.action" field of control region.
+ *   (3) vendor driver loads N'th chunk of device memory snapshot/dirty data
+ *       from region DEVICE_MEMORY.
+ *
+ * 12. get system memory dirty bitmap
+ *   (1) user space calls write system call to specify a range of system
+ *       memory that querying dirty pages.
+ *       system memory's start address --> "system_memory.start_addr" field
+ *       of control region,
+ *       system memory's page count --> "system_memory.page_nr" field of
+ *       control region.
+ *   (2) if device state is not in RUNNING or STOP & LOGGING,
+ *       vendor driver returns empty bitmap; otherwise,
+ *       vendor driver checks the page_nr,
+ *       if it's larger than the size that region DIRTY_BITMAP can support,
+ *       error returns; if not,
+ *       vendor driver returns as bitmap to specify dirty pages that
+ *       device produces since last query in this range of system memory .
+ *   (3) usespace reads back the dirty bitmap from region DIRTY_BITMAP.
+ *
+ */
+
+struct vfio_device_state_ctl {
+	__u32 version;		  /* ro versio of devcie state interfaces*/
+	__u32 device_state;       /* VFIO device state, wo */
+	__u32 caps;		 /* ro */
+        struct {
+		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */
+		__u64 size;    /*rw, total size of device config*/
+	} device_config;
+	struct {
+		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */
+		__u64 size;     /* rw, total size of device memory*/
+        __u64 pos;/*chunk offset in total buffer of device memory*/
+	} device_memory;
+	struct {
+		__u64 start_addr; /* wo */
+		__u64 page_nr;   /* wo */
+	} system_memory;
+}__attribute__((packed));
+
 /* ***************************************************************** */
 
 #endif /* VFIO_H */
Signature · 1 line
2.7.4
#2 lore raw

Re: [PATCH 1/5] vfio/migration: define kernel interfaces

Quoted 8 lines · On Tue, 19 Feb 2019 16:52:14 +0800 Yan Zhao <[email protected]> wrote:

- defined 4 device states regions: one control region and 3 data regions
- defined layout of control region in struct vfio_device_state_ctl
- defined 4 device states: running, stop, running&logging, stop&logging
- define 3 device data categories: device config, device memory, system memory
- defined 2 device data capabilities: device memory and system memory
- defined device state interfaces' version and 12 device state interfaces

Quoted 6 lines
---
 linux-headers/linux/vfio.h | 260 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 260 insertions(+)

[commenting here for convenience; changes obviously need to be done in the Linux patch]

Quoted 6 lines
linux-headers/linux/vfio.h +0 -0
index ceb6453..a124fc1 100644

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 310

Quoted 8 lines
linux-headers/linux/vfio.h +5 -0
@@ -303,6 +303,56 @@ struct vfio_region_info_cap_type {
 #define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG	(2)
 #define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG	(3)
 
+/* Device State region type and sub-type
+ *
+ * A VFIO device driver needs to register up to four device state regions in
+ * total: two mandatory and another two optional, if it plans to support device
+ * state management.

Suggest to rephrase:

"A VFIO device driver that plans to support device state management needs to register..."

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 323

Quoted 13 lines
linux-headers/linux/vfio.h +13 -0
+ *
+ * 1. region CTL :
+ *          Mandatory.
+ *          This is a control region.
+ *          Its layout is defined in struct vfio_device_state_ctl.
+ *          Reading from this region can get version, capabilities and data
+ *          size of device state interfaces.
+ *          Writing to this region can set device state, data size and
+ *          choose which interface to use.
+ * 2. region DEVICE_CONFIG
+ *          Mandatory.
+ *          This is a data region that holds device config data.
+ *          Device config is such kind of data like MMIOs, page tables...

"Device config is data such as..."

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 325

Quoted 2 lines
linux-headers/linux/vfio.h +2 -0
+ *          Every device is supposed to possess device config data.
+ *          Usually the size of device config data is small (no big

s/no big/no bigger/

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 331

Quoted 6 lines
linux-headers/linux/vfio.h +6 -0
+ *          than 10M), and it needs to be loaded in certain strict
+ *          order.
+ *          Therefore no dirty data logging is enabled for device
+ *          config and it must be got/set as a whole.
+ *          Size of device config data is smaller than or equal to that of
+ *          device config region.

Not sure if I understand that sentence correctly... but what if a device has more config state than fits into this region? Is that supposed to be covered by the device memory region? Or is this assumed to be something so exotic that we don't need to plan for it?

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 336

Quoted 5 lines
linux-headers/linux/vfio.h +5 -0
+ *          It is able to be mmaped into user space.
+ * 3. region DEVICE_MEMORY
+ *          Optional.
+ *          This is a data region that holds device memory data.
+ *          Device memory is device's internal memory, standalone and outside

s/outside/distinct from/ ?

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 338

Quoted 2 lines
linux-headers/linux/vfio.h +2 -0
+ *          system memory.  It is usually very big.
+ *          Not all device has device memory. Like IGD only uses system

s/all devices has/all devices have/

s/Like/E.g./

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 340

Quoted 2 lines
linux-headers/linux/vfio.h +2 -0
+ *          memory and has no device memory.
+ *          Size of devie memory is usually larger than that of device

s/devie/device/

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 342

Quoted 2 lines
linux-headers/linux/vfio.h +2 -0
+ *          memory region. qemu needs to save/load it in chunks of size of
+ *          device memory region.

I'd rather not explicitly mention QEMU in this header. Maybe "Userspace"?

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 350

Quoted 8 lines
linux-headers/linux/vfio.h +8 -0
+ *          It is able to be mmaped into user space.
+ * 4. region DIRTY_BITMAP
+ *          Optional.
+ *          This is a data region that holds bitmap of dirty pages in system
+ *          memory that a VFIO devices produces.
+ *          It is able to be mmaped into user space.
+ */
+#define VFIO_REGION_TYPE_DEVICE_STATE           (1 << 1)

Can you make this an explicit number instead?

(FWIW, I plan to add a CCW region as type 2, whatever comes first.)

Quoted 9 lines
linux-headers/linux/vfio.h +5 -0
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL       (1)
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG      (2)
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY      (3)
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP (4)
+
 /*
  * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped
  * which allows direct access to non-MSIX registers which happened to be within

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 870

Quoted 5 lines
linux-headers/linux/vfio.h +2 -0
@@ -816,6 +866,216 @@ struct vfio_iommu_spapr_tce_remove {
 };
 #define VFIO_IOMMU_SPAPR_TCE_REMOVE	_IO(VFIO_TYPE, VFIO_BASE + 20)
 
+/* version number of the device state interface */
+#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1

Hm. Is this supposed to be backwards-compatible, should we need to bump this?

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 873

Quoted 3 lines
linux-headers/linux/vfio.h +3 -0
+
+/*
+ * For devices that have devcie memory, it is required to expose

s/devcie/device/

Quoted 6 lines
linux-headers/linux/vfio.h +6 -0
+ * DEVICE_MEMORY capability.
+ *
+ * For devices producing dirty pages in system memory, it is required to
+ * expose cap SYSTEM_MEMORY in order to get dirty bitmap in certain range
+ * of system memory.
+ */

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 889

Quoted 10 lines
linux-headers/linux/vfio.h +10 -0
+#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
+#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2
+
+/*
+ * DEVICE STATES
+ *
+ * Four states are defined for a VFIO device:
+ * RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP.
+ * They can be set by writing to device_state field of
+ * vfio_device_state_ctl region.

Who controls this? Userspace?

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 896

Quoted 7 lines
linux-headers/linux/vfio.h +7 -0
+ *
+ * RUNNING: In this state, a VFIO device is in active state ready to
+ * receive commands from device driver.
+ * It is the default state that a VFIO device enters initially.
+ *
+ * STOP: In this state, a VFIO device is deactivated to interact with
+ * device driver.

I think 'STOPPED' would read nicer.

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 899

Quoted 3 lines
linux-headers/linux/vfio.h +3 -0
+ *
+ * LOGGING state is a special state that it CANNOT exist
+ * independently.

So it's not a state, but rather a modifier?

Quoted 28 lines
linux-headers/linux/vfio.h +28 -0
+ * It must be set alongside with state RUNNING or STOP, i.e,
+ * RUNNING & LOGGING, STOP & LOGGING.
+ * It is used for dirty data logging both for device memory
+ * and system memory.
+ *
+ * LOGGING only impacts device/system memory. In LOGGING state, get buffer
+ * of device memory returns dirty pages since last call; outside LOGGING
+ * state, get buffer of device memory returns whole snapshot of device
+ * memory. system memory's dirty page is only available in LOGGING state.
+ *
+ * Device config should be always accessible and return whole config snapshot
+ * regardless of LOGGING state.
+ * */
+#define VFIO_DEVICE_STATE_RUNNING 0
+#define VFIO_DEVICE_STATE_STOP 1
+#define VFIO_DEVICE_STATE_LOGGING 2
+
+/* action to get data from device memory or device config
+ * the action is write to device state's control region, and data is read
+ * from device memory region or device config region.
+ * Each time before read device memory region or device config region,
+ * action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action
+ * field in control region. That is because device memory and devie config
+ * region is mmaped into user space. vendor driver has to be notified of
+ * the the GET_BUFFER action in advance.
+ */
+#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
+

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 937

Quoted 10 lines
linux-headers/linux/vfio.h +10 -0
+/* action to set data to device memory or device config
+ * the action is write to device state's control region, and data is
+ * written to device memory region or device config region.
+ * Each time after write to device memory region or device config region,
+ * action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action
+ * field in control region. That is because device memory and devie config
+ * region is mmaped into user space. vendor driver has to be notified of
+ * the the SET_BUFFER action after data written.
+ */
+#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2

Let me describe this in my own words to make sure that I understand this correctly.

- The actions are set by userspace to notify the kernel that it is going to get data or that it has just written data.
- This is needed as a notification that the mmapped data should not be changed resp. just has changed.

So, how does the kernel know whether the read action has finished resp. whether the write action has started? Even if userspace reads/writes it as a whole.

Quoted 111 lines
linux-headers/linux/vfio.h +111 -0
+
+/* layout of device state interfaces' control region
+ * By reading to control region and reading/writing data from device config
+ * region, device memory region, system memory regions, below interface can
+ * be implemented:
+ *
+ * 1. get version
+ *   (1) user space calls read system call on "version" field of control
+ *   region.
+ *   (2) vendor driver writes version number of device state interfaces
+ *   to the "version" field of control region.
+ *
+ * 2. get caps
+ *   (1) user space calls read system call on "caps" field of control region.
+ *   (2) if a VFIO device has huge device memory, vendor driver reports
+ *      VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY in "caps" field of control region.
+ *      if a VFIO device produces dirty pages in system memory, vendor driver
+ *      reports VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY in "caps" field of
+ *      control region.
+ *
+ * 3. set device state
+ *    (1) user space calls write system call on "device_state" field of
+ *    control region.
+ *    (2) device state transitions as:
+ *
+ *    RUNNING -- start dirty data logging --> RUNNING & LOGGING
+ *    RUNNING -- deactivate --> STOP
+ *    RUNNING -- deactivate & start dirty data longging --> STOP & LOGGING
+ *    RUNNING & LOGGING -- stop dirty data logging --> RUNNING
+ *    RUNNING & LOGGING -- deactivate --> STOP & LOGGING
+ *    RUNNING & LOGGING -- deactivate & stop dirty data logging --> STOP
+ *    STOP -- activate --> RUNNING
+ *    STOP -- start dirty data logging --> STOP & LOGGING
+ *    STOP -- activate & start dirty data logging --> RUNNING & LOGGING
+ *    STOP & LOGGING -- stop dirty data logging --> STOP
+ *    STOP & LOGGING -- activate --> RUNNING & LOGGING
+ *    STOP & LOGGING -- activate & stop dirty data logging --> RUNNING
+ *
+ * 4. get device config size
+ *   (1) user space calls read system call on "device_config.size" field of
+ *       control region for the total size of device config snapshot.
+ *   (2) vendor driver writes device config data's total size in
+ *       "device_config.size" field of control region.
+ *
+ * 5. set device config size
+ *   (1) user space calls write system call.
+ *       total size of device config snapshot --> "device_config.size" field
+ *       of control region.
+ *   (2) vendor driver reads device config data's total size from
+ *       "device_config.size" field of control region.
+ *
+ * 6 get device config buffer
+ *   (1) user space calls write system call.
+ *       "GET_BUFFER" --> "device_config.action" field of control region.
+ *   (2) vendor driver
+ *       a. gets whole snapshot for device config
+ *       b. writes whole device config snapshot to region
+ *       DEVICE_CONFIG.
+ *   (3) user space reads the whole of device config snapshot from region
+ *       DEVICE_CONFIG.
+ *
+ * 7. set device config buffer
+ *   (1) user space writes whole of device config data to region
+ *       DEVICE_CONFIG.
+ *   (2) user space calls write system call.
+ *       "SET_BUFFER" --> "device_config.action" field of control region.
+ *   (3) vendor driver loads whole of device config from region DEVICE_CONFIG.
+ *
+ * 8. get device memory size
+ *   (1) user space calls read system call on "device_memory.size" field of
+ *       control region for device memory size.
+ *   (2) vendor driver
+ *       a. gets device memory snapshot (in state RUNNING or STOP), or
+ *          gets device memory dirty data (in state RUNNING & LOGGING or
+ *          state STOP & LOGGING)
+ *       b. writes size in "device_memory.size" field of control region
+ *
+ * 9. set device memory size
+ *   (1) user space calls write system call on "device_memory.size" field of
+ *       control region to set total size of device memory snapshot.
+ *   (2) vendor driver reads device memory's size from "device_memory.size"
+ *       field of control region.
+ *
+ *
+ * 10. get device memory buffer
+ *   (1) user space calls write system.
+ *       pos --> "device_memory.pos" field of control region,
+ *       "GET_BUFFER" --> "device_memory.action" field of control region.
+ *       (pos must be 0 or multiples of length of region DEVICE_MEMORY).
+ *   (2) vendor driver writes N'th chunk of device memory snapshot/dirty data
+ *       to region DEVICE_MEMORY.
+ *       (N equals to pos/(region length of DEVICE_MEMORY))
+ *   (3) user space reads the N'th chunk of device memory snapshot/dirty data
+ *       from region DEVICE_MEMORY.
+ *
+ * 11. set device memory buffer
+ *   (1) user space writes N'th chunk of device memory snapshot/dirty data to
+ *       region DEVICE_MEMORY.
+ *       (N equals to pos/(region length of DEVICE_MEMORY))
+ *   (2) user space writes pos to "device_memory.pos" field and writes
+ *       "SET_BUFFER" to "device_memory.action" field of control region.
+ *   (3) vendor driver loads N'th chunk of device memory snapshot/dirty data
+ *       from region DEVICE_MEMORY.
+ *
+ * 12. get system memory dirty bitmap
+ *   (1) user space calls write system call to specify a range of system
+ *       memory that querying dirty pages.
+ *       system memory's start address --> "system_memory.start_addr" field
+ *       of control region,
+ *       system memory's page count --> "system_memory.page_nr" field of
+ *       control region.

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1058

Quoted 10 lines
linux-headers/linux/vfio.h +10 -0
+ *   (2) if device state is not in RUNNING or STOP & LOGGING,
+ *       vendor driver returns empty bitmap; otherwise,
+ *       vendor driver checks the page_nr,
+ *       if it's larger than the size that region DIRTY_BITMAP can support,
+ *       error returns; if not,
+ *       vendor driver returns as bitmap to specify dirty pages that
+ *       device produces since last query in this range of system memory .
+ *   (3) usespace reads back the dirty bitmap from region DIRTY_BITMAP.
+ *
+ */

It might make sense to extract the explanations above into a separate design document in the kernel Documentation/ directory. You could also add ASCII art there :)

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1061

Quoted 3 lines
linux-headers/linux/vfio.h +3 -0
+
+struct vfio_device_state_ctl {
+	__u32 version;		  /* ro versio of devcie state interfaces*/

s/versio/version/
s/devcie/device/

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1064

Quoted 3 lines
linux-headers/linux/vfio.h +3 -0
+	__u32 device_state;       /* VFIO device state, wo */
+	__u32 caps;		 /* ro */
+        struct {

Indentation looks a bit off.

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1071

Quoted 7 lines
linux-headers/linux/vfio.h +7 -0
+		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */
+		__u64 size;    /*rw, total size of device config*/
+	} device_config;
+	struct {
+		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */
+		__u64 size;     /* rw, total size of device memory*/
+        __u64 pos;/*chunk offset in total buffer of device memory*/

Here as well.

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1077

Quoted 6 lines
linux-headers/linux/vfio.h +6 -0
+	} device_memory;
+	struct {
+		__u64 start_addr; /* wo */
+		__u64 page_nr;   /* wo */
+	} system_memory;
+}__attribute__((packed));

For an interface definition, it's probably better to avoid packed and instead add padding if needed.

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1081

Quoted 4 lines
linux-headers/linux/vfio.h +1 -0
+
 /* ***************************************************************** */
 
 #endif /* VFIO_H */

On the whole, I think this is moving into the right direction.

author
#3 lore raw

Re: [PATCH 1/5] vfio/migration: define kernel interfaces

Quoted 18 lines · Cornelia Huck
Quoted 8 lines · On Tue, 19 Feb 2019 16:52:14 +0800 Yan Zhao <[email protected]> wrote:

- defined 4 device states regions: one control region and 3 data regions
- defined layout of control region in struct vfio_device_state_ctl
- defined 4 device states: running, stop, running&logging, stop&logging
- define 3 device data categories: device config, device memory, system memory
- defined 2 device data capabilities: device memory and system memory
- defined device state interfaces' version and 12 device state interfaces

Quoted 6 lines
---
 linux-headers/linux/vfio.h | 260 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 260 insertions(+)
Quoted 3 lines

[commenting here for convenience; changes obviously need to be done in the Linux patch]

yes, you can find the corresponding kernel part code at
https://patchwork.freedesktop.org/series/56876/

Quoted 17 lines
Quoted 6 lines
linux-headers/linux/vfio.h +0 -0
index ceb6453..a124fc1 100644

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 310

Quoted 8 lines
linux-headers/linux/vfio.h +5 -0
@@ -303,6 +303,56 @@ struct vfio_region_info_cap_type {
 #define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG	(2)
 #define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG	(3)
 
+/* Device State region type and sub-type
+ *
+ * A VFIO device driver needs to register up to four device state regions in
+ * total: two mandatory and another two optional, if it plans to support device
+ * state management.

Suggest to rephrase:

Quoted 3 lines

"A VFIO device driver that plans to support device state management needs to register..."

ok :)

Quoted 14 lines

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 323

Quoted 13 lines
linux-headers/linux/vfio.h +13 -0
+ *
+ * 1. region CTL :
+ *          Mandatory.
+ *          This is a control region.
+ *          Its layout is defined in struct vfio_device_state_ctl.
+ *          Reading from this region can get version, capabilities and data
+ *          size of device state interfaces.
+ *          Writing to this region can set device state, data size and
+ *          choose which interface to use.
+ * 2. region DEVICE_CONFIG
+ *          Mandatory.
+ *          This is a data region that holds device config data.
+ *          Device config is such kind of data like MMIOs, page tables...
Quoted 1 line

"Device config is data such as..."

ok :)

Quoted 5 lines

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 325

Quoted 3 lines
linux-headers/linux/vfio.h +2 -0
+ *          Every device is supposed to possess device config data.
+ *          Usually the size of device config data is small (no big

s/no big/no bigger/

right :)

Quoted 8 lines

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 331

Quoted 7 lines
linux-headers/linux/vfio.h +6 -0
+ *          than 10M), and it needs to be loaded in certain strict
+ *          order.
+ *          Therefore no dirty data logging is enabled for device
+ *          config and it must be got/set as a whole.
+ *          Size of device config data is smaller than or equal to that of
+ *          device config region.
Quoted 5 lines

Not sure if I understand that sentence correctly... but what if a device has more config state than fits into this region? Is that supposed to be covered by the device memory region? Or is this assumed to be something so exotic that we don't need to plan for it?

Device config data and device config region are all provided by vendor driver, so vendor driver is always able to create a large enough device config region to hold device config data.
So, if a device has data that are better to be saved after device stop and saved/loaded in strict order, the data needs to be in device config region. This kind of data is supposed to be small.
If the device data can be saved/loaded several times, it can also be put into device memory region.

Quoted 7 lines

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 336

Quoted 5 lines
linux-headers/linux/vfio.h +5 -0
+ *          It is able to be mmaped into user space.
+ * 3. region DEVICE_MEMORY
+ *          Optional.
+ *          This is a data region that holds device memory data.
+ *          Device memory is device's internal memory, standalone and outside

s/outside/distinct from/ ?

ok.

Quoted 8 lines

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 338

Quoted 3 lines
linux-headers/linux/vfio.h +2 -0
+ *          system memory.  It is usually very big.
+ *          Not all device has device memory. Like IGD only uses system

s/all devices has/all devices have/

s/Like/E.g./

ok :)

Quoted 5 lines

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 340

Quoted 2 lines
linux-headers/linux/vfio.h +2 -0
+ *          memory and has no device memory.
+ *          Size of devie memory is usually larger than that of device

s/devie/device/

thanks:)

Quoted 6 lines

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 342

Quoted 2 lines
linux-headers/linux/vfio.h +2 -0
+ *          memory region. qemu needs to save/load it in chunks of size of
+ *          device memory region.

I'd rather not explicitly mention QEMU in this header. Maybe "Userspace"?

ok.

Quoted 9 lines

linux-headers/linux/vfio.h struct vfio_region_info_cap_type { · line 350

Quoted 8 lines
linux-headers/linux/vfio.h +8 -0
+ *          It is able to be mmaped into user space.
+ * 4. region DIRTY_BITMAP
+ *          Optional.
+ *          This is a data region that holds bitmap of dirty pages in system
+ *          memory that a VFIO devices produces.
+ *          It is able to be mmaped into user space.
+ */
+#define VFIO_REGION_TYPE_DEVICE_STATE           (1 << 1)
Quoted 3 lines

Can you make this an explicit number instead?

(FWIW, I plan to add a CCW region as type 2, whatever comes first.)

ok :)

Quoted 13 lines
Quoted 13 lines
linux-headers/linux/vfio.h +5 -0
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL       (1)
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG      (2)
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY      (3)
+#define VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP (4)
+
 /*
  * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped
  * which allows direct access to non-MSIX registers which happened to be within
@@ -816,6 +866,216 @@ struct vfio_iommu_spapr_tce_remove {
 };
 #define VFIO_IOMMU_SPAPR_TCE_REMOVE	_IO(VFIO_TYPE, VFIO_BASE + 20)
 
Quoted 6 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 870

Quoted 2 lines
linux-headers/linux/vfio.h +2 -0
+/* version number of the device state interface */
+#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1

Hm. Is this supposed to be backwards-compatible, should we need to bump this?

currently no backwords-compatible. we can discuss on that.

Quoted 16 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 873

Quoted 3 lines
linux-headers/linux/vfio.h +3 -0
+
+/*
+ * For devices that have devcie memory, it is required to expose

s/devcie/device/

Quoted 10 lines
linux-headers/linux/vfio.h +10 -0
+ * DEVICE_MEMORY capability.
+ *
+ * For devices producing dirty pages in system memory, it is required to
+ * expose cap SYSTEM_MEMORY in order to get dirty bitmap in certain range
+ * of system memory.
+ */
+#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
+#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2
+
+/*
Quoted 8 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 889

Quoted 6 lines
linux-headers/linux/vfio.h +6 -0
+ * DEVICE STATES
+ *
+ * Four states are defined for a VFIO device:
+ * RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP.
+ * They can be set by writing to device_state field of
+ * vfio_device_state_ctl region.

Who controls this? Userspace?

Yes. Userspace notifies vendor driver to do the state switching.

Quoted 10 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 896

Quoted 7 lines
linux-headers/linux/vfio.h +7 -0
+ *
+ * RUNNING: In this state, a VFIO device is in active state ready to
+ * receive commands from device driver.
+ * It is the default state that a VFIO device enters initially.
+ *
+ * STOP: In this state, a VFIO device is deactivated to interact with
+ * device driver.

I think 'STOPPED' would read nicer.

sounds better :)

Quoted 6 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 899

Quoted 3 lines
linux-headers/linux/vfio.h +3 -0
+ *
+ * LOGGING state is a special state that it CANNOT exist
+ * independently.

So it's not a state, but rather a modifier?

yes. or thinking LOGGING/not LOGGING as bit 1 of a device state, whereas RUNNING/STOPPED is bit 0 of a device state. They have to be got as a whole.

Quoted 42 lines
Quoted 28 lines
linux-headers/linux/vfio.h +28 -0
+ * It must be set alongside with state RUNNING or STOP, i.e,
+ * RUNNING & LOGGING, STOP & LOGGING.
+ * It is used for dirty data logging both for device memory
+ * and system memory.
+ *
+ * LOGGING only impacts device/system memory. In LOGGING state, get buffer
+ * of device memory returns dirty pages since last call; outside LOGGING
+ * state, get buffer of device memory returns whole snapshot of device
+ * memory. system memory's dirty page is only available in LOGGING state.
+ *
+ * Device config should be always accessible and return whole config snapshot
+ * regardless of LOGGING state.
+ * */
+#define VFIO_DEVICE_STATE_RUNNING 0
+#define VFIO_DEVICE_STATE_STOP 1
+#define VFIO_DEVICE_STATE_LOGGING 2
+
+/* action to get data from device memory or device config
+ * the action is write to device state's control region, and data is read
+ * from device memory region or device config region.
+ * Each time before read device memory region or device config region,
+ * action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action
+ * field in control region. That is because device memory and devie config
+ * region is mmaped into user space. vendor driver has to be notified of
+ * the the GET_BUFFER action in advance.
+ */
+#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
+

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 937

Quoted 10 lines
linux-headers/linux/vfio.h +10 -0
+/* action to set data to device memory or device config
+ * the action is write to device state's control region, and data is
+ * written to device memory region or device config region.
+ * Each time after write to device memory region or device config region,
+ * action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action
+ * field in control region. That is because device memory and devie config
+ * region is mmaped into user space. vendor driver has to be notified of
+ * the the SET_BUFFER action after data written.
+ */
+#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2

Let me describe this in my own words to make sure that I understand this correctly.

Quoted 4 lines

- The actions are set by userspace to notify the kernel that it is going to get data or that it has just written data.
- This is needed as a notification that the mmapped data should not be changed resp. just has changed.

we need this notification is because when userspace read the mmapped data, it's from the ptr returned from mmap(). So, when userspace reads that ptr, there will be no page fault or read/write system calls, so vendor driver does not know whether read/write opertation happens or not. Therefore, before userspace reads the ptr from mmap, it first writes action field in control region (through write system call), and vendor driver will not return the write system call until data prepared.

When userspace writes to that ptr from mmap, it writes data to the data region first, then writes the action field in control region (through write system call) to notify vendor driver. vendor driver will return the system call after it copies the buffer completely.

Quoted 5 lines

So, how does the kernel know whether the read action has finished resp. whether the write action has started? Even if userspace reads/writes it as a whole.

kernel does not touch the data region except when in response to the "action" write system call.

Quoted 117 lines
Quoted 117 lines
linux-headers/linux/vfio.h +117 -0
+
+/* layout of device state interfaces' control region
+ * By reading to control region and reading/writing data from device config
+ * region, device memory region, system memory regions, below interface can
+ * be implemented:
+ *
+ * 1. get version
+ *   (1) user space calls read system call on "version" field of control
+ *   region.
+ *   (2) vendor driver writes version number of device state interfaces
+ *   to the "version" field of control region.
+ *
+ * 2. get caps
+ *   (1) user space calls read system call on "caps" field of control region.
+ *   (2) if a VFIO device has huge device memory, vendor driver reports
+ *      VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY in "caps" field of control region.
+ *      if a VFIO device produces dirty pages in system memory, vendor driver
+ *      reports VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY in "caps" field of
+ *      control region.
+ *
+ * 3. set device state
+ *    (1) user space calls write system call on "device_state" field of
+ *    control region.
+ *    (2) device state transitions as:
+ *
+ *    RUNNING -- start dirty data logging --> RUNNING & LOGGING
+ *    RUNNING -- deactivate --> STOP
+ *    RUNNING -- deactivate & start dirty data longging --> STOP & LOGGING
+ *    RUNNING & LOGGING -- stop dirty data logging --> RUNNING
+ *    RUNNING & LOGGING -- deactivate --> STOP & LOGGING
+ *    RUNNING & LOGGING -- deactivate & stop dirty data logging --> STOP
+ *    STOP -- activate --> RUNNING
+ *    STOP -- start dirty data logging --> STOP & LOGGING
+ *    STOP -- activate & start dirty data logging --> RUNNING & LOGGING
+ *    STOP & LOGGING -- stop dirty data logging --> STOP
+ *    STOP & LOGGING -- activate --> RUNNING & LOGGING
+ *    STOP & LOGGING -- activate & stop dirty data logging --> RUNNING
+ *
+ * 4. get device config size
+ *   (1) user space calls read system call on "device_config.size" field of
+ *       control region for the total size of device config snapshot.
+ *   (2) vendor driver writes device config data's total size in
+ *       "device_config.size" field of control region.
+ *
+ * 5. set device config size
+ *   (1) user space calls write system call.
+ *       total size of device config snapshot --> "device_config.size" field
+ *       of control region.
+ *   (2) vendor driver reads device config data's total size from
+ *       "device_config.size" field of control region.
+ *
+ * 6 get device config buffer
+ *   (1) user space calls write system call.
+ *       "GET_BUFFER" --> "device_config.action" field of control region.
+ *   (2) vendor driver
+ *       a. gets whole snapshot for device config
+ *       b. writes whole device config snapshot to region
+ *       DEVICE_CONFIG.
+ *   (3) user space reads the whole of device config snapshot from region
+ *       DEVICE_CONFIG.
+ *
+ * 7. set device config buffer
+ *   (1) user space writes whole of device config data to region
+ *       DEVICE_CONFIG.
+ *   (2) user space calls write system call.
+ *       "SET_BUFFER" --> "device_config.action" field of control region.
+ *   (3) vendor driver loads whole of device config from region DEVICE_CONFIG.
+ *
+ * 8. get device memory size
+ *   (1) user space calls read system call on "device_memory.size" field of
+ *       control region for device memory size.
+ *   (2) vendor driver
+ *       a. gets device memory snapshot (in state RUNNING or STOP), or
+ *          gets device memory dirty data (in state RUNNING & LOGGING or
+ *          state STOP & LOGGING)
+ *       b. writes size in "device_memory.size" field of control region
+ *
+ * 9. set device memory size
+ *   (1) user space calls write system call on "device_memory.size" field of
+ *       control region to set total size of device memory snapshot.
+ *   (2) vendor driver reads device memory's size from "device_memory.size"
+ *       field of control region.
+ *
+ *
+ * 10. get device memory buffer
+ *   (1) user space calls write system.
+ *       pos --> "device_memory.pos" field of control region,
+ *       "GET_BUFFER" --> "device_memory.action" field of control region.
+ *       (pos must be 0 or multiples of length of region DEVICE_MEMORY).
+ *   (2) vendor driver writes N'th chunk of device memory snapshot/dirty data
+ *       to region DEVICE_MEMORY.
+ *       (N equals to pos/(region length of DEVICE_MEMORY))
+ *   (3) user space reads the N'th chunk of device memory snapshot/dirty data
+ *       from region DEVICE_MEMORY.
+ *
+ * 11. set device memory buffer
+ *   (1) user space writes N'th chunk of device memory snapshot/dirty data to
+ *       region DEVICE_MEMORY.
+ *       (N equals to pos/(region length of DEVICE_MEMORY))
+ *   (2) user space writes pos to "device_memory.pos" field and writes
+ *       "SET_BUFFER" to "device_memory.action" field of control region.
+ *   (3) vendor driver loads N'th chunk of device memory snapshot/dirty data
+ *       from region DEVICE_MEMORY.
+ *
+ * 12. get system memory dirty bitmap
+ *   (1) user space calls write system call to specify a range of system
+ *       memory that querying dirty pages.
+ *       system memory's start address --> "system_memory.start_addr" field
+ *       of control region,
+ *       system memory's page count --> "system_memory.page_nr" field of
+ *       control region.
+ *   (2) if device state is not in RUNNING or STOP & LOGGING,
+ *       vendor driver returns empty bitmap; otherwise,
+ *       vendor driver checks the page_nr,
+ *       if it's larger than the size that region DIRTY_BITMAP can support,
+ *       error returns; if not,
+ *       vendor driver returns as bitmap to specify dirty pages that
Quoted 9 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1058

Quoted 4 lines
linux-headers/linux/vfio.h +4 -0
+ *       device produces since last query in this range of system memory .
+ *   (3) usespace reads back the dirty bitmap from region DIRTY_BITMAP.
+ *
+ */

It might make sense to extract the explanations above into a separate design document in the kernel Documentation/ directory. You could also add ASCII art there :)

yes, a diagram is better:)

Quoted 7 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1061

Quoted 3 lines
linux-headers/linux/vfio.h +3 -0
+
+struct vfio_device_state_ctl {
+	__u32 version;		  /* ro versio of devcie state interfaces*/

s/versio/version/
s/devcie/device/

thanks~

Quoted 6 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1064

Quoted 3 lines
linux-headers/linux/vfio.h +3 -0
+	__u32 device_state;       /* VFIO device state, wo */
+	__u32 caps;		 /* ro */
+        struct {

Indentation looks a bit off.

Quoted 10 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1071

Quoted 7 lines
linux-headers/linux/vfio.h +7 -0
+		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */
+		__u64 size;    /*rw, total size of device config*/
+	} device_config;
+	struct {
+		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */
+		__u64 size;     /* rw, total size of device memory*/
+        __u64 pos;/*chunk offset in total buffer of device memory*/

Here as well.

thanks~

Quoted 10 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1077

Quoted 6 lines
linux-headers/linux/vfio.h +6 -0
+	} device_memory;
+	struct {
+		__u64 start_addr; /* wo */
+		__u64 page_nr;   /* wo */
+	} system_memory;
+}__attribute__((packed));

For an interface definition, it's probably better to avoid packed and instead add padding if needed.

ok. so just remove the __attribute__((packed)); is enough for this interface.

Quoted 6 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 1081

Quoted 4 lines
linux-headers/linux/vfio.h +1 -0
+
 /* ***************************************************************** */
 
 #endif /* VFIO_H */

On the whole, I think this is moving into the right direction.

#4 lore raw

Re: [PATCH 1/5] vfio/migration: define kernel interfaces

Quoted 3 lines · On Wed, 20 Feb 2019 02:36:36 -0500 Zhao Yan <[email protected]> wrote:
Quoted 2 lines · Cornelia Huck

On Tue, 19 Feb 2019 16:52:14 +0800
Yan Zhao <[email protected]> wrote:

(...)

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

+ * Size of device config data is smaller than or equal to that of
+ * device config region.

Not sure if I understand that sentence correctly... but what if a device has more config state than fits into this region? Is that supposed to be covered by the device memory region? Or is this assumed to be something so exotic that we don't need to plan for it?

Quoted 8 lines

Device config data and device config region are all provided by vendor driver, so vendor driver is always able to create a large enough device config region to hold device config data.
So, if a device has data that are better to be saved after device stop and saved/loaded in strict order, the data needs to be in device config region. This kind of data is supposed to be small.
If the device data can be saved/loaded several times, it can also be put into device memory region.

So, it is the vendor driver's decision which device information should go via which region? With the device config data supposed to be saved/loaded in one go?

(...)

Quoted 7 lines
Quoted 6 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 870

Quoted 2 lines
linux-headers/linux/vfio.h +2 -0
+/* version number of the device state interface */
+#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1  

Hm. Is this supposed to be backwards-compatible, should we need to bump this?

currently no backwords-compatible. we can discuss on that.

It might be useful if we discover that we need some extensions. But I'm not sure how much work it would be.

(...)

Quoted 11 lines
Quoted 9 lines
Quoted 7 lines

+/*
+ * DEVICE STATES
+ *
+ * Four states are defined for a VFIO device:
+ * RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP.
+ * They can be set by writing to device_state field of
+ * vfio_device_state_ctl region.

Who controls this? Userspace?

Yes. Userspace notifies vendor driver to do the state switching.

Might be good to mention this (just to make it obvious).

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

+ * LOGGING state is a special state that it CANNOT exist
+ * independently.

So it's not a state, but rather a modifier?

yes. or thinking LOGGING/not LOGGING as bit 1 of a device state, whereas RUNNING/STOPPED is bit 0 of a device state. They have to be got as a whole.

So it is (on a bit level):
RUNNING -> 00
STOPPED -> 01
LOGGING/RUNNING -> 10
LOGGING/STOPPED -> 11

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+ * It must be set alongside with state RUNNING or STOP, i.e,
+ * RUNNING & LOGGING, STOP & LOGGING.
+ * It is used for dirty data logging both for device memory
+ * and system memory.
+ *
+ * LOGGING only impacts device/system memory. In LOGGING state, get buffer
+ * of device memory returns dirty pages since last call; outside LOGGING
+ * state, get buffer of device memory returns whole snapshot of device

Quoted 8 lines
Quoted 8 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 915

Quoted 8 lines

+ * memory. system memory's dirty page is only available in LOGGING state.
+ *
+ * Device config should be always accessible and return whole config snapshot
+ * regardless of LOGGING state.
+ * */

linux-headers/linux/vfio.h +3 -0
+#define VFIO_DEVICE_STATE_RUNNING 0
+#define VFIO_DEVICE_STATE_STOP 1
+#define VFIO_DEVICE_STATE_LOGGING 2

This makes it look a bit like LOGGING were an individual state, while 2 is in reality LOGGING/RUNNING... not sure how to make that more obvious. Maybe (as we are dealing with a u32):

#define VFIO_DEVICE_STATE_RUNNING 0x00000000 #define VFIO_DEVICE_STATE_STOPPED 0x00000001 #define VFIO_DEVICE_STATE_LOGGING_RUNNING 0x00000002 #define VFIO_DEVICE_STATE_LOGGING_STOPPED 0x00000003 #define VFIO_DEVICE_STATE_LOGGING_MASK 0x00000002

Quoted 43 lines
Quoted 26 lines
Quoted 12 lines
linux-headers/linux/vfio.h +12 -0
+
+/* action to get data from device memory or device config
+ * the action is write to device state's control region, and data is read
+ * from device memory region or device config region.
+ * Each time before read device memory region or device config region,
+ * action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action
+ * field in control region. That is because device memory and devie config
+ * region is mmaped into user space. vendor driver has to be notified of
+ * the the GET_BUFFER action in advance.
+ */
+#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
+

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 937

Quoted 10 lines
linux-headers/linux/vfio.h +10 -0
+/* action to set data to device memory or device config
+ * the action is write to device state's control region, and data is
+ * written to device memory region or device config region.
+ * Each time after write to device memory region or device config region,
+ * action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action
+ * field in control region. That is because device memory and devie config
+ * region is mmaped into user space. vendor driver has to be notified of
+ * the the SET_BUFFER action after data written.
+ */
+#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2  

Let me describe this in my own words to make sure that I understand this correctly.

Quoted 4 lines

- The actions are set by userspace to notify the kernel that it is going to get data or that it has just written data.
- This is needed as a notification that the mmapped data should not be changed resp. just has changed.

we need this notification is because when userspace read the mmapped data, it's from the ptr returned from mmap(). So, when userspace reads that ptr, there will be no page fault or read/write system calls, so vendor driver does not know whether read/write opertation happens or not. Therefore, before userspace reads the ptr from mmap, it first writes action field in control region (through write system call), and vendor driver will not return the write system call until data prepared.

When userspace writes to that ptr from mmap, it writes data to the data region first, then writes the action field in control region (through write system call) to notify vendor driver. vendor driver will return the system call after it copies the buffer completely.

Quoted 1 line
Quoted 6 lines
Quoted 4 lines

So, how does the kernel know whether the read action has finished resp. whether the write action has started? Even if userspace reads/writes it as a whole.

kernel does not touch the data region except when in response to the "action" write system call.

Thanks for the explanation, that makes sense.
(...)

author
#5 lore raw

Re: [PATCH 1/5] vfio/migration: define kernel interfaces

Quoted 24 lines · Cornelia Huck
Quoted 3 lines · On Wed, 20 Feb 2019 02:36:36 -0500 Zhao Yan <[email protected]> wrote:
Quoted 2 lines · Cornelia Huck

On Tue, 19 Feb 2019 16:52:14 +0800
Yan Zhao <[email protected]> wrote:

(...)

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

+ * Size of device config data is smaller than or equal to that of
+ * device config region.

Not sure if I understand that sentence correctly... but what if a device has more config state than fits into this region? Is that supposed to be covered by the device memory region? Or is this assumed to be something so exotic that we don't need to plan for it?

Quoted 8 lines

Device config data and device config region are all provided by vendor driver, so vendor driver is always able to create a large enough device config region to hold device config data.
So, if a device has data that are better to be saved after device stop and saved/loaded in strict order, the data needs to be in device config region. This kind of data is supposed to be small.
If the device data can be saved/loaded several times, it can also be put into device memory region.

Quoted 3 lines

So, it is the vendor driver's decision which device information should go via which region? With the device config data supposed to be saved/loaded in one go?

Right, exactly.

Quoted 21 lines

(...)

Quoted 7 lines
Quoted 6 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 870

Quoted 2 lines
linux-headers/linux/vfio.h +2 -0
+/* version number of the device state interface */
+#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1  

Hm. Is this supposed to be backwards-compatible, should we need to bump this?

currently no backwords-compatible. we can discuss on that.

It might be useful if we discover that we need some extensions. But I'm not sure how much work it would be.

(...)

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

+/*
+ * DEVICE STATES
+ *
+ * Four states are defined for a VFIO device:
+ * RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP.
+ * They can be set by writing to device_state field of
+ * vfio_device_state_ctl region.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Who controls this? Userspace?

Yes. Userspace notifies vendor driver to do the state switching.

Might be good to mention this (just to make it obvious).

Got it. thanks

Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

+ * LOGGING state is a special state that it CANNOT exist
+ * independently.

So it's not a state, but rather a modifier?

yes. or thinking LOGGING/not LOGGING as bit 1 of a device state, whereas RUNNING/STOPPED is bit 0 of a device state. They have to be got as a whole.

Quoted 6 lines

So it is (on a bit level):
RUNNING -> 00
STOPPED -> 01
LOGGING/RUNNING -> 10
LOGGING/STOPPED -> 11

Yes.

Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+ * It must be set alongside with state RUNNING or STOP, i.e,
+ * RUNNING & LOGGING, STOP & LOGGING.
+ * It is used for dirty data logging both for device memory
+ * and system memory.
+ *
+ * LOGGING only impacts device/system memory. In LOGGING state, get buffer
+ * of device memory returns dirty pages since last call; outside LOGGING
+ * state, get buffer of device memory returns whole snapshot of device

Quoted 8 lines
Quoted 8 lines

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 915

Quoted 8 lines

+ * memory. system memory's dirty page is only available in LOGGING state.
+ *
+ * Device config should be always accessible and return whole config snapshot
+ * regardless of LOGGING state.
+ * */

linux-headers/linux/vfio.h +3 -0
+#define VFIO_DEVICE_STATE_RUNNING 0
+#define VFIO_DEVICE_STATE_STOP 1
+#define VFIO_DEVICE_STATE_LOGGING 2

This makes it look a bit like LOGGING were an individual state, while 2 is in reality LOGGING/RUNNING... not sure how to make that more obvious. Maybe (as we are dealing with a u32):

Quoted 6 lines

#define VFIO_DEVICE_STATE_RUNNING 0x00000000 #define VFIO_DEVICE_STATE_STOPPED 0x00000001 #define VFIO_DEVICE_STATE_LOGGING_RUNNING 0x00000002 #define VFIO_DEVICE_STATE_LOGGING_STOPPED 0x00000003 #define VFIO_DEVICE_STATE_LOGGING_MASK 0x00000002

Yes, yours are better, thanks:)

Quoted 47 lines
Quoted 47 lines
Quoted 26 lines
Quoted 12 lines
linux-headers/linux/vfio.h +12 -0
+
+/* action to get data from device memory or device config
+ * the action is write to device state's control region, and data is read
+ * from device memory region or device config region.
+ * Each time before read device memory region or device config region,
+ * action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action
+ * field in control region. That is because device memory and devie config
+ * region is mmaped into user space. vendor driver has to be notified of
+ * the the GET_BUFFER action in advance.
+ */
+#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
+

linux-headers/linux/vfio.h struct vfio_iommu_spapr_tce_remove { · line 937

Quoted 10 lines
linux-headers/linux/vfio.h +10 -0
+/* action to set data to device memory or device config
+ * the action is write to device state's control region, and data is
+ * written to device memory region or device config region.
+ * Each time after write to device memory region or device config region,
+ * action VFIO_DEVICE_DATA_ACTION_GET_BUFFER is required to write to action
+ * field in control region. That is because device memory and devie config
+ * region is mmaped into user space. vendor driver has to be notified of
+ * the the SET_BUFFER action after data written.
+ */
+#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2  

Let me describe this in my own words to make sure that I understand this correctly.

Quoted 4 lines

- The actions are set by userspace to notify the kernel that it is going to get data or that it has just written data.
- This is needed as a notification that the mmapped data should not be changed resp. just has changed.

we need this notification is because when userspace read the mmapped data, it's from the ptr returned from mmap(). So, when userspace reads that ptr, there will be no page fault or read/write system calls, so vendor driver does not know whether read/write opertation happens or not. Therefore, before userspace reads the ptr from mmap, it first writes action field in control region (through write system call), and vendor driver will not return the write system call until data prepared.

When userspace writes to that ptr from mmap, it writes data to the data region first, then writes the action field in control region (through write system call) to notify vendor driver. vendor driver will return the system call after it copies the buffer completely.

Quoted 5 lines

So, how does the kernel know whether the read action has finished resp. whether the write action has started? Even if userspace reads/writes it as a whole.

Quoted 5 lines
Quoted 2 lines

kernel does not touch the data region except when in response to the "action" write system call.

Thanks for the explanation, that makes sense.
(...)

welcome:)

Quoted 4 lines

_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

author
#6 lore raw

PATCH 2/5 vfio/migration: support device of device config capability

In reply to Yan Zhao

Device config is the default data that every device should have. so device config capability is by default on, no need to set.

- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.

Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state to VFIO_DEVICE_STATE_LOGGING

---
 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/migration.c           | 633 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |   1 -
 hw/vfio/pci.h                 |  25 +-
 include/hw/vfio/vfio-common.h |   1 +
 5 files changed, 659 insertions(+), 3 deletions(-)
 create mode 100644 hw/vfio/migration.c
hw/vfio/Makefile.objs +1 -1
index 8b3f664..f32ff19 100644
@@ -1,6 +1,6 @@
 ifeq ($(CONFIG_LINUX), y)
 obj-$(CONFIG_SOFTMMU) += common.o
-obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o
+obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.o
 obj-$(CONFIG_VFIO_CCW) += ccw.o
 obj-$(CONFIG_SOFTMMU) += platform.o
 obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o
hw/vfio/migration.c new file +633 -0

633 lines not on this page · raw message

hw/vfio/pci.c +0 -1
index c0cb1ec..b8e006b 100644
@@ -37,7 +37,6 @@
 
 #define MSIX_CAP_LENGTH 12
 
-#define TYPE_VFIO_PCI "vfio-pci"
 #define PCI_VFIO(obj)    OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)
 
 static void vfio_disable_interrupts(VFIOPCIDevice *vdev);
hw/vfio/pci.h +24 -1
index b1ae4c0..4b7b1bb 100644
@@ -19,6 +19,7 @@
 #include "qemu/event_notifier.h"
 #include "qemu/queue.h"
 #include "qemu/timer.h"
+#include "sysemu/sysemu.h"
 
 #define PCI_ANY_ID (~0)
 
@@ -56,6 +57,21 @@ typedef struct VFIOBAR {
     QLIST_HEAD(, VFIOQuirk) quirks;
 } VFIOBAR;
 
+enum {
+    VFIO_DEVSTATE_REGION_CTL = 0,
+    VFIO_DEVSTATE_REGION_DATA_CONFIG,
+    VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY,
+    VFIO_DEVSTATE_REGION_DATA_BITMAP,
+    VFIO_DEVSTATE_REGION_NUM,
+};
+typedef struct VFIOMigration {
+    VFIORegion region[VFIO_DEVSTATE_REGION_NUM];
+    uint32_t data_caps;
+    uint32_t device_state;
+    uint64_t devconfig_size;
+    VMChangeStateEntry *vm_state;
+} VFIOMigration;
+
 typedef struct VFIOVGARegion {
     MemoryRegion mem;
     off_t offset;
@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {
     VFIOBAR bars[PCI_NUM_REGIONS - 1]; /* No ROM */
     VFIOVGA *vga; /* 0xa0000, 0x3b0, 0x3c0 */
     void *igd_opregion;
+    VFIOMigration *migration;
+    Error *migration_blocker;
     PCIHostDeviceAddress host;
     EventNotifier err_notifier;
     EventNotifier req_notifier;
@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,
 void vfio_display_reset(VFIOPCIDevice *vdev);
 int vfio_display_probe(VFIOPCIDevice *vdev, Error **errp);
 void vfio_display_finalize(VFIOPCIDevice *vdev);
-
+bool vfio_device_data_cap_system_memory(VFIOPCIDevice *vdev);
+bool vfio_device_data_cap_device_memory(VFIOPCIDevice *vdev);
+int vfio_set_dirty_page_bitmap(VFIOPCIDevice *vdev,
+         uint64_t start_addr, uint64_t page_nr);
+int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp);
+void vfio_migration_finalize(VFIOPCIDevice *vdev);
 #endif /* HW_VFIO_VFIO_PCI_H */
include/hw/vfio/vfio-common.h +1 -0
index 1b434d0..ed43613 100644
@@ -32,6 +32,7 @@
 #endif
 
 #define VFIO_MSG_PREFIX "vfio %s: "
+#define TYPE_VFIO_PCI "vfio-pci"
 
 enum {
     VFIO_DEVICE_TYPE_PCI = 0,
Signature · 1 line
2.7.4
#7 lore raw

Re: [PATCH 2/5] vfio/migration: support device of device config capability

Quoted 184 lines · * Yan Zhao ([email protected]) wrote:

Device config is the default data that every device should have. so device config capability is by default on, no need to set.

- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.

Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state to VFIO_DEVICE_STATE_LOGGING

---
 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/migration.c           | 633 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |   1 -
 hw/vfio/pci.h                 |  25 +-
 include/hw/vfio/vfio-common.h |   1 +
 5 files changed, 659 insertions(+), 3 deletions(-)
 create mode 100644 hw/vfio/migration.c
hw/vfio/Makefile.objs +1 -1
index 8b3f664..f32ff19 100644
@@ -1,6 +1,6 @@
 ifeq ($(CONFIG_LINUX), y)
 obj-$(CONFIG_SOFTMMU) += common.o
-obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o
+obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.o
 obj-$(CONFIG_VFIO_CCW) += ccw.o
 obj-$(CONFIG_SOFTMMU) += platform.o
 obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o
hw/vfio/migration.c new file +137 -0
new file mode 100644
index 0000000..16d6395
@@ -0,0 +1,633 @@
+#include "qemu/osdep.h"
+
+#include "hw/vfio/vfio-common.h"
+#include "migration/blocker.h"
+#include "migration/register.h"
+#include "qapi/error.h"
+#include "pci.h"
+#include "sysemu/kvm.h"
+#include "exec/ram_addr.h"
+
+#define VFIO_SAVE_FLAG_SETUP 0
+#define VFIO_SAVE_FLAG_PCI 1
+#define VFIO_SAVE_FLAG_DEVCONFIG 2
+#define VFIO_SAVE_FLAG_DEVMEMORY 4
+#define VFIO_SAVE_FLAG_CONTINUE 8
+
+static int vfio_device_state_region_setup(VFIOPCIDevice *vdev,
+        VFIORegion *region, uint32_t subtype, const char *name)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    struct vfio_region_info *info;
+    int ret;
+
+    ret = vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_DEVICE_STATE,
+            subtype, &info);
+    if (ret) {
+        error_report("Failed to get info of region %s", name);
+        return ret;
+    }
+
+    if (vfio_region_setup(OBJECT(vdev), vbasedev,
+            region, info->index, name)) {
+        error_report("Failed to setup migrtion region %s", name);
+        return ret;
+    }
+
+    if (vfio_region_mmap(region)) {
+        error_report("Failed to mmap migrtion region %s", name);
+    }
+
+    return 0;
+}
+
+bool vfio_device_data_cap_system_memory(VFIOPCIDevice *vdev)
+{
+   return !!(vdev->migration->data_caps & VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY);
+}
+
+bool vfio_device_data_cap_device_memory(VFIOPCIDevice *vdev)
+{
+   return !!(vdev->migration->data_caps & VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY);
+}
+
+static bool vfio_device_state_region_mmaped(VFIORegion *region)
+{
+    bool mmaped = true;
+    if (region->nr_mmaps != 1 || region->mmaps[0].offset ||
+            (region->size != region->mmaps[0].size) ||
+            (region->mmaps[0].mmap == NULL)) {
+        mmaped = false;
+    }
+
+    return mmaped;
+}
+
+static int vfio_get_device_config_size(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    uint64_t len;
+    int sz;
+
+    sz = sizeof(len);
+    if (pread(vbasedev->fd, &len, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.size))
+            != sz) {
+        error_report("vfio: Failed to get length of device config");
+        return -1;
+    }
+    if (len > region_config->size) {
+        error_report("vfio: Error device config length");
+        return -1;
+    }
+    vdev->migration->devconfig_size = len;
+
+    return 0;
+}
+
+static int vfio_set_device_config_size(VFIOPCIDevice *vdev, uint64_t size)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    int sz;
+
+    if (size > region_config->size) {
+        return -1;
+    }
+
+    sz = sizeof(size);
+    if (pwrite(vbasedev->fd, &size, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.size))
+            != sz) {
+        error_report("vfio: Failed to set length of device config");
+        return -1;
+    }
+    vdev->migration->devconfig_size = size;
+    return 0;
+}
+
+static int vfio_save_data_device_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+    uint64_t len = vdev->migration->devconfig_size;
+
+    qemu_put_be64(f, len);
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.action))
+            != sz) {

hw/vfio/migration.c line 147

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+        error_report("vfio: action failure for device config get buffer");
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_config)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }

g_malloc never returns NULL; it aborts on failure to allocate. So you can either drop the check, or my preference is to use g_try_malloc for large/unknown areas, and it can return NULL.

Quoted 276 lines
hw/vfio/migration.c new file +276 -0
+        if (pread(vbasedev->fd, buf, len, region_config->fd_offset) != len) {
+            error_report("vfio: Failed read device config buffer");
+            return -1;
+        }
+        qemu_put_buffer(f, buf, len);
+        g_free(buf);
+    } else {
+        dest = region_config->mmaps[0].mmap;
+        qemu_put_buffer(f, dest, len);
+    }
+    return 0;
+}
+
+static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
+                            QEMUFile *f, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_SET_BUFFER;
+
+    vfio_set_device_config_size(vdev, len);
+
+    if (!vfio_device_state_region_mmaped(region_config)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        qemu_get_buffer(f, buf, len);
+        if (pwrite(vbasedev->fd, buf, len,
+                    region_config->fd_offset) != len) {
+            error_report("vfio: Failed to write devie config buffer");
+            return -1;
+        }
+        g_free(buf);
+    } else {
+        dest = region_config->mmaps[0].mmap;
+        qemu_get_buffer(f, dest, len);
+    }
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.action))
+            != sz) {
+        error_report("vfio: action failure for device config set buffer");
+        return -1;
+    }
+
+    return 0;
+}
+
+static int vfio_set_dirty_page_bitmap_chunk(VFIOPCIDevice *vdev,
+        uint64_t start_addr, uint64_t page_nr)
+{
+
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_bitmap =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP];
+    unsigned long bitmap_size =
+                    BITS_TO_LONGS(page_nr) * sizeof(unsigned long);
+    uint32_t sz;
+
+    struct {
+        __u64 start_addr;
+        __u64 page_nr;
+    } system_memory;
+    system_memory.start_addr = start_addr;
+    system_memory.page_nr = page_nr;
+    sz = sizeof(system_memory);
+    if (pwrite(vbasedev->fd, &system_memory, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, system_memory))
+            != sz) {
+        error_report("vfio: Failed to set system memory range for dirty pages");
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_bitmap)) {
+        void *bitmap = g_malloc0(bitmap_size);
+
+        if (pread(vbasedev->fd, bitmap, bitmap_size,
+                    region_bitmap->fd_offset) != bitmap_size) {
+            error_report("vfio: Failed to read dirty bitmap data");
+            return -1;
+        }
+
+        cpu_physical_memory_set_dirty_lebitmap(bitmap, start_addr, page_nr);
+
+        g_free(bitmap);
+    } else {
+        cpu_physical_memory_set_dirty_lebitmap(
+                    region_bitmap->mmaps[0].mmap,
+                    start_addr, page_nr);
+    }
+   return 0;
+}
+
+int vfio_set_dirty_page_bitmap(VFIOPCIDevice *vdev,
+        uint64_t start_addr, uint64_t page_nr)
+{
+    VFIORegion *region_bitmap =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP];
+    unsigned long chunk_size = region_bitmap->size;
+    uint64_t chunk_pg_nr = (chunk_size / sizeof(unsigned long)) *
+                                BITS_PER_LONG;
+
+    uint64_t cnt_left;
+    int rc = 0;
+
+    cnt_left = page_nr;
+
+    while (cnt_left >= chunk_pg_nr) {
+        rc = vfio_set_dirty_page_bitmap_chunk(vdev, start_addr, chunk_pg_nr);
+        if (rc) {
+            goto exit;
+        }
+        cnt_left -= chunk_pg_nr;
+        start_addr += start_addr;
+   }
+   rc = vfio_set_dirty_page_bitmap_chunk(vdev, start_addr, cnt_left);
+
+exit:
+   return rc;
+}
+
+static int vfio_set_device_state(VFIOPCIDevice *vdev,
+        uint32_t dev_state)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint32_t sz = sizeof(dev_state);
+
+    if (!vdev->migration) {
+        return -1;
+    }
+
+    if (pwrite(vbasedev->fd, &dev_state, sz,
+              region->fd_offset +
+              offsetof(struct vfio_device_state_ctl, device_state))
+            != sz) {
+        error_report("vfio: Failed to set device state %d", dev_state);
+        return -1;
+    }
+    vdev->migration->device_state = dev_state;
+    return 0;
+}
+
+static int vfio_get_device_data_caps(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+
+    uint32_t caps;
+    uint32_t size = sizeof(caps);
+
+    if (pread(vbasedev->fd, &caps, size,
+                region->fd_offset +
+                offsetof(struct vfio_device_state_ctl, caps))
+            != size) {
+        error_report("%s Failed to read data caps of device states",
+                vbasedev->name);
+        return -1;
+    }
+    vdev->migration->data_caps = caps;
+    return 0;
+}
+
+
+static int vfio_check_devstate_version(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+
+    uint32_t version;
+    uint32_t size = sizeof(version);
+
+    if (pread(vbasedev->fd, &version, size,
+                region->fd_offset +
+                offsetof(struct vfio_device_state_ctl, version))
+            != size) {
+        error_report("%s Failed to read version of device state interfaces",
+                vbasedev->name);
+        return -1;
+    }
+
+    if (version != VFIO_DEVICE_STATE_INTERFACE_VERSION) {
+        error_report("%s migration version mismatch, right version is %d",
+                vbasedev->name, VFIO_DEVICE_STATE_INTERFACE_VERSION);
+        return -1;
+    }
+
+    return 0;
+}
+
+static void vfio_vm_change_state_handler(void *pv, int running, RunState state)
+{
+    VFIOPCIDevice *vdev = pv;
+    uint32_t dev_state = vdev->migration->device_state;
+
+    if (!running) {
+        dev_state |= VFIO_DEVICE_STATE_STOP;
+    } else {
+        dev_state &= ~VFIO_DEVICE_STATE_STOP;
+    }
+
+    vfio_set_device_state(vdev, dev_state);
+}
+
+static void vfio_save_live_pending(QEMUFile *f, void *opaque,
+                                   uint64_t max_size,
+                                   uint64_t *res_precopy_only,
+                                   uint64_t *res_compatible,
+                                   uint64_t *res_post_copy_only)
+{
+    VFIOPCIDevice *vdev = opaque;
+
+    if (!vfio_device_data_cap_device_memory(vdev)) {
+        return;
+    }
+
+    return;
+}
+
+static int vfio_save_iterate(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+
+    if (!vfio_device_data_cap_device_memory(vdev)) {
+        return 0;
+    }
+
+    return 0;
+}
+
+static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    uint32_t ctl, msi_lo, msi_hi, msi_data, bar_cfg, i;
+    bool msi_64bit;
+
+    /* retore pci bar configuration */
+    ctl = pci_default_read_config(pdev, PCI_COMMAND, 2);
+    vfio_pci_write_config(pdev, PCI_COMMAND,
+            ctl & (!(PCI_COMMAND_IO | PCI_COMMAND_MEMORY)), 2);
+    for (i = 0; i < PCI_ROM_SLOT; i++) {
+        bar_cfg = qemu_get_be32(f);
+        vfio_pci_write_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, bar_cfg, 4);
+    }
+    vfio_pci_write_config(pdev, PCI_COMMAND,
+            ctl | PCI_COMMAND_IO | PCI_COMMAND_MEMORY, 2);
+
+    /* restore msi configuration */
+    ctl = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, 2);
+    msi_64bit = !!(ctl & PCI_MSI_FLAGS_64BIT);
+
+    vfio_pci_write_config(&vdev->pdev,
+            pdev->msi_cap + PCI_MSI_FLAGS,
+            ctl & (!PCI_MSI_FLAGS_ENABLE), 2);
+
+    msi_lo = qemu_get_be32(f);
+    vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_LO, msi_lo, 4);
+
+    if (msi_64bit) {
+        msi_hi = qemu_get_be32(f);

hw/vfio/migration.c line 433

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+        vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_HI,
+                msi_hi, 4);
+    }
+    msi_data = qemu_get_be32(f);
+    vfio_pci_write_config(pdev,
+            pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
+            msi_data, 2);
+
+    vfio_pci_write_config(&vdev->pdev, pdev->msi_cap + PCI_MSI_FLAGS,
+            ctl | PCI_MSI_FLAGS_ENABLE, 2);

It would probably be best to use a VMStateDescription and the macros for this if possible; I bet you'll want to add more fields in the future for example.
Also what happens if the data read from the migration stream is bad or doesn't agree with this devices hardware? How does this fail?

Quoted 281 lines
hw/vfio/migration.c new file +199 -0
+}
+
+static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
+{
+    VFIOPCIDevice *vdev = opaque;
+    int flag;
+    uint64_t len;
+    int ret = 0;
+
+    if (version_id != VFIO_DEVICE_STATE_INTERFACE_VERSION) {
+        return -EINVAL;
+    }
+
+    do {
+        flag = qemu_get_byte(f);
+
+        switch (flag & ~VFIO_SAVE_FLAG_CONTINUE) {
+        case VFIO_SAVE_FLAG_SETUP:
+            break;
+        case VFIO_SAVE_FLAG_PCI:
+            vfio_pci_load_config(vdev, f);
+            break;
+        case VFIO_SAVE_FLAG_DEVCONFIG:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_config(vdev, f, len);
+            break;
+        default:
+            ret = -EINVAL;
+        }
+    } while (flag & VFIO_SAVE_FLAG_CONTINUE);
+
+    return ret;
+}
+
+static void vfio_pci_save_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    uint32_t msi_cfg, msi_lo, msi_hi, msi_data, bar_cfg, i;
+    bool msi_64bit;
+
+    for (i = 0; i < PCI_ROM_SLOT; i++) {
+        bar_cfg = pci_default_read_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, 4);
+        qemu_put_be32(f, bar_cfg);
+    }
+
+    msi_cfg = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, 2);
+    msi_64bit = !!(msi_cfg & PCI_MSI_FLAGS_64BIT);
+
+    msi_lo = pci_default_read_config(pdev,
+            pdev->msi_cap + PCI_MSI_ADDRESS_LO, 4);
+    qemu_put_be32(f, msi_lo);
+
+    if (msi_64bit) {
+        msi_hi = pci_default_read_config(pdev,
+                pdev->msi_cap + PCI_MSI_ADDRESS_HI,
+                4);
+        qemu_put_be32(f, msi_hi);
+    }
+
+    msi_data = pci_default_read_config(pdev,
+            pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
+            2);
+    qemu_put_be32(f, msi_data);
+
+}
+
+static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    int rc = 0;
+
+    qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
+    vfio_pci_save_config(vdev, f);
+
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVCONFIG);
+    rc += vfio_get_device_config_size(vdev);
+    rc += vfio_save_data_device_config(vdev, f);
+
+    return rc;
+}
+
+static int vfio_save_setup(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
+            VFIO_DEVICE_STATE_LOGGING);
+    return 0;
+}
+
+static int vfio_load_setup(QEMUFile *f, void *opaque)
+{
+    return 0;
+}
+
+static void vfio_save_cleanup(void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    uint32_t dev_state = vdev->migration->device_state;
+
+    dev_state &= ~VFIO_DEVICE_STATE_LOGGING;
+
+    vfio_set_device_state(vdev, dev_state);
+}
+
+static SaveVMHandlers savevm_vfio_handlers = {
+    .save_setup = vfio_save_setup,
+    .save_live_pending = vfio_save_live_pending,
+    .save_live_iterate = vfio_save_iterate,
+    .save_live_complete_precopy = vfio_save_complete_precopy,
+    .save_cleanup = vfio_save_cleanup,
+    .load_setup = vfio_load_setup,
+    .load_state = vfio_load_state,
+};
+
+int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
+{
+    int ret;
+    Error *local_err = NULL;
+    vdev->migration = g_new0(VFIOMigration, 1);
+
+    if (vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL,
+              "device-state-ctl")) {
+        goto error;
+    }
+
+    if (vfio_check_devstate_version(vdev)) {
+        goto error;
+    }
+
+    if (vfio_get_device_data_caps(vdev)) {
+        goto error;
+    }
+
+    if (vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG,
+              "device-state-data-device-config")) {
+        goto error;
+    }
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        error_report("No suppport of data cap device memory Yet");
+        goto error;
+    }
+
+    if (vfio_device_data_cap_system_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP,
+              "device-state-data-dirtybitmap")) {
+        goto error;
+    }
+
+    vdev->migration->device_state = VFIO_DEVICE_STATE_RUNNING;
+
+    register_savevm_live(NULL, TYPE_VFIO_PCI, -1,
+            VFIO_DEVICE_STATE_INTERFACE_VERSION,
+            &savevm_vfio_handlers,
+            vdev);
+
+    vdev->migration->vm_state =
+        qemu_add_vm_change_state_handler(vfio_vm_change_state_handler, vdev);
+
+    return 0;
+error:
+    error_setg(&vdev->migration_blocker,
+            "VFIO device doesn't support migration");
+    ret = migrate_add_blocker(vdev->migration_blocker, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        error_free(vdev->migration_blocker);
+    }
+
+    g_free(vdev->migration);
+    vdev->migration = NULL;
+
+    return ret;
+}
+
+void vfio_migration_finalize(VFIOPCIDevice *vdev)
+{
+    if (vdev->migration) {
+        int i;
+        qemu_del_vm_change_state_handler(vdev->migration->vm_state);
+        unregister_savevm(NULL, TYPE_VFIO_PCI, vdev);
+        for (i = 0; i < VFIO_DEVSTATE_REGION_NUM; i++) {
+            vfio_region_finalize(&vdev->migration->region[i]);
+        }
+        g_free(vdev->migration);
+        vdev->migration = NULL;
+    } else if (vdev->migration_blocker) {
+        migrate_del_blocker(vdev->migration_blocker);
+        error_free(vdev->migration_blocker);
+    }
+}
hw/vfio/pci.c +0 -1
index c0cb1ec..b8e006b 100644
@@ -37,7 +37,6 @@
 
 #define MSIX_CAP_LENGTH 12
 
-#define TYPE_VFIO_PCI "vfio-pci"
 #define PCI_VFIO(obj)    OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)
 
 static void vfio_disable_interrupts(VFIOPCIDevice *vdev);
hw/vfio/pci.h +24 -1
index b1ae4c0..4b7b1bb 100644
@@ -19,6 +19,7 @@
 #include "qemu/event_notifier.h"
 #include "qemu/queue.h"
 #include "qemu/timer.h"
+#include "sysemu/sysemu.h"
 
 #define PCI_ANY_ID (~0)
 
@@ -56,6 +57,21 @@ typedef struct VFIOBAR {
     QLIST_HEAD(, VFIOQuirk) quirks;
 } VFIOBAR;
 
+enum {
+    VFIO_DEVSTATE_REGION_CTL = 0,
+    VFIO_DEVSTATE_REGION_DATA_CONFIG,
+    VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY,
+    VFIO_DEVSTATE_REGION_DATA_BITMAP,
+    VFIO_DEVSTATE_REGION_NUM,
+};
+typedef struct VFIOMigration {
+    VFIORegion region[VFIO_DEVSTATE_REGION_NUM];
+    uint32_t data_caps;
+    uint32_t device_state;
+    uint64_t devconfig_size;
+    VMChangeStateEntry *vm_state;
+} VFIOMigration;
+
 typedef struct VFIOVGARegion {
     MemoryRegion mem;
     off_t offset;
@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {
     VFIOBAR bars[PCI_NUM_REGIONS - 1]; /* No ROM */
     VFIOVGA *vga; /* 0xa0000, 0x3b0, 0x3c0 */
     void *igd_opregion;
+    VFIOMigration *migration;
+    Error *migration_blocker;
     PCIHostDeviceAddress host;
     EventNotifier err_notifier;
     EventNotifier req_notifier;
@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,
 void vfio_display_reset(VFIOPCIDevice *vdev);
 int vfio_display_probe(VFIOPCIDevice *vdev, Error **errp);
 void vfio_display_finalize(VFIOPCIDevice *vdev);
-
+bool vfio_device_data_cap_system_memory(VFIOPCIDevice *vdev);
+bool vfio_device_data_cap_device_memory(VFIOPCIDevice *vdev);
+int vfio_set_dirty_page_bitmap(VFIOPCIDevice *vdev,
+         uint64_t start_addr, uint64_t page_nr);
+int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp);
+void vfio_migration_finalize(VFIOPCIDevice *vdev);
 #endif /* HW_VFIO_VFIO_PCI_H */
include/hw/vfio/vfio-common.h +1 -0
index 1b434d0..ed43613 100644
@@ -32,6 +32,7 @@
 #endif
 
 #define VFIO_MSG_PREFIX "vfio %s: "
+#define TYPE_VFIO_PCI "vfio-pci"
 
 enum {
     VFIO_DEVICE_TYPE_PCI = 0,
Signature · 1 line
2.7.4

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#8 lore raw

Re: [PATCH 2/5] vfio/migration: support device of device config capability

Quoted 191 lines · Dr. David Alan Gilbert
Quoted 190 lines · * Yan Zhao ([email protected]) wrote:

Device config is the default data that every device should have. so device config capability is by default on, no need to set.

- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.

Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state to VFIO_DEVICE_STATE_LOGGING

---
 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/migration.c           | 633 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |   1 -
 hw/vfio/pci.h                 |  25 +-
 include/hw/vfio/vfio-common.h |   1 +
 5 files changed, 659 insertions(+), 3 deletions(-)
 create mode 100644 hw/vfio/migration.c
hw/vfio/Makefile.objs +1 -1
index 8b3f664..f32ff19 100644
@@ -1,6 +1,6 @@
 ifeq ($(CONFIG_LINUX), y)
 obj-$(CONFIG_SOFTMMU) += common.o
-obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o
+obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.o
 obj-$(CONFIG_VFIO_CCW) += ccw.o
 obj-$(CONFIG_SOFTMMU) += platform.o
 obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o
hw/vfio/migration.c new file +143 -0
new file mode 100644
index 0000000..16d6395
@@ -0,0 +1,633 @@
+#include "qemu/osdep.h"
+
+#include "hw/vfio/vfio-common.h"
+#include "migration/blocker.h"
+#include "migration/register.h"
+#include "qapi/error.h"
+#include "pci.h"
+#include "sysemu/kvm.h"
+#include "exec/ram_addr.h"
+
+#define VFIO_SAVE_FLAG_SETUP 0
+#define VFIO_SAVE_FLAG_PCI 1
+#define VFIO_SAVE_FLAG_DEVCONFIG 2
+#define VFIO_SAVE_FLAG_DEVMEMORY 4
+#define VFIO_SAVE_FLAG_CONTINUE 8
+
+static int vfio_device_state_region_setup(VFIOPCIDevice *vdev,
+        VFIORegion *region, uint32_t subtype, const char *name)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    struct vfio_region_info *info;
+    int ret;
+
+    ret = vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_DEVICE_STATE,
+            subtype, &info);
+    if (ret) {
+        error_report("Failed to get info of region %s", name);
+        return ret;
+    }
+
+    if (vfio_region_setup(OBJECT(vdev), vbasedev,
+            region, info->index, name)) {
+        error_report("Failed to setup migrtion region %s", name);
+        return ret;
+    }
+
+    if (vfio_region_mmap(region)) {
+        error_report("Failed to mmap migrtion region %s", name);
+    }
+
+    return 0;
+}
+
+bool vfio_device_data_cap_system_memory(VFIOPCIDevice *vdev)
+{
+   return !!(vdev->migration->data_caps & VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY);
+}
+
+bool vfio_device_data_cap_device_memory(VFIOPCIDevice *vdev)
+{
+   return !!(vdev->migration->data_caps & VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY);
+}
+
+static bool vfio_device_state_region_mmaped(VFIORegion *region)
+{
+    bool mmaped = true;
+    if (region->nr_mmaps != 1 || region->mmaps[0].offset ||
+            (region->size != region->mmaps[0].size) ||
+            (region->mmaps[0].mmap == NULL)) {
+        mmaped = false;
+    }
+
+    return mmaped;
+}
+
+static int vfio_get_device_config_size(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    uint64_t len;
+    int sz;
+
+    sz = sizeof(len);
+    if (pread(vbasedev->fd, &len, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.size))
+            != sz) {
+        error_report("vfio: Failed to get length of device config");
+        return -1;
+    }
+    if (len > region_config->size) {
+        error_report("vfio: Error device config length");
+        return -1;
+    }
+    vdev->migration->devconfig_size = len;
+
+    return 0;
+}
+
+static int vfio_set_device_config_size(VFIOPCIDevice *vdev, uint64_t size)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    int sz;
+
+    if (size > region_config->size) {
+        return -1;
+    }
+
+    sz = sizeof(size);
+    if (pwrite(vbasedev->fd, &size, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.size))
+            != sz) {
+        error_report("vfio: Failed to set length of device config");
+        return -1;
+    }
+    vdev->migration->devconfig_size = size;
+    return 0;
+}
+
+static int vfio_save_data_device_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+    uint64_t len = vdev->migration->devconfig_size;
+
+    qemu_put_be64(f, len);
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.action))
+            != sz) {
+        error_report("vfio: action failure for device config get buffer");
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_config)) {
+        buf = g_malloc(len);
Quoted 9 lines

hw/vfio/migration.c line 147

Quoted 4 lines
hw/vfio/migration.c new file +4 -0
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }

g_malloc never returns NULL; it aborts on failure to allocate. So you can either drop the check, or my preference is to use g_try_malloc for large/unknown areas, and it can return NULL.

ok. got that. I'll use g_try_malloc next time :)

Quoted 282 lines
Quoted 282 lines
hw/vfio/migration.c new file +282 -0
+        if (pread(vbasedev->fd, buf, len, region_config->fd_offset) != len) {
+            error_report("vfio: Failed read device config buffer");
+            return -1;
+        }
+        qemu_put_buffer(f, buf, len);
+        g_free(buf);
+    } else {
+        dest = region_config->mmaps[0].mmap;
+        qemu_put_buffer(f, dest, len);
+    }
+    return 0;
+}
+
+static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
+                            QEMUFile *f, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_SET_BUFFER;
+
+    vfio_set_device_config_size(vdev, len);
+
+    if (!vfio_device_state_region_mmaped(region_config)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        qemu_get_buffer(f, buf, len);
+        if (pwrite(vbasedev->fd, buf, len,
+                    region_config->fd_offset) != len) {
+            error_report("vfio: Failed to write devie config buffer");
+            return -1;
+        }
+        g_free(buf);
+    } else {
+        dest = region_config->mmaps[0].mmap;
+        qemu_get_buffer(f, dest, len);
+    }
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.action))
+            != sz) {
+        error_report("vfio: action failure for device config set buffer");
+        return -1;
+    }
+
+    return 0;
+}
+
+static int vfio_set_dirty_page_bitmap_chunk(VFIOPCIDevice *vdev,
+        uint64_t start_addr, uint64_t page_nr)
+{
+
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_bitmap =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP];
+    unsigned long bitmap_size =
+                    BITS_TO_LONGS(page_nr) * sizeof(unsigned long);
+    uint32_t sz;
+
+    struct {
+        __u64 start_addr;
+        __u64 page_nr;
+    } system_memory;
+    system_memory.start_addr = start_addr;
+    system_memory.page_nr = page_nr;
+    sz = sizeof(system_memory);
+    if (pwrite(vbasedev->fd, &system_memory, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, system_memory))
+            != sz) {
+        error_report("vfio: Failed to set system memory range for dirty pages");
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_bitmap)) {
+        void *bitmap = g_malloc0(bitmap_size);
+
+        if (pread(vbasedev->fd, bitmap, bitmap_size,
+                    region_bitmap->fd_offset) != bitmap_size) {
+            error_report("vfio: Failed to read dirty bitmap data");
+            return -1;
+        }
+
+        cpu_physical_memory_set_dirty_lebitmap(bitmap, start_addr, page_nr);
+
+        g_free(bitmap);
+    } else {
+        cpu_physical_memory_set_dirty_lebitmap(
+                    region_bitmap->mmaps[0].mmap,
+                    start_addr, page_nr);
+    }
+   return 0;
+}
+
+int vfio_set_dirty_page_bitmap(VFIOPCIDevice *vdev,
+        uint64_t start_addr, uint64_t page_nr)
+{
+    VFIORegion *region_bitmap =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP];
+    unsigned long chunk_size = region_bitmap->size;
+    uint64_t chunk_pg_nr = (chunk_size / sizeof(unsigned long)) *
+                                BITS_PER_LONG;
+
+    uint64_t cnt_left;
+    int rc = 0;
+
+    cnt_left = page_nr;
+
+    while (cnt_left >= chunk_pg_nr) {
+        rc = vfio_set_dirty_page_bitmap_chunk(vdev, start_addr, chunk_pg_nr);
+        if (rc) {
+            goto exit;
+        }
+        cnt_left -= chunk_pg_nr;
+        start_addr += start_addr;
+   }
+   rc = vfio_set_dirty_page_bitmap_chunk(vdev, start_addr, cnt_left);
+
+exit:
+   return rc;
+}
+
+static int vfio_set_device_state(VFIOPCIDevice *vdev,
+        uint32_t dev_state)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint32_t sz = sizeof(dev_state);
+
+    if (!vdev->migration) {
+        return -1;
+    }
+
+    if (pwrite(vbasedev->fd, &dev_state, sz,
+              region->fd_offset +
+              offsetof(struct vfio_device_state_ctl, device_state))
+            != sz) {
+        error_report("vfio: Failed to set device state %d", dev_state);
+        return -1;
+    }
+    vdev->migration->device_state = dev_state;
+    return 0;
+}
+
+static int vfio_get_device_data_caps(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+
+    uint32_t caps;
+    uint32_t size = sizeof(caps);
+
+    if (pread(vbasedev->fd, &caps, size,
+                region->fd_offset +
+                offsetof(struct vfio_device_state_ctl, caps))
+            != size) {
+        error_report("%s Failed to read data caps of device states",
+                vbasedev->name);
+        return -1;
+    }
+    vdev->migration->data_caps = caps;
+    return 0;
+}
+
+
+static int vfio_check_devstate_version(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+
+    uint32_t version;
+    uint32_t size = sizeof(version);
+
+    if (pread(vbasedev->fd, &version, size,
+                region->fd_offset +
+                offsetof(struct vfio_device_state_ctl, version))
+            != size) {
+        error_report("%s Failed to read version of device state interfaces",
+                vbasedev->name);
+        return -1;
+    }
+
+    if (version != VFIO_DEVICE_STATE_INTERFACE_VERSION) {
+        error_report("%s migration version mismatch, right version is %d",
+                vbasedev->name, VFIO_DEVICE_STATE_INTERFACE_VERSION);
+        return -1;
+    }
+
+    return 0;
+}
+
+static void vfio_vm_change_state_handler(void *pv, int running, RunState state)
+{
+    VFIOPCIDevice *vdev = pv;
+    uint32_t dev_state = vdev->migration->device_state;
+
+    if (!running) {
+        dev_state |= VFIO_DEVICE_STATE_STOP;
+    } else {
+        dev_state &= ~VFIO_DEVICE_STATE_STOP;
+    }
+
+    vfio_set_device_state(vdev, dev_state);
+}
+
+static void vfio_save_live_pending(QEMUFile *f, void *opaque,
+                                   uint64_t max_size,
+                                   uint64_t *res_precopy_only,
+                                   uint64_t *res_compatible,
+                                   uint64_t *res_post_copy_only)
+{
+    VFIOPCIDevice *vdev = opaque;
+
+    if (!vfio_device_data_cap_device_memory(vdev)) {
+        return;
+    }
+
+    return;
+}
+
+static int vfio_save_iterate(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+
+    if (!vfio_device_data_cap_device_memory(vdev)) {
+        return 0;
+    }
+
+    return 0;
+}
+
+static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    uint32_t ctl, msi_lo, msi_hi, msi_data, bar_cfg, i;
+    bool msi_64bit;
+
+    /* retore pci bar configuration */
+    ctl = pci_default_read_config(pdev, PCI_COMMAND, 2);
+    vfio_pci_write_config(pdev, PCI_COMMAND,
+            ctl & (!(PCI_COMMAND_IO | PCI_COMMAND_MEMORY)), 2);
+    for (i = 0; i < PCI_ROM_SLOT; i++) {
+        bar_cfg = qemu_get_be32(f);
+        vfio_pci_write_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, bar_cfg, 4);
+    }
+    vfio_pci_write_config(pdev, PCI_COMMAND,
+            ctl | PCI_COMMAND_IO | PCI_COMMAND_MEMORY, 2);
+
+    /* restore msi configuration */
+    ctl = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, 2);
+    msi_64bit = !!(ctl & PCI_MSI_FLAGS_64BIT);
+
+    vfio_pci_write_config(&vdev->pdev,
+            pdev->msi_cap + PCI_MSI_FLAGS,
+            ctl & (!PCI_MSI_FLAGS_ENABLE), 2);
+
+    msi_lo = qemu_get_be32(f);
+    vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_LO, msi_lo, 4);
+
+    if (msi_64bit) {
+        msi_hi = qemu_get_be32(f);
+        vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_HI,
+                msi_hi, 4);
+    }
+    msi_data = qemu_get_be32(f);
+    vfio_pci_write_config(pdev,
+            pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
Quoted 8 lines

hw/vfio/migration.c line 433

Quoted 4 lines
hw/vfio/migration.c new file +4 -0
+            msi_data, 2);
+
+    vfio_pci_write_config(&vdev->pdev, pdev->msi_cap + PCI_MSI_FLAGS,
+            ctl | PCI_MSI_FLAGS_ENABLE, 2);

It would probably be best to use a VMStateDescription and the macros for this if possible; I bet you'll want to add more fields in the future for example.

yes, it is also a good idea to use VMStateDescription for pci general config data, which are saved one time in stop-and-copy phase. But it's a little hard to maintain two type of interfaces. Maybe use the way as what VMStateDescription did, i.e. using field list and macros?

Quoted 3 lines

Also what happens if the data read from the migration stream is bad or doesn't agree with this devices hardware? How does this fail?

right, errors in migration stream needs to be checked. I'll add the error check.
For the hardware incompatiable issue, seems vfio_pci_write_config does not return error if device driver returns error for this write. Seems it needs to be addressed in somewhere else like checking compitibility before lauching migration.

Quoted 287 lines
Quoted 281 lines
hw/vfio/migration.c new file +199 -0
+}
+
+static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
+{
+    VFIOPCIDevice *vdev = opaque;
+    int flag;
+    uint64_t len;
+    int ret = 0;
+
+    if (version_id != VFIO_DEVICE_STATE_INTERFACE_VERSION) {
+        return -EINVAL;
+    }
+
+    do {
+        flag = qemu_get_byte(f);
+
+        switch (flag & ~VFIO_SAVE_FLAG_CONTINUE) {
+        case VFIO_SAVE_FLAG_SETUP:
+            break;
+        case VFIO_SAVE_FLAG_PCI:
+            vfio_pci_load_config(vdev, f);
+            break;
+        case VFIO_SAVE_FLAG_DEVCONFIG:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_config(vdev, f, len);
+            break;
+        default:
+            ret = -EINVAL;
+        }
+    } while (flag & VFIO_SAVE_FLAG_CONTINUE);
+
+    return ret;
+}
+
+static void vfio_pci_save_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    uint32_t msi_cfg, msi_lo, msi_hi, msi_data, bar_cfg, i;
+    bool msi_64bit;
+
+    for (i = 0; i < PCI_ROM_SLOT; i++) {
+        bar_cfg = pci_default_read_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, 4);
+        qemu_put_be32(f, bar_cfg);
+    }
+
+    msi_cfg = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, 2);
+    msi_64bit = !!(msi_cfg & PCI_MSI_FLAGS_64BIT);
+
+    msi_lo = pci_default_read_config(pdev,
+            pdev->msi_cap + PCI_MSI_ADDRESS_LO, 4);
+    qemu_put_be32(f, msi_lo);
+
+    if (msi_64bit) {
+        msi_hi = pci_default_read_config(pdev,
+                pdev->msi_cap + PCI_MSI_ADDRESS_HI,
+                4);
+        qemu_put_be32(f, msi_hi);
+    }
+
+    msi_data = pci_default_read_config(pdev,
+            pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
+            2);
+    qemu_put_be32(f, msi_data);
+
+}
+
+static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    int rc = 0;
+
+    qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
+    vfio_pci_save_config(vdev, f);
+
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVCONFIG);
+    rc += vfio_get_device_config_size(vdev);
+    rc += vfio_save_data_device_config(vdev, f);
+
+    return rc;
+}
+
+static int vfio_save_setup(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
+            VFIO_DEVICE_STATE_LOGGING);
+    return 0;
+}
+
+static int vfio_load_setup(QEMUFile *f, void *opaque)
+{
+    return 0;
+}
+
+static void vfio_save_cleanup(void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    uint32_t dev_state = vdev->migration->device_state;
+
+    dev_state &= ~VFIO_DEVICE_STATE_LOGGING;
+
+    vfio_set_device_state(vdev, dev_state);
+}
+
+static SaveVMHandlers savevm_vfio_handlers = {
+    .save_setup = vfio_save_setup,
+    .save_live_pending = vfio_save_live_pending,
+    .save_live_iterate = vfio_save_iterate,
+    .save_live_complete_precopy = vfio_save_complete_precopy,
+    .save_cleanup = vfio_save_cleanup,
+    .load_setup = vfio_load_setup,
+    .load_state = vfio_load_state,
+};
+
+int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
+{
+    int ret;
+    Error *local_err = NULL;
+    vdev->migration = g_new0(VFIOMigration, 1);
+
+    if (vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL,
+              "device-state-ctl")) {
+        goto error;
+    }
+
+    if (vfio_check_devstate_version(vdev)) {
+        goto error;
+    }
+
+    if (vfio_get_device_data_caps(vdev)) {
+        goto error;
+    }
+
+    if (vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG,
+              "device-state-data-device-config")) {
+        goto error;
+    }
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        error_report("No suppport of data cap device memory Yet");
+        goto error;
+    }
+
+    if (vfio_device_data_cap_system_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP,
+              "device-state-data-dirtybitmap")) {
+        goto error;
+    }
+
+    vdev->migration->device_state = VFIO_DEVICE_STATE_RUNNING;
+
+    register_savevm_live(NULL, TYPE_VFIO_PCI, -1,
+            VFIO_DEVICE_STATE_INTERFACE_VERSION,
+            &savevm_vfio_handlers,
+            vdev);
+
+    vdev->migration->vm_state =
+        qemu_add_vm_change_state_handler(vfio_vm_change_state_handler, vdev);
+
+    return 0;
+error:
+    error_setg(&vdev->migration_blocker,
+            "VFIO device doesn't support migration");
+    ret = migrate_add_blocker(vdev->migration_blocker, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        error_free(vdev->migration_blocker);
+    }
+
+    g_free(vdev->migration);
+    vdev->migration = NULL;
+
+    return ret;
+}
+
+void vfio_migration_finalize(VFIOPCIDevice *vdev)
+{
+    if (vdev->migration) {
+        int i;
+        qemu_del_vm_change_state_handler(vdev->migration->vm_state);
+        unregister_savevm(NULL, TYPE_VFIO_PCI, vdev);
+        for (i = 0; i < VFIO_DEVSTATE_REGION_NUM; i++) {
+            vfio_region_finalize(&vdev->migration->region[i]);
+        }
+        g_free(vdev->migration);
+        vdev->migration = NULL;
+    } else if (vdev->migration_blocker) {
+        migrate_del_blocker(vdev->migration_blocker);
+        error_free(vdev->migration_blocker);
+    }
+}
hw/vfio/pci.c +0 -1
index c0cb1ec..b8e006b 100644
@@ -37,7 +37,6 @@
 
 #define MSIX_CAP_LENGTH 12
 
-#define TYPE_VFIO_PCI "vfio-pci"
 #define PCI_VFIO(obj)    OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)
 
 static void vfio_disable_interrupts(VFIOPCIDevice *vdev);
hw/vfio/pci.h +24 -1
index b1ae4c0..4b7b1bb 100644
@@ -19,6 +19,7 @@
 #include "qemu/event_notifier.h"
 #include "qemu/queue.h"
 #include "qemu/timer.h"
+#include "sysemu/sysemu.h"
 
 #define PCI_ANY_ID (~0)
 
@@ -56,6 +57,21 @@ typedef struct VFIOBAR {
     QLIST_HEAD(, VFIOQuirk) quirks;
 } VFIOBAR;
 
+enum {
+    VFIO_DEVSTATE_REGION_CTL = 0,
+    VFIO_DEVSTATE_REGION_DATA_CONFIG,
+    VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY,
+    VFIO_DEVSTATE_REGION_DATA_BITMAP,
+    VFIO_DEVSTATE_REGION_NUM,
+};
+typedef struct VFIOMigration {
+    VFIORegion region[VFIO_DEVSTATE_REGION_NUM];
+    uint32_t data_caps;
+    uint32_t device_state;
+    uint64_t devconfig_size;
+    VMChangeStateEntry *vm_state;
+} VFIOMigration;
+
 typedef struct VFIOVGARegion {
     MemoryRegion mem;
     off_t offset;
@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {
     VFIOBAR bars[PCI_NUM_REGIONS - 1]; /* No ROM */
     VFIOVGA *vga; /* 0xa0000, 0x3b0, 0x3c0 */
     void *igd_opregion;
+    VFIOMigration *migration;
+    Error *migration_blocker;
     PCIHostDeviceAddress host;
     EventNotifier err_notifier;
     EventNotifier req_notifier;
@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,
 void vfio_display_reset(VFIOPCIDevice *vdev);
 int vfio_display_probe(VFIOPCIDevice *vdev, Error **errp);
 void vfio_display_finalize(VFIOPCIDevice *vdev);
-
+bool vfio_device_data_cap_system_memory(VFIOPCIDevice *vdev);
+bool vfio_device_data_cap_device_memory(VFIOPCIDevice *vdev);
+int vfio_set_dirty_page_bitmap(VFIOPCIDevice *vdev,
+         uint64_t start_addr, uint64_t page_nr);
+int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp);
+void vfio_migration_finalize(VFIOPCIDevice *vdev);
 #endif /* HW_VFIO_VFIO_PCI_H */
include/hw/vfio/vfio-common.h +1 -0
index 1b434d0..ed43613 100644
@@ -32,6 +32,7 @@
 #endif
 
 #define VFIO_MSG_PREFIX "vfio %s: "
+#define TYPE_VFIO_PCI "vfio-pci"
 
 enum {
     VFIO_DEVICE_TYPE_PCI = 0,
Signature · 1 line
2.7.4

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

#9 lore raw

Re: [PATCH 2/5] vfio/migration: support device of device config capability

Quoted 2 lines · * Zhao Yan ([email protected]) wrote:
Quoted 1 line · Dr. David Alan Gilbert

* Yan Zhao ([email protected]) wrote:

<snip>

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
hw/vfio/migration.c new file +7 -0
+    msi_data = qemu_get_be32(f);
+    vfio_pci_write_config(pdev,
+            pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
+            msi_data, 2);
+
+    vfio_pci_write_config(&vdev->pdev, pdev->msi_cap + PCI_MSI_FLAGS,
+            ctl | PCI_MSI_FLAGS_ENABLE, 2);
Quoted 8 lines
Quoted 3 lines

It would probably be best to use a VMStateDescription and the macros for this if possible; I bet you'll want to add more fields in the future for example.

yes, it is also a good idea to use VMStateDescription for pci general config data, which are saved one time in stop-and-copy phase. But it's a little hard to maintain two type of interfaces. Maybe use the way as what VMStateDescription did, i.e. using field list and macros?

Maybe, but if you can actually use the VMStateDescription it would give you the versioning and conditional fields and things like that.

Quoted 7 lines
Quoted 3 lines

Also what happens if the data read from the migration stream is bad or doesn't agree with this devices hardware? How does this fail?

right, errors in migration stream needs to be checked. I'll add the error check.
For the hardware incompatiable issue, seems vfio_pci_write_config does not return error if device driver returns error for this write.

It would be great if we could make that fail properly.

Quoted 2 lines

Seems it needs to be addressed in somewhere else like checking compitibility before lauching migration.

That would be good; but we should still guard against it being wrong if possible - these type of checks for compatibility are probably quite difficult.

Dave

Quoted 276 lines
Quoted 276 lines
Quoted 276 lines
hw/vfio/migration.c new file +199 -0
+}
+
+static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
+{
+    VFIOPCIDevice *vdev = opaque;
+    int flag;
+    uint64_t len;
+    int ret = 0;
+
+    if (version_id != VFIO_DEVICE_STATE_INTERFACE_VERSION) {
+        return -EINVAL;
+    }
+
+    do {
+        flag = qemu_get_byte(f);
+
+        switch (flag & ~VFIO_SAVE_FLAG_CONTINUE) {
+        case VFIO_SAVE_FLAG_SETUP:
+            break;
+        case VFIO_SAVE_FLAG_PCI:
+            vfio_pci_load_config(vdev, f);
+            break;
+        case VFIO_SAVE_FLAG_DEVCONFIG:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_config(vdev, f, len);
+            break;
+        default:
+            ret = -EINVAL;
+        }
+    } while (flag & VFIO_SAVE_FLAG_CONTINUE);
+
+    return ret;
+}
+
+static void vfio_pci_save_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    uint32_t msi_cfg, msi_lo, msi_hi, msi_data, bar_cfg, i;
+    bool msi_64bit;
+
+    for (i = 0; i < PCI_ROM_SLOT; i++) {
+        bar_cfg = pci_default_read_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, 4);
+        qemu_put_be32(f, bar_cfg);
+    }
+
+    msi_cfg = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, 2);
+    msi_64bit = !!(msi_cfg & PCI_MSI_FLAGS_64BIT);
+
+    msi_lo = pci_default_read_config(pdev,
+            pdev->msi_cap + PCI_MSI_ADDRESS_LO, 4);
+    qemu_put_be32(f, msi_lo);
+
+    if (msi_64bit) {
+        msi_hi = pci_default_read_config(pdev,
+                pdev->msi_cap + PCI_MSI_ADDRESS_HI,
+                4);
+        qemu_put_be32(f, msi_hi);
+    }
+
+    msi_data = pci_default_read_config(pdev,
+            pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
+            2);
+    qemu_put_be32(f, msi_data);
+
+}
+
+static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    int rc = 0;
+
+    qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
+    vfio_pci_save_config(vdev, f);
+
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVCONFIG);
+    rc += vfio_get_device_config_size(vdev);
+    rc += vfio_save_data_device_config(vdev, f);
+
+    return rc;
+}
+
+static int vfio_save_setup(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
+            VFIO_DEVICE_STATE_LOGGING);
+    return 0;
+}
+
+static int vfio_load_setup(QEMUFile *f, void *opaque)
+{
+    return 0;
+}
+
+static void vfio_save_cleanup(void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    uint32_t dev_state = vdev->migration->device_state;
+
+    dev_state &= ~VFIO_DEVICE_STATE_LOGGING;
+
+    vfio_set_device_state(vdev, dev_state);
+}
+
+static SaveVMHandlers savevm_vfio_handlers = {
+    .save_setup = vfio_save_setup,
+    .save_live_pending = vfio_save_live_pending,
+    .save_live_iterate = vfio_save_iterate,
+    .save_live_complete_precopy = vfio_save_complete_precopy,
+    .save_cleanup = vfio_save_cleanup,
+    .load_setup = vfio_load_setup,
+    .load_state = vfio_load_state,
+};
+
+int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
+{
+    int ret;
+    Error *local_err = NULL;
+    vdev->migration = g_new0(VFIOMigration, 1);
+
+    if (vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL,
+              "device-state-ctl")) {
+        goto error;
+    }
+
+    if (vfio_check_devstate_version(vdev)) {
+        goto error;
+    }
+
+    if (vfio_get_device_data_caps(vdev)) {
+        goto error;
+    }
+
+    if (vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG,
+              "device-state-data-device-config")) {
+        goto error;
+    }
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        error_report("No suppport of data cap device memory Yet");
+        goto error;
+    }
+
+    if (vfio_device_data_cap_system_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP,
+              "device-state-data-dirtybitmap")) {
+        goto error;
+    }
+
+    vdev->migration->device_state = VFIO_DEVICE_STATE_RUNNING;
+
+    register_savevm_live(NULL, TYPE_VFIO_PCI, -1,
+            VFIO_DEVICE_STATE_INTERFACE_VERSION,
+            &savevm_vfio_handlers,
+            vdev);
+
+    vdev->migration->vm_state =
+        qemu_add_vm_change_state_handler(vfio_vm_change_state_handler, vdev);
+
+    return 0;
+error:
+    error_setg(&vdev->migration_blocker,
+            "VFIO device doesn't support migration");
+    ret = migrate_add_blocker(vdev->migration_blocker, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        error_free(vdev->migration_blocker);
+    }
+
+    g_free(vdev->migration);
+    vdev->migration = NULL;
+
+    return ret;
+}
+
+void vfio_migration_finalize(VFIOPCIDevice *vdev)
+{
+    if (vdev->migration) {
+        int i;
+        qemu_del_vm_change_state_handler(vdev->migration->vm_state);
+        unregister_savevm(NULL, TYPE_VFIO_PCI, vdev);
+        for (i = 0; i < VFIO_DEVSTATE_REGION_NUM; i++) {
+            vfio_region_finalize(&vdev->migration->region[i]);
+        }
+        g_free(vdev->migration);
+        vdev->migration = NULL;
+    } else if (vdev->migration_blocker) {
+        migrate_del_blocker(vdev->migration_blocker);
+        error_free(vdev->migration_blocker);
+    }
+}
hw/vfio/pci.c +0 -1
index c0cb1ec..b8e006b 100644
@@ -37,7 +37,6 @@
 
 #define MSIX_CAP_LENGTH 12
 
-#define TYPE_VFIO_PCI "vfio-pci"
 #define PCI_VFIO(obj)    OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)
 
 static void vfio_disable_interrupts(VFIOPCIDevice *vdev);
hw/vfio/pci.h +24 -1
index b1ae4c0..4b7b1bb 100644
@@ -19,6 +19,7 @@
 #include "qemu/event_notifier.h"
 #include "qemu/queue.h"
 #include "qemu/timer.h"
+#include "sysemu/sysemu.h"
 
 #define PCI_ANY_ID (~0)
 
@@ -56,6 +57,21 @@ typedef struct VFIOBAR {
     QLIST_HEAD(, VFIOQuirk) quirks;
 } VFIOBAR;
 
+enum {
+    VFIO_DEVSTATE_REGION_CTL = 0,
+    VFIO_DEVSTATE_REGION_DATA_CONFIG,
+    VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY,
+    VFIO_DEVSTATE_REGION_DATA_BITMAP,
+    VFIO_DEVSTATE_REGION_NUM,
+};
+typedef struct VFIOMigration {
+    VFIORegion region[VFIO_DEVSTATE_REGION_NUM];
+    uint32_t data_caps;
+    uint32_t device_state;
+    uint64_t devconfig_size;
+    VMChangeStateEntry *vm_state;
+} VFIOMigration;
+
 typedef struct VFIOVGARegion {
     MemoryRegion mem;
     off_t offset;
@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {
     VFIOBAR bars[PCI_NUM_REGIONS - 1]; /* No ROM */
     VFIOVGA *vga; /* 0xa0000, 0x3b0, 0x3c0 */
     void *igd_opregion;
+    VFIOMigration *migration;
+    Error *migration_blocker;
     PCIHostDeviceAddress host;
     EventNotifier err_notifier;
     EventNotifier req_notifier;
@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,
 void vfio_display_reset(VFIOPCIDevice *vdev);
 int vfio_display_probe(VFIOPCIDevice *vdev, Error **errp);
 void vfio_display_finalize(VFIOPCIDevice *vdev);
-
+bool vfio_device_data_cap_system_memory(VFIOPCIDevice *vdev);
+bool vfio_device_data_cap_device_memory(VFIOPCIDevice *vdev);
+int vfio_set_dirty_page_bitmap(VFIOPCIDevice *vdev,
+         uint64_t start_addr, uint64_t page_nr);
+int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp);
+void vfio_migration_finalize(VFIOPCIDevice *vdev);
 #endif /* HW_VFIO_VFIO_PCI_H */
include/hw/vfio/vfio-common.h +1 -0
index 1b434d0..ed43613 100644
@@ -32,6 +32,7 @@
 #endif
 
 #define VFIO_MSG_PREFIX "vfio %s: "
+#define TYPE_VFIO_PCI "vfio-pci"
 
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
include/hw/vfio/vfio-common.h +0 -0
 enum {
     VFIO_DEVICE_TYPE_PCI = 0,
Signature · 1 line
2.7.4

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#10 lore raw

Re: [PATCH 2/5] vfio/migration: support device of device config capability

In reply to Yan Zhao
Quoted 34 lines · On Tue, 19 Feb 2019 16:52:27 +0800 Yan Zhao <[email protected]> wrote:

Device config is the default data that every device should have. so device config capability is by default on, no need to set.

- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.

Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state to VFIO_DEVICE_STATE_LOGGING

---
 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/migration.c           | 633 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |   1 -
 hw/vfio/pci.h                 |  25 +-
 include/hw/vfio/vfio-common.h |   1 +
 5 files changed, 659 insertions(+), 3 deletions(-)
 create mode 100644 hw/vfio/migration.c
hw/vfio/Makefile.objs +0 -0
index 8b3f664..f32ff19 100644

hw/vfio/Makefile.objs line 3

Quoted 4 lines
hw/vfio/Makefile.objs +1 -1
@@ -1,6 +1,6 @@
 ifeq ($(CONFIG_LINUX), y)
 obj-$(CONFIG_SOFTMMU) += common.o
-obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o
+obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.o

I think you want to split the migration code: The type-independent code, and the pci-specific code.

Quoted 17 lines
hw/vfio/Makefile.objs +0 -0
 obj-$(CONFIG_VFIO_CCW) += ccw.o
 obj-$(CONFIG_SOFTMMU) += platform.o
 obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o
hw/vfio/migration.c new file +8 -0
new file mode 100644
index 0000000..16d6395
@@ -0,0 +1,633 @@
+#include "qemu/osdep.h"
+
+#include "hw/vfio/vfio-common.h"
+#include "migration/blocker.h"
+#include "migration/register.h"
+#include "qapi/error.h"
+#include "pci.h"
+#include "sysemu/kvm.h"

hw/vfio/migration.c line 18

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+#include "exec/ram_addr.h"
+
+#define VFIO_SAVE_FLAG_SETUP 0
+#define VFIO_SAVE_FLAG_PCI 1
+#define VFIO_SAVE_FLAG_DEVCONFIG 2
+#define VFIO_SAVE_FLAG_DEVMEMORY 4
+#define VFIO_SAVE_FLAG_CONTINUE 8
+
+static int vfio_device_state_region_setup(VFIOPCIDevice *vdev,
+        VFIORegion *region, uint32_t subtype, const char *name)

This function looks like it should be more generic and e.g. take a VFIODevice instead of a VFIOPCIDevice as argument.

Quoted 24 lines
hw/vfio/migration.c new file +24 -0
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    struct vfio_region_info *info;
+    int ret;
+
+    ret = vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_DEVICE_STATE,
+            subtype, &info);
+    if (ret) {
+        error_report("Failed to get info of region %s", name);
+        return ret;
+    }
+
+    if (vfio_region_setup(OBJECT(vdev), vbasedev,
+            region, info->index, name)) {
+        error_report("Failed to setup migrtion region %s", name);
+        return ret;
+    }
+
+    if (vfio_region_mmap(region)) {
+        error_report("Failed to mmap migrtion region %s", name);
+    }
+
+    return 0;
+}

hw/vfio/migration.c line 52

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+
+bool vfio_device_data_cap_system_memory(VFIOPCIDevice *vdev)
+{
+   return !!(vdev->migration->data_caps & VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY);
+}
+
+bool vfio_device_data_cap_device_memory(VFIOPCIDevice *vdev)
+{
+   return !!(vdev->migration->data_caps & VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY);
+}

These two as well. The migration structure should probably hang off the VFIODevice instead.

hw/vfio/migration.c line 64

Quoted 12 lines
hw/vfio/migration.c new file +12 -0
+
+static bool vfio_device_state_region_mmaped(VFIORegion *region)
+{
+    bool mmaped = true;
+    if (region->nr_mmaps != 1 || region->mmaps[0].offset ||
+            (region->size != region->mmaps[0].size) ||
+            (region->mmaps[0].mmap == NULL)) {
+        mmaped = false;
+    }
+
+    return mmaped;
+}

s/mmaped/mmapped/ ?

hw/vfio/migration.c line 72

Quoted 8 lines
hw/vfio/migration.c new file +8 -0
+
+static int vfio_get_device_config_size(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];

This looks like it should not depend on pci, either.

Quoted 17 lines
hw/vfio/migration.c new file +17 -0
+    uint64_t len;
+    int sz;
+
+    sz = sizeof(len);
+    if (pread(vbasedev->fd, &len, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.size))
+            != sz) {
+        error_report("vfio: Failed to get length of device config");
+        return -1;
+    }
+    if (len > region_config->size) {
+        error_report("vfio: Error device config length");
+        return -1;
+    }
+    vdev->migration->devconfig_size = len;
+

hw/vfio/migration.c line 99

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+    return 0;
+}
+
+static int vfio_set_device_config_size(VFIOPCIDevice *vdev, uint64_t size)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];

Ditto. Also for the functions below.

Quoted 22 lines
hw/vfio/migration.c new file +22 -0
+    int sz;
+
+    if (size > region_config->size) {
+        return -1;
+    }
+
+    sz = sizeof(size);
+    if (pwrite(vbasedev->fd, &size, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.size))
+            != sz) {
+        error_report("vfio: Failed to set length of device config");
+        return -1;
+    }
+    vdev->migration->devconfig_size = size;
+    return 0;
+}
+
+static int vfio_save_data_device_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =

hw/vfio/migration.c line 131

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+    uint64_t len = vdev->migration->devconfig_size;
+
+    qemu_put_be64(f, len);

Why big endian? (Generally, do we need any endianness considerations?)

hw/vfio/migration.c line 140

Quoted 9 lines
hw/vfio/migration.c new file +9 -0
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.action))
+            != sz) {
+        error_report("vfio: action failure for device config get buffer");
+        return -1;
+    }

Might make sense to wrap this into a set_action() helper that takes a SET_BUFFER/GET_BUFFER argument.

Quoted 148 lines
hw/vfio/migration.c new file +148 -0
+
+    if (!vfio_device_state_region_mmaped(region_config)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        if (pread(vbasedev->fd, buf, len, region_config->fd_offset) != len) {
+            error_report("vfio: Failed read device config buffer");
+            return -1;
+        }
+        qemu_put_buffer(f, buf, len);
+        g_free(buf);
+    } else {
+        dest = region_config->mmaps[0].mmap;
+        qemu_put_buffer(f, dest, len);
+    }
+    return 0;
+}
+
+static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
+                            QEMUFile *f, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_SET_BUFFER;
+
+    vfio_set_device_config_size(vdev, len);
+
+    if (!vfio_device_state_region_mmaped(region_config)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        qemu_get_buffer(f, buf, len);
+        if (pwrite(vbasedev->fd, buf, len,
+                    region_config->fd_offset) != len) {
+            error_report("vfio: Failed to write devie config buffer");
+            return -1;
+        }
+        g_free(buf);
+    } else {
+        dest = region_config->mmaps[0].mmap;
+        qemu_get_buffer(f, dest, len);
+    }
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.action))
+            != sz) {
+        error_report("vfio: action failure for device config set buffer");
+        return -1;
+    }
+
+    return 0;
+}
+
+static int vfio_set_dirty_page_bitmap_chunk(VFIOPCIDevice *vdev,
+        uint64_t start_addr, uint64_t page_nr)
+{
+
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_bitmap =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP];
+    unsigned long bitmap_size =
+                    BITS_TO_LONGS(page_nr) * sizeof(unsigned long);
+    uint32_t sz;
+
+    struct {
+        __u64 start_addr;
+        __u64 page_nr;
+    } system_memory;
+    system_memory.start_addr = start_addr;
+    system_memory.page_nr = page_nr;
+    sz = sizeof(system_memory);
+    if (pwrite(vbasedev->fd, &system_memory, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, system_memory))
+            != sz) {
+        error_report("vfio: Failed to set system memory range for dirty pages");
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_bitmap)) {
+        void *bitmap = g_malloc0(bitmap_size);
+
+        if (pread(vbasedev->fd, bitmap, bitmap_size,
+                    region_bitmap->fd_offset) != bitmap_size) {
+            error_report("vfio: Failed to read dirty bitmap data");
+            return -1;
+        }
+
+        cpu_physical_memory_set_dirty_lebitmap(bitmap, start_addr, page_nr);
+
+        g_free(bitmap);
+    } else {
+        cpu_physical_memory_set_dirty_lebitmap(
+                    region_bitmap->mmaps[0].mmap,
+                    start_addr, page_nr);
+    }
+   return 0;
+}
+
+int vfio_set_dirty_page_bitmap(VFIOPCIDevice *vdev,
+        uint64_t start_addr, uint64_t page_nr)
+{
+    VFIORegion *region_bitmap =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP];
+    unsigned long chunk_size = region_bitmap->size;
+    uint64_t chunk_pg_nr = (chunk_size / sizeof(unsigned long)) *
+                                BITS_PER_LONG;
+
+    uint64_t cnt_left;
+    int rc = 0;
+
+    cnt_left = page_nr;
+
+    while (cnt_left >= chunk_pg_nr) {
+        rc = vfio_set_dirty_page_bitmap_chunk(vdev, start_addr, chunk_pg_nr);
+        if (rc) {
+            goto exit;
+        }
+        cnt_left -= chunk_pg_nr;
+        start_addr += start_addr;
+   }
+   rc = vfio_set_dirty_page_bitmap_chunk(vdev, start_addr, cnt_left);
+
+exit:
+   return rc;
+}
+
+static int vfio_set_device_state(VFIOPCIDevice *vdev,
+        uint32_t dev_state)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint32_t sz = sizeof(dev_state);

hw/vfio/migration.c line 298

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+
+    if (!vdev->migration) {
+        return -1;
+    }
+
+    if (pwrite(vbasedev->fd, &dev_state, sz,
+              region->fd_offset +
+              offsetof(struct vfio_device_state_ctl, device_state))
+            != sz) {
+        error_report("vfio: Failed to set device state %d", dev_state);

Can the kernel reject this if a state transition is not allowed (or are all transitions allowed?)

Quoted 39 lines
hw/vfio/migration.c new file +39 -0
+        return -1;
+    }
+    vdev->migration->device_state = dev_state;
+    return 0;
+}
+
+static int vfio_get_device_data_caps(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+
+    uint32_t caps;
+    uint32_t size = sizeof(caps);
+
+    if (pread(vbasedev->fd, &caps, size,
+                region->fd_offset +
+                offsetof(struct vfio_device_state_ctl, caps))
+            != size) {
+        error_report("%s Failed to read data caps of device states",
+                vbasedev->name);
+        return -1;
+    }
+    vdev->migration->data_caps = caps;
+    return 0;
+}
+
+
+static int vfio_check_devstate_version(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+
+    uint32_t version;
+    uint32_t size = sizeof(version);
+
+    if (pread(vbasedev->fd, &version, size,
+                region->fd_offset +

hw/vfio/migration.c line 347

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+                offsetof(struct vfio_device_state_ctl, version))
+            != size) {
+        error_report("%s Failed to read version of device state interfaces",
+                vbasedev->name);
+        return -1;
+    }
+
+    if (version != VFIO_DEVICE_STATE_INTERFACE_VERSION) {
+        error_report("%s migration version mismatch, right version is %d",
+                vbasedev->name, VFIO_DEVICE_STATE_INTERFACE_VERSION);

So, we require an exact match... or should we allow to extend the interface in an backwards-compatible way, in which case we'd require (QEMU interface version) <= (kernel interface version)?

Quoted 35 lines
hw/vfio/migration.c new file +35 -0
+        return -1;
+    }
+
+    return 0;
+}
+
+static void vfio_vm_change_state_handler(void *pv, int running, RunState state)
+{
+    VFIOPCIDevice *vdev = pv;
+    uint32_t dev_state = vdev->migration->device_state;
+
+    if (!running) {
+        dev_state |= VFIO_DEVICE_STATE_STOP;
+    } else {
+        dev_state &= ~VFIO_DEVICE_STATE_STOP;
+    }
+
+    vfio_set_device_state(vdev, dev_state);
+}
+
+static void vfio_save_live_pending(QEMUFile *f, void *opaque,
+                                   uint64_t max_size,
+                                   uint64_t *res_precopy_only,
+                                   uint64_t *res_compatible,
+                                   uint64_t *res_post_copy_only)
+{
+    VFIOPCIDevice *vdev = opaque;
+
+    if (!vfio_device_data_cap_device_memory(vdev)) {
+        return;
+    }
+
+    return;
+}
+

hw/vfio/migration.c line 392

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+static int vfio_save_iterate(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+
+    if (!vfio_device_data_cap_device_memory(vdev)) {
+        return 0;
+    }
+
+    return 0;
+}

These look a bit weird...

Quoted 32 lines
hw/vfio/migration.c new file +32 -0
+
+static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    uint32_t ctl, msi_lo, msi_hi, msi_data, bar_cfg, i;
+    bool msi_64bit;
+
+    /* retore pci bar configuration */
+    ctl = pci_default_read_config(pdev, PCI_COMMAND, 2);
+    vfio_pci_write_config(pdev, PCI_COMMAND,
+            ctl & (!(PCI_COMMAND_IO | PCI_COMMAND_MEMORY)), 2);
+    for (i = 0; i < PCI_ROM_SLOT; i++) {
+        bar_cfg = qemu_get_be32(f);
+        vfio_pci_write_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, bar_cfg, 4);
+    }
+    vfio_pci_write_config(pdev, PCI_COMMAND,
+            ctl | PCI_COMMAND_IO | PCI_COMMAND_MEMORY, 2);
+
+    /* restore msi configuration */
+    ctl = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, 2);
+    msi_64bit = !!(ctl & PCI_MSI_FLAGS_64BIT);
+
+    vfio_pci_write_config(&vdev->pdev,
+            pdev->msi_cap + PCI_MSI_FLAGS,
+            ctl & (!PCI_MSI_FLAGS_ENABLE), 2);
+
+    msi_lo = qemu_get_be32(f);
+    vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_LO, msi_lo, 4);
+
+    if (msi_64bit) {
+        msi_hi = qemu_get_be32(f);
+        vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_HI,

hw/vfio/migration.c line 434

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+                msi_hi, 4);
+    }
+    msi_data = qemu_get_be32(f);
+    vfio_pci_write_config(pdev,
+            pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
+            msi_data, 2);
+
+    vfio_pci_write_config(&vdev->pdev, pdev->msi_cap + PCI_MSI_FLAGS,
+            ctl | PCI_MSI_FLAGS_ENABLE, 2);
+

Ok, this function is indeed pci-specific and probably should be moved to the vfio-pci code (other types could hook themselves up in the same place, then).

Quoted 95 lines
hw/vfio/migration.c new file +95 -0
+}
+
+static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
+{
+    VFIOPCIDevice *vdev = opaque;
+    int flag;
+    uint64_t len;
+    int ret = 0;
+
+    if (version_id != VFIO_DEVICE_STATE_INTERFACE_VERSION) {
+        return -EINVAL;
+    }
+
+    do {
+        flag = qemu_get_byte(f);
+
+        switch (flag & ~VFIO_SAVE_FLAG_CONTINUE) {
+        case VFIO_SAVE_FLAG_SETUP:
+            break;
+        case VFIO_SAVE_FLAG_PCI:
+            vfio_pci_load_config(vdev, f);
+            break;
+        case VFIO_SAVE_FLAG_DEVCONFIG:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_config(vdev, f, len);
+            break;
+        default:
+            ret = -EINVAL;
+        }
+    } while (flag & VFIO_SAVE_FLAG_CONTINUE);
+
+    return ret;
+}
+
+static void vfio_pci_save_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    uint32_t msi_cfg, msi_lo, msi_hi, msi_data, bar_cfg, i;
+    bool msi_64bit;
+
+    for (i = 0; i < PCI_ROM_SLOT; i++) {
+        bar_cfg = pci_default_read_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, 4);
+        qemu_put_be32(f, bar_cfg);
+    }
+
+    msi_cfg = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, 2);
+    msi_64bit = !!(msi_cfg & PCI_MSI_FLAGS_64BIT);
+
+    msi_lo = pci_default_read_config(pdev,
+            pdev->msi_cap + PCI_MSI_ADDRESS_LO, 4);
+    qemu_put_be32(f, msi_lo);
+
+    if (msi_64bit) {
+        msi_hi = pci_default_read_config(pdev,
+                pdev->msi_cap + PCI_MSI_ADDRESS_HI,
+                4);
+        qemu_put_be32(f, msi_hi);
+    }
+
+    msi_data = pci_default_read_config(pdev,
+            pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
+            2);
+    qemu_put_be32(f, msi_data);
+
+}
+
+static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    int rc = 0;
+
+    qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
+    vfio_pci_save_config(vdev, f);
+
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVCONFIG);
+    rc += vfio_get_device_config_size(vdev);
+    rc += vfio_save_data_device_config(vdev, f);
+
+    return rc;
+}
+
+static int vfio_save_setup(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
+            VFIO_DEVICE_STATE_LOGGING);
+    return 0;
+}
+
+static int vfio_load_setup(QEMUFile *f, void *opaque)
+{
+    return 0;
+}

hw/vfio/migration.c line 539

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+
+static void vfio_save_cleanup(void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    uint32_t dev_state = vdev->migration->device_state;
+
+    dev_state &= ~VFIO_DEVICE_STATE_LOGGING;
+
+    vfio_set_device_state(vdev, dev_state);
+}

These look like they should be type-independent, again.

Quoted 99 lines
hw/vfio/migration.c new file +94 -0
+
+static SaveVMHandlers savevm_vfio_handlers = {
+    .save_setup = vfio_save_setup,
+    .save_live_pending = vfio_save_live_pending,
+    .save_live_iterate = vfio_save_iterate,
+    .save_live_complete_precopy = vfio_save_complete_precopy,
+    .save_cleanup = vfio_save_cleanup,
+    .load_setup = vfio_load_setup,
+    .load_state = vfio_load_state,
+};
+
+int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
+{
+    int ret;
+    Error *local_err = NULL;
+    vdev->migration = g_new0(VFIOMigration, 1);
+
+    if (vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL,
+              "device-state-ctl")) {
+        goto error;
+    }
+
+    if (vfio_check_devstate_version(vdev)) {
+        goto error;
+    }
+
+    if (vfio_get_device_data_caps(vdev)) {
+        goto error;
+    }
+
+    if (vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG,
+              "device-state-data-device-config")) {
+        goto error;
+    }
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        error_report("No suppport of data cap device memory Yet");
+        goto error;
+    }
+
+    if (vfio_device_data_cap_system_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP,
+              "device-state-data-dirtybitmap")) {
+        goto error;
+    }
+
+    vdev->migration->device_state = VFIO_DEVICE_STATE_RUNNING;
+
+    register_savevm_live(NULL, TYPE_VFIO_PCI, -1,
+            VFIO_DEVICE_STATE_INTERFACE_VERSION,
+            &savevm_vfio_handlers,
+            vdev);
+
+    vdev->migration->vm_state =
+        qemu_add_vm_change_state_handler(vfio_vm_change_state_handler, vdev);
+
+    return 0;
+error:
+    error_setg(&vdev->migration_blocker,
+            "VFIO device doesn't support migration");
+    ret = migrate_add_blocker(vdev->migration_blocker, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        error_free(vdev->migration_blocker);
+    }
+
+    g_free(vdev->migration);
+    vdev->migration = NULL;
+
+    return ret;
+}
+
+void vfio_migration_finalize(VFIOPCIDevice *vdev)
+{
+    if (vdev->migration) {
+        int i;
+        qemu_del_vm_change_state_handler(vdev->migration->vm_state);
+        unregister_savevm(NULL, TYPE_VFIO_PCI, vdev);
+        for (i = 0; i < VFIO_DEVSTATE_REGION_NUM; i++) {
+            vfio_region_finalize(&vdev->migration->region[i]);
+        }
+        g_free(vdev->migration);
+        vdev->migration = NULL;
+    } else if (vdev->migration_blocker) {
+        migrate_del_blocker(vdev->migration_blocker);
+        error_free(vdev->migration_blocker);
+    }
+}
hw/vfio/pci.c +0 -0
index c0cb1ec..b8e006b 100644

hw/vfio/pci.c line 40

Quoted 4 lines
hw/vfio/pci.c +0 -1
@@ -37,7 +37,6 @@
 
 #define MSIX_CAP_LENGTH 12
 
-#define TYPE_VFIO_PCI "vfio-pci"

Why do you need to move this? That looks like a sign that the layering needs work.

Quoted 38 lines
hw/vfio/pci.c +0 -0
 #define PCI_VFIO(obj)    OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)
 
 static void vfio_disable_interrupts(VFIOPCIDevice *vdev);
hw/vfio/pci.h +16 -0
index b1ae4c0..4b7b1bb 100644
@@ -19,6 +19,7 @@
 #include "qemu/event_notifier.h"
 #include "qemu/queue.h"
 #include "qemu/timer.h"
+#include "sysemu/sysemu.h"
 
 #define PCI_ANY_ID (~0)
 
@@ -56,6 +57,21 @@ typedef struct VFIOBAR {
     QLIST_HEAD(, VFIOQuirk) quirks;
 } VFIOBAR;
 
+enum {
+    VFIO_DEVSTATE_REGION_CTL = 0,
+    VFIO_DEVSTATE_REGION_DATA_CONFIG,
+    VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY,
+    VFIO_DEVSTATE_REGION_DATA_BITMAP,
+    VFIO_DEVSTATE_REGION_NUM,
+};
+typedef struct VFIOMigration {
+    VFIORegion region[VFIO_DEVSTATE_REGION_NUM];
+    uint32_t data_caps;
+    uint32_t device_state;
+    uint64_t devconfig_size;
+    VMChangeStateEntry *vm_state;
+} VFIOMigration;
+
 typedef struct VFIOVGARegion {
     MemoryRegion mem;
     off_t offset;

hw/vfio/pci.h typedef struct VFIOPCIDevice { · line 151

Quoted 4 lines
hw/vfio/pci.h +1 -0
@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {
     VFIOBAR bars[PCI_NUM_REGIONS - 1]; /* No ROM */
     VFIOVGA *vga; /* 0xa0000, 0x3b0, 0x3c0 */
     void *igd_opregion;
+    VFIOMigration *migration;

As said, it would probably be better to hang this off VFIODevice.

Quoted 5 lines
hw/vfio/pci.h +1 -0
+    Error *migration_blocker;
     PCIHostDeviceAddress host;
     EventNotifier err_notifier;
     EventNotifier req_notifier;

hw/vfio/pci.h int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev, · line 224

Quoted 10 lines
hw/vfio/pci.h +6 -1
@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,
 void vfio_display_reset(VFIOPCIDevice *vdev);
 int vfio_display_probe(VFIOPCIDevice *vdev, Error **errp);
 void vfio_display_finalize(VFIOPCIDevice *vdev);
-
+bool vfio_device_data_cap_system_memory(VFIOPCIDevice *vdev);
+bool vfio_device_data_cap_device_memory(VFIOPCIDevice *vdev);
+int vfio_set_dirty_page_bitmap(VFIOPCIDevice *vdev,
+         uint64_t start_addr, uint64_t page_nr);
+int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp);
+void vfio_migration_finalize(VFIOPCIDevice *vdev);

And the interfaces should be in vfio-common.

Quoted 13 lines
hw/vfio/pci.h +0 -0
 #endif /* HW_VFIO_VFIO_PCI_H */
include/hw/vfio/vfio-common.h +1 -0
index 1b434d0..ed43613 100644
@@ -32,6 +32,7 @@
 #endif
 
 #define VFIO_MSG_PREFIX "vfio %s: "
+#define TYPE_VFIO_PCI "vfio-pci"
 
 enum {
     VFIO_DEVICE_TYPE_PCI = 0,
author
#11 lore raw

Re: [PATCH 2/5] vfio/migration: support device of device config capability

Quoted 32 lines · Cornelia Huck
Quoted 29 lines · On Tue, 19 Feb 2019 16:52:27 +0800 Yan Zhao <[email protected]> wrote:

Device config is the default data that every device should have. so device config capability is by default on, no need to set.

- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.

Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state to VFIO_DEVICE_STATE_LOGGING

---
 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/migration.c           | 633 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |   1 -
 hw/vfio/pci.h                 |  25 +-
 include/hw/vfio/vfio-common.h |   1 +
 5 files changed, 659 insertions(+), 3 deletions(-)
 create mode 100644 hw/vfio/migration.c
Quoted 13 lines

hw/vfio/Makefile.objs line 3

Quoted 9 lines
hw/vfio/Makefile.objs +1 -1
index 8b3f664..f32ff19 100644
@@ -1,6 +1,6 @@
 ifeq ($(CONFIG_LINUX), y)
 obj-$(CONFIG_SOFTMMU) += common.o
-obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o
+obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.o

I think you want to split the migration code: The type-independent code, and the pci-specific code.

ok. actually, now only saving/loading of pci generic config data is pci-specific. the data getting/setting through device state interfaces are type-independent.

Quoted 60 lines
Quoted 17 lines
hw/vfio/Makefile.objs +0 -0
 obj-$(CONFIG_VFIO_CCW) += ccw.o
 obj-$(CONFIG_SOFTMMU) += platform.o
 obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o
hw/vfio/migration.c new file +8 -0
new file mode 100644
index 0000000..16d6395
@@ -0,0 +1,633 @@
+#include "qemu/osdep.h"
+
+#include "hw/vfio/vfio-common.h"
+#include "migration/blocker.h"
+#include "migration/register.h"
+#include "qapi/error.h"
+#include "pci.h"
+#include "sysemu/kvm.h"

hw/vfio/migration.c line 18

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+#include "exec/ram_addr.h"
+
+#define VFIO_SAVE_FLAG_SETUP 0
+#define VFIO_SAVE_FLAG_PCI 1
+#define VFIO_SAVE_FLAG_DEVCONFIG 2
+#define VFIO_SAVE_FLAG_DEVMEMORY 4
+#define VFIO_SAVE_FLAG_CONTINUE 8
+
+static int vfio_device_state_region_setup(VFIOPCIDevice *vdev,
+        VFIORegion *region, uint32_t subtype, const char *name)

This function looks like it should be more generic and e.g. take a VFIODevice instead of a VFIOPCIDevice as argument.

Quoted 29 lines
hw/vfio/migration.c new file +29 -0
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    struct vfio_region_info *info;
+    int ret;
+
+    ret = vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_DEVICE_STATE,
+            subtype, &info);
+    if (ret) {
+        error_report("Failed to get info of region %s", name);
+        return ret;
+    }
+
+    if (vfio_region_setup(OBJECT(vdev), vbasedev,
+            region, info->index, name)) {
+        error_report("Failed to setup migrtion region %s", name);
+        return ret;
+    }
+
+    if (vfio_region_mmap(region)) {
+        error_report("Failed to mmap migrtion region %s", name);
+    }
+
+    return 0;
+}
+
+bool vfio_device_data_cap_system_memory(VFIOPCIDevice *vdev)
+{
+   return !!(vdev->migration->data_caps & VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY);
+}
Quoted 9 lines

hw/vfio/migration.c line 52

Quoted 5 lines
hw/vfio/migration.c new file +5 -0
+
+bool vfio_device_data_cap_device_memory(VFIOPCIDevice *vdev)
+{
+   return !!(vdev->migration->data_caps & VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY);
+}

These two as well. The migration structure should probably hang off the VFIODevice instead.

ok.

Quoted 6 lines
Quoted 6 lines
hw/vfio/migration.c new file +6 -0
+
+static bool vfio_device_state_region_mmaped(VFIORegion *region)
+{
+    bool mmaped = true;
+    if (region->nr_mmaps != 1 || region->mmaps[0].offset ||
+            (region->size != region->mmaps[0].size) ||
Quoted 8 lines

hw/vfio/migration.c line 64

Quoted 6 lines
hw/vfio/migration.c new file +6 -0
+            (region->mmaps[0].mmap == NULL)) {
+        mmaped = false;
+    }
+
+    return mmaped;
+}

s/mmaped/mmapped/ ?

yes :)

Quoted 12 lines

hw/vfio/migration.c line 72

Quoted 9 lines
hw/vfio/migration.c new file +8 -0
+
+static int vfio_get_device_config_size(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];

This looks like it should not depend on pci, either.

right.

Quoted 56 lines
Quoted 17 lines
hw/vfio/migration.c new file +17 -0
+    uint64_t len;
+    int sz;
+
+    sz = sizeof(len);
+    if (pread(vbasedev->fd, &len, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.size))
+            != sz) {
+        error_report("vfio: Failed to get length of device config");
+        return -1;
+    }
+    if (len > region_config->size) {
+        error_report("vfio: Error device config length");
+        return -1;
+    }
+    vdev->migration->devconfig_size = len;
+

hw/vfio/migration.c line 99

Quoted 10 lines
hw/vfio/migration.c new file +10 -0
+    return 0;
+}
+
+static int vfio_set_device_config_size(VFIOPCIDevice *vdev, uint64_t size)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];

Ditto. Also for the functions below.

Quoted 26 lines
hw/vfio/migration.c new file +26 -0
+    int sz;
+
+    if (size > region_config->size) {
+        return -1;
+    }
+
+    sz = sizeof(size);
+    if (pwrite(vbasedev->fd, &size, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.size))
+            != sz) {
+        error_report("vfio: Failed to set length of device config");
+        return -1;
+    }
+    vdev->migration->devconfig_size = size;
+    return 0;
+}
+
+static int vfio_save_data_device_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    void *dest;
Quoted 9 lines

hw/vfio/migration.c line 131

Quoted 6 lines
hw/vfio/migration.c new file +6 -0
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+    uint64_t len = vdev->migration->devconfig_size;
+
+    qemu_put_be64(f, len);

Why big endian? (Generally, do we need any endianness considerations?)

I think big endian is the endian for qemu to save file. as long as qemu_put and qemu_get use the same endian, it will be no problem.
do you think so?

Quoted 10 lines

hw/vfio/migration.c line 140

Quoted 9 lines
hw/vfio/migration.c new file +9 -0
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.action))
+            != sz) {
+        error_report("vfio: action failure for device config get buffer");
+        return -1;
+    }
Quoted 3 lines

Might make sense to wrap this into a set_action() helper that takes a SET_BUFFER/GET_BUFFER argument.

right. it makes sense :)

Quoted 153 lines
Quoted 153 lines
hw/vfio/migration.c new file +153 -0
+
+    if (!vfio_device_state_region_mmaped(region_config)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        if (pread(vbasedev->fd, buf, len, region_config->fd_offset) != len) {
+            error_report("vfio: Failed read device config buffer");
+            return -1;
+        }
+        qemu_put_buffer(f, buf, len);
+        g_free(buf);
+    } else {
+        dest = region_config->mmaps[0].mmap;
+        qemu_put_buffer(f, dest, len);
+    }
+    return 0;
+}
+
+static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
+                            QEMUFile *f, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_SET_BUFFER;
+
+    vfio_set_device_config_size(vdev, len);
+
+    if (!vfio_device_state_region_mmaped(region_config)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        qemu_get_buffer(f, buf, len);
+        if (pwrite(vbasedev->fd, buf, len,
+                    region_config->fd_offset) != len) {
+            error_report("vfio: Failed to write devie config buffer");
+            return -1;
+        }
+        g_free(buf);
+    } else {
+        dest = region_config->mmaps[0].mmap;
+        qemu_get_buffer(f, dest, len);
+    }
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_config.action))
+            != sz) {
+        error_report("vfio: action failure for device config set buffer");
+        return -1;
+    }
+
+    return 0;
+}
+
+static int vfio_set_dirty_page_bitmap_chunk(VFIOPCIDevice *vdev,
+        uint64_t start_addr, uint64_t page_nr)
+{
+
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_bitmap =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP];
+    unsigned long bitmap_size =
+                    BITS_TO_LONGS(page_nr) * sizeof(unsigned long);
+    uint32_t sz;
+
+    struct {
+        __u64 start_addr;
+        __u64 page_nr;
+    } system_memory;
+    system_memory.start_addr = start_addr;
+    system_memory.page_nr = page_nr;
+    sz = sizeof(system_memory);
+    if (pwrite(vbasedev->fd, &system_memory, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, system_memory))
+            != sz) {
+        error_report("vfio: Failed to set system memory range for dirty pages");
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_bitmap)) {
+        void *bitmap = g_malloc0(bitmap_size);
+
+        if (pread(vbasedev->fd, bitmap, bitmap_size,
+                    region_bitmap->fd_offset) != bitmap_size) {
+            error_report("vfio: Failed to read dirty bitmap data");
+            return -1;
+        }
+
+        cpu_physical_memory_set_dirty_lebitmap(bitmap, start_addr, page_nr);
+
+        g_free(bitmap);
+    } else {
+        cpu_physical_memory_set_dirty_lebitmap(
+                    region_bitmap->mmaps[0].mmap,
+                    start_addr, page_nr);
+    }
+   return 0;
+}
+
+int vfio_set_dirty_page_bitmap(VFIOPCIDevice *vdev,
+        uint64_t start_addr, uint64_t page_nr)
+{
+    VFIORegion *region_bitmap =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP];
+    unsigned long chunk_size = region_bitmap->size;
+    uint64_t chunk_pg_nr = (chunk_size / sizeof(unsigned long)) *
+                                BITS_PER_LONG;
+
+    uint64_t cnt_left;
+    int rc = 0;
+
+    cnt_left = page_nr;
+
+    while (cnt_left >= chunk_pg_nr) {
+        rc = vfio_set_dirty_page_bitmap_chunk(vdev, start_addr, chunk_pg_nr);
+        if (rc) {
+            goto exit;
+        }
+        cnt_left -= chunk_pg_nr;
+        start_addr += start_addr;
+   }
+   rc = vfio_set_dirty_page_bitmap_chunk(vdev, start_addr, cnt_left);
+
+exit:
+   return rc;
+}
+
+static int vfio_set_device_state(VFIOPCIDevice *vdev,
+        uint32_t dev_state)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint32_t sz = sizeof(dev_state);
+
+    if (!vdev->migration) {
+        return -1;
+    }
+
Quoted 9 lines

hw/vfio/migration.c line 298

Quoted 5 lines
hw/vfio/migration.c new file +5 -0
+    if (pwrite(vbasedev->fd, &dev_state, sz,
+              region->fd_offset +
+              offsetof(struct vfio_device_state_ctl, device_state))
+            != sz) {
+        error_report("vfio: Failed to set device state %d", dev_state);

Can the kernel reject this if a state transition is not allowed (or are all transitions allowed?)

yes, kernel can reject some state transition if it's not allowed. But currently all transitions are allowed.
Maybe a check of self-to-self transition is needed in kernel.

Quoted 45 lines
Quoted 45 lines
hw/vfio/migration.c new file +45 -0
+        return -1;
+    }
+    vdev->migration->device_state = dev_state;
+    return 0;
+}
+
+static int vfio_get_device_data_caps(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+
+    uint32_t caps;
+    uint32_t size = sizeof(caps);
+
+    if (pread(vbasedev->fd, &caps, size,
+                region->fd_offset +
+                offsetof(struct vfio_device_state_ctl, caps))
+            != size) {
+        error_report("%s Failed to read data caps of device states",
+                vbasedev->name);
+        return -1;
+    }
+    vdev->migration->data_caps = caps;
+    return 0;
+}
+
+
+static int vfio_check_devstate_version(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+
+    uint32_t version;
+    uint32_t size = sizeof(version);
+
+    if (pread(vbasedev->fd, &version, size,
+                region->fd_offset +
+                offsetof(struct vfio_device_state_ctl, version))
+            != size) {
+        error_report("%s Failed to read version of device state interfaces",
+                vbasedev->name);
+        return -1;
+    }
Quoted 9 lines

hw/vfio/migration.c line 347

Quoted 4 lines
hw/vfio/migration.c new file +4 -0
+
+    if (version != VFIO_DEVICE_STATE_INTERFACE_VERSION) {
+        error_report("%s migration version mismatch, right version is %d",
+                vbasedev->name, VFIO_DEVICE_STATE_INTERFACE_VERSION);

So, we require an exact match... or should we allow to extend the interface in an backwards-compatible way, in which case we'd require (QEMU interface version) <= (kernel interface version)?

currently yes. we can discuss on that.

Quoted 39 lines
Quoted 39 lines
hw/vfio/migration.c new file +39 -0
+        return -1;
+    }
+
+    return 0;
+}
+
+static void vfio_vm_change_state_handler(void *pv, int running, RunState state)
+{
+    VFIOPCIDevice *vdev = pv;
+    uint32_t dev_state = vdev->migration->device_state;
+
+    if (!running) {
+        dev_state |= VFIO_DEVICE_STATE_STOP;
+    } else {
+        dev_state &= ~VFIO_DEVICE_STATE_STOP;
+    }
+
+    vfio_set_device_state(vdev, dev_state);
+}
+
+static void vfio_save_live_pending(QEMUFile *f, void *opaque,
+                                   uint64_t max_size,
+                                   uint64_t *res_precopy_only,
+                                   uint64_t *res_compatible,
+                                   uint64_t *res_post_copy_only)
+{
+    VFIOPCIDevice *vdev = opaque;
+
+    if (!vfio_device_data_cap_device_memory(vdev)) {
+        return;
+    }
+
+    return;
+}
+
+static int vfio_save_iterate(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+
Quoted 9 lines

hw/vfio/migration.c line 392

Quoted 6 lines
hw/vfio/migration.c new file +6 -0
+    if (!vfio_device_data_cap_device_memory(vdev)) {
+        return 0;
+    }
+
+    return 0;
+}

These look a bit weird...

patch 5 (adding device memory cap support ) will make it look better :)

Quoted 38 lines
Quoted 38 lines
hw/vfio/migration.c new file +38 -0
+
+static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    uint32_t ctl, msi_lo, msi_hi, msi_data, bar_cfg, i;
+    bool msi_64bit;
+
+    /* retore pci bar configuration */
+    ctl = pci_default_read_config(pdev, PCI_COMMAND, 2);
+    vfio_pci_write_config(pdev, PCI_COMMAND,
+            ctl & (!(PCI_COMMAND_IO | PCI_COMMAND_MEMORY)), 2);
+    for (i = 0; i < PCI_ROM_SLOT; i++) {
+        bar_cfg = qemu_get_be32(f);
+        vfio_pci_write_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, bar_cfg, 4);
+    }
+    vfio_pci_write_config(pdev, PCI_COMMAND,
+            ctl | PCI_COMMAND_IO | PCI_COMMAND_MEMORY, 2);
+
+    /* restore msi configuration */
+    ctl = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, 2);
+    msi_64bit = !!(ctl & PCI_MSI_FLAGS_64BIT);
+
+    vfio_pci_write_config(&vdev->pdev,
+            pdev->msi_cap + PCI_MSI_FLAGS,
+            ctl & (!PCI_MSI_FLAGS_ENABLE), 2);
+
+    msi_lo = qemu_get_be32(f);
+    vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_LO, msi_lo, 4);
+
+    if (msi_64bit) {
+        msi_hi = qemu_get_be32(f);
+        vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_HI,
+                msi_hi, 4);
+    }
+    msi_data = qemu_get_be32(f);
+    vfio_pci_write_config(pdev,
+            pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
+            msi_data, 2);
Quoted 9 lines

hw/vfio/migration.c line 434

Quoted 4 lines
hw/vfio/migration.c new file +4 -0
+
+    vfio_pci_write_config(&vdev->pdev, pdev->msi_cap + PCI_MSI_FLAGS,
+            ctl | PCI_MSI_FLAGS_ENABLE, 2);
+

Ok, this function is indeed pci-specific and probably should be moved to the vfio-pci code (other types could hook themselves up in the same place, then).

yes, this is the only pci-specific code.
maybe using VFIO_DEVICE_TYPE_PCI as a sign to decide whether to save/load pci config data?
or as Dave said, put saving/loading of pci config data into VMStateDescription interface?

Quoted 99 lines
Quoted 99 lines
hw/vfio/migration.c new file +99 -0
+}
+
+static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
+{
+    VFIOPCIDevice *vdev = opaque;
+    int flag;
+    uint64_t len;
+    int ret = 0;
+
+    if (version_id != VFIO_DEVICE_STATE_INTERFACE_VERSION) {
+        return -EINVAL;
+    }
+
+    do {
+        flag = qemu_get_byte(f);
+
+        switch (flag & ~VFIO_SAVE_FLAG_CONTINUE) {
+        case VFIO_SAVE_FLAG_SETUP:
+            break;
+        case VFIO_SAVE_FLAG_PCI:
+            vfio_pci_load_config(vdev, f);
+            break;
+        case VFIO_SAVE_FLAG_DEVCONFIG:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_config(vdev, f, len);
+            break;
+        default:
+            ret = -EINVAL;
+        }
+    } while (flag & VFIO_SAVE_FLAG_CONTINUE);
+
+    return ret;
+}
+
+static void vfio_pci_save_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    uint32_t msi_cfg, msi_lo, msi_hi, msi_data, bar_cfg, i;
+    bool msi_64bit;
+
+    for (i = 0; i < PCI_ROM_SLOT; i++) {
+        bar_cfg = pci_default_read_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, 4);
+        qemu_put_be32(f, bar_cfg);
+    }
+
+    msi_cfg = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, 2);
+    msi_64bit = !!(msi_cfg & PCI_MSI_FLAGS_64BIT);
+
+    msi_lo = pci_default_read_config(pdev,
+            pdev->msi_cap + PCI_MSI_ADDRESS_LO, 4);
+    qemu_put_be32(f, msi_lo);
+
+    if (msi_64bit) {
+        msi_hi = pci_default_read_config(pdev,
+                pdev->msi_cap + PCI_MSI_ADDRESS_HI,
+                4);
+        qemu_put_be32(f, msi_hi);
+    }
+
+    msi_data = pci_default_read_config(pdev,
+            pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
+            2);
+    qemu_put_be32(f, msi_data);
+
+}
+
+static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    int rc = 0;
+
+    qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
+    vfio_pci_save_config(vdev, f);
+
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVCONFIG);
+    rc += vfio_get_device_config_size(vdev);
+    rc += vfio_save_data_device_config(vdev, f);
+
+    return rc;
+}
+
+static int vfio_save_setup(QEMUFile *f, void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
+            VFIO_DEVICE_STATE_LOGGING);
+    return 0;
+}
+
+static int vfio_load_setup(QEMUFile *f, void *opaque)
+{
+    return 0;
+}
+
+static void vfio_save_cleanup(void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
Quoted 9 lines

hw/vfio/migration.c line 539

Quoted 6 lines
hw/vfio/migration.c new file +6 -0
+    uint32_t dev_state = vdev->migration->device_state;
+
+    dev_state &= ~VFIO_DEVICE_STATE_LOGGING;
+
+    vfio_set_device_state(vdev, dev_state);
+}

These look like they should be type-independent, again.

right:)

Quoted 100 lines
Quoted 100 lines
hw/vfio/migration.c new file +94 -0
+
+static SaveVMHandlers savevm_vfio_handlers = {
+    .save_setup = vfio_save_setup,
+    .save_live_pending = vfio_save_live_pending,
+    .save_live_iterate = vfio_save_iterate,
+    .save_live_complete_precopy = vfio_save_complete_precopy,
+    .save_cleanup = vfio_save_cleanup,
+    .load_setup = vfio_load_setup,
+    .load_state = vfio_load_state,
+};
+
+int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
+{
+    int ret;
+    Error *local_err = NULL;
+    vdev->migration = g_new0(VFIOMigration, 1);
+
+    if (vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_CTL,
+              "device-state-ctl")) {
+        goto error;
+    }
+
+    if (vfio_check_devstate_version(vdev)) {
+        goto error;
+    }
+
+    if (vfio_get_device_data_caps(vdev)) {
+        goto error;
+    }
+
+    if (vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_CONFIG,
+              "device-state-data-device-config")) {
+        goto error;
+    }
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        error_report("No suppport of data cap device memory Yet");
+        goto error;
+    }
+
+    if (vfio_device_data_cap_system_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_BITMAP],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP,
+              "device-state-data-dirtybitmap")) {
+        goto error;
+    }
+
+    vdev->migration->device_state = VFIO_DEVICE_STATE_RUNNING;
+
+    register_savevm_live(NULL, TYPE_VFIO_PCI, -1,
+            VFIO_DEVICE_STATE_INTERFACE_VERSION,
+            &savevm_vfio_handlers,
+            vdev);
+
+    vdev->migration->vm_state =
+        qemu_add_vm_change_state_handler(vfio_vm_change_state_handler, vdev);
+
+    return 0;
+error:
+    error_setg(&vdev->migration_blocker,
+            "VFIO device doesn't support migration");
+    ret = migrate_add_blocker(vdev->migration_blocker, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        error_free(vdev->migration_blocker);
+    }
+
+    g_free(vdev->migration);
+    vdev->migration = NULL;
+
+    return ret;
+}
+
+void vfio_migration_finalize(VFIOPCIDevice *vdev)
+{
+    if (vdev->migration) {
+        int i;
+        qemu_del_vm_change_state_handler(vdev->migration->vm_state);
+        unregister_savevm(NULL, TYPE_VFIO_PCI, vdev);
+        for (i = 0; i < VFIO_DEVSTATE_REGION_NUM; i++) {
+            vfio_region_finalize(&vdev->migration->region[i]);
+        }
+        g_free(vdev->migration);
+        vdev->migration = NULL;
+    } else if (vdev->migration_blocker) {
+        migrate_del_blocker(vdev->migration_blocker);
+        error_free(vdev->migration_blocker);
+    }
+}
hw/vfio/pci.c +0 -0
index c0cb1ec..b8e006b 100644
@@ -37,7 +37,6 @@
 
Quoted 7 lines

hw/vfio/pci.c line 40

Quoted 3 lines
hw/vfio/pci.c +0 -1
 #define MSIX_CAP_LENGTH 12
 
-#define TYPE_VFIO_PCI "vfio-pci"

Why do you need to move this? That looks like a sign that the layering needs work.

I moved it for passing it as an idstr for register_savevm_live. Seems it's not necessary if we make it pci independent. Thanks:)

Quoted 36 lines
Quoted 36 lines
hw/vfio/pci.c +0 -0
 #define PCI_VFIO(obj)    OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI)
 
 static void vfio_disable_interrupts(VFIOPCIDevice *vdev);
hw/vfio/pci.h +16 -0
index b1ae4c0..4b7b1bb 100644
@@ -19,6 +19,7 @@
 #include "qemu/event_notifier.h"
 #include "qemu/queue.h"
 #include "qemu/timer.h"
+#include "sysemu/sysemu.h"
 
 #define PCI_ANY_ID (~0)
 
@@ -56,6 +57,21 @@ typedef struct VFIOBAR {
     QLIST_HEAD(, VFIOQuirk) quirks;
 } VFIOBAR;
 
+enum {
+    VFIO_DEVSTATE_REGION_CTL = 0,
+    VFIO_DEVSTATE_REGION_DATA_CONFIG,
+    VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY,
+    VFIO_DEVSTATE_REGION_DATA_BITMAP,
+    VFIO_DEVSTATE_REGION_NUM,
+};
+typedef struct VFIOMigration {
+    VFIORegion region[VFIO_DEVSTATE_REGION_NUM];
+    uint32_t data_caps;
+    uint32_t device_state;
+    uint64_t devconfig_size;
+    VMChangeStateEntry *vm_state;
+} VFIOMigration;
+
 typedef struct VFIOVGARegion {
     MemoryRegion mem;
Quoted 9 lines

hw/vfio/pci.h typedef struct VFIOPCIDevice { · line 151

Quoted 6 lines
hw/vfio/pci.h +1 -0
     off_t offset;
@@ -132,6 +148,8 @@ typedef struct VFIOPCIDevice {
     VFIOBAR bars[PCI_NUM_REGIONS - 1]; /* No ROM */
     VFIOVGA *vga; /* 0xa0000, 0x3b0, 0x3c0 */
     void *igd_opregion;
+    VFIOMigration *migration;

As said, it would probably be better to hang this off VFIODevice.

ok.

Quoted 9 lines
Quoted 9 lines
hw/vfio/pci.h +1 -1
+    Error *migration_blocker;
     PCIHostDeviceAddress host;
     EventNotifier err_notifier;
     EventNotifier req_notifier;
@@ -198,5 +216,10 @@ int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,
 void vfio_display_reset(VFIOPCIDevice *vdev);
 int vfio_display_probe(VFIOPCIDevice *vdev, Error **errp);
 void vfio_display_finalize(VFIOPCIDevice *vdev);
-
Quoted 9 lines

hw/vfio/pci.h int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev, · line 224

Quoted 6 lines
hw/vfio/pci.h +6 -0
+bool vfio_device_data_cap_system_memory(VFIOPCIDevice *vdev);
+bool vfio_device_data_cap_device_memory(VFIOPCIDevice *vdev);
+int vfio_set_dirty_page_bitmap(VFIOPCIDevice *vdev,
+         uint64_t start_addr, uint64_t page_nr);
+int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp);
+void vfio_migration_finalize(VFIOPCIDevice *vdev);

And the interfaces should be in vfio-common.

ok.

Quoted 14 lines
Quoted 13 lines
hw/vfio/pci.h +0 -0
 #endif /* HW_VFIO_VFIO_PCI_H */
include/hw/vfio/vfio-common.h +1 -0
index 1b434d0..ed43613 100644
@@ -32,6 +32,7 @@
 #endif
 
 #define VFIO_MSG_PREFIX "vfio %s: "
+#define TYPE_VFIO_PCI "vfio-pci"
 
 enum {
     VFIO_DEVICE_TYPE_PCI = 0,
#12 lore raw

Re: [PATCH 2/5] vfio/migration: support device of device config capability

Quoted 46 lines · On Wed, 20 Feb 2019 17:54:00 -0500 Zhao Yan <[email protected]> wrote:
Quoted 32 lines · Cornelia Huck
Quoted 29 lines · On Tue, 19 Feb 2019 16:52:27 +0800 Yan Zhao <[email protected]> wrote:

Device config is the default data that every device should have. so device config capability is by default on, no need to set.

- Currently two type of resources are saved/loaded for device of device config capability:
General PCI config data, and Device config data.
They are copies as a whole when precopy is stopped.

Migration setup flow:
- Setup device state regions, check its device state version and capabilities. Mmap Device Config Region and Dirty Bitmap Region, if available.
- If device state regions are failed to get setup, a migration blocker is registered instead.
- Added SaveVMHandlers to register device state save/load handlers.
- Register VM state change handler to set device's running/stop states.
- On migration startup on source machine, set device's state to VFIO_DEVICE_STATE_LOGGING

---
 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/migration.c           | 633 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |   1 -
 hw/vfio/pci.h                 |  25 +-
 include/hw/vfio/vfio-common.h |   1 +
 5 files changed, 659 insertions(+), 3 deletions(-)
 create mode 100644 hw/vfio/migration.c
Quoted 13 lines

hw/vfio/Makefile.objs line 3

Quoted 9 lines
hw/vfio/Makefile.objs +1 -1
index 8b3f664..f32ff19 100644
@@ -1,6 +1,6 @@
 ifeq ($(CONFIG_LINUX), y)
 obj-$(CONFIG_SOFTMMU) += common.o
-obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o
+obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o migration.o  

I think you want to split the migration code: The type-independent code, and the pci-specific code.

Quoted 3 lines

ok. actually, now only saving/loading of pci generic config data is pci-specific. the data getting/setting through device state interfaces are type-independent.

Yes. If it has capability chains, it can probably be made to work.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
hw/vfio/Makefile.objs +0 -0
 obj-$(CONFIG_VFIO_CCW) += ccw.o
 obj-$(CONFIG_SOFTMMU) += platform.o
 obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o

(...)

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
hw/vfio/Makefile.objs +8 -0
+static int vfio_save_data_device_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_config =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_CONFIG];
+    void *dest;
Quoted 9 lines

hw/vfio/Makefile.objs

Quoted 6 lines
hw/vfio/Makefile.objs +6 -0
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+    uint64_t len = vdev->migration->devconfig_size;
+
+    qemu_put_be64(f, len);  

Why big endian? (Generally, do we need any endianness considerations?)

Quoted 4 lines

I think big endian is the endian for qemu to save file. as long as qemu_put and qemu_get use the same endian, it will be no problem.
do you think so?

Yes, as long we are explicit about the endianness. I'm not sure whether e.g. power even has the ability to mix endianness.

(...)

Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
hw/vfio/Makefile.objs +12 -0
+static int vfio_set_device_state(VFIOPCIDevice *vdev,
+        uint32_t dev_state)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint32_t sz = sizeof(dev_state);
+
+    if (!vdev->migration) {
+        return -1;
+    }
+
Quoted 9 lines

hw/vfio/Makefile.objs

Quoted 5 lines
hw/vfio/Makefile.objs +5 -0
+    if (pwrite(vbasedev->fd, &dev_state, sz,
+              region->fd_offset +
+              offsetof(struct vfio_device_state_ctl, device_state))
+            != sz) {
+        error_report("vfio: Failed to set device state %d", dev_state);  

Can the kernel reject this if a state transition is not allowed (or are all transitions allowed?)

Quoted 3 lines

yes, kernel can reject some state transition if it's not allowed. But currently all transitions are allowed.
Maybe a check of self-to-self transition is needed in kernel.

Self-to-self looks benign enough to simply return success early.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
hw/vfio/Makefile.objs +5 -0
+        return -1;
+    }
+    vdev->migration->device_state = dev_state;
+    return 0;
+}

(...)

Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
hw/vfio/Makefile.objs +17 -0
+static int vfio_check_devstate_version(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+
+    uint32_t version;
+    uint32_t size = sizeof(version);
+
+    if (pread(vbasedev->fd, &version, size,
+                region->fd_offset +
+                offsetof(struct vfio_device_state_ctl, version))
+            != size) {
+        error_report("%s Failed to read version of device state interfaces",
+                vbasedev->name);
+        return -1;
+    }
Quoted 9 lines

hw/vfio/Makefile.objs

Quoted 4 lines
hw/vfio/Makefile.objs +4 -0
+
+    if (version != VFIO_DEVICE_STATE_INTERFACE_VERSION) {
+        error_report("%s migration version mismatch, right version is %d",
+                vbasedev->name, VFIO_DEVICE_STATE_INTERFACE_VERSION);  

So, we require an exact match... or should we allow to extend the interface in an backwards-compatible way, in which case we'd require (QEMU interface version) <= (kernel interface version)?

Quoted 1 line

currently yes. we can discuss on that.

If we want to allow that, we need to have a strictly monotonous progression of versions here (which means dragging along old compatibility code for basically forever). Maintaining a table about which version is compatible with which other version would get insane pretty quickly.

Can we somehow accommodate more optional regions via capabilities? Maybe via optional vmstates?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
hw/vfio/Makefile.objs +5 -0
+        return -1;
+    }
+
+    return 0;
+}

(...)

Quoted 46 lines
Quoted 37 lines
Quoted 37 lines
hw/vfio/Makefile.objs +37 -0
+static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    uint32_t ctl, msi_lo, msi_hi, msi_data, bar_cfg, i;
+    bool msi_64bit;
+
+    /* retore pci bar configuration */
+    ctl = pci_default_read_config(pdev, PCI_COMMAND, 2);
+    vfio_pci_write_config(pdev, PCI_COMMAND,
+            ctl & (!(PCI_COMMAND_IO | PCI_COMMAND_MEMORY)), 2);
+    for (i = 0; i < PCI_ROM_SLOT; i++) {
+        bar_cfg = qemu_get_be32(f);
+        vfio_pci_write_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, bar_cfg, 4);
+    }
+    vfio_pci_write_config(pdev, PCI_COMMAND,
+            ctl | PCI_COMMAND_IO | PCI_COMMAND_MEMORY, 2);
+
+    /* restore msi configuration */
+    ctl = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, 2);
+    msi_64bit = !!(ctl & PCI_MSI_FLAGS_64BIT);
+
+    vfio_pci_write_config(&vdev->pdev,
+            pdev->msi_cap + PCI_MSI_FLAGS,
+            ctl & (!PCI_MSI_FLAGS_ENABLE), 2);
+
+    msi_lo = qemu_get_be32(f);
+    vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_LO, msi_lo, 4);
+
+    if (msi_64bit) {
+        msi_hi = qemu_get_be32(f);
+        vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_HI,
+                msi_hi, 4);
+    }
+    msi_data = qemu_get_be32(f);
+    vfio_pci_write_config(pdev,
+            pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32),
+            msi_data, 2);
Quoted 9 lines

hw/vfio/Makefile.objs

Quoted 4 lines
hw/vfio/Makefile.objs +4 -0
+
+    vfio_pci_write_config(&vdev->pdev, pdev->msi_cap + PCI_MSI_FLAGS,
+            ctl | PCI_MSI_FLAGS_ENABLE, 2);
+  

Ok, this function is indeed pci-specific and probably should be moved to the vfio-pci code (other types could hook themselves up in the same place, then).

Quoted 5 lines

yes, this is the only pci-specific code.
maybe using VFIO_DEVICE_TYPE_PCI as a sign to decide whether to save/load pci config data?
or as Dave said, put saving/loading of pci config data into VMStateDescription interface?

I like having an interface like vmstate where other types can register themselves better than introducing conditional handling based on hard-coded type values.

author
#13 lore raw

PATCH 3/5 vfio/migration: tracking of dirty page in system memory

In reply to Yan Zhao

register the log_sync interface to hook into ram's live migration callbacks.

ram_save_pending
|->migration_bitmap_sync

       |->memory_global_dirty_log_sync
           |->memory_region_sync_dirty_bitmap
               |->listener->log_sync(listener, &mrs);

So, the dirty page produced by vfio device in system memory will be save/load by ram's live migration code iteratively.

Bitmap of device's dirty page in system memory is retrieved from Dirty Bitmap Region

---
 hw/vfio/common.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
hw/vfio/common.c +26 -0
index 7c185e5a..719e750 100644
@@ -27,6 +27,7 @@
 
 #include "hw/vfio/vfio-common.h"
 #include "hw/vfio/vfio.h"
+#include "hw/vfio/pci.h"
 #include "exec/address-spaces.h"
 #include "exec/memory.h"
 #include "hw/hw.h"
@@ -698,9 +699,34 @@ static void vfio_listener_region_del(MemoryListener *listener,
     }
 }
 
+static void vfio_log_sync(MemoryListener *listener,
+                          MemoryRegionSection *section)
+{
+    VFIOContainer *container = container_of(listener, VFIOContainer, listener);
+    VFIOGroup *group = QLIST_FIRST(&container->group_list);
+    VFIODevice *vbasedev;
+    VFIOPCIDevice *vdev;
+
+    ram_addr_t size = int128_get64(section->size);
+    uint64_t page_nr = size >> TARGET_PAGE_BITS;
+    uint64_t start_addr = section->offset_within_address_space;
+
+    QLIST_FOREACH(vbasedev, &group->device_list, next) {
+        vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev);
+        if (!vdev->migration ||
+                !vfio_device_data_cap_system_memory(vdev) ||
+                !(vdev->migration->device_state & VFIO_DEVICE_STATE_LOGGING)) {
+            continue;
+        }
+
+        vfio_set_dirty_page_bitmap(vdev, start_addr, page_nr);
+    }
+}
+
 static const MemoryListener vfio_memory_listener = {
     .region_add = vfio_listener_region_add,
     .region_del = vfio_listener_region_del,
+    .log_sync = vfio_log_sync,
 };
 
 static void vfio_listener_release(VFIOContainer *container)
Signature · 1 line
2.7.4
author
#14 lore raw

PATCH 4/5 vfio/migration: turn on migration

In reply to Yan Zhao

init vfio migration in vfio_realize() and register migraton blocker if failure met.
finalize all migration resources when vfio_instance_finalize().

---
 hw/vfio/pci.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
hw/vfio/pci.c +3 -6
index b8e006b..8bf625e 100644
@@ -3068,6 +3068,8 @@ static void vfio_realize(PCIDevice *pdev, Error **errp)
         goto out_teardown;
     }
 
+    vfio_migration_init(vdev, errp);
+
     vfio_register_err_notifier(vdev);
     vfio_register_req_notifier(vdev);
     vfio_setup_resetfn_quirk(vdev);
@@ -3089,6 +3091,7 @@ static void vfio_instance_finalize(Object *obj)
 
     vfio_display_finalize(vdev);
     vfio_bars_finalize(vdev);
+    vfio_migration_finalize(vdev);
     g_free(vdev->emulated_config_bits);
     g_free(vdev->rom);
     /*
@@ -3221,11 +3224,6 @@ static Property vfio_pci_dev_properties[] = {
     DEFINE_PROP_END_OF_LIST(),
 };
 
-static const VMStateDescription vfio_pci_vmstate = {
-    .name = "vfio-pci",
-    .unmigratable = 1,
-};
-
 static void vfio_pci_dev_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
@@ -3233,7 +3231,6 @@ static void vfio_pci_dev_class_init(ObjectClass *klass, void *data)
 
     dc->reset = vfio_pci_reset;
     dc->props = vfio_pci_dev_properties;
-    dc->vmsd = &vfio_pci_vmstate;
     dc->desc = "VFIO-based PCI device assignment";
     set_bit(DEVICE_CATEGORY_MISC, dc->categories);
     pdc->realize = vfio_realize;
Signature · 1 line
2.7.4
author
#15 lore raw

PATCH 5/5 vfio/migration: support device memory capability

In reply to Yan Zhao

If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.

LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.

Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.

---
 hw/vfio/migration.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 hw/vfio/pci.h       |   1 +
 2 files changed, 231 insertions(+), 5 deletions(-)
hw/vfio/migration.c +230 -5
index 16d6395..f1e9309 100644
@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
     return 0;
 }
 
+static int vfio_get_device_memory_size(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint64_t len;
+    int sz;
+
+    sz = sizeof(len);
+    if (pread(vbasedev->fd, &len, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to get length of device memory");
+        return -1;
+    }
+    vdev->migration->devmem_size = len;
+    return 0;
+}
+
+static int vfio_set_device_memory_size(VFIOPCIDevice *vdev, uint64_t size)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    int sz;
+
+    sz = sizeof(size);
+    if (pwrite(vbasedev->fd, &size, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to set length of device comemory");
+        return -1;
+    }
+    vdev->migration->devmem_size = size;
+    return 0;
+}
+
+static
+int vfio_save_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                    uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+
+    if (len > region_devmem->size) {
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set save buffer pos");
+        return -1;
+    }
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set save buffer action");
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        if (pread(vbasedev->fd, buf, len, region_devmem->fd_offset) != len) {
+            error_report("vfio: error load device memory buffer");
+            return -1;
+        }
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, buf, len);
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, dest, len);
+    }
+    return 0;
+}
+
+static int vfio_save_data_device_memory(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+    uint64_t total_len = vdev->migration->devmem_size;
+    uint64_t pos = 0;
+
+    qemu_put_be64(f, total_len);
+    while (pos < total_len) {
+        uint64_t len = region_devmem->size;
+
+        if (pos + len >= total_len) {
+            len = total_len - pos;
+        }
+        if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) {
+            return -1;
+        }
+    }
+
+    return 0;
+}
+
+static
+int vfio_load_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_SET_BUFFER;
+
+    if (len > region_devmem->size) {
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set device memory buffer pos");
+        return -1;
+    }
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        qemu_get_buffer(f, buf, len);
+        if (pwrite(vbasedev->fd, buf, len,
+                    region_devmem->fd_offset) != len) {
+            error_report("vfio: Failed to load devie memory buffer");
+            return -1;
+        }
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_get_buffer(f, dest, len);
+    }
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set load device memory buffer action");
+        return -1;
+    }
+
+    return 0;
+
+}
+
+static int vfio_load_data_device_memory(VFIOPCIDevice *vdev,
+                        QEMUFile *f, uint64_t total_len)
+{
+    uint64_t pos = 0, len = 0;
+
+    vfio_set_device_memory_size(vdev, total_len);
+
+    while (pos + len < total_len) {
+        len = qemu_get_be64(f);
+        pos = qemu_get_be64(f);
+
+        vfio_load_data_device_memory_chunk(vdev, f, pos, len);
+    }
+
+    return 0;
+}
+
+
 static int vfio_set_dirty_page_bitmap_chunk(VFIOPCIDevice *vdev,
         uint64_t start_addr, uint64_t page_nr)
 {
@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,
         return;
     }
 
+    /* get dirty data size of device memory */
+    vfio_get_device_memory_size(vdev);
+
+    *res_precopy_only += vdev->migration->devmem_size;
     return;
 }
 
@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)
         return 0;
     }
 
-    return 0;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+    /* get dirty data of device memory */
+    return vfio_save_data_device_memory(vdev, f);
 }
 
 static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
             len = qemu_get_be64(f);
             vfio_load_data_device_config(vdev, f, len);
             break;
+        case VFIO_SAVE_FLAG_DEVMEMORY:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_memory(vdev, f, len);
+            break;
         default:
             ret = -EINVAL;
         }
@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
     VFIOPCIDevice *vdev = opaque;
     int rc = 0;
 
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY | VFIO_SAVE_FLAG_CONTINUE);
+        /* get dirty data of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    }
+
     qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
     vfio_pci_save_config(vdev, f);
 
@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
 
 static int vfio_save_setup(QEMUFile *f, void *opaque)
 {
+    int rc = 0;
     VFIOPCIDevice *vdev = opaque;
-    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP | VFIO_SAVE_FLAG_CONTINUE);
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+        /* get whole snapshot of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    } else {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+    }
 
     vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
             VFIO_DEVICE_STATE_LOGGING);
-    return 0;
+    return rc;
 }
 
 static int vfio_load_setup(QEMUFile *f, void *opaque)
@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
         goto error;
     }
 
-    if (vfio_device_data_cap_device_memory(vdev)) {
-        error_report("No suppport of data cap device memory Yet");
+    if (vfio_device_data_cap_device_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY,
+              "device-state-data-device-memory")) {
         goto error;
     }
 
hw/vfio/pci.h +1 -0
index 4b7b1bb..a2cc64b 100644
@@ -69,6 +69,7 @@ typedef struct VFIOMigration {
     uint32_t data_caps;
     uint32_t device_state;
     uint64_t devconfig_size;
+    uint64_t devmem_size;
     VMChangeStateEntry *vm_state;
 } VFIOMigration;
 
Signature · 1 line
2.7.4
#16 lore raw

Re: [PATCH 5/5] vfio/migration: support device memory capability

Quoted 98 lines · * Yan Zhao ([email protected]) wrote:

If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.

LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.

Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.

---
 hw/vfio/migration.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 hw/vfio/pci.h       |   1 +
 2 files changed, 231 insertions(+), 5 deletions(-)
hw/vfio/migration.c +72 -0
index 16d6395..f1e9309 100644
@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
     return 0;
 }
 
+static int vfio_get_device_memory_size(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint64_t len;
+    int sz;
+
+    sz = sizeof(len);
+    if (pread(vbasedev->fd, &len, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to get length of device memory");
+        return -1;
+    }
+    vdev->migration->devmem_size = len;
+    return 0;
+}
+
+static int vfio_set_device_memory_size(VFIOPCIDevice *vdev, uint64_t size)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    int sz;
+
+    sz = sizeof(size);
+    if (pwrite(vbasedev->fd, &size, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to set length of device comemory");
+        return -1;
+    }
+    vdev->migration->devmem_size = size;
+    return 0;
+}
+
+static
+int vfio_save_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                    uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+
+    if (len > region_devmem->size) {
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set save buffer pos");
+        return -1;
+    }
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set save buffer action");
+        return -1;

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 287

Quoted 10 lines
hw/vfio/migration.c +10 -0
+    }
+
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        if (pread(vbasedev->fd, buf, len, region_devmem->fd_offset) != len) {
+            error_report("vfio: error load device memory buffer");

That's forgotten to g_free(buf)

Quoted 64 lines
hw/vfio/migration.c +64 -0
+            return -1;
+        }
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, buf, len);
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, dest, len);
+    }
+    return 0;
+}
+
+static int vfio_save_data_device_memory(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+    uint64_t total_len = vdev->migration->devmem_size;
+    uint64_t pos = 0;
+
+    qemu_put_be64(f, total_len);
+    while (pos < total_len) {
+        uint64_t len = region_devmem->size;
+
+        if (pos + len >= total_len) {
+            len = total_len - pos;
+        }
+        if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) {
+            return -1;
+        }
+    }
+
+    return 0;
+}
+
+static
+int vfio_load_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_SET_BUFFER;
+
+    if (len > region_devmem->size) {
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set device memory buffer pos");
+        return -1;
+    }

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 361

Quoted 10 lines
hw/vfio/migration.c +10 -0
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        qemu_get_buffer(f, buf, len);
+        if (pwrite(vbasedev->fd, buf, len,
+                    region_devmem->fd_offset) != len) {
+            error_report("vfio: Failed to load devie memory buffer");

Again, failed to free buf

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 368

Quoted 7 lines
hw/vfio/migration.c +7 -0
+            return -1;
+        }
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_get_buffer(f, dest, len);
+    }

You might want to use qemu_file_get_error(f) before writing the data to the device, to check for the case of a read error on the migration stream that happened somewhere in the pevious qemu_get's

Quoted 13 lines
hw/vfio/migration.c +13 -0
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set load device memory buffer action");
+        return -1;
+    }
+
+    return 0;
+
+}
+

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392

Quoted 10 lines
hw/vfio/migration.c +10 -0
+static int vfio_load_data_device_memory(VFIOPCIDevice *vdev,
+                        QEMUFile *f, uint64_t total_len)
+{
+    uint64_t pos = 0, len = 0;
+
+    vfio_set_device_memory_size(vdev, total_len);
+
+    while (pos + len < total_len) {
+        len = qemu_get_be64(f);
+        pos = qemu_get_be64(f);

Please check len/pos - always assume that the migration stream could be (maliciously or accidentally) corrupt.

Quoted 111 lines
hw/vfio/migration.c +42 -5
+        vfio_load_data_device_memory_chunk(vdev, f, pos, len);
+    }
+
+    return 0;
+}
+
+
 static int vfio_set_dirty_page_bitmap_chunk(VFIOPCIDevice *vdev,
         uint64_t start_addr, uint64_t page_nr)
 {
@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,
         return;
     }
 
+    /* get dirty data size of device memory */
+    vfio_get_device_memory_size(vdev);
+
+    *res_precopy_only += vdev->migration->devmem_size;
     return;
 }
 
@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)
         return 0;
     }
 
-    return 0;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+    /* get dirty data of device memory */
+    return vfio_save_data_device_memory(vdev, f);
 }
 
 static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
             len = qemu_get_be64(f);
             vfio_load_data_device_config(vdev, f, len);
             break;
+        case VFIO_SAVE_FLAG_DEVMEMORY:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_memory(vdev, f, len);
+            break;
         default:
             ret = -EINVAL;
         }
@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
     VFIOPCIDevice *vdev = opaque;
     int rc = 0;
 
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY | VFIO_SAVE_FLAG_CONTINUE);
+        /* get dirty data of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    }
+
     qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
     vfio_pci_save_config(vdev, f);
 
@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
 
 static int vfio_save_setup(QEMUFile *f, void *opaque)
 {
+    int rc = 0;
     VFIOPCIDevice *vdev = opaque;
-    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP | VFIO_SAVE_FLAG_CONTINUE);
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+        /* get whole snapshot of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    } else {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+    }
 
     vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
             VFIO_DEVICE_STATE_LOGGING);
-    return 0;
+    return rc;
 }
 
 static int vfio_load_setup(QEMUFile *f, void *opaque)
@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
         goto error;
     }
 
-    if (vfio_device_data_cap_device_memory(vdev)) {
-        error_report("No suppport of data cap device memory Yet");
+    if (vfio_device_data_cap_device_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY,
+              "device-state-data-device-memory")) {
         goto error;
     }
 
hw/vfio/pci.h +1 -0
index 4b7b1bb..a2cc64b 100644
@@ -69,6 +69,7 @@ typedef struct VFIOMigration {
     uint32_t data_caps;
     uint32_t device_state;
     uint64_t devconfig_size;
+    uint64_t devmem_size;
     VMChangeStateEntry *vm_state;
 } VFIOMigration;
 
Signature · 1 line
2.7.4

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#17 lore raw

Re: [PATCH 5/5] vfio/migration: support device memory capability

Quoted 103 lines · Dr. David Alan Gilbert
Quoted 102 lines · * Yan Zhao ([email protected]) wrote:

If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.

LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.

Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.

---
 hw/vfio/migration.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 hw/vfio/pci.h       |   1 +
 2 files changed, 231 insertions(+), 5 deletions(-)
hw/vfio/migration.c +76 -0
index 16d6395..f1e9309 100644
@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
     return 0;
 }
 
+static int vfio_get_device_memory_size(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint64_t len;
+    int sz;
+
+    sz = sizeof(len);
+    if (pread(vbasedev->fd, &len, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to get length of device memory");
+        return -1;
+    }
+    vdev->migration->devmem_size = len;
+    return 0;
+}
+
+static int vfio_set_device_memory_size(VFIOPCIDevice *vdev, uint64_t size)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    int sz;
+
+    sz = sizeof(size);
+    if (pwrite(vbasedev->fd, &size, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to set length of device comemory");
+        return -1;
+    }
+    vdev->migration->devmem_size = size;
+    return 0;
+}
+
+static
+int vfio_save_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                    uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+
+    if (len > region_devmem->size) {
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set save buffer pos");
+        return -1;
+    }
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set save buffer action");
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
Quoted 9 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 287

Quoted 6 lines
hw/vfio/migration.c +6 -0
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        if (pread(vbasedev->fd, buf, len, region_devmem->fd_offset) != len) {
+            error_report("vfio: error load device memory buffer");

That's forgotten to g_free(buf)

Right, I'll correct that.

Quoted 68 lines
Quoted 68 lines
hw/vfio/migration.c +68 -0
+            return -1;
+        }
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, buf, len);
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, dest, len);
+    }
+    return 0;
+}
+
+static int vfio_save_data_device_memory(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+    uint64_t total_len = vdev->migration->devmem_size;
+    uint64_t pos = 0;
+
+    qemu_put_be64(f, total_len);
+    while (pos < total_len) {
+        uint64_t len = region_devmem->size;
+
+        if (pos + len >= total_len) {
+            len = total_len - pos;
+        }
+        if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) {
+            return -1;
+        }
+    }
+
+    return 0;
+}
+
+static
+int vfio_load_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_SET_BUFFER;
+
+    if (len > region_devmem->size) {
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set device memory buffer pos");
+        return -1;
+    }
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
Quoted 9 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 361

Quoted 6 lines
hw/vfio/migration.c +6 -0
+            return -1;
+        }
+        qemu_get_buffer(f, buf, len);
+        if (pwrite(vbasedev->fd, buf, len,
+                    region_devmem->fd_offset) != len) {
+            error_report("vfio: Failed to load devie memory buffer");

Again, failed to free buf

Right, I'll correct that.

Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 368

Quoted 7 lines
hw/vfio/migration.c +7 -0
+            return -1;
+        }
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_get_buffer(f, dest, len);
+    }
Quoted 4 lines

You might want to use qemu_file_get_error(f) before writing the data to the device, to check for the case of a read error on the migration stream that happened somewhere in the pevious qemu_get's

ok.

Quoted 18 lines
Quoted 18 lines
hw/vfio/migration.c +18 -0
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set load device memory buffer action");
+        return -1;
+    }
+
+    return 0;
+
+}
+
+static int vfio_load_data_device_memory(VFIOPCIDevice *vdev,
+                        QEMUFile *f, uint64_t total_len)
+{
+    uint64_t pos = 0, len = 0;
+
Quoted 9 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392

Quoted 5 lines
hw/vfio/migration.c +5 -0
+    vfio_set_device_memory_size(vdev, total_len);
+
+    while (pos + len < total_len) {
+        len = qemu_get_be64(f);
+        pos = qemu_get_be64(f);

Please check len/pos - always assume that the migration stream could be (maliciously or accidentally) corrupt.

ok.

Quoted 117 lines
Quoted 111 lines
hw/vfio/migration.c +42 -5
+        vfio_load_data_device_memory_chunk(vdev, f, pos, len);
+    }
+
+    return 0;
+}
+
+
 static int vfio_set_dirty_page_bitmap_chunk(VFIOPCIDevice *vdev,
         uint64_t start_addr, uint64_t page_nr)
 {
@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,
         return;
     }
 
+    /* get dirty data size of device memory */
+    vfio_get_device_memory_size(vdev);
+
+    *res_precopy_only += vdev->migration->devmem_size;
     return;
 }
 
@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)
         return 0;
     }
 
-    return 0;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+    /* get dirty data of device memory */
+    return vfio_save_data_device_memory(vdev, f);
 }
 
 static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
             len = qemu_get_be64(f);
             vfio_load_data_device_config(vdev, f, len);
             break;
+        case VFIO_SAVE_FLAG_DEVMEMORY:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_memory(vdev, f, len);
+            break;
         default:
             ret = -EINVAL;
         }
@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
     VFIOPCIDevice *vdev = opaque;
     int rc = 0;
 
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY | VFIO_SAVE_FLAG_CONTINUE);
+        /* get dirty data of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    }
+
     qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
     vfio_pci_save_config(vdev, f);
 
@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
 
 static int vfio_save_setup(QEMUFile *f, void *opaque)
 {
+    int rc = 0;
     VFIOPCIDevice *vdev = opaque;
-    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP | VFIO_SAVE_FLAG_CONTINUE);
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+        /* get whole snapshot of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    } else {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+    }
 
     vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
             VFIO_DEVICE_STATE_LOGGING);
-    return 0;
+    return rc;
 }
 
 static int vfio_load_setup(QEMUFile *f, void *opaque)
@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
         goto error;
     }
 
-    if (vfio_device_data_cap_device_memory(vdev)) {
-        error_report("No suppport of data cap device memory Yet");
+    if (vfio_device_data_cap_device_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY,
+              "device-state-data-device-memory")) {
         goto error;
     }
 
hw/vfio/pci.h +1 -0
index 4b7b1bb..a2cc64b 100644
@@ -69,6 +69,7 @@ typedef struct VFIOMigration {
     uint32_t data_caps;
     uint32_t device_state;
     uint64_t devconfig_size;
+    uint64_t devmem_size;
     VMChangeStateEntry *vm_state;
 } VFIOMigration;
 
Signature · 1 line
2.7.4

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

#18 lore raw

Re: [PATCH 5/5] vfio/migration: support device memory capability

In reply to Yan Zhao
Quoted 34 lines · Yan Zhao

If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.

LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.

Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.

---
hw/vfio/migration.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++--
hw/vfio/pci.h | 1 +

2 files changed, 231 insertions(+), 5 deletions(-)
hw/vfio/migration.c +4 -0
index 16d6395..f1e9309 100644
@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
    return 0;
 }
 
+static int vfio_get_device_memory_size(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 219

Quoted 10 lines
hw/vfio/migration.c +10 -0
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint64_t len;
+    int sz;
+
+    sz = sizeof(len);
+    if (pread(vbasedev->fd, &len, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to get length of device memory”);

s/length/size/ ? (to be consistent with function name)

Quoted 9 lines
hw/vfio/migration.c +9 -0
+        return -1;
+    }
+    vdev->migration->devmem_size = len;
+    return 0;
+}
+
+static int vfio_set_device_memory_size(VFIOPCIDevice *vdev, uint64_t size)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 10 lines
hw/vfio/migration.c +10 -0
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    int sz;
+
+    sz = sizeof(size);
+    if (pwrite(vbasedev->fd, &size, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to set length of device comemory”);

What is comemory? Typo?

Same comment about length vs size

Quoted 11 lines
hw/vfio/migration.c +11 -0
+        return -1;
+    }
+    vdev->migration->devmem_size = size;
+    return 0;
+}
+
+static
+int vfio_save_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                    uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 259

Quoted 10 lines
hw/vfio/migration.c +10 -0
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+
+    if (len > region_devmem->size) {

Is it intentional that there is no error_report here?

Quoted 14 lines
hw/vfio/migration.c +14 -0
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set save buffer pos");
+        return -1;
+    }
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 10 lines
hw/vfio/migration.c +10 -0
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set save buffer action");
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate”);

s/migrate/migration/ ?

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 4 lines
hw/vfio/migration.c +4 -0
+            return -1;
+        }
+        if (pread(vbasedev->fd, buf, len, region_devmem->fd_offset) != len) {
+            error_report("vfio: error load device memory buffer”);

s/load/loading/ ?

Quoted 22 lines
hw/vfio/migration.c +22 -0
+            return -1;
+        }
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, buf, len);
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, dest, len);
+    }
+    return 0;
+}
+
+static int vfio_save_data_device_memory(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+    uint64_t total_len = vdev->migration->devmem_size;
+    uint64_t pos = 0;
+

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 319

Quoted 10 lines
hw/vfio/migration.c +10 -0
+    qemu_put_be64(f, total_len);
+    while (pos < total_len) {
+        uint64_t len = region_devmem->size;
+
+        if (pos + len >= total_len) {
+            len = total_len - pos;
+        }
+        if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) {
+            return -1;
+        }

I don’t see where pos is incremented in this loop

Quoted 11 lines
hw/vfio/migration.c +11 -0
+    }
+
+    return 0;
+}
+
+static
+int vfio_load_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 340

Quoted 10 lines
hw/vfio/migration.c +10 -0
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_SET_BUFFER;
+
+    if (len > region_devmem->size) {

error_report?

Quoted 42 lines
hw/vfio/migration.c +42 -0
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set device memory buffer pos");
+        return -1;
+    }
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        qemu_get_buffer(f, buf, len);
+        if (pwrite(vbasedev->fd, buf, len,
+                    region_devmem->fd_offset) != len) {
+            error_report("vfio: Failed to load devie memory buffer");
+            return -1;
+        }
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_get_buffer(f, dest, len);
+    }
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set load device memory buffer action");
+        return -1;
+    }
+
+    return 0;
+
+}
+

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392

Quoted 10 lines
hw/vfio/migration.c +10 -0
+static int vfio_load_data_device_memory(VFIOPCIDevice *vdev,
+                        QEMUFile *f, uint64_t total_len)
+{
+    uint64_t pos = 0, len = 0;
+
+    vfio_set_device_memory_size(vdev, total_len);
+
+    while (pos + len < total_len) {
+        len = qemu_get_be64(f);
+        pos = qemu_get_be64(f);

Nit: load reads len/pos in the loop, whereas save does it in the inner function (vfio_save_data_device_memory_chunk)

Quoted 116 lines
hw/vfio/migration.c +43 -5
+
+        vfio_load_data_device_memory_chunk(vdev, f, pos, len);
+    }
+
+    return 0;
+}
+
+
 static int vfio_set_dirty_page_bitmap_chunk(VFIOPCIDevice *vdev,
        uint64_t start_addr, uint64_t page_nr)
 {
@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,
        return;
    }
 
+    /* get dirty data size of device memory */
+    vfio_get_device_memory_size(vdev);
+
+    *res_precopy_only += vdev->migration->devmem_size;
    return;
 }
 
@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)
        return 0;
    }
 
-    return 0;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+    /* get dirty data of device memory */
+    return vfio_save_data_device_memory(vdev, f);
 }
 
 static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
            len = qemu_get_be64(f);
            vfio_load_data_device_config(vdev, f, len);
            break;
+        case VFIO_SAVE_FLAG_DEVMEMORY:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_memory(vdev, f, len);
+            break;
        default:
            ret = -EINVAL;
        }
@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
    VFIOPCIDevice *vdev = opaque;
    int rc = 0;
 
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY | VFIO_SAVE_FLAG_CONTINUE);
+        /* get dirty data of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    }
+
    qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
    vfio_pci_save_config(vdev, f);
 
@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
 
 static int vfio_save_setup(QEMUFile *f, void *opaque)
 {
+    int rc = 0;
    VFIOPCIDevice *vdev = opaque;
-    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP | VFIO_SAVE_FLAG_CONTINUE);
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+        /* get whole snapshot of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    } else {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+    }
 
    vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
            VFIO_DEVICE_STATE_LOGGING);
-    return 0;
+    return rc;
 }
 
 static int vfio_load_setup(QEMUFile *f, void *opaque)
@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
        goto error;
    }
 
-    if (vfio_device_data_cap_device_memory(vdev)) {
-        error_report("No suppport of data cap device memory Yet");
+    if (vfio_device_data_cap_device_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY,
+              "device-state-data-device-memory")) {
        goto error;
    }
 
hw/vfio/pci.h +1 -0
index 4b7b1bb..a2cc64b 100644
@@ -69,6 +69,7 @@ typedef struct VFIOMigration {
    uint32_t data_caps;
    uint32_t device_state;
    uint64_t devconfig_size;
+    uint64_t devmem_size;
    VMChangeStateEntry *vm_state;
 } VFIOMigration;
 
Signature · 6 lines
2.7.4

_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
author
#19 lore raw

Re: [PATCH 5/5] vfio/migration: support device memory capability

Quoted 30 lines · Christophe de Dinechin
Quoted 30 lines · Yan Zhao

If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.

LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.

Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.

---
hw/vfio/migration.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++--
hw/vfio/pci.h | 1 +

2 files changed, 231 insertions(+), 5 deletions(-)
hw/vfio/migration.c +0 -0
index 16d6395..f1e9309 100644
@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
    return 0;
 }
 
Quoted 16 lines
Quoted 4 lines
hw/vfio/migration.c +4 -0
+static int vfio_get_device_memory_size(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 10 lines
hw/vfio/migration.c +10 -0
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint64_t len;
+    int sz;
+
+    sz = sizeof(len);
+    if (pread(vbasedev->fd, &len, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to get length of device memory”);

s/length/size/ ? (to be consistent with function name)

ok. thanks

Quoted 13 lines
Quoted 13 lines
hw/vfio/migration.c +13 -0
+        return -1;
+    }
+    vdev->migration->devmem_size = len;
+    return 0;
+}
+
+static int vfio_set_device_memory_size(VFIOPCIDevice *vdev, uint64_t size)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    int sz;
+
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 6 lines
hw/vfio/migration.c +6 -0
+    sz = sizeof(size);
+    if (pwrite(vbasedev->fd, &size, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to set length of device comemory”);

What is comemory? Typo?

Right, typo. should be "memory" :)

Quoted 3 lines

Same comment about length vs size

got it. thanks

Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15 -0
+        return -1;
+    }
+    vdev->migration->devmem_size = size;
+    return 0;
+}
+
+static
+int vfio_save_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                    uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
Quoted 9 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 259

Quoted 6 lines
hw/vfio/migration.c +6 -0
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+
+    if (len > region_devmem->size) {

Is it intentional that there is no error_report here?

an error_report here may be better.

Quoted 17 lines
Quoted 17 lines
hw/vfio/migration.c +17 -0
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set save buffer pos");
+        return -1;
+    }
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set save buffer action");
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 7 lines
hw/vfio/migration.c +7 -0
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate”);

s/migrate/migration/ ?

yes, thanks

Quoted 5 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 4 lines
hw/vfio/migration.c +4 -0
+            return -1;
+        }
+        if (pread(vbasedev->fd, buf, len, region_devmem->fd_offset) != len) {
+            error_report("vfio: error load device memory buffer”);

s/load/loading/ ?

error to load? :)

Quoted 26 lines
Quoted 26 lines
hw/vfio/migration.c +26 -0
+            return -1;
+        }
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, buf, len);
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, dest, len);
+    }
+    return 0;
+}
+
+static int vfio_save_data_device_memory(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+    uint64_t total_len = vdev->migration->devmem_size;
+    uint64_t pos = 0;
+
+    qemu_put_be64(f, total_len);
+    while (pos < total_len) {
+        uint64_t len = region_devmem->size;
+
Quoted 9 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 319

Quoted 6 lines
hw/vfio/migration.c +6 -0
+        if (pos + len >= total_len) {
+            len = total_len - pos;
+        }
+        if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) {
+            return -1;
+        }

I don’t see where pos is incremented in this loop

yes, missing one line "pos += len;"
Currently, code is not verified in hardware with device memory cap on. Thanks:)

Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15 -0
+    }
+
+    return 0;
+}
+
+static
+int vfio_load_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 340

Quoted 6 lines
hw/vfio/migration.c +6 -0
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_SET_BUFFER;
+
+    if (len > region_devmem->size) {

error_report?

seems better to add error_report.

Quoted 47 lines
Quoted 47 lines
hw/vfio/migration.c +47 -0
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set device memory buffer pos");
+        return -1;
+    }
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        qemu_get_buffer(f, buf, len);
+        if (pwrite(vbasedev->fd, buf, len,
+                    region_devmem->fd_offset) != len) {
+            error_report("vfio: Failed to load devie memory buffer");
+            return -1;
+        }
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_get_buffer(f, dest, len);
+    }
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set load device memory buffer action");
+        return -1;
+    }
+
+    return 0;
+
+}
+
+static int vfio_load_data_device_memory(VFIOPCIDevice *vdev,
+                        QEMUFile *f, uint64_t total_len)
+{
+    uint64_t pos = 0, len = 0;
+
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392

Quoted 5 lines
hw/vfio/migration.c +5 -0
+    vfio_set_device_memory_size(vdev, total_len);
+
+    while (pos + len < total_len) {
+        len = qemu_get_be64(f);
+        pos = qemu_get_be64(f);

Nit: load reads len/pos in the loop, whereas save does it in the inner function (vfio_save_data_device_memory_chunk)

right, load has to read len/pos in the loop.

Quoted 122 lines
Quoted 117 lines
hw/vfio/migration.c +43 -5
+
+        vfio_load_data_device_memory_chunk(vdev, f, pos, len);
+    }
+
+    return 0;
+}
+
+
 static int vfio_set_dirty_page_bitmap_chunk(VFIOPCIDevice *vdev,
        uint64_t start_addr, uint64_t page_nr)
 {
@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,
        return;
    }
 
+    /* get dirty data size of device memory */
+    vfio_get_device_memory_size(vdev);
+
+    *res_precopy_only += vdev->migration->devmem_size;
    return;
 }
 
@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)
        return 0;
    }
 
-    return 0;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+    /* get dirty data of device memory */
+    return vfio_save_data_device_memory(vdev, f);
 }
 
 static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
            len = qemu_get_be64(f);
            vfio_load_data_device_config(vdev, f, len);
            break;
+        case VFIO_SAVE_FLAG_DEVMEMORY:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_memory(vdev, f, len);
+            break;
        default:
            ret = -EINVAL;
        }
@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
    VFIOPCIDevice *vdev = opaque;
    int rc = 0;
 
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY | VFIO_SAVE_FLAG_CONTINUE);
+        /* get dirty data of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    }
+
    qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
    vfio_pci_save_config(vdev, f);
 
@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
 
 static int vfio_save_setup(QEMUFile *f, void *opaque)
 {
+    int rc = 0;
    VFIOPCIDevice *vdev = opaque;
-    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP | VFIO_SAVE_FLAG_CONTINUE);
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+        /* get whole snapshot of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    } else {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+    }
 
    vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
            VFIO_DEVICE_STATE_LOGGING);
-    return 0;
+    return rc;
 }
 
 static int vfio_load_setup(QEMUFile *f, void *opaque)
@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
        goto error;
    }
 
-    if (vfio_device_data_cap_device_memory(vdev)) {
-        error_report("No suppport of data cap device memory Yet");
+    if (vfio_device_data_cap_device_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY,
+              "device-state-data-device-memory")) {
        goto error;
    }
 
hw/vfio/pci.h +1 -0
index 4b7b1bb..a2cc64b 100644
@@ -69,6 +69,7 @@ typedef struct VFIOMigration {
    uint32_t data_caps;
    uint32_t device_state;
    uint64_t devconfig_size;
+    uint64_t devmem_size;
    VMChangeStateEntry *vm_state;
 } VFIOMigration;
 
Signature · 6 lines
2.7.4

_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

#20 lore raw

Re: [PATCH 5/5] vfio/migration: support device memory capability

Quoted 132 lines · Zhao Yan
Quoted 30 lines · Christophe de Dinechin
Quoted 30 lines · Yan Zhao

If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.

LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.

Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.

---
hw/vfio/migration.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++--
hw/vfio/pci.h | 1 +

2 files changed, 231 insertions(+), 5 deletions(-)
hw/vfio/migration.c +0 -0
index 16d6395..f1e9309 100644
@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
   return 0;
 }
 
Quoted 16 lines
Quoted 4 lines
hw/vfio/migration.c +4 -0
+static int vfio_get_device_memory_size(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 10 lines
hw/vfio/migration.c +10 -0
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint64_t len;
+    int sz;
+
+    sz = sizeof(len);
+    if (pread(vbasedev->fd, &len, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to get length of device memory”);

s/length/size/ ? (to be consistent with function name)

ok. thanks

Quoted 13 lines
Quoted 13 lines
hw/vfio/migration.c +13 -0
+        return -1;
+    }
+    vdev->migration->devmem_size = len;
+    return 0;
+}
+
+static int vfio_set_device_memory_size(VFIOPCIDevice *vdev, uint64_t size)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    int sz;
+
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 6 lines
hw/vfio/migration.c +6 -0
+    sz = sizeof(size);
+    if (pwrite(vbasedev->fd, &size, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to set length of device comemory”);

What is comemory? Typo?

Right, typo. should be "memory" :)

Quoted 3 lines

Same comment about length vs size

got it. thanks

Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15 -0
+        return -1;
+    }
+    vdev->migration->devmem_size = size;
+    return 0;
+}
+
+static
+int vfio_save_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                    uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
Quoted 9 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 259

Quoted 6 lines
hw/vfio/migration.c +6 -0
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+
+    if (len > region_devmem->size) {

Is it intentional that there is no error_report here?

an error_report here may be better.

Quoted 17 lines
Quoted 17 lines
hw/vfio/migration.c +17 -0
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set save buffer pos");
+        return -1;
+    }
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set save buffer action");
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 7 lines
hw/vfio/migration.c +7 -0
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate”);

s/migrate/migration/ ?

Quoted 7 lines

yes, thanks

Quoted 5 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 4 lines
hw/vfio/migration.c +4 -0
+            return -1;
+        }
+        if (pread(vbasedev->fd, buf, len, region_devmem->fd_offset) != len) {
+            error_report("vfio: error load device memory buffer”);

s/load/loading/ ?

error to load? :)

I’d check with a native speaker, but I believe it’s “error loading”.

To me (to be checked), the two sentences don’t have the same meaning:

“It is an error to load device memory buffer” -> “You are not allowed to do that” “I had an error loading device memory buffer” -> “I tried, but it failed"

Quoted 242 lines
Quoted 27 lines
Quoted 27 lines
hw/vfio/migration.c +26 -0
+            return -1;
+        }
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, buf, len);
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, dest, len);
+    }
+    return 0;
+}
+
+static int vfio_save_data_device_memory(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+    uint64_t total_len = vdev->migration->devmem_size;
+    uint64_t pos = 0;
+
+    qemu_put_be64(f, total_len);
+    while (pos < total_len) {
+        uint64_t len = region_devmem->size;
+
Quoted 9 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 319

Quoted 6 lines
hw/vfio/migration.c +6 -0
+        if (pos + len >= total_len) {
+            len = total_len - pos;
+        }
+        if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) {
+            return -1;
+        }

I don’t see where pos is incremented in this loop

yes, missing one line "pos += len;"
Currently, code is not verified in hardware with device memory cap on. Thanks:)

Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15 -0
+    }
+
+    return 0;
+}
+
+static
+int vfio_load_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 340

Quoted 6 lines
hw/vfio/migration.c +6 -0
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_SET_BUFFER;
+
+    if (len > region_devmem->size) {

error_report?

seems better to add error_report.

Quoted 47 lines
Quoted 47 lines
hw/vfio/migration.c +47 -0
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set device memory buffer pos");
+        return -1;
+    }
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        qemu_get_buffer(f, buf, len);
+        if (pwrite(vbasedev->fd, buf, len,
+                    region_devmem->fd_offset) != len) {
+            error_report("vfio: Failed to load devie memory buffer");
+            return -1;
+        }
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_get_buffer(f, dest, len);
+    }
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set load device memory buffer action");
+        return -1;
+    }
+
+    return 0;
+
+}
+
+static int vfio_load_data_device_memory(VFIOPCIDevice *vdev,
+                        QEMUFile *f, uint64_t total_len)
+{
+    uint64_t pos = 0, len = 0;
+
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392

Quoted 5 lines
hw/vfio/migration.c +5 -0
+    vfio_set_device_memory_size(vdev, total_len);
+
+    while (pos + len < total_len) {
+        len = qemu_get_be64(f);
+        pos = qemu_get_be64(f);

Nit: load reads len/pos in the loop, whereas save does it in the inner function (vfio_save_data_device_memory_chunk)

right, load has to read len/pos in the loop.

Quoted 122 lines
Quoted 117 lines
hw/vfio/migration.c +43 -5
+
+        vfio_load_data_device_memory_chunk(vdev, f, pos, len);
+    }
+
+    return 0;
+}
+
+
 static int vfio_set_dirty_page_bitmap_chunk(VFIOPCIDevice *vdev,
       uint64_t start_addr, uint64_t page_nr)
 {
@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,
       return;
   }
 
+    /* get dirty data size of device memory */
+    vfio_get_device_memory_size(vdev);
+
+    *res_precopy_only += vdev->migration->devmem_size;
   return;
 }
 
@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)
       return 0;
   }
 
-    return 0;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+    /* get dirty data of device memory */
+    return vfio_save_data_device_memory(vdev, f);
 }
 
 static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
           len = qemu_get_be64(f);
           vfio_load_data_device_config(vdev, f, len);
           break;
+        case VFIO_SAVE_FLAG_DEVMEMORY:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_memory(vdev, f, len);
+            break;
       default:
           ret = -EINVAL;
       }
@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
   VFIOPCIDevice *vdev = opaque;
   int rc = 0;
 
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY | VFIO_SAVE_FLAG_CONTINUE);
+        /* get dirty data of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    }
+
   qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
   vfio_pci_save_config(vdev, f);
 
@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
 
 static int vfio_save_setup(QEMUFile *f, void *opaque)
 {
+    int rc = 0;
   VFIOPCIDevice *vdev = opaque;
-    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP | VFIO_SAVE_FLAG_CONTINUE);
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+        /* get whole snapshot of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    } else {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+    }
 
   vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
           VFIO_DEVICE_STATE_LOGGING);
-    return 0;
+    return rc;
 }
 
 static int vfio_load_setup(QEMUFile *f, void *opaque)
@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
       goto error;
   }
 
-    if (vfio_device_data_cap_device_memory(vdev)) {
-        error_report("No suppport of data cap device memory Yet");
+    if (vfio_device_data_cap_device_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY,
+              "device-state-data-device-memory")) {
       goto error;
   }
 
hw/vfio/pci.h +1 -0
index 4b7b1bb..a2cc64b 100644
@@ -69,6 +69,7 @@ typedef struct VFIOMigration {
   uint32_t data_caps;
   uint32_t device_state;
   uint64_t devconfig_size;
+    uint64_t devmem_size;
   VMChangeStateEntry *vm_state;
 } VFIOMigration;
 
Signature · 6 lines
2.7.4

_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

author
#21 lore raw

Re: [PATCH 5/5] vfio/migration: support device memory capability

Quoted 138 lines · Christophe de Dinechin
Quoted 138 lines · Zhao Yan
Quoted 30 lines · Christophe de Dinechin
Quoted 30 lines · Yan Zhao

If a device has device memory capability, save/load data from device memory in pre-copy and stop-and-copy phases.

LOGGING state is set for device memory for dirty page logging:
in LOGGING state, get device memory returns whole device memory snapshot; outside LOGGING state, get device memory returns dirty data since last get operation.

Usually, device memory is very big, qemu needs to chunk it into several pieces each with size of device memory region.

---
hw/vfio/migration.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++++++--
hw/vfio/pci.h | 1 +

2 files changed, 231 insertions(+), 5 deletions(-)
hw/vfio/migration.c +0 -0
index 16d6395..f1e9309 100644
@@ -203,6 +203,201 @@ static int vfio_load_data_device_config(VFIOPCIDevice *vdev,
   return 0;
 }
 
Quoted 16 lines
Quoted 4 lines
hw/vfio/migration.c +4 -0
+static int vfio_get_device_memory_size(VFIOPCIDevice *vdev)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 10 lines
hw/vfio/migration.c +10 -0
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    uint64_t len;
+    int sz;
+
+    sz = sizeof(len);
+    if (pread(vbasedev->fd, &len, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to get length of device memory”);

s/length/size/ ? (to be consistent with function name)

ok. thanks

Quoted 13 lines
Quoted 13 lines
hw/vfio/migration.c +13 -0
+        return -1;
+    }
+    vdev->migration->devmem_size = len;
+    return 0;
+}
+
+static int vfio_set_device_memory_size(VFIOPCIDevice *vdev, uint64_t size)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    int sz;
+
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 6 lines
hw/vfio/migration.c +6 -0
+    sz = sizeof(size);
+    if (pwrite(vbasedev->fd, &size, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.size))
+            != sz) {
+        error_report("vfio: Failed to set length of device comemory”);

What is comemory? Typo?

Right, typo. should be "memory" :)

Quoted 3 lines

Same comment about length vs size

got it. thanks

Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15 -0
+        return -1;
+    }
+    vdev->migration->devmem_size = size;
+    return 0;
+}
+
+static
+int vfio_save_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                    uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
Quoted 9 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 259

Quoted 6 lines
hw/vfio/migration.c +6 -0
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_GET_BUFFER;
+
+    if (len > region_devmem->size) {

Is it intentional that there is no error_report here?

an error_report here may be better.

Quoted 17 lines
Quoted 17 lines
hw/vfio/migration.c +17 -0
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set save buffer pos");
+        return -1;
+    }
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set save buffer action");
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 7 lines
hw/vfio/migration.c +7 -0
+        return -1;
+    }
+
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate”);

s/migrate/migration/ ?

yes, thanks

Quoted 5 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev,

Quoted 4 lines
hw/vfio/migration.c +4 -0
+            return -1;
+        }
+        if (pread(vbasedev->fd, buf, len, region_devmem->fd_offset) != len) {
+            error_report("vfio: error load device memory buffer”);

s/load/loading/ ?

Quoted 9 lines
Quoted 1 line

error to load? :)

I’d check with a native speaker, but I believe it’s “error loading”.

To me (to be checked), the two sentences don’t have the same meaning:

“It is an error to load device memory buffer” -> “You are not allowed to do that” “I had an error loading device memory buffer” -> “I tried, but it failed"

haha, ok, I'll change it to loading, thanks :)

Quoted 243 lines
Quoted 242 lines
Quoted 27 lines
Quoted 27 lines
hw/vfio/migration.c +26 -0
+            return -1;
+        }
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, buf, len);
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_put_be64(f, len);
+        qemu_put_be64(f, pos);
+        qemu_put_buffer(f, dest, len);
+    }
+    return 0;
+}
+
+static int vfio_save_data_device_memory(VFIOPCIDevice *vdev, QEMUFile *f)
+{
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+    uint64_t total_len = vdev->migration->devmem_size;
+    uint64_t pos = 0;
+
+    qemu_put_be64(f, total_len);
+    while (pos < total_len) {
+        uint64_t len = region_devmem->size;
+
Quoted 9 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 319

Quoted 6 lines
hw/vfio/migration.c +6 -0
+        if (pos + len >= total_len) {
+            len = total_len - pos;
+        }
+        if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) {
+            return -1;
+        }

I don’t see where pos is incremented in this loop

yes, missing one line "pos += len;"
Currently, code is not verified in hardware with device memory cap on. Thanks:)

Quoted 15 lines
Quoted 15 lines
hw/vfio/migration.c +15 -0
+    }
+
+    return 0;
+}
+
+static
+int vfio_load_data_device_memory_chunk(VFIOPCIDevice *vdev, QEMUFile *f,
+                                uint64_t pos, uint64_t len)
+{
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIORegion *region_ctl =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_CTL];
+    VFIORegion *region_devmem =
+        &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY];
+
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 340

Quoted 6 lines
hw/vfio/migration.c +6 -0
+    void *dest;
+    uint32_t sz;
+    uint8_t *buf = NULL;
+    uint32_t action = VFIO_DEVICE_DATA_ACTION_SET_BUFFER;
+
+    if (len > region_devmem->size) {

error_report?

seems better to add error_report.

Quoted 47 lines
Quoted 47 lines
hw/vfio/migration.c +47 -0
+        return -1;
+    }
+
+    sz = sizeof(pos);
+    if (pwrite(vbasedev->fd, &pos, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.pos))
+            != sz) {
+        error_report("vfio: Failed to set device memory buffer pos");
+        return -1;
+    }
+    if (!vfio_device_state_region_mmaped(region_devmem)) {
+        buf = g_malloc(len);
+        if (buf == NULL) {
+            error_report("vfio: Failed to allocate memory for migrate");
+            return -1;
+        }
+        qemu_get_buffer(f, buf, len);
+        if (pwrite(vbasedev->fd, buf, len,
+                    region_devmem->fd_offset) != len) {
+            error_report("vfio: Failed to load devie memory buffer");
+            return -1;
+        }
+        g_free(buf);
+    } else {
+        dest = region_devmem->mmaps[0].mmap;
+        qemu_get_buffer(f, dest, len);
+    }
+
+    sz = sizeof(action);
+    if (pwrite(vbasedev->fd, &action, sz,
+                region_ctl->fd_offset +
+                offsetof(struct vfio_device_state_ctl, device_memory.action))
+            != sz) {
+        error_report("vfio: Failed to set load device memory buffer action");
+        return -1;
+    }
+
+    return 0;
+
+}
+
+static int vfio_load_data_device_memory(VFIOPCIDevice *vdev,
+                        QEMUFile *f, uint64_t total_len)
+{
+    uint64_t pos = 0, len = 0;
+
Quoted 8 lines

hw/vfio/migration.c static int vfio_load_data_device_config(VFIOPCIDevice *vdev, · line 392

Quoted 5 lines
hw/vfio/migration.c +5 -0
+    vfio_set_device_memory_size(vdev, total_len);
+
+    while (pos + len < total_len) {
+        len = qemu_get_be64(f);
+        pos = qemu_get_be64(f);

Nit: load reads len/pos in the loop, whereas save does it in the inner function (vfio_save_data_device_memory_chunk)

right, load has to read len/pos in the loop.

Quoted 122 lines
Quoted 117 lines
hw/vfio/migration.c +43 -5
+
+        vfio_load_data_device_memory_chunk(vdev, f, pos, len);
+    }
+
+    return 0;
+}
+
+
 static int vfio_set_dirty_page_bitmap_chunk(VFIOPCIDevice *vdev,
       uint64_t start_addr, uint64_t page_nr)
 {
@@ -377,6 +572,10 @@ static void vfio_save_live_pending(QEMUFile *f, void *opaque,
       return;
   }
 
+    /* get dirty data size of device memory */
+    vfio_get_device_memory_size(vdev);
+
+    *res_precopy_only += vdev->migration->devmem_size;
   return;
 }
 
@@ -388,7 +587,9 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)
       return 0;
   }
 
-    return 0;
+    qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+    /* get dirty data of device memory */
+    return vfio_save_data_device_memory(vdev, f);
 }
 
 static void vfio_pci_load_config(VFIOPCIDevice *vdev, QEMUFile *f)
@@ -458,6 +659,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id)
           len = qemu_get_be64(f);
           vfio_load_data_device_config(vdev, f, len);
           break;
+        case VFIO_SAVE_FLAG_DEVMEMORY:
+            len = qemu_get_be64(f);
+            vfio_load_data_device_memory(vdev, f, len);
+            break;
       default:
           ret = -EINVAL;
       }
@@ -503,6 +708,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
   VFIOPCIDevice *vdev = opaque;
   int rc = 0;
 
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY | VFIO_SAVE_FLAG_CONTINUE);
+        /* get dirty data of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    }
+
   qemu_put_byte(f, VFIO_SAVE_FLAG_PCI | VFIO_SAVE_FLAG_CONTINUE);
   vfio_pci_save_config(vdev, f);
 
@@ -515,12 +727,22 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
 
 static int vfio_save_setup(QEMUFile *f, void *opaque)
 {
+    int rc = 0;
   VFIOPCIDevice *vdev = opaque;
-    qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+
+    if (vfio_device_data_cap_device_memory(vdev)) {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP | VFIO_SAVE_FLAG_CONTINUE);
+        qemu_put_byte(f, VFIO_SAVE_FLAG_DEVMEMORY);
+        /* get whole snapshot of device memory */
+        vfio_get_device_memory_size(vdev);
+        rc = vfio_save_data_device_memory(vdev, f);
+    } else {
+        qemu_put_byte(f, VFIO_SAVE_FLAG_SETUP);
+    }
 
   vfio_set_device_state(vdev, VFIO_DEVICE_STATE_RUNNING |
           VFIO_DEVICE_STATE_LOGGING);
-    return 0;
+    return rc;
 }
 
 static int vfio_load_setup(QEMUFile *f, void *opaque)
@@ -576,8 +798,11 @@ int vfio_migration_init(VFIOPCIDevice *vdev, Error **errp)
       goto error;
   }
 
-    if (vfio_device_data_cap_device_memory(vdev)) {
-        error_report("No suppport of data cap device memory Yet");
+    if (vfio_device_data_cap_device_memory(vdev) &&
+            vfio_device_state_region_setup(vdev,
+              &vdev->migration->region[VFIO_DEVSTATE_REGION_DATA_DEVICE_MEMORY],
+              VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_MEMORY,
+              "device-state-data-device-memory")) {
       goto error;
   }
 
hw/vfio/pci.h +1 -0
index 4b7b1bb..a2cc64b 100644
@@ -69,6 +69,7 @@ typedef struct VFIOMigration {
   uint32_t data_caps;
   uint32_t device_state;
   uint64_t devconfig_size;
+    uint64_t devmem_size;
   VMChangeStateEntry *vm_state;
 } VFIOMigration;
 
Signature · 6 lines
2.7.4

_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

#22 lore raw
In reply to Yan Zhao
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Hi,
I've sent minor comments to later patches; but some minor general
comments:

a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.
b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?
c) Consider using the trace_ mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.

Dave

(P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'

Quoted 264 lines

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;    /*rw*/
	} device_config;
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;     /* rw */  
                __u64 pos; /*the offset in total buffer of device memory*/
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING & VFIO_DEVICE_STATE_LOGGING |
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

Quoted 13 lines
 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c
Signature · 1 line
2.7.4

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#23 lore raw
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Quoted 6 lines

Hi,
I've sent minor comments to later patches; but some minor general
comments:

a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.

hi Dave
Thanks for this suggestion. I'll add more checks for migration streams.

Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
This patchset is intended to settle down the main device state interfaces for VFIO migration. So that we can work on that and improve it.

Quoted 5 lines

c) Consider using the trace_ mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.

Dave

Got it. many thanks~~

Quoted 2 lines

(P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'

sorry :)

Quoted 284 lines
Quoted 265 lines

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;    /*rw*/
	} device_config;
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;     /* rw */  
                __u64 pos; /*the offset in total buffer of device memory*/
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING & VFIO_DEVICE_STATE_LOGGING |
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

Quoted 13 lines
 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c
Signature · 1 line
2.7.4

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

#24 lore raw
Quoted 34 lines · * Zhao Yan ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Quoted 6 lines

Hi,
I've sent minor comments to later patches; but some minor general
comments:

a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.

hi Dave
Thanks for this suggestion. I'll add more checks for migration streams.

Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
This patchset is intended to settle down the main device state interfaces for VFIO migration. So that we can work on that and improve it.

Quoted 5 lines

c) Consider using the trace_ mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.

Dave

Quoted 7 lines

Got it. many thanks~~

Quoted 2 lines

(P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'

sorry :)

No problem.

Given the mails, I'm guessing you've mostly tested this on graphics devices? Have you also checked with VFIO network cards?

Also see the mail I sent in reply to Kirti's series; we need to boil these down to one solution.

Dave

Quoted 278 lines
Quoted 278 lines
Quoted 278 lines

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;    /*rw*/
	} device_config;
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;     /* rw */  
                __u64 pos; /*the offset in total buffer of device memory*/
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING & VFIO_DEVICE_STATE_LOGGING |
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c
Signature · 1 line
2.7.4
Quoted 6 lines
Quoted 6 lines

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#25 lore raw
Quoted 16 lines

-----Original Message-----
From: Dr. David Alan Gilbert [mailto:[email protected]]
Sent: Wednesday, February 20, 2019 7:02 PM
To: Zhao Yan <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 26 lines
Quoted 25 lines · * Zhao Yan ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Quoted 6 lines

Hi,
I've sent minor comments to later patches; but some minor general
comments:

a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.

hi Dave
Thanks for this suggestion. I'll add more checks for migration streams.

Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Quoted 348 lines
Quoted 9 lines

This patchset is intended to settle down the main device state interfaces for VFIO migration. So that we can work on that and improve it.

Quoted 5 lines

c) Consider using the trace_ mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.

Dave

Quoted 7 lines

Got it. many thanks~~

Quoted 2 lines

(P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'

sorry :)

No problem.

Given the mails, I'm guessing you've mostly tested this on graphics devices? Have you also checked with VFIO network cards?

Also see the mail I sent in reply to Kirti's series; we need to boil these down to one solution.

Dave

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Device Memory: device's internal memory, standalone and outside

system

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system

memory.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of

system

Quoted 1 line
Quoted 1 line
Quoted 1 line

memory. This dirty bitmap is queried in pre-copy and

stop-and-copy

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by

ram's

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

live migration code.
The dirty bitmap of system memory is held in dirty bitmap

region.

Quoted 1 line
Quoted 1 line
Quoted 1 line

If system memory range is larger than that dirty bitmap region

can

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another

two

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

optional regions if it plans to support device state management.

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system

memory

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

dirty pages

(The reason why four seperate regions are defined is that the unit of

mmap

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big

region

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines

padded and sparse mmaped).

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
Quoted 8 lines
Quoted 8 lines
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */
		__u64 size;    /*rw*/
	} device_config;
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */
		__u64 size;     /* rw */
                __u64 pos; /*the offset in total buffer of device

memory*/

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING &

LOGGING,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then

vendor

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return

whole

Quoted 1 line
Quoted 1 line
Quoted 1 line

snapshot outside LOGGING and dirty data since last get

operation

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

inside LOGGING.
Device config should be always accessible and return whole

config

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Then it will probe VFIO device's data capability by reading caps field of control region.

        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load

data of

Quoted 1 line
Quoted 1 line
Quoted 1 line

device memory in pre-copy and stop-and-copy phase. The data

of

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty

pages

Quoted 1 line
Quoted 1 line
Quoted 1 line

produced by VFIO device during pre-copy and stop-and-copy

phase.

Quoted 1 line
Quoted 1 line
Quoted 1 line

The dirty bitmap of system memory is held in dirty bitmap

region.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE

STATE)

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size) get device memory buffer (whole snapshot data) set device state --> VFIO_DEVICE_STATE_RUNNING &

VFIO_DEVICE_STATE_LOGGING

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

|
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state -->

VFIO_DEVICE_STATE_STOP

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE

STATE)

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config

size,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of

control

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action

fields of

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy

phase

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

In precopy phase, if a device has

VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

.log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr",

"system_memory.page_nr", and

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

"system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback

just

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will

be

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858

++++++++++++++++++++++++++++++++++++++++++

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c

--
2.7.4

Quoted 6 lines
Quoted 6 lines

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#26 lore raw
Quoted 28 lines · On Wed, 20 Feb 2019 11:28:46 +0000 "Gonglei (Arei)" <[email protected]>…
Quoted 28 lines

-----Original Message-----
From: Dr. David Alan Gilbert [mailto:[email protected]]
Sent: Wednesday, February 20, 2019 7:02 PM
To: Zhao Yan <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 12 lines · * Zhao Yan ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

#27 lore raw
Quoted 63 lines

-----Original Message-----
From: Cornelia Huck [mailto:[email protected]]
Sent: Wednesday, February 20, 2019 7:43 PM
To: Gonglei (Arei) <[email protected]>
Cc: Dr. David Alan Gilbert <[email protected]>; Zhao Yan <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 5 lines · On Wed, 20 Feb 2019 11:28:46 +0000 "Gonglei (Arei)" <[email protected]>…
Quoted 5 lines

-----Original Message-----
From: Dr. David Alan Gilbert [mailto:[email protected]]
Sent: Wednesday, February 20, 2019 7:02 PM
To: Zhao Yan <[email protected]>
Cc: [email protected]; [email protected]; [email protected];

Quoted 8 lines
Quoted 8 lines

[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected]

Quoted 9 lines
Quoted 9 lines

Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 6 lines · * Zhao Yan ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Yan Zhao ([email protected]) wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration

patches,

Quoted 11 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

including device states, VFIO device state region layout, dirty bitmap's query.

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public

cloud

Quoted 1 line

providers. But we still think about the live migration between from the the

lower

Quoted 1 line

generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

Quoted 8 lines

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

We can provide an interface to query if the VM support live migration or not in prepare phase of Libvirt.

Can we get the revision_id from the vendor driver ? before invoking

register_savevm_live(NULL, TYPE_VFIO_PCI, -1,

            revision_id,
            &savevm_vfio_handlers,
            vdev);

then limit the live migration form higher gens to lower gens?

Regards,
-Gonglei

author
#28 lore raw
In reply to Cornelia Huck
Quoted 14 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

Quoted 8 lines

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".

struct vfio_device_state_ctl {
     __u32 version;            /* ro */
     __u8 device_version[MAX_DEVICE_VERSION_LEN];            /* ro */
     struct {
     	__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
	...
     }data;
     ...

};

Then, an action IS_COMPATIBLE is added to check device compatibility.

The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.

The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.

Thanks
Yan

#29 lore raw
Quoted 57 lines · * Zhao Yan ([email protected]) wrote:
Quoted 14 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

Quoted 8 lines

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.
2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".

struct vfio_device_state_ctl {
     __u32 version;            /* ro */
     __u8 device_version[MAX_DEVICE_VERSION_LEN];            /* ro */
     struct {
     	__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
	...
     }data;
     ...

};

Then, an action IS_COMPATIBLE is added to check device compatibility.

The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.

Quoted 6 lines

The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.

It would also be good if the 'IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.

Dave

Quoted 5 lines

Thanks
Yan

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#30 lore raw
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

Quoted 7 lines
Quoted 7 lines

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

Quoted 3 lines
Quoted 3 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

For vendor specific regions.

Quoted 5 lines
Quoted 5 lines

2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".

Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl {
     __u32 version;            /* ro */
     __u8 device_version[MAX_DEVICE_VERSION_LEN];            /* ro */
     struct {
     	__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
	...
     }data;
     ...

};

We have a buffer area where we can read and write data from the vendor driver, why would we have this IS_COMPATIBLE protocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's use GET_VERSION, CHECK_VERSION actions that make use of the buffer area and allow vendor specific version information length.

Quoted 10 lines
Quoted 10 lines

Then, an action IS_COMPATIBLE is added to check device compatibility.

The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.

Quoted 6 lines
Quoted 6 lines

The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.

I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.

Quoted 4 lines

It would also be good if the 'IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.

I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,

Alex

author
#31 lore raw

hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.

Quoted 31 lines · Alex Williamson
Quoted 28 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 27 lines · * Zhao Yan ([email protected]) wrote:
Quoted 14 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

Quoted 8 lines

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

yes, good idea!
so, what about declaring below new cap?

    #define VFIO_REGION_INFO_CAP_MIGRATION 4
    struct vfio_region_info_cap_migration {
        struct vfio_info_cap_header header;
        __u32 device_version_len;
        __u8  device_version[];
    };
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Quoted 8 lines

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

For vendor specific regions.

Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what is VFIO_REGION_TYPE_PCI_VENDOR_MASK used for? why it's 0xffff? I searched QEMU and kernel code and did not find anywhere uses it.

Quoted 15 lines
Quoted 5 lines
Quoted 5 lines

2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".

Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl {
     __u32 version;            /* ro */
     __u8 device_version[MAX_DEVICE_VERSION_LEN];            /* ro */
     struct {
     	__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
	...
     }data;
     ...

};

Quoted 6 lines

We have a buffer area where we can read and write data from the vendor driver, why would we have this IS_COMPATIBLE protocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's use GET_VERSION, CHECK_VERSION actions that make use of the buffer area and allow vendor specific version information length.

you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. use GET_VERSION action, and read from region buffer
2. get it when querying cap VFIO_REGION_INFO_CAP_MIGRATION

seems approach 1 is better, and cap VFIO_REGION_INFO_CAP_MIGRATION is only for checking migration interface's version?

Quoted 17 lines
Quoted 10 lines
Quoted 10 lines

Then, an action IS_COMPATIBLE is added to check device compatibility.

The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.

Quoted 6 lines
Quoted 6 lines

The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.

Quoted 3 lines

I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.

:)

Quoted 4 lines
Quoted 4 lines

It would also be good if the 'IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.

so qemu needs to expose two qmp/sysfs interfaces: GET_VERSION and CHECK_VERSION. GET_VERSION returns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?

Quoted 8 lines

I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex

do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version

Thanks
Yan

#32 lore raw
Quoted 145 lines · Zhao Yan

hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.

Quoted 31 lines · Alex Williamson
Quoted 28 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 27 lines · * Zhao Yan ([email protected]) wrote:
Quoted 14 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

Quoted 8 lines

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

yes, good idea!
so, what about declaring below new cap?

    #define VFIO_REGION_INFO_CAP_MIGRATION 4
    struct vfio_region_info_cap_migration {
        struct vfio_info_cap_header header;
        __u32 device_version_len;
        __u8  device_version[];
    };
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Quoted 8 lines

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

For vendor specific regions.

Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what is VFIO_REGION_TYPE_PCI_VENDOR_MASK used for? why it's 0xffff? I searched QEMU and kernel code and did not find anywhere uses it.

Quoted 15 lines
Quoted 5 lines
Quoted 5 lines

2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".

Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl {
     __u32 version;            /* ro */
     __u8 device_version[MAX_DEVICE_VERSION_LEN];            /* ro */
     struct {
     	__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
	...
     }data;
     ...

};

Quoted 6 lines

We have a buffer area where we can read and write data from the vendor driver, why would we have this IS_COMPATIBLE protocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's use GET_VERSION, CHECK_VERSION actions that make use of the buffer area and allow vendor specific version information length.

you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. use GET_VERSION action, and read from region buffer
2. get it when querying cap VFIO_REGION_INFO_CAP_MIGRATION

seems approach 1 is better, and cap VFIO_REGION_INFO_CAP_MIGRATION is only for checking migration interface's version?

Quoted 17 lines
Quoted 10 lines
Quoted 10 lines

Then, an action IS_COMPATIBLE is added to check device compatibility.

The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.

Quoted 6 lines
Quoted 6 lines

The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.

Quoted 3 lines

I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.

:)

Quoted 4 lines
Quoted 4 lines

It would also be good if the 'IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.

so qemu needs to expose two qmp/sysfs interfaces: GET_VERSION and CHECK_VERSION. GET_VERSION returns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?

Quoted 8 lines

I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex

Quoted 4 lines

do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version

Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?

Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.

On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.

So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.

Regards,
Erik

#33 lore raw
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan

hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.

Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

Quoted 7 lines
Quoted 7 lines

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

Quoted 8 lines
Quoted 8 lines

yes, good idea!
so, what about declaring below new cap?

    #define VFIO_REGION_INFO_CAP_MIGRATION 4
    struct vfio_region_info_cap_migration {
        struct vfio_info_cap_header header;
        __u32 device_version_len;
        __u8  device_version[];
    };

I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

For vendor specific regions.

Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what is VFIO_REGION_TYPE_PCI_VENDOR_MASK used for? why it's 0xffff? I searched QEMU and kernel code and did not find anywhere uses it.

PCI vendor IDs are 16bits, it's just indicating that when the PCI_VENDOR_TYPE bit is set the valid data is the lower 16bits.

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 5 lines
Quoted 5 lines

2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".

Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl {
     __u32 version;            /* ro */
     __u8 device_version[MAX_DEVICE_VERSION_LEN];            /* ro */
     struct {
     	__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
	...
     }data;
     ...

};

We have a buffer area where we can read and write data from the vendor driver, why would we have this IS_COMPATIBLE protocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's use GET_VERSION, CHECK_VERSION actions that make use of the buffer area and allow vendor specific version information length.

Quoted 7 lines
Quoted 7 lines

you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. use GET_VERSION action, and read from region buffer
2. get it when querying cap VFIO_REGION_INFO_CAP_MIGRATION

seems approach 1 is better, and cap VFIO_REGION_INFO_CAP_MIGRATION is only for checking migration interface's version?

I think 1 provides the most flexibility to the vendor driver.

Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines

Then, an action IS_COMPATIBLE is added to check device compatibility.

The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.

Quoted 6 lines
Quoted 6 lines

The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.

Quoted 3 lines

I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.

:)

Quoted 4 lines
Quoted 4 lines

It would also be good if the 'IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.

Quoted 5 lines
Quoted 5 lines

so qemu needs to expose two qmp/sysfs interfaces: GET_VERSION and CHECK_VERSION. GET_VERSION returns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?

That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex

Quoted 4 lines
Quoted 4 lines

do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version

I'd think it might live more in the mdev_support_types area, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:

/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version

Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.

Quoted 16 lines

Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?

Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.

Quoted 8 lines

On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.

So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.

Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,

Alex

author
#34 lore raw
Quoted 57 lines · Alex Williamson
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan

hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.

Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

Quoted 7 lines
Quoted 7 lines

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

Quoted 8 lines
Quoted 8 lines

yes, good idea!
so, what about declaring below new cap?

    #define VFIO_REGION_INFO_CAP_MIGRATION 4
    struct vfio_region_info_cap_migration {
        struct vfio_info_cap_header header;
        __u32 device_version_len;
        __u8  device_version[];
    };
Quoted 4 lines

I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.

so, to get version of VFIO live migration device state interface (simply call it migration interface?),
a new cap looks like this:
#define VFIO_REGION_INFO_CAP_MIGRATION 4
it contains struct vfio_info_cap_header only.
when get region info of the migration region, we query this cap and get migration interface's version. right?

or just directly use VFIO_REGION_INFO_CAP_TYPE is fine?

Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

For vendor specific regions.

Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what is VFIO_REGION_TYPE_PCI_VENDOR_MASK used for? why it's 0xffff? I searched QEMU and kernel code and did not find anywhere uses it.

Quoted 2 lines

PCI vendor IDs are 16bits, it's just indicating that when the PCI_VENDOR_TYPE bit is set the valid data is the lower 16bits.

thanks:)

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines

2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".

Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl {
     __u32 version;            /* ro */
     __u8 device_version[MAX_DEVICE_VERSION_LEN];            /* ro */
     struct {
     	__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
	...
     }data;
     ...

};

Quoted 6 lines

We have a buffer area where we can read and write data from the vendor driver, why would we have this IS_COMPATIBLE protocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's use GET_VERSION, CHECK_VERSION actions that make use of the buffer area and allow vendor specific version information length.

you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. use GET_VERSION action, and read from region buffer
2. get it when querying cap VFIO_REGION_INFO_CAP_MIGRATION

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

seems approach 1 is better, and cap VFIO_REGION_INFO_CAP_MIGRATION is only for checking migration interface's version?

I think 1 provides the most flexibility to the vendor driver.

Got it.
For VFIO live migration, compared to reuse device state region (which takes GET_BUFFER/SET_BUFFER actions),
could we create a new region for GET_VERSION & CHECK_VERSION ?

Quoted 49 lines
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines

Then, an action IS_COMPATIBLE is added to check device compatibility.

The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.

Quoted 6 lines
Quoted 6 lines

The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.

Quoted 3 lines

I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.

:)

Quoted 4 lines
Quoted 4 lines

It would also be good if the 'IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.

Quoted 5 lines
Quoted 5 lines

so qemu needs to expose two qmp/sysfs interfaces: GET_VERSION and CHECK_VERSION. GET_VERSION returns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?

That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex

Quoted 4 lines
Quoted 4 lines

do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version

Quoted 8 lines

I'd think it might live more in the mdev_support_types area, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:

/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version

Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.

yes, knowing if a device is compatible before it's created is good. but do you think check whether a device is compatible after it's created is also required? For live migration, user usually only queries this information when it's really required, i.e. when a device has been created. maybe we can add this version get/check at both places?

Quoted 26 lines
Quoted 17 lines

Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?

Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.

Quoted 8 lines

On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.

So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.

Quoted 11 lines

Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,

So QEMU vfio uses sysfs to check device compatiblity in migration's save_setup phase?

Thanks
Yan

#35 lore raw
Quoted 70 lines · On Thu, 28 Mar 2019 22:47:04 -0400 Zhao Yan <[email protected]> wrote:
Quoted 57 lines · Alex Williamson
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan

hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.

Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

Quoted 7 lines
Quoted 7 lines

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

Quoted 8 lines
Quoted 8 lines

yes, good idea!
so, what about declaring below new cap?

    #define VFIO_REGION_INFO_CAP_MIGRATION 4
    struct vfio_region_info_cap_migration {
        struct vfio_info_cap_header header;
        __u32 device_version_len;
        __u8  device_version[];
    };  
Quoted 4 lines

I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.

so, to get version of VFIO live migration device state interface (simply call it migration interface?),
a new cap looks like this:
#define VFIO_REGION_INFO_CAP_MIGRATION 4
it contains struct vfio_info_cap_header only.
when get region info of the migration region, we query this cap and get migration interface's version. right?

Quoted 1 line

or just directly use VFIO_REGION_INFO_CAP_TYPE is fine?

Again, why a new region. I'm imagining we have one region and this is just asking for a slightly different thing from it. But TBH, I'm not sure we need it at all vs the sysfs interface.

Quoted 53 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

For vendor specific regions.

Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what is VFIO_REGION_TYPE_PCI_VENDOR_MASK used for? why it's 0xffff? I searched QEMU and kernel code and did not find anywhere uses it.

Quoted 2 lines

PCI vendor IDs are 16bits, it's just indicating that when the PCI_VENDOR_TYPE bit is set the valid data is the lower 16bits.

thanks:)

Quoted 29 lines
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines

2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".

Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl {
     __u32 version;            /* ro */
     __u8 device_version[MAX_DEVICE_VERSION_LEN];            /* ro */
     struct {
     	__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
	...
     }data;
     ...

};

Quoted 6 lines

We have a buffer area where we can read and write data from the vendor driver, why would we have this IS_COMPATIBLE protocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's use GET_VERSION, CHECK_VERSION actions that make use of the buffer area and allow vendor specific version information length.

you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. use GET_VERSION action, and read from region buffer
2. get it when querying cap VFIO_REGION_INFO_CAP_MIGRATION

seems approach 1 is better, and cap VFIO_REGION_INFO_CAP_MIGRATION is only for checking migration interface's version?

Quoted 6 lines
Quoted 1 line

I think 1 provides the most flexibility to the vendor driver.

Got it.
For VFIO live migration, compared to reuse device state region (which takes GET_BUFFER/SET_BUFFER actions),
could we create a new region for GET_VERSION & CHECK_VERSION ?

Why?

Quoted 55 lines
Quoted 55 lines
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines

Then, an action IS_COMPATIBLE is added to check device compatibility.

The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.

Quoted 6 lines
Quoted 6 lines

The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.

Quoted 3 lines

I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.

:)

Quoted 4 lines
Quoted 4 lines

It would also be good if the 'IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.

Quoted 5 lines
Quoted 5 lines

so qemu needs to expose two qmp/sysfs interfaces: GET_VERSION and CHECK_VERSION. GET_VERSION returns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?

That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex

Quoted 4 lines
Quoted 4 lines

do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version

I'd think it might live more in the mdev_support_types area, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:

/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version

Quoted 7 lines
Quoted 2 lines

Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.

yes, knowing if a device is compatible before it's created is good. but do you think check whether a device is compatible after it's created is also required? For live migration, user usually only queries this information when it's really required, i.e. when a device has been created. maybe we can add this version get/check at both places?

Why does an instantiated device suddenly not follow the version and compatibility rules of an uninstantiated device? IOW, if the version and compatibility check are on the mdev type, why can't we trace back from the device to the mdev type and make use of that same interface? Seems the only question is whether we require an interface through the vfio API directly or if sysfs is sufficient.

Quoted 36 lines
Quoted 25 lines
Quoted 16 lines

Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?

Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.

Quoted 8 lines

On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.

So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.

Quoted 11 lines

Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,

Quoted 2 lines

So QEMU vfio uses sysfs to check device compatiblity in migration's save_setup phase?

The migration stream between source and target device are the ultimate test of compatibility, the vendor driver should never rely on userspace validating compatibility of the migration. At the point it could do so, the migration has already begun, so we're only testing how quickly we can fail the migration. The management layer setting up the migration can test via sysfs for compatibility and the migration stream itself needs to be self validating, so what value is added for QEMU to perform a version compatibility test? Thanks,

Alex

author
#36 lore raw
Quoted 204 lines · Alex Williamson
Quoted 70 lines · On Thu, 28 Mar 2019 22:47:04 -0400 Zhao Yan <[email protected]> wrote:
Quoted 57 lines · Alex Williamson
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan

hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.

Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps
a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

Quoted 7 lines
Quoted 7 lines

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

Quoted 8 lines
Quoted 8 lines

yes, good idea!
so, what about declaring below new cap?

    #define VFIO_REGION_INFO_CAP_MIGRATION 4
    struct vfio_region_info_cap_migration {
        struct vfio_info_cap_header header;
        __u32 device_version_len;
        __u8  device_version[];
    };  
Quoted 4 lines

I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.

so, to get version of VFIO live migration device state interface (simply call it migration interface?),
a new cap looks like this:
#define VFIO_REGION_INFO_CAP_MIGRATION 4
it contains struct vfio_info_cap_header only.
when get region info of the migration region, we query this cap and get migration interface's version. right?

Quoted 1 line

or just directly use VFIO_REGION_INFO_CAP_TYPE is fine?

Again, why a new region. I'm imagining we have one region and this is just asking for a slightly different thing from it. But TBH, I'm not sure we need it at all vs the sysfs interface.

Quoted 53 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

For vendor specific regions.

Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what is VFIO_REGION_TYPE_PCI_VENDOR_MASK used for? why it's 0xffff? I searched QEMU and kernel code and did not find anywhere uses it.

Quoted 2 lines

PCI vendor IDs are 16bits, it's just indicating that when the PCI_VENDOR_TYPE bit is set the valid data is the lower 16bits.

thanks:)

Quoted 29 lines
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines

2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".

Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl {
     __u32 version;            /* ro */
     __u8 device_version[MAX_DEVICE_VERSION_LEN];            /* ro */
     struct {
     	__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
	...
     }data;
     ...

};

Quoted 6 lines

We have a buffer area where we can read and write data from the vendor driver, why would we have this IS_COMPATIBLE protocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's use GET_VERSION, CHECK_VERSION actions that make use of the buffer area and allow vendor specific version information length.

you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. use GET_VERSION action, and read from region buffer
2. get it when querying cap VFIO_REGION_INFO_CAP_MIGRATION

seems approach 1 is better, and cap VFIO_REGION_INFO_CAP_MIGRATION is only for checking migration interface's version?

Quoted 6 lines
Quoted 1 line

I think 1 provides the most flexibility to the vendor driver.

Got it.
For VFIO live migration, compared to reuse device state region (which takes GET_BUFFER/SET_BUFFER actions),
could we create a new region for GET_VERSION & CHECK_VERSION ?

Why?

Quoted 55 lines
Quoted 55 lines
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines

Then, an action IS_COMPATIBLE is added to check device compatibility.

The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.

Quoted 6 lines
Quoted 6 lines

The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.

Quoted 3 lines

I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.

:)

Quoted 4 lines
Quoted 4 lines

It would also be good if the 'IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.

Quoted 5 lines
Quoted 5 lines

so qemu needs to expose two qmp/sysfs interfaces: GET_VERSION and CHECK_VERSION. GET_VERSION returns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?

That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex

Quoted 4 lines
Quoted 4 lines

do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version

I'd think it might live more in the mdev_support_types area, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:

/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version

Quoted 7 lines
Quoted 2 lines

Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.

yes, knowing if a device is compatible before it's created is good. but do you think check whether a device is compatible after it's created is also required? For live migration, user usually only queries this information when it's really required, i.e. when a device has been created. maybe we can add this version get/check at both places?

Quoted 6 lines

Why does an instantiated device suddenly not follow the version and compatibility rules of an uninstantiated device? IOW, if the version and compatibility check are on the mdev type, why can't we trace back from the device to the mdev type and make use of that same interface? Seems the only question is whether we require an interface through the vfio API directly or if sysfs is sufficient.

ok. got it.

Quoted 39 lines
Quoted 36 lines
Quoted 25 lines
Quoted 16 lines

Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?

Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.

Quoted 8 lines

On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.

So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.

Quoted 11 lines

Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,

Quoted 2 lines

So QEMU vfio uses sysfs to check device compatiblity in migration's save_setup phase?

Quoted 8 lines

The migration stream between source and target device are the ultimate test of compatibility, the vendor driver should never rely on userspace validating compatibility of the migration. At the point it could do so, the migration has already begun, so we're only testing how quickly we can fail the migration. The management layer setting up the migration can test via sysfs for compatibility and the migration stream itself needs to be self validating, so what value is added for QEMU to perform a version compatibility test? Thanks,

oh, do you mean vendor driver should embed source device's version in migration stream, which is opaque to qemu?
otherwise, I can't think of a quick way for vendor driver to determine whether source device is an incompatible device.

Thanks
Yan

#37 lore raw
Quoted 260 lines · On Fri, 29 Mar 2019 19:10:50 -0400 Zhao Yan <[email protected]> wrote:
Quoted 204 lines · Alex Williamson
Quoted 70 lines · On Thu, 28 Mar 2019 22:47:04 -0400 Zhao Yan <[email protected]> wrote:
Quoted 57 lines · Alex Williamson
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan

hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.

Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps
a device that has less device memory ?
Actually it's still an open for VFIO migration. Need to think about
whether it's better to check that in libvirt or qemu (like a device magic
along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

Quoted 7 lines
Quoted 7 lines

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

Quoted 8 lines
Quoted 8 lines

yes, good idea!
so, what about declaring below new cap?

    #define VFIO_REGION_INFO_CAP_MIGRATION 4
    struct vfio_region_info_cap_migration {
        struct vfio_info_cap_header header;
        __u32 device_version_len;
        __u8  device_version[];
    };    
Quoted 4 lines

I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.

so, to get version of VFIO live migration device state interface (simply call it migration interface?),
a new cap looks like this:
#define VFIO_REGION_INFO_CAP_MIGRATION 4
it contains struct vfio_info_cap_header only.
when get region info of the migration region, we query this cap and get migration interface's version. right?

Quoted 1 line

or just directly use VFIO_REGION_INFO_CAP_TYPE is fine?

Again, why a new region. I'm imagining we have one region and this is just asking for a slightly different thing from it. But TBH, I'm not sure we need it at all vs the sysfs interface.

Quoted 53 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

For vendor specific regions.

Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what is VFIO_REGION_TYPE_PCI_VENDOR_MASK used for? why it's 0xffff? I searched QEMU and kernel code and did not find anywhere uses it.

Quoted 2 lines

PCI vendor IDs are 16bits, it's just indicating that when the PCI_VENDOR_TYPE bit is set the valid data is the lower 16bits.

thanks:)

Quoted 29 lines
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines

2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".

Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl {
     __u32 version;            /* ro */
     __u8 device_version[MAX_DEVICE_VERSION_LEN];            /* ro */
     struct {
     	__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
	...
     }data;
     ...

};

Quoted 6 lines

We have a buffer area where we can read and write data from the vendor driver, why would we have this IS_COMPATIBLE protocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's use GET_VERSION, CHECK_VERSION actions that make use of the buffer area and allow vendor specific version information length.

you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. use GET_VERSION action, and read from region buffer
2. get it when querying cap VFIO_REGION_INFO_CAP_MIGRATION

seems approach 1 is better, and cap VFIO_REGION_INFO_CAP_MIGRATION is only for checking migration interface's version?

Quoted 6 lines
Quoted 1 line

I think 1 provides the most flexibility to the vendor driver.

Got it.
For VFIO live migration, compared to reuse device state region (which takes GET_BUFFER/SET_BUFFER actions),
could we create a new region for GET_VERSION & CHECK_VERSION ?

Why?

Quoted 55 lines
Quoted 55 lines
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines

Then, an action IS_COMPATIBLE is added to check device compatibility.

The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.

Quoted 6 lines
Quoted 6 lines

The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.

Quoted 3 lines

I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.

:)

Quoted 4 lines
Quoted 4 lines

It would also be good if the 'IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.

Quoted 5 lines
Quoted 5 lines

so qemu needs to expose two qmp/sysfs interfaces: GET_VERSION and CHECK_VERSION. GET_VERSION returns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?

That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex

Quoted 4 lines
Quoted 4 lines

do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version

I'd think it might live more in the mdev_support_types area, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:

/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version

Quoted 7 lines
Quoted 2 lines

Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.

yes, knowing if a device is compatible before it's created is good. but do you think check whether a device is compatible after it's created is also required? For live migration, user usually only queries this information when it's really required, i.e. when a device has been created. maybe we can add this version get/check at both places?

Quoted 6 lines

Why does an instantiated device suddenly not follow the version and compatibility rules of an uninstantiated device? IOW, if the version and compatibility check are on the mdev type, why can't we trace back from the device to the mdev type and make use of that same interface? Seems the only question is whether we require an interface through the vfio API directly or if sysfs is sufficient.

ok. got it.

Quoted 39 lines
Quoted 36 lines
Quoted 25 lines
Quoted 16 lines

Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?

Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.

Quoted 8 lines

On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.

So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.

Quoted 11 lines

Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,

Quoted 2 lines

So QEMU vfio uses sysfs to check device compatiblity in migration's save_setup phase?

Quoted 8 lines

The migration stream between source and target device are the ultimate test of compatibility, the vendor driver should never rely on userspace validating compatibility of the migration. At the point it could do so, the migration has already begun, so we're only testing how quickly we can fail the migration. The management layer setting up the migration can test via sysfs for compatibility and the migration stream itself needs to be self validating, so what value is added for QEMU to perform a version compatibility test? Thanks,

Quoted 4 lines

oh, do you mean vendor driver should embed source device's version in migration stream, which is opaque to qemu?
otherwise, I can't think of a quick way for vendor driver to determine whether source device is an incompatible device.

Yes, the vendor driver cannot rely on the user to make sure the incoming migration stream is compatible, the vendor driver must take responsibility for this. Therefore, regardless of what other interfaces we have for the user to test the compatibility between devices, the vendor driver must make no assumptions about the validity or integrity of the data stream. Plan for and protect against a malicious or incompetent user. Thanks,

Alex

author
#38 lore raw
Quoted 268 lines · Alex Williamson
Quoted 260 lines · On Fri, 29 Mar 2019 19:10:50 -0400 Zhao Yan <[email protected]> wrote:
Quoted 204 lines · Alex Williamson
Quoted 70 lines · On Thu, 28 Mar 2019 22:47:04 -0400 Zhao Yan <[email protected]> wrote:
Quoted 57 lines · Alex Williamson
Quoted 45 lines · On Thu, 28 Mar 2019 10:21:38 +0100 Erik Skultety <[email protected]> wrote:
Quoted 44 lines · Zhao Yan

hi Alex and Dave,
Thanks for your replies.
Please see my comments inline.

Quoted 39 lines · Alex Williamson
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps
a device that has less device memory ?
Actually it's still an open for VFIO migration. Need to think about
whether it's better to check that in libvirt or qemu (like a device magic
along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

Quoted 7 lines
Quoted 7 lines

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

Quoted 8 lines
Quoted 8 lines

yes, good idea!
so, what about declaring below new cap?

    #define VFIO_REGION_INFO_CAP_MIGRATION 4
    struct vfio_region_info_cap_migration {
        struct vfio_info_cap_header header;
        __u32 device_version_len;
        __u8  device_version[];
    };    
Quoted 4 lines

I'm not sure why we need a new region for everything, it seems this could fit within the protocol of a single region. This could simply be a new action to retrieve the version where the protocol would report the number of bytes available, just like the migration stream itself.

so, to get version of VFIO live migration device state interface (simply call it migration interface?),
a new cap looks like this:
#define VFIO_REGION_INFO_CAP_MIGRATION 4
it contains struct vfio_info_cap_header only.
when get region info of the migration region, we query this cap and get migration interface's version. right?

Quoted 1 line

or just directly use VFIO_REGION_INFO_CAP_TYPE is fine?

Again, why a new region. I'm imagining we have one region and this is just asking for a slightly different thing from it. But TBH, I'm not sure we need it at all vs the sysfs interface.

Quoted 53 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

For vendor specific regions.

Yes, use PCI vendor ID.
you are right, we need to use highest bit (VFIO_REGION_TYPE_PCI_VENDOR_TYPE) to identify it's a PCI ID.
Thanks for pointing it out.
But, I have a question. what is VFIO_REGION_TYPE_PCI_VENDOR_MASK used for? why it's 0xffff? I searched QEMU and kernel code and did not find anywhere uses it.

Quoted 2 lines

PCI vendor IDs are 16bits, it's just indicating that when the PCI_VENDOR_TYPE bit is set the valid data is the lower 16bits.

thanks:)

Quoted 29 lines
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines

2. vendor proprietary string: it can be any string that a vendor driver thinks can identify a source device. e.g. pciid + mdev type. "vendor id" is to avoid overlap of "vendor proprietary string".

Quoted 9 lines
Quoted 9 lines
struct vfio_device_state_ctl {
     __u32 version;            /* ro */
     __u8 device_version[MAX_DEVICE_VERSION_LEN];            /* ro */
     struct {
     	__u32 action; /* GET_BUFFER, SET_BUFFER, IS_COMPATIBLE*/
	...
     }data;
     ...

};

Quoted 6 lines

We have a buffer area where we can read and write data from the vendor driver, why would we have this IS_COMPATIBLE protocol to write the device version string but use a static fixed length version string in the control header to read it? IOW, let's use GET_VERSION, CHECK_VERSION actions that make use of the buffer area and allow vendor specific version information length.

you are right, such static fixed length version string is bad :) To get device version, do you think which approach below is better?
1. use GET_VERSION action, and read from region buffer
2. get it when querying cap VFIO_REGION_INFO_CAP_MIGRATION

seems approach 1 is better, and cap VFIO_REGION_INFO_CAP_MIGRATION is only for checking migration interface's version?

Quoted 6 lines
Quoted 1 line

I think 1 provides the most flexibility to the vendor driver.

Got it.
For VFIO live migration, compared to reuse device state region (which takes GET_BUFFER/SET_BUFFER actions),
could we create a new region for GET_VERSION & CHECK_VERSION ?

Why?

Quoted 55 lines
Quoted 55 lines
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines

Then, an action IS_COMPATIBLE is added to check device compatibility.

The flow to figure out whether a source device is migratable to target device is like that:
1. in source side's .save_setup, save source device's device_version string
2. in target side's .load_state, load source device's device version string and write it to data region, and call IS_COMPATIBLE action to ask vendor driver to check whether the source device is compatible to it.

Quoted 6 lines
Quoted 6 lines

The advantage of adding an IS_COMPATIBLE action is that, vendor driver can maintain a compatibility table and decide whether source device is compatible to target device according to its proprietary table.
In device_version string, vendor driver only has to describe the source device as elaborately as possible and resorts to vendor driver in target side to figure out whether they are compatible.

Quoted 3 lines

I agree, it's too complicated and restrictive to try to create an interface for the user to determine compatibility, let the driver declare it compatible or not.

:)

Quoted 4 lines
Quoted 4 lines

It would also be good if the 'IS_COMPATIBLE' was somehow callable externally - so we could be able to answer a question like 'can we migrate this VM to this host' - from the management layer before it actually starts the migration.

Quoted 5 lines
Quoted 5 lines

so qemu needs to expose two qmp/sysfs interfaces: GET_VERSION and CHECK_VERSION. GET_VERSION returns a vm's device's version string.
CHECK_VERSION's input is device version string and return compatible/non-compatible.
Do you think it's good?

That's the idea, but note that QEMU can only provide the QMP interface, the sysfs interface would of course be provided as more of a direct path from the vendor driver or mdev kernel layer.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I think we'd need to mirror this capability in sysfs to support that, or create a qmp interface through QEMU that the device owner could make the request on behalf of the management layer. Getting access to the vfio device requires an iommu context that's already in use by the device owner, we have no intention of supporting a model that allows independent tasks access to a device. Thanks,
Alex

Quoted 4 lines
Quoted 4 lines

do you think two sysfs nodes under a device node is ok?
e.g. /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/get_version /sys/devices/pci0000\:00/0000\:00\:02.0/882cc4da-dede-11e7-9180-078a62063ab1/check_version

I'd think it might live more in the mdev_support_types area, wouldn't we ideally like to know if a device is compatible even before it's created? For example maybe:

/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_4/version

Quoted 7 lines
Quoted 2 lines

Where reading the sysfs attribute returns the version string and writing a string into the attribute return an errno for incompatibility.

yes, knowing if a device is compatible before it's created is good. but do you think check whether a device is compatible after it's created is also required? For live migration, user usually only queries this information when it's really required, i.e. when a device has been created. maybe we can add this version get/check at both places?

Quoted 6 lines

Why does an instantiated device suddenly not follow the version and compatibility rules of an uninstantiated device? IOW, if the version and compatibility check are on the mdev type, why can't we trace back from the device to the mdev type and make use of that same interface? Seems the only question is whether we require an interface through the vfio API directly or if sysfs is sufficient.

ok. got it.

Quoted 39 lines
Quoted 36 lines
Quoted 25 lines
Quoted 16 lines

Why do you need both sysfs and QMP at the same time? I can see it gives us some flexibility, but is there something more to that?

Normally, I'd prefer a QMP interface from libvirt's perspective (with an
appropriate capability that libvirt can check for QEMU support) because I imagine large nodes having a bunch of GPUs with different revisions which might not be backwards compatible. Libvirt might query the version string on source and check it on dest via the
QMP in a way that QEMU, talking to the driver, would return either a list or a
single physical device to which we can migrate, because neither QEMU nor
libvirt know that, only the driver does, so that's an important information
rather than looping through all the devices and trying to find one that is compatible. However, you might have a hard time making all the necessary
changes in QMP introspectable, a new command would be fine, but if you also
wanted to extend say vfio-pci options, IIRC that would not appear in the QAPI
schema and libvirt would not be able to detect support for it.

Quoted 8 lines

On the other hand, the presence of a QMP interface IMO doesn't help mgmt apps much, as it still carries the burden of being able to check this only at the time of migration, which e.g. OpenStack would like to know long before that.

So, having sysfs attributes would work for both libvirt (even though libvirt would benefit from a QMP much more) and OpenStack. OpenStack would IMO then have to figure out how to create the mappings between compatible devices across several nodes which are non-uniform.

Quoted 11 lines

Yep, vfio encompasses more than just QEMU, so a sysfs interface has more utility than a QMP interface. For instance we couldn't predetermine if an mdev type on a host is compatible if we need to first create the device and launch a QEMU instance on it to get access to QMP. So maybe the question is whether we should bother with any sort of VFIO API to do this comparison, perhaps only a sysfs interface is sufficient for a complete solution. The downside of not having a version API in the user interface might be that QEMU on its own can only try a migration and see if it fails, it wouldn't have the ability to test expected compatibility without access to sysfs. And maybe that's fine. Thanks,

Quoted 2 lines

So QEMU vfio uses sysfs to check device compatiblity in migration's save_setup phase?

Quoted 8 lines

The migration stream between source and target device are the ultimate test of compatibility, the vendor driver should never rely on userspace validating compatibility of the migration. At the point it could do so, the migration has already begun, so we're only testing how quickly we can fail the migration. The management layer setting up the migration can test via sysfs for compatibility and the migration stream itself needs to be self validating, so what value is added for QEMU to perform a version compatibility test? Thanks,

Quoted 4 lines

oh, do you mean vendor driver should embed source device's version in migration stream, which is opaque to qemu?
otherwise, I can't think of a quick way for vendor driver to determine whether source device is an incompatible device.

Quoted 8 lines

Yes, the vendor driver cannot rely on the user to make sure the incoming migration stream is compatible, the vendor driver must take responsibility for this. Therefore, regardless of what other interfaces we have for the user to test the compatibility between devices, the vendor driver must make no assumptions about the validity or integrity of the data stream. Plan for and protect against a malicious or incompetent user. Thanks,

ok. got it.
Thank you :)

Yan

#39 lore raw
In reply to Alex Williamson
Quoted 44 lines · On Wed, 27 Mar 2019 16:10:20 -0600 Alex Williamson…
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

Quoted 7 lines
Quoted 7 lines

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

Quoted 3 lines
Quoted 3 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Quoted 8 lines

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

For vendor specific regions.

Just browsing through the thread... if I don't misunderstand, we could use a vfio-ccw region type id here for ccw, couldn't we? Just to make sure that this is not pci-specific.

author
#40 lore raw
Quoted 51 lines · Cornelia Huck
Quoted 48 lines · On Wed, 27 Mar 2019 16:10:20 -0600 Alex Williamson…
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

Quoted 7 lines
Quoted 7 lines

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

Quoted 3 lines
Quoted 3 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

Quoted 8 lines
Quoted 4 lines

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

For vendor specific regions.

Just browsing through the thread... if I don't misunderstand, we could use a vfio-ccw region type id here for ccw, couldn't we? Just to make sure that this is not pci-specific.

CCW could use another bit other than bit 31?
e.g.
#define VFIO_REGION_TYPE_CCW_VENDOR_TYPE (1 << 30)
then ccw device use (VFIO_REGION_TYPE_CCW_VENDOR_TYPE | vendor id) as its first 32 bit for device version string.

But as Alex said we'll not provide an extra region to get device version, and device version is only exported in sysfs, probably we should define them as below:
#define VFIO_DEVICE_VERSION_TYPE_PCI (1<<31)
#define VFIO_DEVICE_VERSION_TYPE_CCW (1<<30)

Do you think it's ok?

Thanks
Yan

#41 lore raw
Quoted 66 lines · On Mon, 1 Apr 2019 04:40:03 -0400 Yan Zhao <[email protected]> wrote:
Quoted 51 lines · Cornelia Huck
Quoted 48 lines · On Wed, 27 Mar 2019 16:10:20 -0600 Alex Williamson…
Quoted 24 lines · On Wed, 27 Mar 2019 20:18:54 +0000 "Dr. David Alan Gilbert"…
Quoted 23 lines · * Zhao Yan ([email protected]) wrote:
Quoted 22 lines · Cornelia Huck
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).

We must keep the hardware generation is the same with one POD of public cloud providers. But we still think about the live migration between from the the lower generation of hardware migrated to the higher generation.

Agreed, lower->higher is the one direction that might make sense to support.

But regardless of that, I think we need to make sure that incompatible devices/versions fail directly instead of failing in a subtle, hard to debug way. Might be useful to do some initial sanity checks in libvirt as well.

How easy is it to obtain that information in a form that can be consumed by higher layers? Can we find out the device type at least? What about some kind of revision?

Quoted 7 lines
Quoted 7 lines

hi Alex and Cornelia
for device compatibility, do you think it's a good idea to use "version" and "device version" fields?

version field: identify live migration interface's version. it can have a sort of backward compatibility, like target machine's version >= source machine's version. something like that.

Don't we essentially already have this via the device specific region? The struct vfio_info_cap_header includes id and version fields, so we can declare a migration id and increment the version for any incompatible changes to the protocol.

Quoted 3 lines
Quoted 3 lines

device_version field consists two parts:
1. vendor id : it takes 32 bits. e.g. 0x8086.

Who allocates IDs? If we're going to use PCI vendor IDs, then I'd suggest we use a bit to flag it as such so we can reserve that portion of the 32bit address space. See for example:

Quoted 8 lines
Quoted 4 lines

#define VFIO_REGION_TYPE_PCI_VENDOR_TYPE (1 << 31)
#define VFIO_REGION_TYPE_PCI_VENDOR_MASK (0xffff)

For vendor specific regions.

Just browsing through the thread... if I don't misunderstand, we could use a vfio-ccw region type id here for ccw, couldn't we? Just to make sure that this is not pci-specific.

CCW could use another bit other than bit 31?
e.g.
#define VFIO_REGION_TYPE_CCW_VENDOR_TYPE (1 << 30)
then ccw device use (VFIO_REGION_TYPE_CCW_VENDOR_TYPE | vendor id) as its first 32 bit for device version string.

Quoted 7 lines

But as Alex said we'll not provide an extra region to get device version, and device version is only exported in sysfs, probably we should define them as below:
#define VFIO_DEVICE_VERSION_TYPE_PCI (1<<31)
#define VFIO_DEVICE_VERSION_TYPE_CCW (1<<30)

Do you think it's ok?

We already had this discussion for device specific regions and decided that CCW doesn't have enough vendors to justify a full subset of the available address space. Also, this doesn't need to imply the device interface, we're simply specifying a vendor registrar such that we can give each vendor their own namespace, so I don't think it would be a problem for a CCW to specify a namespace using a PCI vendor ID. Finally, if we have such need for this in the future, because I'm not sure where we stand with this in the current proposals, maybe we should make use of an IEEE OUI rather than a PCI database to avoid this sort of confusion and mis-association if we have further need. Thanks,

Alex

author
#42 lore raw
In reply to Dr. David Alan Gilbert
Quoted 41 lines · Dr. David Alan Gilbert
Quoted 40 lines · * Zhao Yan ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Quoted 6 lines

Hi,
I've sent minor comments to later patches; but some minor general
comments:

a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.

hi Dave
Thanks for this suggestion. I'll add more checks for migration streams.

Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
This patchset is intended to settle down the main device state interfaces for VFIO migration. So that we can work on that and improve it.

Quoted 5 lines

c) Consider using the trace_ mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.

Dave

Got it. many thanks~~

Quoted 2 lines

(P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'

Quoted 7 lines
Quoted 1 line

sorry :)

No problem.

Given the mails, I'm guessing you've mostly tested this on graphics devices? Have you also checked with VFIO network cards?

yes, I tested it on Intel's graphics devices which do not have device memory. so the cap of device-memory is off.
I believe this patchset can work well on VFIO network cards as well, because Gonglei once said their NIC can work well on our previous code (i.e. device-memory cap off).

Quoted 3 lines

Also see the mail I sent in reply to Kirti's series; we need to boil these down to one solution.

Maybe Kirti can merge their implementaion into the code for device-memory cap (like in my patch 5 for device-memory).

Quoted 292 lines

Dave

Quoted 278 lines
Quoted 278 lines
Quoted 278 lines

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;    /*rw*/
	} device_config;
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;     /* rw */  
                __u64 pos; /*the offset in total buffer of device memory*/
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING & VFIO_DEVICE_STATE_LOGGING |
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c
Signature · 1 line
2.7.4
Quoted 6 lines
Quoted 6 lines

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

#43 lore raw
Quoted 46 lines · * Zhao Yan ([email protected]) wrote:
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 34 lines · * Zhao Yan ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Yan Zhao ([email protected]) wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Quoted 6 lines

Hi,
I've sent minor comments to later patches; but some minor general
comments:

a) Never trust the incoming migrations stream - it might be corrupt, so check when you can.

hi Dave
Thanks for this suggestion. I'll add more checks for migration streams.

Quoted 3 lines

b) How do we detect if we're migrating from/to the wrong device or
version of device? Or say to a device with older firmware or perhaps a device that has less device memory ?

Actually it's still an open for VFIO migration. Need to think about whether it's better to check that in libvirt or qemu (like a device magic along with verion ?).
This patchset is intended to settle down the main device state interfaces for VFIO migration. So that we can work on that and improve it.

Quoted 5 lines

c) Consider using the trace_ mechanism - it's really useful to
add to loops writing/reading data so that you can see when it fails.

Dave

Quoted 7 lines

Got it. many thanks~~

Quoted 2 lines

(P.S. You have a few typo's grep your code for 'devcie', 'devie' and 'migrtion'

sorry :)

No problem.

Quoted 8 lines
Quoted 3 lines

Given the mails, I'm guessing you've mostly tested this on graphics devices? Have you also checked with VFIO network cards?

yes, I tested it on Intel's graphics devices which do not have device memory. so the cap of device-memory is off.
I believe this patchset can work well on VFIO network cards as well, because Gonglei once said their NIC can work well on our previous code (i.e. device-memory cap off).

It would be great if you could find some Intel NIC people to help test it out.

Quoted 6 lines
Quoted 4 lines

Also see the mail I sent in reply to Kirti's series; we need to boil these down to one solution.

Maybe Kirti can merge their implementaion into the code for device-memory cap (like in my patch 5 for device-memory).

It would be great to come up with one patchset between yourself and Kirti that was tested for Intel and Nvidia GPUs and Intel NICs (and anyone else who wants to jump on board!).

Dave

Quoted 286 lines
Quoted 286 lines

Dave

Quoted 284 lines
Quoted 284 lines
Quoted 265 lines

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;    /*rw*/
	} device_config;
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;     /* rw */  
                __u64 pos; /*the offset in total buffer of device memory*/
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING & VFIO_DEVICE_STATE_LOGGING |
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

Quoted 13 lines
 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c
Signature · 1 line
2.7.4

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

Quoted 6 lines
Quoted 6 lines

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
_______________________________________________
intel-gvt-dev mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#44 lore raw
In reply to Yan Zhao

Hi yan,

Thanks for your work.

I have some suggestions or questions:

1) Would you add msix mode support,? if not, pls add a check in vfio_pci_save_config(), likes Nvidia's solution.
2) We should start vfio devices before vcpu resumes, so we can't rely on vm start change handler completely.
3) We'd better support live migration rollback since have many failure scenarios,
register a migration notifier is a good choice.
4) Four memory region for live migration is too complicated IMHO.
5) About log sync, why not register log_global_start/stop in vfio_memory_listener?

Regards,
-Gonglei

Quoted 305 lines

-----Original Message-----
From: Yan Zhao [mailto:[email protected]]
Sent: Tuesday, February 19, 2019 4:51 PM
To: [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; [email protected]; Yan Zhao <[email protected]>
Subject: [PATCH 0/5] QEMU VFIO live migration

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */
		__u64 size;    /*rw*/
	} device_config;
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */
		__u64 size;     /* rw */
                __u64 pos; /*the offset in total buffer of device memory*/
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING &
LOGGING,
STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING &
VFIO_DEVICE_STATE_LOGGING
|
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase
like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on,
.log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and
"system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858
++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c

--
2.7.4

author
#45 lore raw
Quoted 7 lines · Gonglei (Arei)

Hi yan,

Thanks for your work.

I have some suggestions or questions:

1) Would you add msix mode support,? if not, pls add a check in vfio_pci_save_config(), likes Nvidia's solution.

ok.

Quoted 1 line

2) We should start vfio devices before vcpu resumes, so we can't rely on vm start change handler completely.

vfio devices is by default set to running state.
In the target machine, its state transition flow is running->stop->running. so, maybe you can ignore the stop notification in kernel?

Quoted 2 lines

3) We'd better support live migration rollback since have many failure scenarios, register a migration notifier is a good choice.

I think this patchset can also handle the failure case well. if migration failure or cancelling happens,
in cleanup handler, LOGGING state is cleared. device state(running or stopped) keeps as it is).
then,
if vm switches back to running, device state will be set to running; if vm stayes at stopped state, device state is also stopped (it has no meaning to let it in running state).
Do you think so ?

Quoted 1 line

4) Four memory region for live migration is too complicated IMHO.

one big region requires the sub-regions well padded.
like for the first control fields, they have to be padded to 4K. the same for other data fields.
Otherwise, mmap simply fails, because the start-offset and size for mmap both need to be PAGE aligned.

Also, 4 regions is clearer in my view :)

Quoted 3 lines

5) About log sync, why not register log_global_start/stop in vfio_memory_listener?

seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?

Quoted 310 lines

Regards,
-Gonglei

Quoted 305 lines

-----Original Message-----
From: Yan Zhao [mailto:[email protected]]
Sent: Tuesday, February 19, 2019 4:51 PM
To: [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; [email protected]; Yan Zhao <[email protected]>
Subject: [PATCH 0/5] QEMU VFIO live migration

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

Device Memory: device's internal memory, standalone and outside system
memory. It is usually very big.
This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.
device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */
		__u64 size;    /*rw*/
	} device_config;
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */
		__u64 size;     /* rw */
                __u64 pos; /*the offset in total buffer of device memory*/
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING &
LOGGING,
STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING &
VFIO_DEVICE_STATE_LOGGING
|
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.
Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).
.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase
like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on,
.log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and
"system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858
++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c

--
2.7.4

#46 lore raw
Quoted 24 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 8:25 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 6 lines · Gonglei (Arei)

Hi yan,

Thanks for your work.

I have some suggestions or questions:

Quoted 8 lines
Quoted 1 line

1) Would you add msix mode support,? if not, pls add a check in

vfio_pci_save_config(), likes Nvidia's solution.
ok.

Quoted 1 line

2) We should start vfio devices before vcpu resumes, so we can't rely on vm

start change handler completely.
vfio devices is by default set to running state.
In the target machine, its state transition flow is running->stop->running.

That's confusing. We should start vfio devices after vfio_load_state, otherwise how can you keep the devices' information are the same between source side and destination side?

Quoted 8 lines

so, maybe you can ignore the stop notification in kernel?

Quoted 1 line

3) We'd better support live migration rollback since have many failure

scenarios,

Quoted 1 line

register a migration notifier is a good choice.

I think this patchset can also handle the failure case well. if migration failure or cancelling happens,
in cleanup handler, LOGGING state is cleared. device state(running or stopped) keeps as it is).

IIRC there're many failure paths don't calling cleanup handler.

Quoted 6 lines

then,
if vm switches back to running, device state will be set to running; if vm stayes at stopped state, device state is also stopped (it has no meaning to let it in running state).
Do you think so ?

IF the underlying state machine is complicated,
We should tell the canceling state to vendor driver proactively.

Quoted 7 lines
Quoted 1 line

4) Four memory region for live migration is too complicated IMHO.

one big region requires the sub-regions well padded.
like for the first control fields, they have to be padded to 4K. the same for other data fields.
Otherwise, mmap simply fails, because the start-offset and size for mmap both need to be PAGE aligned.

But if we don't need use mmap for control filed and device state, they are small basically. The performance is enough using pread/pwrite.

Quoted 10 lines

Also, 4 regions is clearer in my view :)

Quoted 1 line

5) About log sync, why not register log_global_start/stop in

vfio_memory_listener?

Quoted 2 lines

seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?

We just need invoking only once for start and stop logging. Why we need to call them literately? See memory_listener of vhost.

Regards,
-Gonglei

author
#47 lore raw
Quoted 28 lines · Gonglei (Arei)
Quoted 28 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 8:25 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 7 lines · Gonglei (Arei)

Hi yan,

Thanks for your work.

I have some suggestions or questions:

1) Would you add msix mode support,? if not, pls add a check in

vfio_pci_save_config(), likes Nvidia's solution.
ok.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

2) We should start vfio devices before vcpu resumes, so we can't rely on vm

start change handler completely.
vfio devices is by default set to running state.
In the target machine, its state transition flow is running->stop->running.

That's confusing. We should start vfio devices after vfio_load_state, otherwise how can you keep the devices' information are the same between source side and destination side?

so, your meaning is to set device state to running in the first call to vfio_load_state?

Quoted 11 lines
Quoted 8 lines

so, maybe you can ignore the stop notification in kernel?

Quoted 1 line

3) We'd better support live migration rollback since have many failure

scenarios,

Quoted 1 line

register a migration notifier is a good choice.

I think this patchset can also handle the failure case well. if migration failure or cancelling happens,
in cleanup handler, LOGGING state is cleared. device state(running or stopped) keeps as it is).

IIRC there're many failure paths don't calling cleanup handler.

could you take an example?

Quoted 9 lines
Quoted 6 lines

then,
if vm switches back to running, device state will be set to running; if vm stayes at stopped state, device state is also stopped (it has no meaning to let it in running state).
Do you think so ?

IF the underlying state machine is complicated,
We should tell the canceling state to vendor driver proactively.

That makes sense.

Quoted 10 lines
Quoted 7 lines
Quoted 1 line

4) Four memory region for live migration is too complicated IMHO.

one big region requires the sub-regions well padded.
like for the first control fields, they have to be padded to 4K. the same for other data fields.
Otherwise, mmap simply fails, because the start-offset and size for mmap both need to be PAGE aligned.

But if we don't need use mmap for control filed and device state, they are small basically. The performance is enough using pread/pwrite.

we don't mmap control fields. but if data fields going immedately after control fields (e.g. just 64 bytes), we can't mmap data fields successfully because its start offset is 64. Therefore control fields have to be padded to 4k to let data fields start from 4k.
That's the drawback of one big region holding both control and data fields.

Quoted 19 lines
Quoted 10 lines

Also, 4 regions is clearer in my view :)

Quoted 1 line

5) About log sync, why not register log_global_start/stop in

vfio_memory_listener?

Quoted 2 lines

seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?

We just need invoking only once for start and stop logging. Why we need to call them literately? See memory_listener of vhost.

Regards,
-Gonglei

#48 lore raw
Quoted 53 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 9:59 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 10 lines · Gonglei (Arei)
Quoted 10 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 8:25 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 3 lines
Quoted 3 lines

[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 9 lines
Quoted 9 lines

[email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 4 lines · Gonglei (Arei)

Hi yan,

Thanks for your work.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

I have some suggestions or questions:

1) Would you add msix mode support,? if not, pls add a check in

vfio_pci_save_config(), likes Nvidia's solution.
ok.

Quoted 1 line

2) We should start vfio devices before vcpu resumes, so we can't rely on

vm

Quoted 4 lines
Quoted 3 lines

start change handler completely.
vfio devices is by default set to running state.
In the target machine, its state transition flow is running->stop->running.

Quoted 8 lines
Quoted 1 line

That's confusing. We should start vfio devices after vfio_load_state,

otherwise

Quoted 3 lines

how can you keep the devices' information are the same between source side and destination side?

so, your meaning is to set device state to running in the first call to vfio_load_state?

No, it should start devices after vfio_load_state and before vcpu resuming.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

so, maybe you can ignore the stop notification in kernel?

Quoted 1 line

3) We'd better support live migration rollback since have many failure

scenarios,

Quoted 1 line

register a migration notifier is a good choice.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

I think this patchset can also handle the failure case well. if migration failure or cancelling happens,
in cleanup handler, LOGGING state is cleared. device state(running or stopped) keeps as it is).

IIRC there're many failure paths don't calling cleanup handler.

could you take an example?

Never mind, that's another bug I think.

Quoted 48 lines
Quoted 9 lines
Quoted 6 lines

then,
if vm switches back to running, device state will be set to running; if vm stayes at stopped state, device state is also stopped (it has no meaning to let it in running state).
Do you think so ?

IF the underlying state machine is complicated,
We should tell the canceling state to vendor driver proactively.

That makes sense.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

4) Four memory region for live migration is too complicated IMHO.

one big region requires the sub-regions well padded.
like for the first control fields, they have to be padded to 4K. the same for other data fields.
Otherwise, mmap simply fails, because the start-offset and size for mmap both need to be PAGE aligned.

But if we don't need use mmap for control filed and device state, they are

small basically.

Quoted 2 lines

The performance is enough using pread/pwrite.

we don't mmap control fields. but if data fields going immedately after control fields (e.g. just 64 bytes), we can't mmap data fields successfully because its start offset is 64. Therefore control fields have to be padded to 4k to let data fields start from 4k.
That's the drawback of one big region holding both control and data fields.

Quoted 6 lines
Quoted 6 lines

Also, 4 regions is clearer in my view :)

Quoted 1 line

5) About log sync, why not register log_global_start/stop in

vfio_memory_listener?

Quoted 2 lines
Quoted 6 lines
Quoted 4 lines

seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?

We just need invoking only once for start and stop logging. Why we need to

call

Quoted 7 lines

them literately? See memory_listener of vhost.

Regards,
-Gonglei

author
#49 lore raw
Quoted 54 lines · Gonglei (Arei)
Quoted 54 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 9:59 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 10 lines · Gonglei (Arei)
Quoted 10 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 8:25 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 3 lines
Quoted 3 lines

[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 9 lines
Quoted 9 lines

[email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 4 lines · Gonglei (Arei)

Hi yan,

Thanks for your work.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

I have some suggestions or questions:

1) Would you add msix mode support,? if not, pls add a check in

vfio_pci_save_config(), likes Nvidia's solution.
ok.

Quoted 1 line

2) We should start vfio devices before vcpu resumes, so we can't rely on

vm

Quoted 5 lines
Quoted 3 lines

start change handler completely.
vfio devices is by default set to running state.
In the target machine, its state transition flow is running->stop->running.

That's confusing. We should start vfio devices after vfio_load_state,

Quoted 9 lines
Quoted 7 lines

otherwise

Quoted 3 lines

how can you keep the devices' information are the same between source side and destination side?

so, your meaning is to set device state to running in the first call to vfio_load_state?

No, it should start devices after vfio_load_state and before vcpu resuming.

What about set device state to running in load_cleanup handler ?

Quoted 67 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

so, maybe you can ignore the stop notification in kernel?

Quoted 1 line

3) We'd better support live migration rollback since have many failure

scenarios,

Quoted 1 line

register a migration notifier is a good choice.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

I think this patchset can also handle the failure case well. if migration failure or cancelling happens,
in cleanup handler, LOGGING state is cleared. device state(running or stopped) keeps as it is).

IIRC there're many failure paths don't calling cleanup handler.

could you take an example?

Never mind, that's another bug I think.

Quoted 34 lines
Quoted 9 lines
Quoted 6 lines

then,
if vm switches back to running, device state will be set to running; if vm stayes at stopped state, device state is also stopped (it has no meaning to let it in running state).
Do you think so ?

IF the underlying state machine is complicated,
We should tell the canceling state to vendor driver proactively.

That makes sense.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

4) Four memory region for live migration is too complicated IMHO.

one big region requires the sub-regions well padded.
like for the first control fields, they have to be padded to 4K. the same for other data fields.
Otherwise, mmap simply fails, because the start-offset and size for mmap both need to be PAGE aligned.

But if we don't need use mmap for control filed and device state, they are

small basically.

Quoted 2 lines

The performance is enough using pread/pwrite.

we don't mmap control fields. but if data fields going immedately after control fields (e.g. just 64 bytes), we can't mmap data fields successfully because its start offset is 64. Therefore control fields have to be padded to 4k to let data fields start from 4k.
That's the drawback of one big region holding both control and data fields.

Quoted 6 lines
Quoted 6 lines

Also, 4 regions is clearer in my view :)

Quoted 1 line

5) About log sync, why not register log_global_start/stop in

vfio_memory_listener?

Quoted 2 lines
Quoted 14 lines
Quoted 6 lines
Quoted 4 lines

seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?

We just need invoking only once for start and stop logging. Why we need to

call

Quoted 7 lines

them literately? See memory_listener of vhost.

Regards,
-Gonglei

_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

#50 lore raw
Quoted 85 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 12:08 PM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 9 lines · Gonglei (Arei)
Quoted 9 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 9:59 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 3 lines
Quoted 3 lines

[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 29 lines
Quoted 29 lines

[email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 10 lines · Gonglei (Arei)
Quoted 10 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 8:25 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 3 lines
Quoted 3 lines

[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 9 lines
Quoted 9 lines

[email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 4 lines · Gonglei (Arei)

Hi yan,

Thanks for your work.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

I have some suggestions or questions:

1) Would you add msix mode support,? if not, pls add a check in

vfio_pci_save_config(), likes Nvidia's solution. ok.

Quoted 1 line

2) We should start vfio devices before vcpu resumes, so we can't rely

on

Quoted 4 lines
Quoted 4 lines

vm

Quoted 3 lines
Quoted 3 lines

start change handler completely.
vfio devices is by default set to running state. In the target machine, its state transition flow is

running->stop->running.

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

That's confusing. We should start vfio devices after vfio_load_state,

otherwise

Quoted 1 line

how can you keep the devices' information are the same between source

side

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

and destination side?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

so, your meaning is to set device state to running in the first call to vfio_load_state?

No, it should start devices after vfio_load_state and before vcpu resuming.

What about set device state to running in load_cleanup handler ?

The timing is fine, but you should also think about if should set device state to running in failure branches when calling load_cleanup handler.

Regards,
-Gonglei

author
#51 lore raw
In reply to Gonglei (Arei)
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

5) About log sync, why not register log_global_start/stop in

vfio_memory_listener?

Quoted 2 lines
Quoted 8 lines
Quoted 4 lines

seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?

We just need invoking only once for start and stop logging. Why we need to call them literately? See memory_listener of vhost.

the dirty pages in system memory produces by device is incremental. if it can be got iteratively, the dirty pages in stop-and-copy phase can be minimal.
:)

Quoted 2 lines

Regards,
-Gonglei

#52 lore raw
Quoted 28 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 10:05 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 10 lines
Quoted 9 lines
Quoted 2 lines

5) About log sync, why not register log_global_start/stop in

vfio_memory_listener?

Quoted 2 lines

seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?

Quoted 9 lines
Quoted 1 line

We just need invoking only once for start and stop logging. Why we need to

call

Quoted 2 lines

them literately? See memory_listener of vhost.

the dirty pages in system memory produces by device is incremental. if it can be got iteratively, the dirty pages in stop-and-copy phase can be minimal.
:)

I mean starting or stopping the capability of logging, not log sync.

We register the below callbacks:

.log_sync = vfio_log_sync,
.log_global_start = vfio_log_global_start,
.log_global_stop = vfio_log_global_stop,

Regards,
-Gonglei

author
#53 lore raw
Quoted 37 lines · Gonglei (Arei)
Quoted 28 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 10:05 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 10 lines
Quoted 9 lines
Quoted 2 lines

5) About log sync, why not register log_global_start/stop in

vfio_memory_listener?

Quoted 2 lines

seems log_global_start/stop cannot be iterately called in pre-copy phase? for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?

Quoted 9 lines
Quoted 1 line

We just need invoking only once for start and stop logging. Why we need to

call

Quoted 2 lines

them literately? See memory_listener of vhost.

the dirty pages in system memory produces by device is incremental. if it can be got iteratively, the dirty pages in stop-and-copy phase can be minimal.
:)

Quoted 8 lines

I mean starting or stopping the capability of logging, not log sync.

We register the below callbacks:

.log_sync = vfio_log_sync,
.log_global_start = vfio_log_global_start,
.log_global_stop = vfio_log_global_stop,

.log_global_start is also a good point to notify logging state. But if notifying in .save_setup handler, we can do fine-grained control of when to notify of logging starting together with get_buffer operation.
Is there any special benifit by registering to .log_global_start/stop?

Quoted 2 lines

Regards,
-Gonglei

#54 lore raw
Quoted 47 lines
Quoted 9 lines
Quoted 9 lines

-----Original Message-----
From: Zhao Yan [mailto:[email protected]]
Sent: Thursday, February 21, 2019 10:05 AM
To: Gonglei (Arei) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 3 lines
Quoted 3 lines

[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 10 lines
Quoted 10 lines

[email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/5] QEMU VFIO live migration

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

5) About log sync, why not register log_global_start/stop in

vfio_memory_listener?

Quoted 2 lines

seems log_global_start/stop cannot be iterately called in pre-copy

phase?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

for dirty pages in system memory, it's better to transfer dirty data iteratively to reduce down time, right?

We just need invoking only once for start and stop logging. Why we need

to

Quoted 8 lines
Quoted 8 lines

call

Quoted 2 lines

them literately? See memory_listener of vhost.

the dirty pages in system memory produces by device is incremental. if it can be got iteratively, the dirty pages in stop-and-copy phase can be minimal.
:)

Quoted 8 lines

I mean starting or stopping the capability of logging, not log sync.

We register the below callbacks:

.log_sync = vfio_log_sync,
.log_global_start = vfio_log_global_start,
.log_global_stop = vfio_log_global_stop,

Quoted 6 lines

.log_global_start is also a good point to notify logging state. But if notifying in .save_setup handler, we can do fine-grained control of when to notify of logging starting together with get_buffer operation.
Is there any special benifit by registering to .log_global_start/stop?

Performance benefit when one VM has multiple same vfio devices.

Regards,
-Gonglei

#55 lore raw
In reply to Yan Zhao

Hi Yan,

Thanks for working on this!

Quoted 7 lines · On Tue, 19 Feb 2019 16:50:54 +0800 Yan Zhao <[email protected]> wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Quoted 8 lines

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it

I'm not sure how we can really impose a limit here, it is what it is for a device. A smaller state is obviously desirable to reduce downtime, but some devices could have very large states.

Quoted 6 lines

needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

So the intention here is that this is the last data read from the device and it's done in one pass, so the region needs to be large enough to expose all config data at once. On restore it's the last data written before switching the device to the run state.

Quoted 2 lines

Device Memory: device's internal memory, standalone and outside system

s/system/VM/

Quoted 1 line

memory. It is usually very big.

Or it doesn't exist. Not sure we should be setting expectations since it will vary per device.

Quoted 7 lines

This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

It seems a little gratuitous to me that this is a separate region or that this data is handled separately. All of this data is opaque to QEMU, so why do we need to separate it?

Quoted 17 lines

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

Quoted 5 lines

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.

Is mmap mandatory? I would think this would be defined by the mdev device what access they want to support per region. We don't want to impose a more complicated interface if the device doesn't require it.

Quoted 9 lines

device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

It's not obvious to me how this is better, a big region isn't padded, there's simply a gap in the file descriptor. Is having a sub-PAGE_SIZE gap in a file really of any consequence? Each region beyond the header is more than likely larger than PAGE_SIZE, therefore they can be nicely aligned together. We still need fields to tell us how much data is available in each area, so another to tell us the start of each area is a minor detail. And I think we still want to allow drivers to specify which parts of which areas support mmap, so I don't think we're getting away from sparse mmap support.

Quoted 5 lines

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If we were to go with this multi-region solution, isn't it evident from the regions exposed that device memory and a dirty bitmap are provided? Alternatively, I believe Kirti's proposal doesn't require this distinction between device memory and device config, a device not requiring runtime migration data would simply report no data until the device moved to the stopped state, making it consistent path for userspace. Likewise, the dirty bitmap could report a zero page count in the bitmap rather than branching based on device support.

Note that it seems that cap VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY implies VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP, but there's no consistency in the naming.

Quoted 3 lines

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

It looks like these are being defined as bits, since patch 1 talks about RUNNING & LOGGING and STOP & LOGGING. I think Connie already posted some comments about this. I'm not sure anything prevents us from defining RUNNING a 1 and STOPPED as 0 so we don't have the polarity flip vs LOGGING though.

The state "STOP & LOGGING" also feels like a strange "device state", if the device is stopped, it's not logging any new state, so I think this is more that the device state is STOP, but the LOGGING feature is active. Maybe we should consider these independent bits. LOGGING is active as we stop a device so that we can fetch the last dirtied pages, but disabled as we load the state of the device into the target.

Quoted 4 lines

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

Quoted 8 lines
struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;    /*rw*/
	} device_config;

Patch 1 indicates that to get the config buffer we write GET_BUFFER to action and read from the config region. The size is previously read and apparently constant. To set the config buffer, the config region is written followed by writing SET_BUFFER to action. Why is size listed as read-write?

Doesn't this protocol also require that the mdev driver consume each full region's worth of host kernel memory for backing pages in anticipation of a rare event like migration? This might be a strike against separate regions if the driver needs to provide backing pages for 3 separate regions vs 1. To avoid this runtime overhead, would it be expected that the user only mmap the regions during migration and the mdev driver allocate backing pages on mmap? Should the mmap be restricted to the LOGGING feature being enabled? Maybe I'm mistaken in how the mdev driver would back these mmap'd pages.

Quoted 4 lines
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;     /* rw */  
                __u64 pos; /*the offset in total buffer of device memory*/

Patch 1 outlines the protocol here that getting device memory begins with writing the position field, followed by reading from the device memory region. Setting device memory begins with writing the data to the device memory region, followed by writing the position field. Why does the user need to have visibility of data position? This is opaque data to the user, the device should manage how the chunks fit together.

How does the user know when they reach the end?

Bullets 8 and 9 in patch 1 also discuss setting and getting the device memory size, but these aren't well integrated into the protocol for getting and setting the memory buffer. Is getting the device memory really started by reading the size, which triggers the vendor driver to snapshot the state in an internal buffer which the user then iterates through using GET_BUFFER? Therefore re-reading the size field could corrupt the data stream? Wouldn't it be easier to report bytes available and countdown as the user marks them read? What does position mean when we switch from snapshot to dirty data?

Quoted 6 lines
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Why is one specified as an address and the other as pages? Note that Kirti's implementation has an optimization to know how many pages are set within a range to avoid unnecessary buffer reads.

Quoted 42 lines

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

Quoted 8 lines
        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

As above, these capabilities seem redundant to the existence of the device specific regions in this implementation.

Quoted 65 lines

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING & VFIO_DEVICE_STATE_LOGGING |
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

Quoted 7 lines

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.

This requires iterative reads of device memory buffer but the protocol is unclear (to me) how the user knows how to do this or interact with the position field.

Quoted 5 lines

Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).

What if there's not enough dirty data to fill the region? The data is always padded to fill the region?

Quoted 6 lines

.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

Therefore through the entire precopy phase we have no data from source to target to begin a compatibility check :-\ I think both proposals currently still lack any sort of device compatibility or data versioning check between source and target. Thanks,

Alex

Quoted 49 lines

In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c
author
#56 lore raw
Quoted 57 lines · Alex Williamson

Hi Yan,

Thanks for working on this!

Quoted 7 lines · On Tue, 19 Feb 2019 16:50:54 +0800 Yan Zhao <[email protected]> wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Quoted 8 lines

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it

I'm not sure how we can really impose a limit here, it is what it is for a device. A smaller state is obviously desirable to reduce downtime, but some devices could have very large states.

Quoted 6 lines

needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

So the intention here is that this is the last data read from the device and it's done in one pass, so the region needs to be large enough to expose all config data at once. On restore it's the last data written before switching the device to the run state.

Quoted 2 lines

Device Memory: device's internal memory, standalone and outside system

s/system/VM/

Quoted 1 line

memory. It is usually very big.

Or it doesn't exist. Not sure we should be setting expectations since it will vary per device.

Quoted 7 lines

This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

Quoted 3 lines

It seems a little gratuitous to me that this is a separate region or that this data is handled separately. All of this data is opaque to QEMU, so why do we need to separate it?

hi Alex,
as the device state interfaces are provided by kernel, it is expected to meet as general needs as possible. So, do you think there are such use cases from user space that user space knows well of the device, and it wants kernel to return desired data back to it.
E.g. It just wants to get whole device config data including all mmios, page tables, pci config data...
or, It just wants to get current device memory snapshot, not including any dirty data.
Or, It just needs the dirty pages in device memory or system memory. With all this accurate query, quite a lot of useful features can be developped in user space.

If all of this data is opaque to user app, seems the only use case is for live migration.

Quoted 1 line

From another aspect, if the final solution is to let the data opaque to

user space, like what NV did, kernel side's implementation will be more complicated, and actually a little challenge to vendor driver. in that case, in pre-copy phase,
1. in not LOGGING state, vendor driver first returns full data including full device memory snapshot
2. user space reads some data (you can't expect it to finish reading all data)
3. then userspace set the device state to LOGGING to start dirty data logging
4. vendor driver starts dirty data logging, and appends the dirty data to the tail of remaining unread full data and increase the pending data size?
5. user space keeps reading data.
6. vendor driver keeps appending new dirty data to the tail of remaining unread full data/dirty data and increase the pending data size?

in stop-and-copy phase
1. user space sets device state to exit LOGGING state,
2. vendor driver stops data logging. it has to append device config data at the tail of remaining dirty data unread by userspace.

during this flow, when vendor driver should get dirty data? just keeps logging and appends to tail? how to ensure dirty data is refresh new before LOGGING state exit? how does vendor driver know whether certain dirty data is copied or not?

I've no idea how NVidia handle this problem, and they don't open their kernel side code.
just feel it's a bit hard for other vendor drivers to follow:)

Quoted 23 lines
Quoted 17 lines

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

Quoted 5 lines

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.

Quoted 3 lines

Is mmap mandatory? I would think this would be defined by the mdev device what access they want to support per region. We don't want to impose a more complicated interface if the device doesn't require it.

I think it's "mmap is preferred, but allowed to fail".
just like a normal region with MMAP flag on (like bar regions), we also allow its mmap to fail, right?

Quoted 10 lines
Quoted 9 lines

device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

Quoted 9 lines

It's not obvious to me how this is better, a big region isn't padded, there's simply a gap in the file descriptor. Is having a sub-PAGE_SIZE gap in a file really of any consequence? Each region beyond the header is more than likely larger than PAGE_SIZE, therefore they can be nicely aligned together. We still need fields to tell us how much data is available in each area, so another to tell us the start of each area is a minor detail. And I think we still want to allow drivers to specify which parts of which areas support mmap, so I don't think we're getting away from sparse mmap support.

with seperate regions and sub-region type defined, user space can explictly know which region is which region after vfio_get_dev_region_info(). along with it, user space knows region offset and size. mmap is allowed to fail and falls back to normal read/write to the region.

But with one big region and sparse mmapped subregions (1 data subregion or 3 data subregions, whatever), userspace can't tell which subregion is which one.
So, if using one big region, I think we need to explictly define subregions' sequence (like index 0 is dedicated to control subregion, index 1 is for device config data subregion ...). Vendor driver cannot freely change the sequence.
Then keep data offset the same as region->mmaps[i].offset, and data size the same as region->mmaps[i].size. (i.e. let actual data starts immediatly from first byte of its data subregion)
Also, mmaps for sparse mmaped subregions are not allowed to fail.

With one big region, we also need to consider the case when vendor driver does not want the data subregion to be mmaped.
so, what is the data layout for that case?
data subregion immedately follows control subregion, or not?
Of couse, for this condition, we can specify the data filed's start offset and size through control region. And we must not expect the data start offset in source and target are equal.
(because the big region's fd_offset
may vary in source and target. consider the case when both source and target have one opregion and one device state region, but source has opregion in the first and target has device state region in the first. If we think this case is illegal, we must be able to detect it in the first place).
Also, we must keep the start offset and size consistent with the above mmap case.

Quoted 10 lines
Quoted 5 lines

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If we were to go with this multi-region solution, isn't it evident from the regions exposed that device memory and a dirty bitmap are provided? Alternatively, I believe Kirti's proposal doesn't require

Quoted 5 lines

this distinction between device memory and device config, a device not requiring runtime migration data would simply report no data until the device moved to the stopped state, making it consistent path for userspace. Likewise, the dirty bitmap could report a zero page count in the bitmap rather than branching based on device support.

If the path in userspace is consistent for device config and device memory, there will be many unnecessary call of getting data size into vendor driver.

Quoted 34 lines

Note that it seems that cap VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY implies VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP, but there's no consistency in the naming.

Quoted 3 lines

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

It looks like these are being defined as bits, since patch 1 talks about RUNNING & LOGGING and STOP & LOGGING. I think Connie already posted some comments about this. I'm not sure anything prevents us from defining RUNNING a 1 and STOPPED as 0 so we don't have the polarity flip vs LOGGING though.

The state "STOP & LOGGING" also feels like a strange "device state", if the device is stopped, it's not logging any new state, so I think this is more that the device state is STOP, but the LOGGING feature is active. Maybe we should consider these independent bits. LOGGING is active as we stop a device so that we can fetch the last dirtied pages, but disabled as we load the state of the device into the target.

Quoted 4 lines

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

Quoted 8 lines
struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;    /*rw*/
	} device_config;
Quoted 5 lines

Patch 1 indicates that to get the config buffer we write GET_BUFFER to action and read from the config region. The size is previously read and apparently constant. To set the config buffer, the config region is written followed by writing SET_BUFFER to action. Why is size listed as read-write?

this is the size of config data.
size of config data <= size of config data region.

Quoted 10 lines

Doesn't this protocol also require that the mdev driver consume each full region's worth of host kernel memory for backing pages in anticipation of a rare event like migration? This might be a strike against separate regions if the driver needs to provide backing pages for 3 separate regions vs 1. To avoid this runtime overhead, would it be expected that the user only mmap the regions during migration and the mdev driver allocate backing pages on mmap? Should the mmap be restricted to the LOGGING feature being enabled? Maybe I'm mistaken in how the mdev driver would back these mmap'd pages.

yes, 3 seperate regions consume a little more memory than 1 region. but it's just a little overhead.
As in intel's kernel implementation,
device config region's size is 9M, dirty bitmap region's size is 16k. if there is device memory region, its size can be defined as 100M? so it's 109M vs 100M ?

Quoted 5 lines
Quoted 4 lines
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;     /* rw */  
                __u64 pos; /*the offset in total buffer of device memory*/
Quoted 8 lines

Patch 1 outlines the protocol here that getting device memory begins with writing the position field, followed by reading from the device memory region. Setting device memory begins with writing the data to the device memory region, followed by writing the position field. Why does the user need to have visibility of data position? This is opaque data to the user, the device should manage how the chunks fit together.

How does the user know when they reach the end?

sorry, maybe I didn't explain clearly here.

device  ________________________________________
memory  |    |    |////|    |    |    |    |    |
data:   |____|____|////|____|____|____|____|____|
                  :pos :
                  :    :
device            :____:
memory            |    |
region:           |____|

the whole sequence is like this:

1. user space reads device_memory.size
2. driver gets device memory's data(full snapshot or dirty data, depending on it's in LOGGING state or not), and return the total size of this data.
3. user space finishes reading device_memory.size (>= device memory region's size)

4. user space starts a loop like

   while (pos < total_len) {
        uint64_t len = region_devmem->size;
        if (pos + len >= total_len) {
            len = total_len - pos;
        }
        if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) {
            return -1;
        }
	pos += len;
    }

vfio_save_data_device_memory_chunk reads each chunk from device memory region by writing GET_BUFFER to device_memory.action, and pos to device_memory.pos.

so. each time, userspace will finish getting device memory data in one time.

specifying "pos" is just like the "lseek" before "write".

Quoted 9 lines

Bullets 8 and 9 in patch 1 also discuss setting and getting the device memory size, but these aren't well integrated into the protocol for getting and setting the memory buffer. Is getting the device memory really started by reading the size, which triggers the vendor driver to snapshot the state in an internal buffer which the user then iterates through using GET_BUFFER? Therefore re-reading the size field could corrupt the data stream? Wouldn't it be easier to report bytes available and countdown as the user marks them read? What does position mean when we switch from snapshot to dirty data?

when switching to device memory's dirty data, pos means the pos in whole dirty data.

.save_live_pending ==> driver gets dirty data in device memory and returns total size.

.save_live_iterate ==> userspace reads all dirty data from device memory region chunk by chunk

So, in an iteration, all dirty data are saved. then in next iteration, dirty data is recalculated.

Quoted 9 lines
Quoted 7 lines
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Why is one specified as an address and the other as pages? Note that

Yes, start_addr ==> start pfn is better

Quoted 3 lines

Kirti's implementation has an optimization to know how many pages are set within a range to avoid unnecessary buffer reads.

Let's use start_pfn_all, page_nr_all to represent the start pfn and page_nr passed in from qemu .log_sync interface.

and use start_pfn_i, page_nr_i to the value passed to driver.

start_pfn_all

  |         start_pfn_i
  |         |
 \ /_______\_/_____________________________
  |    |    |////|    |    |    |    |    |
  |____|____|////|____|____|____|____|____|
            :    :
            :    :
            :____:
bitmap      |    |
region:     |____|

1. Each time QEMU queries dirty bitmap from driver, it passes in start_pfn_i, and page_nr_i. (page_nr_i is the biggest page number the bitmap region can hold).
2. driver queries memory range from start_pfn_i with size page_nr_i.
3. driver return a bitmap (if no dirty data, the bitmap is all 0).
4. QEMU saves the pages according to the bitmap

If there's no dirty data found in step 2, step 4 can be skipped. (I'll add this check before step 4 in future, thanks)
but if there's even 1 bit set in the bitmap, no step from 1-4 can be skipped.

Honestly, after reviewing Kirti's implementation, I don't think it's an optimization. As in below pseudo code for Kirti's code, I would think the copied_pfns corresponds to the page_nr_i in my case. so, the case of copied_pfns equaling to 0 is for the tail chunk? don't think it's working..

write start_pfn to driver
write page_size to driver
write pfn_count to driver

do {
    read copied_pfns from driver.
    if (copied_pfns == 0) {
       break;
    }
   bitmap_size = (BITS_TO_LONGS(copied_pfns) + 1) * sizeof(unsigned long);
   buf = get bitmap from driver
   cpu_physical_memory_set_dirty_lebitmap((unsigned long *)buf,
                                           (start_pfn + count) * page_size,
                                                copied_pfns);

     count +=  copied_pfns;

} while (count < pfn_count);

Quoted 51 lines
Quoted 42 lines

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

Quoted 8 lines
        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

Quoted 3 lines

As above, these capabilities seem redundant to the existence of the device specific regions in this implementation.

seems so :)

Quoted 77 lines
Quoted 65 lines

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING & VFIO_DEVICE_STATE_LOGGING |
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

Quoted 7 lines

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.

This requires iterative reads of device memory buffer but the protocol is unclear (to me) how the user knows how to do this or interact with the position field.

Quoted 9 lines
Quoted 5 lines

Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).

What if there's not enough dirty data to fill the region? The data is always padded to fill the region?

I think dirty data in vendor driver is orgnaized in a format like:
(addr0, len0, data0, addr1, len1, data1, addr2, len2, data2,....addrN, lenN, dataN).
for full snapshot, it's like (addr0, len0, data0).
so, to userspace and data region, it doesn't matter whether it's full snapshot or dirty data.

Quoted 11 lines
Quoted 6 lines

.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

Therefore through the entire precopy phase we have no data from source to target to begin a compatibility check :-\ I think both proposals currently still lack any sort of device compatibility or data versioning check between source and target. Thanks,

I checked the compatibility, though not good enough:)

in migration_init, vfio_check_devstate_version() checked version from kernel with VFIO_DEVICE_STATE_INTERFACE_VERSION in both source and target, and in target side, vfio_load_state() checked source side version.

int vfio_load_state(QEMUFile *f, void *opaque, int version_id)

{       
    ...
    if (version_id != VFIO_DEVICE_STATE_INTERFACE_VERSION) {                                   
        return -EINVAL;
    } 
    ...
}

Thanks
Yan

Quoted 52 lines

Alex

Quoted 49 lines

In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c
author
#57 lore raw

hi Alex
we still have some opens as below. could you kindly help review on that? :)

Thanks
Yan

Quoted 659 lines · Zhao Yan
Quoted 57 lines · Alex Williamson

Hi Yan,

Thanks for working on this!

Quoted 7 lines · On Tue, 19 Feb 2019 16:50:54 +0800 Yan Zhao <[email protected]> wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Quoted 8 lines

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it

I'm not sure how we can really impose a limit here, it is what it is for a device. A smaller state is obviously desirable to reduce downtime, but some devices could have very large states.

Quoted 6 lines

needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

So the intention here is that this is the last data read from the device and it's done in one pass, so the region needs to be large enough to expose all config data at once. On restore it's the last data written before switching the device to the run state.

Quoted 2 lines

Device Memory: device's internal memory, standalone and outside system

s/system/VM/

Quoted 1 line

memory. It is usually very big.

Or it doesn't exist. Not sure we should be setting expectations since it will vary per device.

Quoted 7 lines

This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

Quoted 3 lines

It seems a little gratuitous to me that this is a separate region or that this data is handled separately. All of this data is opaque to QEMU, so why do we need to separate it?

hi Alex,
as the device state interfaces are provided by kernel, it is expected to meet as general needs as possible. So, do you think there are such use cases from user space that user space knows well of the device, and it wants kernel to return desired data back to it.
E.g. It just wants to get whole device config data including all mmios, page tables, pci config data...
or, It just wants to get current device memory snapshot, not including any dirty data.
Or, It just needs the dirty pages in device memory or system memory. With all this accurate query, quite a lot of useful features can be developped in user space.

If all of this data is opaque to user app, seems the only use case is for live migration.

From another aspect, if the final solution is to let the data opaque to user space, like what NV did, kernel side's implementation will be more complicated, and actually a little challenge to vendor driver. in that case, in pre-copy phase,
1. in not LOGGING state, vendor driver first returns full data including full device memory snapshot
2. user space reads some data (you can't expect it to finish reading all data)
3. then userspace set the device state to LOGGING to start dirty data logging
4. vendor driver starts dirty data logging, and appends the dirty data to the tail of remaining unread full data and increase the pending data size?
5. user space keeps reading data.
6. vendor driver keeps appending new dirty data to the tail of remaining unread full data/dirty data and increase the pending data size?

in stop-and-copy phase
1. user space sets device state to exit LOGGING state,
2. vendor driver stops data logging. it has to append device config data at the tail of remaining dirty data unread by userspace.

during this flow, when vendor driver should get dirty data? just keeps logging and appends to tail? how to ensure dirty data is refresh new before LOGGING state exit? how does vendor driver know whether certain dirty data is copied or not?

I've no idea how NVidia handle this problem, and they don't open their kernel side code.
just feel it's a bit hard for other vendor drivers to follow:)

Quoted 23 lines
Quoted 17 lines

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

Quoted 5 lines

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.

Quoted 3 lines

Is mmap mandatory? I would think this would be defined by the mdev device what access they want to support per region. We don't want to impose a more complicated interface if the device doesn't require it.

I think it's "mmap is preferred, but allowed to fail".
just like a normal region with MMAP flag on (like bar regions), we also allow its mmap to fail, right?

Quoted 10 lines
Quoted 9 lines

device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

Quoted 9 lines

It's not obvious to me how this is better, a big region isn't padded, there's simply a gap in the file descriptor. Is having a sub-PAGE_SIZE gap in a file really of any consequence? Each region beyond the header is more than likely larger than PAGE_SIZE, therefore they can be nicely aligned together. We still need fields to tell us how much data is available in each area, so another to tell us the start of each area is a minor detail. And I think we still want to allow drivers to specify which parts of which areas support mmap, so I don't think we're getting away from sparse mmap support.

with seperate regions and sub-region type defined, user space can explictly know which region is which region after vfio_get_dev_region_info(). along with it, user space knows region offset and size. mmap is allowed to fail and falls back to normal read/write to the region.

But with one big region and sparse mmapped subregions (1 data subregion or 3 data subregions, whatever), userspace can't tell which subregion is which one.
So, if using one big region, I think we need to explictly define subregions' sequence (like index 0 is dedicated to control subregion, index 1 is for device config data subregion ...). Vendor driver cannot freely change the sequence.
Then keep data offset the same as region->mmaps[i].offset, and data size the same as region->mmaps[i].size. (i.e. let actual data starts immediatly from first byte of its data subregion)
Also, mmaps for sparse mmaped subregions are not allowed to fail.

With one big region, we also need to consider the case when vendor driver does not want the data subregion to be mmaped.
so, what is the data layout for that case?
data subregion immedately follows control subregion, or not?
Of couse, for this condition, we can specify the data filed's start offset and size through control region. And we must not expect the data start offset in source and target are equal.
(because the big region's fd_offset
may vary in source and target. consider the case when both source and target have one opregion and one device state region, but source has opregion in the first and target has device state region in the first. If we think this case is illegal, we must be able to detect it in the first place).
Also, we must keep the start offset and size consistent with the above mmap case.

Quoted 10 lines
Quoted 5 lines

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If we were to go with this multi-region solution, isn't it evident from the regions exposed that device memory and a dirty bitmap are provided? Alternatively, I believe Kirti's proposal doesn't require

Quoted 5 lines

this distinction between device memory and device config, a device not requiring runtime migration data would simply report no data until the device moved to the stopped state, making it consistent path for userspace. Likewise, the dirty bitmap could report a zero page count in the bitmap rather than branching based on device support.

If the path in userspace is consistent for device config and device memory, there will be many unnecessary call of getting data size into vendor driver.

Quoted 34 lines

Note that it seems that cap VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY implies VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP, but there's no consistency in the naming.

Quoted 3 lines

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

It looks like these are being defined as bits, since patch 1 talks about RUNNING & LOGGING and STOP & LOGGING. I think Connie already posted some comments about this. I'm not sure anything prevents us from defining RUNNING a 1 and STOPPED as 0 so we don't have the polarity flip vs LOGGING though.

The state "STOP & LOGGING" also feels like a strange "device state", if the device is stopped, it's not logging any new state, so I think this is more that the device state is STOP, but the LOGGING feature is active. Maybe we should consider these independent bits. LOGGING is active as we stop a device so that we can fetch the last dirtied pages, but disabled as we load the state of the device into the target.

Quoted 4 lines

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

Quoted 8 lines
struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;    /*rw*/
	} device_config;
Quoted 5 lines

Patch 1 indicates that to get the config buffer we write GET_BUFFER to action and read from the config region. The size is previously read and apparently constant. To set the config buffer, the config region is written followed by writing SET_BUFFER to action. Why is size listed as read-write?

this is the size of config data.
size of config data <= size of config data region.

Quoted 10 lines

Doesn't this protocol also require that the mdev driver consume each full region's worth of host kernel memory for backing pages in anticipation of a rare event like migration? This might be a strike against separate regions if the driver needs to provide backing pages for 3 separate regions vs 1. To avoid this runtime overhead, would it be expected that the user only mmap the regions during migration and the mdev driver allocate backing pages on mmap? Should the mmap be restricted to the LOGGING feature being enabled? Maybe I'm mistaken in how the mdev driver would back these mmap'd pages.

yes, 3 seperate regions consume a little more memory than 1 region. but it's just a little overhead.
As in intel's kernel implementation,
device config region's size is 9M, dirty bitmap region's size is 16k. if there is device memory region, its size can be defined as 100M? so it's 109M vs 100M ?

Quoted 5 lines
Quoted 4 lines
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;     /* rw */  
                __u64 pos; /*the offset in total buffer of device memory*/
Quoted 8 lines

Patch 1 outlines the protocol here that getting device memory begins with writing the position field, followed by reading from the device memory region. Setting device memory begins with writing the data to the device memory region, followed by writing the position field. Why does the user need to have visibility of data position? This is opaque data to the user, the device should manage how the chunks fit together.

How does the user know when they reach the end?

sorry, maybe I didn't explain clearly here.

device  ________________________________________
memory  |    |    |////|    |    |    |    |    |
data:   |____|____|////|____|____|____|____|____|
                  :pos :
                  :    :
device            :____:
memory            |    |
region:           |____|

the whole sequence is like this:

1. user space reads device_memory.size
2. driver gets device memory's data(full snapshot or dirty data, depending on it's in LOGGING state or not), and return the total size of this data.
3. user space finishes reading device_memory.size (>= device memory region's size)

4. user space starts a loop like

   while (pos < total_len) {
        uint64_t len = region_devmem->size;
        if (pos + len >= total_len) {
            len = total_len - pos;
        }
        if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) {
            return -1;
        }
	pos += len;
    }

vfio_save_data_device_memory_chunk reads each chunk from device memory region by writing GET_BUFFER to device_memory.action, and pos to device_memory.pos.

so. each time, userspace will finish getting device memory data in one time.

specifying "pos" is just like the "lseek" before "write".

Quoted 9 lines

Bullets 8 and 9 in patch 1 also discuss setting and getting the device memory size, but these aren't well integrated into the protocol for getting and setting the memory buffer. Is getting the device memory really started by reading the size, which triggers the vendor driver to snapshot the state in an internal buffer which the user then iterates through using GET_BUFFER? Therefore re-reading the size field could corrupt the data stream? Wouldn't it be easier to report bytes available and countdown as the user marks them read? What does position mean when we switch from snapshot to dirty data?

when switching to device memory's dirty data, pos means the pos in whole dirty data.

.save_live_pending ==> driver gets dirty data in device memory and returns total size.

.save_live_iterate ==> userspace reads all dirty data from device memory region chunk by chunk

So, in an iteration, all dirty data are saved. then in next iteration, dirty data is recalculated.

Quoted 9 lines
Quoted 7 lines
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};

Why is one specified as an address and the other as pages? Note that

Yes, start_addr ==> start pfn is better

Quoted 3 lines

Kirti's implementation has an optimization to know how many pages are set within a range to avoid unnecessary buffer reads.

Let's use start_pfn_all, page_nr_all to represent the start pfn and page_nr passed in from qemu .log_sync interface.

and use start_pfn_i, page_nr_i to the value passed to driver.

start_pfn_all

  |         start_pfn_i
  |         |
 \ /_______\_/_____________________________
  |    |    |////|    |    |    |    |    |
  |____|____|////|____|____|____|____|____|
            :    :
            :    :
            :____:
bitmap      |    |
region:     |____|

1. Each time QEMU queries dirty bitmap from driver, it passes in start_pfn_i, and page_nr_i. (page_nr_i is the biggest page number the bitmap region can hold).
2. driver queries memory range from start_pfn_i with size page_nr_i.
3. driver return a bitmap (if no dirty data, the bitmap is all 0).
4. QEMU saves the pages according to the bitmap

If there's no dirty data found in step 2, step 4 can be skipped. (I'll add this check before step 4 in future, thanks)
but if there's even 1 bit set in the bitmap, no step from 1-4 can be skipped.

Honestly, after reviewing Kirti's implementation, I don't think it's an optimization. As in below pseudo code for Kirti's code, I would think the copied_pfns corresponds to the page_nr_i in my case. so, the case of copied_pfns equaling to 0 is for the tail chunk? don't think it's working..

write start_pfn to driver
write page_size to driver
write pfn_count to driver

do {
    read copied_pfns from driver.
    if (copied_pfns == 0) {
       break;
    }
   bitmap_size = (BITS_TO_LONGS(copied_pfns) + 1) * sizeof(unsigned long);
   buf = get bitmap from driver
   cpu_physical_memory_set_dirty_lebitmap((unsigned long *)buf,
                                           (start_pfn + count) * page_size,
                                                copied_pfns);

     count +=  copied_pfns;

} while (count < pfn_count);

Quoted 51 lines
Quoted 42 lines

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

Quoted 8 lines
        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

Quoted 3 lines

As above, these capabilities seem redundant to the existence of the device specific regions in this implementation.

seems so :)

Quoted 77 lines
Quoted 65 lines

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING & VFIO_DEVICE_STATE_LOGGING |
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

Quoted 7 lines

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.

This requires iterative reads of device memory buffer but the protocol is unclear (to me) how the user knows how to do this or interact with the position field.

Quoted 9 lines
Quoted 5 lines

Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).

What if there's not enough dirty data to fill the region? The data is always padded to fill the region?

I think dirty data in vendor driver is orgnaized in a format like:
(addr0, len0, data0, addr1, len1, data1, addr2, len2, data2,....addrN, lenN, dataN).
for full snapshot, it's like (addr0, len0, data0).
so, to userspace and data region, it doesn't matter whether it's full snapshot or dirty data.

Quoted 11 lines
Quoted 6 lines

.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

Therefore through the entire precopy phase we have no data from source to target to begin a compatibility check :-\ I think both proposals currently still lack any sort of device compatibility or data versioning check between source and target. Thanks,

I checked the compatibility, though not good enough:)

in migration_init, vfio_check_devstate_version() checked version from kernel with VFIO_DEVICE_STATE_INTERFACE_VERSION in both source and target, and in target side, vfio_load_state() checked source side version.

int vfio_load_state(QEMUFile *f, void *opaque, int version_id)

{       
    ...
    if (version_id != VFIO_DEVICE_STATE_INTERFACE_VERSION) {                                   
        return -EINVAL;
    } 
    ...
}

Thanks
Yan

Quoted 52 lines

Alex

Quoted 49 lines

In precopy phase, if a device has VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY on, .log_sync callback will get system memory dirty bitmap from dirty bitmap region by writing system memory's start address, page count and action (GET_BITMAP) to "system_memory.start_addr", "system_memory.page_nr", and "system_memory.action" fields of control region.
If page count passed in .log_sync callback is larger than the bitmap size the dirty bitmap region supports, Qemu will cut it into chunks and call vendor driver's get system memory dirty bitmap interface.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is off, .log_sync callback just returns without call to vendor driver.

In stop-and-copy phase, device state will be set to STOP & LOGGING first. in save_live_complete_precopy callback,
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on,
get device memory size and get device memory buffer will be called again. After that,
device config data is get from device config region by reading devcie_config.size of control region and writing action (GET_BITMAP) to device_config.action of control region.
Then after migration completes, in cleanup handler, LOGGING state will be cleared (i.e. deivce state is set to STOP).
Clearing LOGGING state in cleanup handler is in consideration of the case of "migration failed" and "migration cancelled". They can also leverage the cleanup handler to unset LOGGING state.

References
----------
1. kernel side implementation of Device state interfaces:
https://patchwork.freedesktop.org/series/56876/

Yan Zhao (5):
vfio/migration: define kernel interfaces vfio/migration: support device of device config capability vfio/migration: tracking of dirty page in system memory vfio/migration: turn on migration vfio/migration: support device memory capability

 hw/vfio/Makefile.objs         |   2 +-
 hw/vfio/common.c              |  26 ++
 hw/vfio/migration.c           | 858 ++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 |  10 +-
 hw/vfio/pci.h                 |  26 +-
 include/hw/vfio/vfio-common.h |   1 +
 linux-headers/linux/vfio.h    | 260 +++++++++++++
 7 files changed, 1174 insertions(+), 9 deletions(-)
 create mode 100644 hw/vfio/migration.c

_______________________________________________
intel-gvt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev

#58 lore raw
In reply to Zhao Yan

Hi Yan,

Sorry for the delay, I've been on PTO...

Quoted 74 lines · On Sun, 24 Feb 2019 21:22:56 -0500 Zhao Yan <[email protected]> wrote:
Quoted 57 lines · Alex Williamson

Hi Yan,

Thanks for working on this!

Quoted 7 lines · On Tue, 19 Feb 2019 16:50:54 +0800 Yan Zhao <[email protected]> wrote:

This patchset enables VFIO devices to have live migration capability. Currently it does not support post-copy phase.

It follows Alex's comments on last version of VFIO live migration patches, including device states, VFIO device state region layout, dirty bitmap's query.

Quoted 8 lines

Device Data
-----------
Device data is divided into three types: device memory, device config, and system memory dirty pages produced by device.

Device config: data like MMIOs, page tables...
Every device is supposed to possess device config data. Usually device config's size is small (no big than 10M), and it

I'm not sure how we can really impose a limit here, it is what it is for a device. A smaller state is obviously desirable to reduce downtime, but some devices could have very large states.

Quoted 6 lines

needs to be loaded in certain strict order. Therefore, device config only needs to be saved/loaded in stop-and-copy phase.
The data of device config is held in device config region. Size of device config data is smaller than or equal to that of device config region.

So the intention here is that this is the last data read from the device and it's done in one pass, so the region needs to be large enough to expose all config data at once. On restore it's the last data written before switching the device to the run state.

Quoted 2 lines

Device Memory: device's internal memory, standalone and outside system

s/system/VM/

Quoted 1 line

memory. It is usually very big.

Or it doesn't exist. Not sure we should be setting expectations since it will vary per device.

Quoted 7 lines

This kind of data needs to be saved / loaded in pre-copy and stop-and-copy phase.
The data of device memory is held in device memory region. Size of devie memory is usually larger than that of device memory region. qemu needs to save/load it in chunks of size of device memory region.
Not all device has device memory. Like IGD only uses system memory.

Quoted 3 lines

It seems a little gratuitous to me that this is a separate region or that this data is handled separately. All of this data is opaque to QEMU, so why do we need to separate it?

hi Alex,
as the device state interfaces are provided by kernel, it is expected to meet as general needs as possible. So, do you think there are such use cases from user space that user space knows well of the device, and it wants kernel to return desired data back to it.
E.g. It just wants to get whole device config data including all mmios, page tables, pci config data...
or, It just wants to get current device memory snapshot, not including any dirty data.
Or, It just needs the dirty pages in device memory or system memory. With all this accurate query, quite a lot of useful features can be developped in user space.

Quoted 2 lines

If all of this data is opaque to user app, seems the only use case is for live migration.

I can certainly appreciate a more versatile interface, but I think we're also trying to create the most simple interface we can, with the primary target being live migration. As soon as we start defining this type of device memory and that type of device memory, we're going to have another device come along that needs yet another because they have a slightly different requirement. Even without that, we're going to have vendor drivers implement it differently, so what works for one device for a more targeted approach may not work for all devices. Can you enumerate some specific examples of the use cases you imagine your design to enable?

Quoted 6 lines

From another aspect, if the final solution is to let the data opaque to user space, like what NV did, kernel side's implementation will be more complicated, and actually a little challenge to vendor driver. in that case, in pre-copy phase,
1. in not LOGGING state, vendor driver first returns full data including full device memory snapshot

When we're not LOGGING, does the vendor driver need to return anything? It seems that LOGGING could be considered an enable switch for the interface.

Quoted 6 lines

2. user space reads some data (you can't expect it to finish reading all data)
3. then userspace set the device state to LOGGING to start dirty data logging
4. vendor driver starts dirty data logging, and appends the dirty data to the tail of remaining unread full data and increase the pending data size?

It seems a lot of overhead to expect the vendor driver to consider state read by the user prior to LOGGING being enabled. Does it log those changes forever? It seems like we should consider LOGGING enabled to be a "session".

Quoted 3 lines

5. user space keeps reading data.
6. vendor driver keeps appending new dirty data to the tail of remaining unread full data/dirty data and increase the pending data size?

Until the device is stopped it can always generate new pending date and the size of that pending data needs to be considered volatile by the user, right? What's different here? This all seems to factor into when the user decides whether the migration is converting and whether to transition to stopped phase to force that convergence.

Quoted 9 lines

in stop-and-copy phase
1. user space sets device state to exit LOGGING state,
2. vendor driver stops data logging. it has to append device config data at the tail of remaining dirty data unread by userspace.

during this flow, when vendor driver should get dirty data? just keeps logging and appends to tail? how to ensure dirty data is refresh new before LOGGING state exit? how does vendor driver know whether certain dirty data is copied or not?

At stop-and-copy, I'd assume LOGGING remains enabled, only adding STOP, such that the device does not generate new data, but perhaps I've forgotten the details on vacation. As above, I'd think we'd want to bound any sort of dirty state tracking to a session bounded by the LOGGING state. The protocol defined with userspace needs to account for determining what the user has and has not read, for instance to support mmap'd data, a trapped interface needs to be used to setup the data and acknowledge a read of that data.

Quoted 3 lines

I've no idea how NVidia handle this problem, and they don't open their kernel side code.
just feel it's a bit hard for other vendor drivers to follow:)

Their interface proposal is available on the list, I don't have access to their proprietary driver either, but I expect the best ideas from each proposal to be combined into a unified solution.

Quoted 23 lines
Quoted 23 lines
Quoted 22 lines

System memory dirty pages: If a device produces dirty pages in system
memory, it is able to get dirty bitmap for certain range of system memory. This dirty bitmap is queried in pre-copy and stop-and-copy phase in .log_sync callback. By setting dirty bitmap in .log_sync callback, dirty pages in system memory will be save/loaded by ram's live migration code.
The dirty bitmap of system memory is held in dirty bitmap region. If system memory range is larger than that dirty bitmap region can hold, qemu will cut it into several chunks and get dirty bitmap in succession.

Device State Regions
--------------------
Vendor driver is required to expose two mandatory regions and another two optional regions if it plans to support device state management.

So, there are up to four regions in total.
One control region: mandatory.
Get access via read/write system call.
Its layout is defined in struct vfio_device_state_ctl
Three data regions: mmaped into qemu.

Quoted 6 lines
Quoted 3 lines

Is mmap mandatory? I would think this would be defined by the mdev device what access they want to support per region. We don't want to impose a more complicated interface if the device doesn't require it.

I think it's "mmap is preferred, but allowed to fail".
just like a normal region with MMAP flag on (like bar regions), we also allow its mmap to fail, right?

Currently mmap support for any region is optional both from the vendor driver and the user. The vendor driver may or may not support mmap of a region (or subset of region with sparse mmap) and the user may or may not make use of mmap if it is available. The question here was whether this interface requires the vendor driver to support mmap of these device specific regions.

Quoted 20 lines
Quoted 10 lines
Quoted 9 lines

device config region: mandatory, holding data of device config device memory region: optional, holding data of device memory dirty bitmap region: optional, holding bitmap of system memory
dirty pages

(The reason why four seperate regions are defined is that the unit of mmap system call is PAGE_SIZE, i.e. 4k bytes. So one read/write region for control and three mmaped regions for data seems better than one big region padded and sparse mmaped).

Quoted 9 lines

It's not obvious to me how this is better, a big region isn't padded, there's simply a gap in the file descriptor. Is having a sub-PAGE_SIZE gap in a file really of any consequence? Each region beyond the header is more than likely larger than PAGE_SIZE, therefore they can be nicely aligned together. We still need fields to tell us how much data is available in each area, so another to tell us the start of each area is a minor detail. And I think we still want to allow drivers to specify which parts of which areas support mmap, so I don't think we're getting away from sparse mmap support.

Quoted 8 lines

with seperate regions and sub-region type defined, user space can explictly know which region is which region after vfio_get_dev_region_info(). along with it, user space knows region offset and size. mmap is allowed to fail and falls back to normal read/write to the region.

But with one big region and sparse mmapped subregions (1 data subregion or 3 data subregions, whatever), userspace can't tell which subregion is which one.

Of course they can, this is part of defining the header structure. One region could define a header including config_offset, config_size, memory_offset, memory_size, dirty_offset, dirty_size. Notice how Kirti even uses the same area to support all of these (which leaves some issues with vendor driver flexibility, but at least shows this is possible).

Quoted 8 lines

So, if using one big region, I think we need to explictly define subregions' sequence (like index 0 is dedicated to control subregion, index 1 is for device config data subregion ...). Vendor driver cannot freely change the sequence.
Then keep data offset the same as region->mmaps[i].offset, and data size the same as region->mmaps[i].size. (i.e. let actual data starts immediatly from first byte of its data subregion)
Also, mmaps for sparse mmaped subregions are not allowed to fail.

This doesn't make any sense to me, the vendor driver can define the start and size of each area within the region with simple header fields. We don't need fixed sequence fields. Likewise the sparse mmap capability for the region can define which of those areas within the region support mmap. The mmap can be optional for both vendor driver and user, just as it is elsewhere. The header fields defining the sub-areas can be read-only to the user, the sparse mmap only needs to match what the vendor driver defines and supports.

Quoted 3 lines

With one big region, we also need to consider the case when vendor driver does not want the data subregion to be mmaped.
so, what is the data layout for that case?

Vendor driver defines data_offset and data_size, sparse mmap capability does not list that area as mmap capable.

Quoted 1 line

data subregion immedately follows control subregion, or not?

The header needs to begin at offset zero, the layout of the rest is defined by the vendor driver within this header. I believe this is (mostly) implemented in Kirti's version.

Quoted 11 lines

Of couse, for this condition, we can specify the data filed's start offset and size through control region. And we must not expect the data start offset in source and target are equal.
(because the big region's fd_offset
may vary in source and target. consider the case when both source and target have one opregion and one device state region, but source has opregion in the first and target has device state region in the first. If we think this case is illegal, we must be able to detect it in the first place).
Also, we must keep the start offset and size consistent with the above mmap case.

AFAICT, these are all non-issues. Please look at Kirti's proposal. The (one) migration region can define a header at offset zero that allows the vendor driver to define where within that region the data, config, and dirty bitmap areas are and the sparse mmap capability defines which of those are mmap capable. Clearly this migration region offset within the vfio device file descriptor is independent between source and target, as are the offsets of the sub-areas within the migration region. These all need to be defined as part of the migration protocol where the source and target implement the same protocol but have no requirement to be absolutely identical.

Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

kernel device state interface [1]
--------------------------------------
#define VFIO_DEVICE_STATE_INTERFACE_VERSION 1
#define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
#define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If we were to go with this multi-region solution, isn't it evident from the regions exposed that device memory and a dirty bitmap are provided? Alternatively, I believe Kirti's proposal doesn't require

Quoted 8 lines
Quoted 5 lines

this distinction between device memory and device config, a device not requiring runtime migration data would simply report no data until the device moved to the stopped state, making it consistent path for userspace. Likewise, the dirty bitmap could report a zero page count in the bitmap rather than branching based on device support.

If the path in userspace is consistent for device config and device memory, there will be many unnecessary call of getting data size into vendor driver.

Consistency seems like a good thing, it makes code more simple, we don't behave differently in one case versus another. If the vendor reports no data, skip. It also provides versatility. Are the "many unnecessary call[s]" quantifiable?

Quoted 34 lines
Quoted 34 lines

Note that it seems that cap VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY implies VFIO_REGION_SUBTYPE_DEVICE_STATE_DATA_DIRTYBITMAP, but there's no consistency in the naming.

Quoted 3 lines

#define VFIO_DEVICE_STATE_RUNNING 0
#define VFIO_DEVICE_STATE_STOP 1
#define VFIO_DEVICE_STATE_LOGGING 2

It looks like these are being defined as bits, since patch 1 talks about RUNNING & LOGGING and STOP & LOGGING. I think Connie already posted some comments about this. I'm not sure anything prevents us from defining RUNNING a 1 and STOPPED as 0 so we don't have the polarity flip vs LOGGING though.

The state "STOP & LOGGING" also feels like a strange "device state", if the device is stopped, it's not logging any new state, so I think this is more that the device state is STOP, but the LOGGING feature is active. Maybe we should consider these independent bits. LOGGING is active as we stop a device so that we can fetch the last dirtied pages, but disabled as we load the state of the device into the target.

Quoted 12 lines

#define VFIO_DEVICE_DATA_ACTION_GET_BUFFER 1
#define VFIO_DEVICE_DATA_ACTION_SET_BUFFER 2
#define VFIO_DEVICE_DATA_ACTION_GET_BITMAP 3

struct vfio_device_state_ctl {
	__u32 version;		  /* ro */
	__u32 device_state;       /* VFIO device state, wo */
	__u32 caps;		 /* ro */
        struct {
		__u32 action;  /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;    /*rw*/
	} device_config;  
Quoted 7 lines
Quoted 5 lines

Patch 1 indicates that to get the config buffer we write GET_BUFFER to action and read from the config region. The size is previously read and apparently constant. To set the config buffer, the config region is written followed by writing SET_BUFFER to action. Why is size listed as read-write?

this is the size of config data.
size of config data <= size of config data region.

Where in the usage protocol does the user WRITE the config data size?

Quoted 10 lines
Quoted 10 lines

Doesn't this protocol also require that the mdev driver consume each full region's worth of host kernel memory for backing pages in anticipation of a rare event like migration? This might be a strike against separate regions if the driver needs to provide backing pages for 3 separate regions vs 1. To avoid this runtime overhead, would it be expected that the user only mmap the regions during migration and the mdev driver allocate backing pages on mmap? Should the mmap be restricted to the LOGGING feature being enabled? Maybe I'm mistaken in how the mdev driver would back these mmap'd pages.

Quoted 6 lines

yes, 3 seperate regions consume a little more memory than 1 region. but it's just a little overhead.
As in intel's kernel implementation,
device config region's size is 9M, dirty bitmap region's size is 16k. if there is device memory region, its size can be defined as 100M? so it's 109M vs 100M ?

But what if it's 100M config with no device memory? This proposal requires 100M in-kernel backing due to the definition of the config region when it could be implemented with significantly less by allowing a small data area to be read multiple times until a bytes remaining counter becomes zero.

Quoted 52 lines
Quoted 5 lines
Quoted 4 lines
	struct {
		__u32 action;    /* wo, GET_BUFFER or SET_BUFFER */ 
		__u64 size;     /* rw */  
                __u64 pos; /*the offset in total buffer of device memory*/  
Quoted 8 lines

Patch 1 outlines the protocol here that getting device memory begins with writing the position field, followed by reading from the device memory region. Setting device memory begins with writing the data to the device memory region, followed by writing the position field. Why does the user need to have visibility of data position? This is opaque data to the user, the device should manage how the chunks fit together.

How does the user know when they reach the end?

sorry, maybe I didn't explain clearly here.

device  ________________________________________
memory  |    |    |////|    |    |    |    |    |
data:   |____|____|////|____|____|____|____|____|
                  :pos :
                  :    :
device            :____:
memory            |    |
region:           |____|

the whole sequence is like this:

1. user space reads device_memory.size
2. driver gets device memory's data(full snapshot or dirty data, depending on it's in LOGGING state or not), and return the total size of this data.
3. user space finishes reading device_memory.size (>= device memory region's size)

4. user space starts a loop like

   while (pos < total_len) {
        uint64_t len = region_devmem->size;
        if (pos + len >= total_len) {
            len = total_len - pos;
        }
        if (vfio_save_data_device_memory_chunk(vdev, f, pos, len)) {
            return -1;
        }
	pos += len;
    }

vfio_save_data_device_memory_chunk reads each chunk from device memory region by writing GET_BUFFER to device_memory.action, and pos to device_memory.pos.

Quoted 4 lines

so. each time, userspace will finish getting device memory data in one time.

specifying "pos" is just like the "lseek" before "write".

This could also be implemented as a remaining bytes counter in the interface where the vendor driver wouldn't rely on the user to manage the position. What internal consistency checking is going to protect the host kernel when the user writes data to the wrong position? If we consider the data to be opaque, the vendor driver can embed that sort of meta data into the data blob the user reads and reassemble it correctly or generate a consistency failure itself.

Quoted 88 lines
Quoted 9 lines

Bullets 8 and 9 in patch 1 also discuss setting and getting the device memory size, but these aren't well integrated into the protocol for getting and setting the memory buffer. Is getting the device memory really started by reading the size, which triggers the vendor driver to snapshot the state in an internal buffer which the user then iterates through using GET_BUFFER? Therefore re-reading the size field could corrupt the data stream? Wouldn't it be easier to report bytes available and countdown as the user marks them read? What does position mean when we switch from snapshot to dirty data?

when switching to device memory's dirty data, pos means the pos in whole dirty data.

.save_live_pending ==> driver gets dirty data in device memory and returns total size.

.save_live_iterate ==> userspace reads all dirty data from device memory region chunk by chunk

So, in an iteration, all dirty data are saved. then in next iteration, dirty data is recalculated.

Quoted 8 lines
Quoted 6 lines
	} device_memory;
	struct {
		__u64 start_addr; /* wo */
		__u64 page_nr;   /* wo */
	} system_memory;
};  

Why is one specified as an address and the other as pages? Note that

Yes, start_addr ==> start pfn is better

Quoted 3 lines

Kirti's implementation has an optimization to know how many pages are set within a range to avoid unnecessary buffer reads.

Let's use start_pfn_all, page_nr_all to represent the start pfn and page_nr passed in from qemu .log_sync interface.

and use start_pfn_i, page_nr_i to the value passed to driver.

start_pfn_all

  |         start_pfn_i
  |         |
 \ /_______\_/_____________________________
  |    |    |////|    |    |    |    |    |
  |____|____|////|____|____|____|____|____|
            :    :
            :    :
            :____:
bitmap      |    |
region:     |____|

1. Each time QEMU queries dirty bitmap from driver, it passes in start_pfn_i, and page_nr_i. (page_nr_i is the biggest page number the bitmap region can hold).
2. driver queries memory range from start_pfn_i with size page_nr_i.
3. driver return a bitmap (if no dirty data, the bitmap is all 0).
4. QEMU saves the pages according to the bitmap

If there's no dirty data found in step 2, step 4 can be skipped. (I'll add this check before step 4 in future, thanks)
but if there's even 1 bit set in the bitmap, no step from 1-4 can be skipped.

Honestly, after reviewing Kirti's implementation, I don't think it's an optimization. As in below pseudo code for Kirti's code, I would think the copied_pfns corresponds to the page_nr_i in my case. so, the case of copied_pfns equaling to 0 is for the tail chunk? don't think it's working..

write start_pfn to driver
write page_size to driver
write pfn_count to driver

do {
    read copied_pfns from driver.
    if (copied_pfns == 0) {
       break;
    }
   bitmap_size = (BITS_TO_LONGS(copied_pfns) + 1) * sizeof(unsigned long);
   buf = get bitmap from driver
   cpu_physical_memory_set_dirty_lebitmap((unsigned long *)buf,
                                           (start_pfn + count) * page_size,
                                                copied_pfns);

     count +=  copied_pfns;
Quoted 1 line

} while (count < pfn_count);

The intent of Kirti's copied_pfns is clearly to avoid unnecessarily reading pages from the kernel when nothing has changed. Perhaps the implementation still requires work, but I don't see from above how that's not considered an optimization.

Quoted 168 lines
Quoted 51 lines
Quoted 42 lines

Devcie States
-------------
After migration is initialzed, it will set device state via writing to device_state field of control region.

Four states are defined for a VFIO device:
RUNNING, RUNNING & LOGGING, STOP & LOGGING, STOP

RUNNING: In this state, a VFIO device is in active state ready to receive
commands from device driver.
It is the default state that a VFIO device enters initially.

STOP: In this state, a VFIO device is deactivated to interact with
device driver.

LOGGING: a special state that it CANNOT exist independently. It must be
set alongside with state RUNNING or STOP (i.e. RUNNING & LOGGING, STOP & LOGGING).
Qemu will set LOGGING state on in .save_setup callbacks, then vendor driver can start dirty data logging for device memory and system memory.
LOGGING only impacts device/system memory. They return whole snapshot outside LOGGING and dirty data since last get operation inside LOGGING.
Device config should be always accessible and return whole config snapshot regardless of LOGGING state.

Note:
The reason why RUNNING is the default state is that device's active state must not depend on device state interface.
It is possible that region vfio_device_state_ctl fails to get registered. In that condition, a device needs be in active state by default.

Get Version & Get Caps
----------------------
On migration init phase, qemu will probe the existence of device state regions of vendor driver, then get version of the device state interface from the r/w control region.

Then it will probe VFIO device's data capability by reading caps field of control region.

Quoted 8 lines
        #define VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY 1
        #define VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY 2

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is on, it will save/load data of
device memory in pre-copy and stop-and-copy phase. The data of device memory is held in device memory region.
If VFIO_DEVICE_DATA_CAP_SYSTEM_MEMORY is on, it will query of dirty pages
produced by VFIO device during pre-copy and stop-and-copy phase. The dirty bitmap of system memory is held in dirty bitmap region.

Quoted 3 lines

As above, these capabilities seem redundant to the existence of the device specific regions in this implementation.

seems so :)

Quoted 77 lines
Quoted 65 lines

If failing to find two mandatory regions and optional data regions corresponding to data caps or version mismatching, it will setup a migration blocker and disable live migration for VFIO device.

Flows to call device state interface for VFIO live migration
------------------------------------------------------------

Live migration save path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_SAVE_SETUP
|
.save_setup callback -->
get device memory size (whole snapshot size)
get device memory buffer (whole snapshot data)
set device state --> VFIO_DEVICE_STATE_RUNNING & VFIO_DEVICE_STATE_LOGGING |
MIGRATION_STATUS_ACTIVE
|
.save_live_pending callback --> get device memory size (dirty data) .save_live_iteration callback --> get device memory buffer (dirty data) .log_sync callback --> get system memory dirty bitmap
|
(vcpu stops) --> set device state -->
VFIO_DEVICE_STATE_STOP & VFIO_DEVICE_STATE_LOGGING
|
.save_live_complete_precopy callback -->
get device memory size (dirty data)
get device memory buffer (dirty data)
get device config size (whole snapshot size)
get device config buffer (whole snapshot data)
|
.save_cleanup callback --> set device state --> VFIO_DEVICE_STATE_STOP MIGRATION_STATUS_COMPLETED

MIGRATION_STATUS_CANCELLED or
MIGRATION_STATUS_FAILED
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING

Live migration load path:

(QEMU LIVE MIGRATION STATE --> DEVICE STATE INTERFACE --> DEVICE STATE)

MIGRATION_STATUS_NONE --> VFIO_DEVICE_STATE_RUNNING
|
(vcpu stops) --> set device state --> VFIO_DEVICE_STATE_STOP
|
MIGRATION_STATUS_ACTIVE
|
.load state callback -->
set device memory size, set device memory buffer, set device config size, set device config buffer
|
(vcpu starts) --> set device state --> VFIO_DEVICE_STATE_RUNNING
|
MIGRATION_STATUS_COMPLETED

Quoted 7 lines

In source VM side,
In precopy phase,
if a device has VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY on,
qemu will first get whole snapshot of device memory in .save_setup callback, and then it will get total size of dirty data in device memory in .save_live_pending callback by reading device_memory.size field of control region.

This requires iterative reads of device memory buffer but the protocol is unclear (to me) how the user knows how to do this or interact with the position field.

Quoted 9 lines
Quoted 5 lines

Then in .save_live_iteration callback, it will get buffer of device memory's dirty data chunk by chunk from device memory region by writing pos & action (GET_BUFFER) to device_memory.pos & device_memory.action fields of control region. (size of each chunk is the size of device memory data region).

What if there's not enough dirty data to fill the region? The data is always padded to fill the region?

I think dirty data in vendor driver is orgnaized in a format like:
(addr0, len0, data0, addr1, len1, data1, addr2, len2, data2,....addrN, lenN, dataN).
for full snapshot, it's like (addr0, len0, data0).
so, to userspace and data region, it doesn't matter whether it's full snapshot or dirty data.

Quoted 11 lines
Quoted 6 lines

.save_live_pending and .save_live_iteration may be called several times in precopy phase to get dirty data in device memory.

If VFIO_DEVICE_DATA_CAP_DEVICE_MEMORY is off, callbacks in precopy phase like .save_setup, .save_live_pending, .save_live_iteration will not call vendor driver's device state interface to get data from devcie memory.

Therefore through the entire precopy phase we have no data from source to target to begin a compatibility check :-\ I think both proposals currently still lack any sort of device compatibility or data versioning check between source and target. Thanks,

I checked the compatibility, though not good enough:)

in migration_init, vfio_check_devstate_version() checked version from kernel with VFIO_DEVICE_STATE_INTERFACE_VERSION in both source and target, and in target side, vfio_load_state() checked source side version.

Quoted 8 lines

int vfio_load_state(QEMUFile *f, void *opaque, int version_id)

{       
    ...
    if (version_id != VFIO_DEVICE_STATE_INTERFACE_VERSION) {                                   
        return -EINVAL;
    } 
    ...
}

But this only checks that both source and target are using the same migration interface, how do we know that they're compatible devices and that the vendor data stream is compatible between source and target? Whether both ends use the same migration interface is potentially not relevant if the data stream is compatible. Thanks,

Alex