Skip to contentSkip to conversation

docs/devel: Add VFIO device migration documentation

PATCH In review Kirti Wankhede 3 participants 15 messages Oct 29, 2020 – Nov 6, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 docs/devel: Add VFIO device migration documentation

Document interfaces used for VFIO device migration. Added flow of state changes during live migration with VFIO device.

---
 MAINTAINERS                   |   1 +
 docs/devel/vfio-migration.rst | 119 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 120 insertions(+)
 create mode 100644 docs/devel/vfio-migration.rst
MAINTAINERS +1 -0
index 6a197bd358d6..6f3fcffc6b3d 100644
@@ -1728,6 +1728,7 @@ M: Alex Williamson <[email protected]>
 S: Supported
 F: hw/vfio/*
 F: include/hw/vfio/
+F: docs/devel/vfio-migration.rst
 
 vfio-ccw
 M: Cornelia Huck <[email protected]>
docs/devel/vfio-migration.rst new file +119 -0

119 lines not on this page · raw message

Signature · 1 line
2.7.0
#1 lore raw
Quoted 7 lines · On Thu, 29 Oct 2020 11:23:11 +0530 Kirti Wankhede <[email protected]> wrote:

Document interfaces used for VFIO device migration. Added flow of state changes during live migration with VFIO device.

---
 MAINTAINERS                   |   1 +
 docs/devel/vfio-migration.rst | 119 ++++++++++++++++++++++++++++++++++++++++++

You probably want to include this into the Developer's Guide via index.rst.

Quoted 21 lines
 2 files changed, 120 insertions(+)
 create mode 100644 docs/devel/vfio-migration.rst
MAINTAINERS +1 -0
index 6a197bd358d6..6f3fcffc6b3d 100644
@@ -1728,6 +1728,7 @@ M: Alex Williamson <[email protected]>
 S: Supported
 F: hw/vfio/*
 F: include/hw/vfio/
+F: docs/devel/vfio-migration.rst
 
 vfio-ccw
 M: Cornelia Huck <[email protected]>
docs/devel/vfio-migration.rst new file +0 -0
new file mode 100644
index 000000000000..dab9127825e4

docs/devel/vfio-migration.rst line 5

Quoted 5 lines
docs/devel/vfio-migration.rst new file +5 -0
@@ -0,0 +1,119 @@
+=====================
+VFIO device Migration
+=====================
+
+VFIO devices use iterative approach for migration because certain VFIO devices

s/use/use an/ ?

docs/devel/vfio-migration.rst line 8

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+(e.g. GPU) have large amount of data to be transfered. The iterative pre-copy
+phase of migration allows for the guest to continue whilst the VFIO device state
+is transferred to destination, this helps to reduce the total downtime of the

s/to destination,/to the destination;/

docs/devel/vfio-migration.rst line 10

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+VM. VFIO devices can choose to skip the pre-copy phase of migration by returning
+pending_bytes as zero during pre-copy phase.

s/during/during the/

docs/devel/vfio-migration.rst line 14

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+
+Detailed description of UAPI for VFIO device for migration is in the comment
+above ``vfio_device_migration_info`` structure definition in header file
+linux-headers/linux/vfio.h.

I think I'd copy that to this file. If I'm looking at the documentation, I'd rather not go hunting for source code to find out what structure you are talking about. Plus, as it's UAPI, I don't expect it to change much, so it should be easy to keep the definitions in sync (famous last words).

docs/devel/vfio-migration.rst line 17

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+
+VFIO device hooks for iterative approach:
+-  A ``save_setup`` function that setup migration region, sets _SAVING flag in

s/setup/sets up the/
s/in/in the/

docs/devel/vfio-migration.rst line 18

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+VFIO device state and inform VFIO IOMMU module to start dirty page tracking.

s/inform/informs the/

docs/devel/vfio-migration.rst line 20

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- A ``load_setup`` function that setup migration region on the destination and

s/setup/sets up the/

docs/devel/vfio-migration.rst line 21

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+sets _RESUMING flag in VFIO device state.

s/in/in the/

docs/devel/vfio-migration.rst line 25

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+
+- A ``save_live_pending`` function that reads pending_bytes from vendor driver
+that indicate how much more data the vendor driver yet to save for the VFIO
+device.

"A ``save_live_pending`` function that reads pending_bytes from the vendor driver, which indicates the amount of data that the vendor driver has yet to save for the VFIO device." ?

docs/devel/vfio-migration.rst line 27

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- A ``save_live_iterate`` function that reads VFIO device's data from vendor

s/reads/reads the/
s/from/from the/

docs/devel/vfio-migration.rst line 28

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+driver through migration region during iterative phase.

s/through/through the/

docs/devel/vfio-migration.rst line 30

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- A ``save_live_complete_precopy`` function that resets _RUNNING flag from VFIO

s/from/from the/

docs/devel/vfio-migration.rst line 31

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+device state, saves device config space, if any, and iteratively copies

s/saves/saves the/

docs/devel/vfio-migration.rst line 33

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+remaining data for VFIO device till pending_bytes returned by vendor driver
+is zero.

"...and interactively copies the remaining data for the VFIO device until the vendor driver indicates that no data remains (pending_bytes is zero)." ?

docs/devel/vfio-migration.rst line 36

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+
+- A ``load_state`` function loads config section and data sections generated by
+above save functions.

"A ``load_state`` function that loads the config section and the data sections that are generated by the save functions above." ?

docs/devel/vfio-migration.rst line 38

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- ``cleanup`` functions for both save and load that unmap migration region.

..."that perform any migration-related cleanup, including unmapping the migration region." ?

docs/devel/vfio-migration.rst line 41

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+
+VM state change handler is registered to change VFIO device state based on VM
+state change.

"A VM state change handler is registered to change the VFIO device state when the VM state changes." ?

docs/devel/vfio-migration.rst line 43

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+Similarly, a migration state change notifier is added to get a notification on

s/added/registered/ ?

docs/devel/vfio-migration.rst line 50

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+migration state change. These states are translated to VFIO device state and
+conveyed to vendor driver.
+
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages

s/is added to mark/marks those/

docs/devel/vfio-migration.rst line 51

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

docs/devel/vfio-migration.rst line 52

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

docs/devel/vfio-migration.rst line 55

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device

s/by/by the/ (x2)

docs/devel/vfio-migration.rst line 62

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

docs/devel/vfio-migration.rst line 66

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+
+System memory dirty pages tracking when vIOMMU is enabled
+---------------------------------------------------------
+With vIOMMU, IO virtual address range can get unmapped while in pre-copy phase

s/IO/an I/O/

docs/devel/vfio-migration.rst line 67

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+of migration. In that case, unmap ioctl returns pages pinned in that range and

s/unmap ioctl returns pages pinned/the unmap ioctl returns any pinned pages/ ?

docs/devel/vfio-migration.rst line 77

Quoted 10 lines
docs/devel/vfio-migration.rst new file +10 -0
+QEMU reports corresponding guest physical pages dirty.
+During stop-and-copy phase, an IOMMU notifier is used to get a callback for
+mapped pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for
+those mapped ranges.
+
+Flow of state changes during Live migration
+===========================================
+Below is the flow of state change during live migration where states in brackets
+represent VM state, migration state and VFIO device state as:
+                (VM state, MIGRATION_STATUS, VFIO_DEVICE_STATE)

"The values in the brackets represent the VM state, the migration state, and the VFIO device state, respectively." ?

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+
+Live migration save path
+------------------------
+                        QEMU normal running state
+                        (RUNNING, _NONE, _RUNNING)
+                                    |
+                       migrate_init spawns migration_thread

docs/devel/vfio-migration.rst line 94

Quoted 10 lines
docs/devel/vfio-migration.rst new file +10 -0
+                Migration thread then calls each device's .save_setup()
+                        (RUNNING, _SETUP, _RUNNING|_SAVING)
+                                    |
+                        (RUNNING, _ACTIVE, _RUNNING|_SAVING)
+            If device is active, get pending_bytes by .save_live_pending()
+         if total pending_bytes >= threshold_size, call .save_live_iterate()
+                  Data of VFIO device for pre-copy phase is copied
+     Iterate till total pending bytes converge and are less than threshold
+                                    |
+   On migration completion, vCPUs stops and calls .save_live_complete_precopy

s/vCPUs/vCPU/ ?

docs/devel/vfio-migration.rst line 95

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+   for each active device. VFIO device is then transitioned in _SAVING state

s/VFIO device/The VFIO device/

s/in/into/

docs/devel/vfio-migration.rst line 98

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+                    (FINISH_MIGRATE, _DEVICE, _SAVING)
+                                    |
+For VFIO device, iterate in .save_live_complete_precopy until pending data is 0

s/For/For the/

docs/devel/vfio-migration.rst line 102

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+                    (FINISH_MIGRATE, _DEVICE, _STOPPED)
+                                    |
+                    (FINISH_MIGRATE, _COMPLETED, _STOPPED)
+                Migraton thread schedule cleanup bottom half and exit

s/schedule/schedules/
s/exit/exits/

docs/devel/vfio-migration.rst line 113

Quoted 11 lines
docs/devel/vfio-migration.rst new file +11 -0
+
+Live migration resume path
+--------------------------
+
+             Incoming migration calls .load_setup for each device
+                        (RESTORE_VM, _ACTIVE, _STOPPED)
+                                    |
+    For each device, .load_state is called for that device section data
+                        (RESTORE_VM, _ACTIVE, _RESUMING)
+                                    |
+    At the end, called .load_cleanup for each device and vCPUs are started                        |

Stray '|', probably should go on the next line?

s/called .load_cleanup/.load_cleanup is called/

docs/devel/vfio-migration.rst line 119

Quoted 6 lines
docs/devel/vfio-migration.rst new file +6 -0
+                        (RUNNING, _NONE, _RUNNING)
+
+
+Postcopy
+========
+Postcopy migration is not supported for VFIO devices.

s/not/not yet/, I think? (Disregarding any time frame here.)

Generally, I think this is useful information to have in the docs.

#2 lore raw

Thanks for corrections Cornelia. I had done the corrections you suggested I had not replied, see my comments on couple of places where I disagree.

Quoted 6 lines · Cornelia Huck
Quoted 3 lines · On Thu, 29 Oct 2020 11:23:11 +0530 Kirti Wankhede <[email protected]> wrote:

Document interfaces used for VFIO device migration. Added flow of state changes during live migration with VFIO device.

Quoted 8 lines
Quoted 4 lines
---
  MAINTAINERS                   |   1 +
  docs/devel/vfio-migration.rst | 119 ++++++++++++++++++++++++++++++++++++++++++

You probably want to include this into the Developer's Guide via index.rst.

Ok.

Quoted 40 lines
Quoted 21 lines
  2 files changed, 120 insertions(+)

create mode 100644 docs/devel/vfio-migration.rst

MAINTAINERS +1 -0
index 6a197bd358d6..6f3fcffc6b3d 100644
@@ -1728,6 +1728,7 @@ M: Alex Williamson <[email protected]>
  S: Supported
  F: hw/vfio/*
  F: include/hw/vfio/
+F: docs/devel/vfio-migration.rst
 
  vfio-ccw
  M: Cornelia Huck <[email protected]>
docs/devel/vfio-migration.rst new file +0 -0
new file mode 100644
index 000000000000..dab9127825e4

docs/devel/vfio-migration.rst line 5

Quoted 5 lines
docs/devel/vfio-migration.rst new file +5 -0
@@ -0,0 +1,119 @@
+=====================
+VFIO device Migration
+=====================
+
+VFIO devices use iterative approach for migration because certain VFIO devices

s/use/use an/ ?

docs/devel/vfio-migration.rst line 8

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+(e.g. GPU) have large amount of data to be transfered. The iterative pre-copy
+phase of migration allows for the guest to continue whilst the VFIO device state
+is transferred to destination, this helps to reduce the total downtime of the

s/to destination,/to the destination;/

docs/devel/vfio-migration.rst line 10

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+VM. VFIO devices can choose to skip the pre-copy phase of migration by returning
+pending_bytes as zero during pre-copy phase.

s/during/during the/

Quoted 11 lines

docs/devel/vfio-migration.rst line 14

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+
+Detailed description of UAPI for VFIO device for migration is in the comment
+above ``vfio_device_migration_info`` structure definition in header file
+linux-headers/linux/vfio.h.

I think I'd copy that to this file. If I'm looking at the documentation, I'd rather not go hunting for source code to find out what structure you are talking about. Plus, as it's UAPI, I don't expect it to change much, so it should be easy to keep the definitions in sync (famous last words).

I feel its duplication of documentation. I would like to know others views as well.

Quoted 116 lines

docs/devel/vfio-migration.rst line 17

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+
+VFIO device hooks for iterative approach:
+-  A ``save_setup`` function that setup migration region, sets _SAVING flag in

s/setup/sets up the/
s/in/in the/

docs/devel/vfio-migration.rst line 18

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+VFIO device state and inform VFIO IOMMU module to start dirty page tracking.

s/inform/informs the/

docs/devel/vfio-migration.rst line 20

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- A ``load_setup`` function that setup migration region on the destination and

s/setup/sets up the/

docs/devel/vfio-migration.rst line 21

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+sets _RESUMING flag in VFIO device state.

s/in/in the/

docs/devel/vfio-migration.rst line 25

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+
+- A ``save_live_pending`` function that reads pending_bytes from vendor driver
+that indicate how much more data the vendor driver yet to save for the VFIO
+device.

"A ``save_live_pending`` function that reads pending_bytes from the vendor driver, which indicates the amount of data that the vendor driver has yet to save for the VFIO device." ?

docs/devel/vfio-migration.rst line 27

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- A ``save_live_iterate`` function that reads VFIO device's data from vendor

s/reads/reads the/
s/from/from the/

docs/devel/vfio-migration.rst line 28

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+driver through migration region during iterative phase.

s/through/through the/

docs/devel/vfio-migration.rst line 30

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- A ``save_live_complete_precopy`` function that resets _RUNNING flag from VFIO

s/from/from the/

docs/devel/vfio-migration.rst line 31

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+device state, saves device config space, if any, and iteratively copies

s/saves/saves the/

docs/devel/vfio-migration.rst line 33

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+remaining data for VFIO device till pending_bytes returned by vendor driver
+is zero.

"...and interactively copies the remaining data for the VFIO device until the vendor driver indicates that no data remains (pending_bytes is zero)." ?

docs/devel/vfio-migration.rst line 36

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+
+- A ``load_state`` function loads config section and data sections generated by
+above save functions.

"A ``load_state`` function that loads the config section and the data sections that are generated by the save functions above." ?

docs/devel/vfio-migration.rst line 38

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- ``cleanup`` functions for both save and load that unmap migration region.

..."that perform any migration-related cleanup, including unmapping the migration region." ?

docs/devel/vfio-migration.rst line 41

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+
+VM state change handler is registered to change VFIO device state based on VM
+state change.

"A VM state change handler is registered to change the VFIO device state when the VM state changes." ?

docs/devel/vfio-migration.rst line 43

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+Similarly, a migration state change notifier is added to get a notification on

s/added/registered/ ?

docs/devel/vfio-migration.rst line 50

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+migration state change. These states are translated to VFIO device state and
+conveyed to vendor driver.
+
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages

s/is added to mark/marks those/

docs/devel/vfio-migration.rst line 51

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

docs/devel/vfio-migration.rst line 52

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

docs/devel/vfio-migration.rst line 55

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device

s/by/by the/ (x2)

docs/devel/vfio-migration.rst line 62

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Thanks,
Kirti

Quoted 93 lines

docs/devel/vfio-migration.rst line 66

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+
+System memory dirty pages tracking when vIOMMU is enabled
+---------------------------------------------------------
+With vIOMMU, IO virtual address range can get unmapped while in pre-copy phase

s/IO/an I/O/

docs/devel/vfio-migration.rst line 67

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+of migration. In that case, unmap ioctl returns pages pinned in that range and

s/unmap ioctl returns pages pinned/the unmap ioctl returns any pinned pages/ ?

docs/devel/vfio-migration.rst line 77

Quoted 10 lines
docs/devel/vfio-migration.rst new file +10 -0
+QEMU reports corresponding guest physical pages dirty.
+During stop-and-copy phase, an IOMMU notifier is used to get a callback for
+mapped pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for
+those mapped ranges.
+
+Flow of state changes during Live migration
+===========================================
+Below is the flow of state change during live migration where states in brackets
+represent VM state, migration state and VFIO device state as:
+                (VM state, MIGRATION_STATUS, VFIO_DEVICE_STATE)

"The values in the brackets represent the VM state, the migration state, and the VFIO device state, respectively." ?

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+
+Live migration save path
+------------------------
+                        QEMU normal running state
+                        (RUNNING, _NONE, _RUNNING)
+                                    |
+                       migrate_init spawns migration_thread

docs/devel/vfio-migration.rst line 94

Quoted 10 lines
docs/devel/vfio-migration.rst new file +10 -0
+                Migration thread then calls each device's .save_setup()
+                        (RUNNING, _SETUP, _RUNNING|_SAVING)
+                                    |
+                        (RUNNING, _ACTIVE, _RUNNING|_SAVING)
+            If device is active, get pending_bytes by .save_live_pending()
+         if total pending_bytes >= threshold_size, call .save_live_iterate()
+                  Data of VFIO device for pre-copy phase is copied
+     Iterate till total pending bytes converge and are less than threshold
+                                    |
+   On migration completion, vCPUs stops and calls .save_live_complete_precopy

s/vCPUs/vCPU/ ?

docs/devel/vfio-migration.rst line 95

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+   for each active device. VFIO device is then transitioned in _SAVING state

s/VFIO device/The VFIO device/

s/in/into/

docs/devel/vfio-migration.rst line 98

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+                    (FINISH_MIGRATE, _DEVICE, _SAVING)
+                                    |
+For VFIO device, iterate in .save_live_complete_precopy until pending data is 0

s/For/For the/

docs/devel/vfio-migration.rst line 102

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+                    (FINISH_MIGRATE, _DEVICE, _STOPPED)
+                                    |
+                    (FINISH_MIGRATE, _COMPLETED, _STOPPED)
+                Migraton thread schedule cleanup bottom half and exit

s/schedule/schedules/
s/exit/exits/

docs/devel/vfio-migration.rst line 113

Quoted 11 lines
docs/devel/vfio-migration.rst new file +11 -0
+
+Live migration resume path
+--------------------------
+
+             Incoming migration calls .load_setup for each device
+                        (RESTORE_VM, _ACTIVE, _STOPPED)
+                                    |
+    For each device, .load_state is called for that device section data
+                        (RESTORE_VM, _ACTIVE, _RESUMING)
+                                    |
+    At the end, called .load_cleanup for each device and vCPUs are started                        |

Stray '|', probably should go on the next line?

s/called .load_cleanup/.load_cleanup is called/

docs/devel/vfio-migration.rst line 119

Quoted 6 lines
docs/devel/vfio-migration.rst new file +6 -0
+                        (RUNNING, _NONE, _RUNNING)
+
+
+Postcopy
+========
+Postcopy migration is not supported for VFIO devices.

s/not/not yet/, I think? (Disregarding any time frame here.)

Generally, I think this is useful information to have in the docs.

#3 lore raw
Quoted 75 lines · On Thu, 29 Oct 2020 23:11:16 +0530 Kirti Wankhede <[email protected]> wrote:

Thanks for corrections Cornelia. I had done the corrections you suggested I had not replied, see my comments on couple of places where I disagree.

Quoted 6 lines · Cornelia Huck
Quoted 3 lines · On Thu, 29 Oct 2020 11:23:11 +0530 Kirti Wankhede <[email protected]> wrote:

Document interfaces used for VFIO device migration. Added flow of state changes during live migration with VFIO device.

Quoted 8 lines
Quoted 4 lines
---
  MAINTAINERS                   |   1 +
  docs/devel/vfio-migration.rst | 119 ++++++++++++++++++++++++++++++++++++++++++  

You probably want to include this into the Developer's Guide via index.rst.

Ok.

Quoted 40 lines
Quoted 21 lines
  2 files changed, 120 insertions(+)

create mode 100644 docs/devel/vfio-migration.rst

MAINTAINERS +1 -0
index 6a197bd358d6..6f3fcffc6b3d 100644
@@ -1728,6 +1728,7 @@ M: Alex Williamson <[email protected]>
  S: Supported
  F: hw/vfio/*
  F: include/hw/vfio/
+F: docs/devel/vfio-migration.rst
 
  vfio-ccw
  M: Cornelia Huck <[email protected]>
docs/devel/vfio-migration.rst new file +0 -0
new file mode 100644
index 000000000000..dab9127825e4

docs/devel/vfio-migration.rst line 5

Quoted 5 lines
docs/devel/vfio-migration.rst new file +5 -0
@@ -0,0 +1,119 @@
+=====================
+VFIO device Migration
+=====================
+
+VFIO devices use iterative approach for migration because certain VFIO devices  

s/use/use an/ ?

docs/devel/vfio-migration.rst line 8

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+(e.g. GPU) have large amount of data to be transfered. The iterative pre-copy
+phase of migration allows for the guest to continue whilst the VFIO device state
+is transferred to destination, this helps to reduce the total downtime of the  

s/to destination,/to the destination;/

docs/devel/vfio-migration.rst line 10

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+VM. VFIO devices can choose to skip the pre-copy phase of migration by returning
+pending_bytes as zero during pre-copy phase.  

s/during/during the/

Quoted 11 lines

docs/devel/vfio-migration.rst line 14

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+
+Detailed description of UAPI for VFIO device for migration is in the comment
+above ``vfio_device_migration_info`` structure definition in header file
+linux-headers/linux/vfio.h.  

I think I'd copy that to this file. If I'm looking at the documentation, I'd rather not go hunting for source code to find out what structure you are talking about. Plus, as it's UAPI, I don't expect it to change much, so it should be easy to keep the definitions in sync (famous last words).

Quoted 2 lines

I feel its duplication of documentation. I would like to know others views as well.

TBH I don't think it's necessary here either, we're documenting the QEMU interaction with the uAPI, the uAPI itself is documented in the kernel header. I don't think it would be unreasonable to ask someone trying to understand this to look at both sources together.

Quoted 46 lines
Quoted 46 lines

docs/devel/vfio-migration.rst line 17

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+
+VFIO device hooks for iterative approach:
+-  A ``save_setup`` function that setup migration region, sets _SAVING flag in  

s/setup/sets up the/
s/in/in the/

docs/devel/vfio-migration.rst line 18

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+VFIO device state and inform VFIO IOMMU module to start dirty page tracking.  

s/inform/informs the/

docs/devel/vfio-migration.rst line 20

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- A ``load_setup`` function that setup migration region on the destination and  

s/setup/sets up the/

docs/devel/vfio-migration.rst line 21

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+sets _RESUMING flag in VFIO device state.  

s/in/in the/

docs/devel/vfio-migration.rst line 25

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+
+- A ``save_live_pending`` function that reads pending_bytes from vendor driver
+that indicate how much more data the vendor driver yet to save for the VFIO
+device.  

"A ``save_live_pending`` function that reads pending_bytes from the vendor driver, which indicates the amount of data that the vendor driver has yet to save for the VFIO device." ?

docs/devel/vfio-migration.rst line 27

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- A ``save_live_iterate`` function that reads VFIO device's data from vendor  

s/reads/reads the/
s/from/from the/

docs/devel/vfio-migration.rst line 28

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+driver through migration region during iterative phase.  

s/through/through the/

docs/devel/vfio-migration.rst line 30

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- A ``save_live_complete_precopy`` function that resets _RUNNING flag from VFIO  

s/from/from the/

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+device state, saves device config space, if any, and iteratively copies  
Quoted 8 lines
Quoted 8 lines

s/saves/saves the/

docs/devel/vfio-migration.rst line 33

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+remaining data for VFIO device till pending_bytes returned by vendor driver
+is zero.  

"...and interactively copies the remaining data for the VFIO device until the vendor driver indicates that no data remains (pending_bytes is zero)." ?

Connie, was that intentional to replace "iteratively" with "interactively"? Iteratively seems correct to me.

Quoted 71 lines
Quoted 63 lines

docs/devel/vfio-migration.rst line 36

Quoted 4 lines
docs/devel/vfio-migration.rst new file +3 -0
+
+- A ``load_state`` function loads config section and data sections generated by
+above save functions.  

"A ``load_state`` function that loads the config section and the data sections that are generated by the save functions above." ?

docs/devel/vfio-migration.rst line 38

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+- ``cleanup`` functions for both save and load that unmap migration region.  

..."that perform any migration-related cleanup, including unmapping the migration region." ?

docs/devel/vfio-migration.rst line 41

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+
+VM state change handler is registered to change VFIO device state based on VM
+state change.  

"A VM state change handler is registered to change the VFIO device state when the VM state changes." ?

docs/devel/vfio-migration.rst line 43

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+
+Similarly, a migration state change notifier is added to get a notification on  

s/added/registered/ ?

docs/devel/vfio-migration.rst line 50

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+migration state change. These states are translated to VFIO device state and
+conveyed to vendor driver.
+
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages  

s/is added to mark/marks those/

docs/devel/vfio-migration.rst line 51

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried  

s/by/by the/
s/Dirty/The dirty/

docs/devel/vfio-migration.rst line 52

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+per container. All pages pinned by vendor driver through vfio_pin_pages()  

s/by/by the/

docs/devel/vfio-migration.rst line 55

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device  

s/by/by the/ (x2)

docs/devel/vfio-migration.rst line 62

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.  

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

Quoted 3 lines

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Now I'm confused, I thought we had abandoned the idea that we can only report pinned pages during stop-and-copy. Doesn't the device needs to expose its dirty memory footprint during the iterative phase regardless of whether that causes repeat copies? If QEMU iterates and sees that all memory is still dirty, it may have transferred more data, but it can actually predict if it can achieve its downtime tolerances. Which is more important, less data transfer or predictability? Thanks,

Alex

Quoted 94 lines
Quoted 93 lines

docs/devel/vfio-migration.rst line 66

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+
+System memory dirty pages tracking when vIOMMU is enabled
+---------------------------------------------------------
+With vIOMMU, IO virtual address range can get unmapped while in pre-copy phase  

s/IO/an I/O/

docs/devel/vfio-migration.rst line 67

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+of migration. In that case, unmap ioctl returns pages pinned in that range and  

s/unmap ioctl returns pages pinned/the unmap ioctl returns any pinned pages/ ?

docs/devel/vfio-migration.rst line 77

Quoted 10 lines
docs/devel/vfio-migration.rst new file +10 -0
+QEMU reports corresponding guest physical pages dirty.
+During stop-and-copy phase, an IOMMU notifier is used to get a callback for
+mapped pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for
+those mapped ranges.
+
+Flow of state changes during Live migration
+===========================================
+Below is the flow of state change during live migration where states in brackets
+represent VM state, migration state and VFIO device state as:
+                (VM state, MIGRATION_STATUS, VFIO_DEVICE_STATE)  

"The values in the brackets represent the VM state, the migration state, and the VFIO device state, respectively." ?

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+
+Live migration save path
+------------------------
+                        QEMU normal running state
+                        (RUNNING, _NONE, _RUNNING)
+                                    |
+                       migrate_init spawns migration_thread

docs/devel/vfio-migration.rst line 94

Quoted 10 lines
docs/devel/vfio-migration.rst new file +10 -0
+                Migration thread then calls each device's .save_setup()
+                        (RUNNING, _SETUP, _RUNNING|_SAVING)
+                                    |
+                        (RUNNING, _ACTIVE, _RUNNING|_SAVING)
+            If device is active, get pending_bytes by .save_live_pending()
+         if total pending_bytes >= threshold_size, call .save_live_iterate()
+                  Data of VFIO device for pre-copy phase is copied
+     Iterate till total pending bytes converge and are less than threshold
+                                    |
+   On migration completion, vCPUs stops and calls .save_live_complete_precopy  

s/vCPUs/vCPU/ ?

docs/devel/vfio-migration.rst line 95

Quoted 1 line
docs/devel/vfio-migration.rst new file +1 -0
+   for each active device. VFIO device is then transitioned in _SAVING state  

s/VFIO device/The VFIO device/

s/in/into/

docs/devel/vfio-migration.rst line 98

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+                    (FINISH_MIGRATE, _DEVICE, _SAVING)
+                                    |
+For VFIO device, iterate in .save_live_complete_precopy until pending data is 0  

s/For/For the/

docs/devel/vfio-migration.rst line 102

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+                    (FINISH_MIGRATE, _DEVICE, _STOPPED)
+                                    |
+                    (FINISH_MIGRATE, _COMPLETED, _STOPPED)
+                Migraton thread schedule cleanup bottom half and exit  

s/schedule/schedules/
s/exit/exits/

docs/devel/vfio-migration.rst line 113

Quoted 11 lines
docs/devel/vfio-migration.rst new file +11 -0
+
+Live migration resume path
+--------------------------
+
+             Incoming migration calls .load_setup for each device
+                        (RESTORE_VM, _ACTIVE, _STOPPED)
+                                    |
+    For each device, .load_state is called for that device section data
+                        (RESTORE_VM, _ACTIVE, _RESUMING)
+                                    |
+    At the end, called .load_cleanup for each device and vCPUs are started                        |  

Stray '|', probably should go on the next line?

s/called .load_cleanup/.load_cleanup is called/

docs/devel/vfio-migration.rst line 119

Quoted 6 lines
docs/devel/vfio-migration.rst new file +6 -0
+                        (RUNNING, _NONE, _RUNNING)
+
+
+Postcopy
+========
+Postcopy migration is not supported for VFIO devices.  

s/not/not yet/, I think? (Disregarding any time frame here.)

Generally, I think this is useful information to have in the docs.

#4 lore raw
Quoted 23 lines · On Thu, 29 Oct 2020 13:05:19 -0600 Alex Williamson…
Quoted 20 lines · On Thu, 29 Oct 2020 23:11:16 +0530 Kirti Wankhede <[email protected]> wrote:

Thanks for corrections Cornelia. I had done the corrections you suggested I had not replied, see my comments on couple of places where I disagree.

Quoted 13 lines · Cornelia Huck

docs/devel/vfio-migration.rst line 14

Quoted 3 lines · On Thu, 29 Oct 2020 11:23:11 +0530 Kirti Wankhede <[email protected]> wrote:
docs/devel/vfio-migration.rst new file +3 -0
+Detailed description of UAPI for VFIO device for migration is in the comment
+above ``vfio_device_migration_info`` structure definition in header file
+linux-headers/linux/vfio.h.    

I think I'd copy that to this file. If I'm looking at the documentation, I'd rather not go hunting for source code to find out what structure you are talking about. Plus, as it's UAPI, I don't expect it to change much, so it should be easy to keep the definitions in sync (famous last words).

Quoted 8 lines
Quoted 2 lines

I feel its duplication of documentation. I would like to know others views as well.

TBH I don't think it's necessary here either, we're documenting the QEMU interaction with the uAPI, the uAPI itself is documented in the kernel header. I don't think it would be unreasonable to ask someone trying to understand this to look at both sources together.

Ok, I can live with that. But let me correct some nits :)

"A detailed description of the UAPI for VFIO device migration can be found in the comment for the ``vfio_device_migration_info`` structure in the header file linux-headers/linux/vfio.h."

(...)

Quoted 10 lines
Quoted 6 lines
Quoted 6 lines

docs/devel/vfio-migration.rst line 33

Quoted 2 lines
docs/devel/vfio-migration.rst new file +2 -0
+remaining data for VFIO device till pending_bytes returned by vendor driver
+is zero.    

"...and interactively copies the remaining data for the VFIO device until the vendor driver indicates that no data remains (pending_bytes is zero)." ?

Connie, was that intentional to replace "iteratively" with "interactively"? Iteratively seems correct to me.

Eh, should be "iteratively", of course. Too many meetings :/

#5 lore raw
In reply to Alex Williamson
Quoted 3 lines · Alex Williamson

On Thu, 29 Oct 2020 23:11:16 +0530
Kirti Wankhede <[email protected]> wrote:

<snip>

Quoted 47 lines
Quoted 42 lines
Quoted 34 lines

docs/devel/vfio-migration.rst line 50

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages

s/is added to mark/marks those/

Quoted 1 line

+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

Quoted 1 line

+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

docs/devel/vfio-migration.rst line 55

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device

s/by/by the/ (x2)

docs/devel/vfio-migration.rst line 62

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

Quoted 3 lines

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Quoted 8 lines

Now I'm confused, I thought we had abandoned the idea that we can only report pinned pages during stop-and-copy. Doesn't the device needs to expose its dirty memory footprint during the iterative phase regardless of whether that causes repeat copies? If QEMU iterates and sees that all memory is still dirty, it may have transferred more data, but it can actually predict if it can achieve its downtime tolerances. Which is more important, less data transfer or predictability? Thanks,

Even if QEMU copies and transfers content of all sys mem pages during pre-copy (worst case with IOMMU backed mdev device when its vendor driver is not smart to pin pages explicitly and all sys mem pages are marked dirty), then also its prediction about downtime tolerance will not be correct, because during stop-and-copy again all pages need to be copied as device can write to any of those pinned pages.

Thanks,
Kirti

#6 lore raw
Quoted 63 lines · On Wed, 4 Nov 2020 01:18:12 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 3 lines · Alex Williamson

On Thu, 29 Oct 2020 23:11:16 +0530
Kirti Wankhede <[email protected]> wrote:

<snip>

Quoted 47 lines
Quoted 42 lines
Quoted 34 lines

docs/devel/vfio-migration.rst line 50

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages  

s/is added to mark/marks those/

Quoted 1 line

+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

Quoted 1 line

+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

docs/devel/vfio-migration.rst line 55

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device  

s/by/by the/ (x2)

docs/devel/vfio-migration.rst line 62

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.  

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

Quoted 3 lines

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Quoted 8 lines

Now I'm confused, I thought we had abandoned the idea that we can only report pinned pages during stop-and-copy. Doesn't the device needs to expose its dirty memory footprint during the iterative phase regardless of whether that causes repeat copies? If QEMU iterates and sees that all memory is still dirty, it may have transferred more data, but it can actually predict if it can achieve its downtime tolerances. Which is more important, less data transfer or predictability? Thanks,

Quoted 6 lines

Even if QEMU copies and transfers content of all sys mem pages during pre-copy (worst case with IOMMU backed mdev device when its vendor driver is not smart to pin pages explicitly and all sys mem pages are marked dirty), then also its prediction about downtime tolerance will not be correct, because during stop-and-copy again all pages need to be copied as device can write to any of those pinned pages.

I think you're only reiterating my point. If QEMU copies all of guest memory during the iterative phase and each time it sees that all memory is dirty, such as if CPUs or devices (including assigned devices) are dirtying pages as fast as it copies them (or continuously marks them dirty), then QEMU can predict that downtime will require copying all pages. If instead devices don't mark dirty pages until the VM is stopped, then QEMU might iterate through memory copy and predict a short downtime because not much memory is dirty, only to be surprised that all of memory is suddenly dirty. At that point it's too late, the VM is already stopped, the predicted short downtime takes far longer than expected. This is exactly why we made the kernel interface mark pinned pages persistently dirty when it was proposed that we only report pinned pages once. Thanks,

Alex

#7 lore raw
Quoted 73 lines · Alex Williamson
Quoted 63 lines · On Wed, 4 Nov 2020 01:18:12 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 3 lines · Alex Williamson

On Thu, 29 Oct 2020 23:11:16 +0530
Kirti Wankhede <[email protected]> wrote:

<snip>

Quoted 47 lines
Quoted 42 lines
Quoted 34 lines

docs/devel/vfio-migration.rst line 50

Quoted 4 lines
docs/devel/vfio-migration.rst new file +4 -0
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages

s/is added to mark/marks those/

Quoted 1 line

+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

Quoted 1 line

+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

docs/devel/vfio-migration.rst line 55

Quoted 3 lines
docs/devel/vfio-migration.rst new file +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device

s/by/by the/ (x2)

docs/devel/vfio-migration.rst line 62

Quoted 7 lines
docs/devel/vfio-migration.rst new file +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

Quoted 3 lines

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Quoted 8 lines

Now I'm confused, I thought we had abandoned the idea that we can only report pinned pages during stop-and-copy. Doesn't the device needs to expose its dirty memory footprint during the iterative phase regardless of whether that causes repeat copies? If QEMU iterates and sees that all memory is still dirty, it may have transferred more data, but it can actually predict if it can achieve its downtime tolerances. Which is more important, less data transfer or predictability? Thanks,

Quoted 6 lines

Even if QEMU copies and transfers content of all sys mem pages during pre-copy (worst case with IOMMU backed mdev device when its vendor driver is not smart to pin pages explicitly and all sys mem pages are marked dirty), then also its prediction about downtime tolerance will not be correct, because during stop-and-copy again all pages need to be copied as device can write to any of those pinned pages.

Quoted 6 lines

I think you're only reiterating my point. If QEMU copies all of guest memory during the iterative phase and each time it sees that all memory is dirty, such as if CPUs or devices (including assigned devices) are dirtying pages as fast as it copies them (or continuously marks them dirty), then QEMU can predict that downtime will require copying all pages.

But as of now there is no way to know if device has dirtied pages during iterative phase.

Quoted 9 lines

If instead devices don't mark dirty pages until the VM is stopped, then QEMU might iterate through memory copy and predict a short downtime because not much memory is dirty, only to be surprised that all of memory is suddenly dirty. At that point it's too late, the VM is already stopped, the predicted short downtime takes far longer than expected. This is exactly why we made the kernel interface mark pinned pages persistently dirty when it was proposed that we only report pinned pages once. Thanks,

Since there is no way to know if device dirtied pages during iterative phase, QEMU should query pinned pages in stop-and-copy phase.

Whenever there will be hardware support or some software mechanism to report pages dirtied by device then we will add a capability bit in migration capability and based on that capability bit qemu/user space app should decide to query dirty pages in iterative phase.

Thanks,
Kirti

#8 lore raw
Quoted 81 lines · On Wed, 4 Nov 2020 13:25:40 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 73 lines · Alex Williamson
Quoted 63 lines · On Wed, 4 Nov 2020 01:18:12 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 3 lines · Alex Williamson

On Thu, 29 Oct 2020 23:11:16 +0530
Kirti Wankhede <[email protected]> wrote:

<snip>

Quoted 47 lines
Quoted 42 lines
Quoted 34 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages  

s/is added to mark/marks those/

Quoted 1 line

+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

Quoted 1 line

+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

Review

Quoted 3 lines
(quoted diff) +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device  

s/by/by the/ (x2)

Review

Quoted 7 lines
(quoted diff) +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.  

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

Quoted 3 lines

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Quoted 8 lines

Now I'm confused, I thought we had abandoned the idea that we can only report pinned pages during stop-and-copy. Doesn't the device needs to expose its dirty memory footprint during the iterative phase regardless of whether that causes repeat copies? If QEMU iterates and sees that all memory is still dirty, it may have transferred more data, but it can actually predict if it can achieve its downtime tolerances. Which is more important, less data transfer or predictability? Thanks,

Quoted 6 lines

Even if QEMU copies and transfers content of all sys mem pages during pre-copy (worst case with IOMMU backed mdev device when its vendor driver is not smart to pin pages explicitly and all sys mem pages are marked dirty), then also its prediction about downtime tolerance will not be correct, because during stop-and-copy again all pages need to be copied as device can write to any of those pinned pages.

Quoted 6 lines

I think you're only reiterating my point. If QEMU copies all of guest memory during the iterative phase and each time it sees that all memory is dirty, such as if CPUs or devices (including assigned devices) are dirtying pages as fast as it copies them (or continuously marks them dirty), then QEMU can predict that downtime will require copying all pages.

Quoted 2 lines

But as of now there is no way to know if device has dirtied pages during iterative phase.

This claim doesn't make any sense, pinned pages are considered persistently dirtied, during the iterative phase and while stopped.

Quoted 10 lines
Quoted 9 lines

If instead devices don't mark dirty pages until the VM is stopped, then QEMU might iterate through memory copy and predict a short downtime because not much memory is dirty, only to be surprised that all of memory is suddenly dirty. At that point it's too late, the VM is already stopped, the predicted short downtime takes far longer than expected. This is exactly why we made the kernel interface mark pinned pages persistently dirty when it was proposed that we only report pinned pages once. Thanks,

Quoted 2 lines

Since there is no way to know if device dirtied pages during iterative phase, QEMU should query pinned pages in stop-and-copy phase.

As above, I don't believe this is true.

Quoted 4 lines

Whenever there will be hardware support or some software mechanism to report pages dirtied by device then we will add a capability bit in migration capability and based on that capability bit qemu/user space app should decide to query dirty pages in iterative phase.

Yes, we could advertise support for fine granularity dirty page tracking, but I completely disagree that we should consider pinned pages clean until suddenly exposing them as dirty once the VM is stopped. Thanks,

Alex

#9 lore raw
Quoted 115 lines · Alex Williamson
Quoted 81 lines · On Wed, 4 Nov 2020 13:25:40 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 73 lines · Alex Williamson
Quoted 63 lines · On Wed, 4 Nov 2020 01:18:12 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 3 lines · Alex Williamson

On Thu, 29 Oct 2020 23:11:16 +0530
Kirti Wankhede <[email protected]> wrote:

<snip>

Quoted 47 lines
Quoted 42 lines
Quoted 34 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages

s/is added to mark/marks those/

Quoted 1 line

+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

Quoted 1 line

+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

Review

Quoted 3 lines
(quoted diff) +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device

s/by/by the/ (x2)

Review

Quoted 7 lines
(quoted diff) +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

Quoted 3 lines

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Quoted 8 lines

Now I'm confused, I thought we had abandoned the idea that we can only report pinned pages during stop-and-copy. Doesn't the device needs to expose its dirty memory footprint during the iterative phase regardless of whether that causes repeat copies? If QEMU iterates and sees that all memory is still dirty, it may have transferred more data, but it can actually predict if it can achieve its downtime tolerances. Which is more important, less data transfer or predictability? Thanks,

Quoted 6 lines

Even if QEMU copies and transfers content of all sys mem pages during pre-copy (worst case with IOMMU backed mdev device when its vendor driver is not smart to pin pages explicitly and all sys mem pages are marked dirty), then also its prediction about downtime tolerance will not be correct, because during stop-and-copy again all pages need to be copied as device can write to any of those pinned pages.

Quoted 6 lines

I think you're only reiterating my point. If QEMU copies all of guest memory during the iterative phase and each time it sees that all memory is dirty, such as if CPUs or devices (including assigned devices) are dirtying pages as fast as it copies them (or continuously marks them dirty), then QEMU can predict that downtime will require copying all pages.

Quoted 2 lines

But as of now there is no way to know if device has dirtied pages during iterative phase.

This claim doesn't make any sense, pinned pages are considered persistently dirtied, during the iterative phase and while stopped.

Quoted 10 lines
Quoted 9 lines

If instead devices don't mark dirty pages until the VM is stopped, then QEMU might iterate through memory copy and predict a short downtime because not much memory is dirty, only to be surprised that all of memory is suddenly dirty. At that point it's too late, the VM is already stopped, the predicted short downtime takes far longer than expected. This is exactly why we made the kernel interface mark pinned pages persistently dirty when it was proposed that we only report pinned pages once. Thanks,

Quoted 2 lines

Since there is no way to know if device dirtied pages during iterative phase, QEMU should query pinned pages in stop-and-copy phase.

As above, I don't believe this is true.

Quoted 4 lines

Whenever there will be hardware support or some software mechanism to report pages dirtied by device then we will add a capability bit in migration capability and based on that capability bit qemu/user space app should decide to query dirty pages in iterative phase.

Quoted 5 lines

Yes, we could advertise support for fine granularity dirty page tracking, but I completely disagree that we should consider pinned pages clean until suddenly exposing them as dirty once the VM is stopped. Thanks,

Should QEMU copy dirtied pages twice, during iterative phase and then when VM is stopped?

Thanks,
Kirti

#10 lore raw
Quoted 116 lines · On Fri, 6 Nov 2020 00:29:36 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 115 lines · Alex Williamson
Quoted 81 lines · On Wed, 4 Nov 2020 13:25:40 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 73 lines · Alex Williamson
Quoted 63 lines · On Wed, 4 Nov 2020 01:18:12 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 3 lines · Alex Williamson

On Thu, 29 Oct 2020 23:11:16 +0530
Kirti Wankhede <[email protected]> wrote:

<snip>

Quoted 47 lines
Quoted 42 lines
Quoted 34 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages  

s/is added to mark/marks those/

Quoted 1 line

+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

Quoted 1 line

+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

Review

Quoted 3 lines
(quoted diff) +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device  

s/by/by the/ (x2)

Review

Quoted 7 lines
(quoted diff) +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.  

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

Quoted 3 lines

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Quoted 8 lines

Now I'm confused, I thought we had abandoned the idea that we can only report pinned pages during stop-and-copy. Doesn't the device needs to expose its dirty memory footprint during the iterative phase regardless of whether that causes repeat copies? If QEMU iterates and sees that all memory is still dirty, it may have transferred more data, but it can actually predict if it can achieve its downtime tolerances. Which is more important, less data transfer or predictability? Thanks,

Quoted 6 lines

Even if QEMU copies and transfers content of all sys mem pages during pre-copy (worst case with IOMMU backed mdev device when its vendor driver is not smart to pin pages explicitly and all sys mem pages are marked dirty), then also its prediction about downtime tolerance will not be correct, because during stop-and-copy again all pages need to be copied as device can write to any of those pinned pages.

Quoted 6 lines

I think you're only reiterating my point. If QEMU copies all of guest memory during the iterative phase and each time it sees that all memory is dirty, such as if CPUs or devices (including assigned devices) are dirtying pages as fast as it copies them (or continuously marks them dirty), then QEMU can predict that downtime will require copying all pages.

Quoted 2 lines

But as of now there is no way to know if device has dirtied pages during iterative phase.

This claim doesn't make any sense, pinned pages are considered persistently dirtied, during the iterative phase and while stopped.

Quoted 10 lines
Quoted 9 lines

If instead devices don't mark dirty pages until the VM is stopped, then QEMU might iterate through memory copy and predict a short downtime because not much memory is dirty, only to be surprised that all of memory is suddenly dirty. At that point it's too late, the VM is already stopped, the predicted short downtime takes far longer than expected. This is exactly why we made the kernel interface mark pinned pages persistently dirty when it was proposed that we only report pinned pages once. Thanks,

Quoted 2 lines

Since there is no way to know if device dirtied pages during iterative phase, QEMU should query pinned pages in stop-and-copy phase.

As above, I don't believe this is true.

Quoted 4 lines

Whenever there will be hardware support or some software mechanism to report pages dirtied by device then we will add a capability bit in migration capability and based on that capability bit qemu/user space app should decide to query dirty pages in iterative phase.

Quoted 8 lines
Quoted 5 lines

Yes, we could advertise support for fine granularity dirty page tracking, but I completely disagree that we should consider pinned pages clean until suddenly exposing them as dirty once the VM is stopped. Thanks,

Should QEMU copy dirtied pages twice, during iterative phase and then when VM is stopped?

I don't understand why this is controversial. We cannot decide within the vfio device to only expose device dirtied pages in the final stage of migration. It's not our job to minimize the number of pages copied beyond the hardware granularity. If core QEMU migration code asks for dirty pages, we provide them, regardless of how many times we report a page as dirty. So yes, if that migration code asks for dirty pages in the iterative stage and the stopped stage, we provide them both times. If someone wants to skip the iterative phase altogether, I imagine there are migration parameters that allow it, but we should not be determining that policy at the device level. Thanks,

Alex

#11 lore raw
Quoted 128 lines · Alex Williamson
Quoted 116 lines · On Fri, 6 Nov 2020 00:29:36 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 115 lines · Alex Williamson
Quoted 81 lines · On Wed, 4 Nov 2020 13:25:40 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 73 lines · Alex Williamson
Quoted 63 lines · On Wed, 4 Nov 2020 01:18:12 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 3 lines · Alex Williamson

On Thu, 29 Oct 2020 23:11:16 +0530
Kirti Wankhede <[email protected]> wrote:

<snip>

Quoted 47 lines
Quoted 42 lines
Quoted 34 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages

s/is added to mark/marks those/

Quoted 1 line

+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

Quoted 1 line

+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

Review

Quoted 3 lines
(quoted diff) +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device

s/by/by the/ (x2)

Review

Quoted 7 lines
(quoted diff) +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

Quoted 3 lines

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Quoted 8 lines

Now I'm confused, I thought we had abandoned the idea that we can only report pinned pages during stop-and-copy. Doesn't the device needs to expose its dirty memory footprint during the iterative phase regardless of whether that causes repeat copies? If QEMU iterates and sees that all memory is still dirty, it may have transferred more data, but it can actually predict if it can achieve its downtime tolerances. Which is more important, less data transfer or predictability? Thanks,

Quoted 6 lines

Even if QEMU copies and transfers content of all sys mem pages during pre-copy (worst case with IOMMU backed mdev device when its vendor driver is not smart to pin pages explicitly and all sys mem pages are marked dirty), then also its prediction about downtime tolerance will not be correct, because during stop-and-copy again all pages need to be copied as device can write to any of those pinned pages.

Quoted 6 lines

I think you're only reiterating my point. If QEMU copies all of guest memory during the iterative phase and each time it sees that all memory is dirty, such as if CPUs or devices (including assigned devices) are dirtying pages as fast as it copies them (or continuously marks them dirty), then QEMU can predict that downtime will require copying all pages.

Quoted 2 lines

But as of now there is no way to know if device has dirtied pages during iterative phase.

This claim doesn't make any sense, pinned pages are considered persistently dirtied, during the iterative phase and while stopped.

Quoted 10 lines
Quoted 9 lines

If instead devices don't mark dirty pages until the VM is stopped, then QEMU might iterate through memory copy and predict a short downtime because not much memory is dirty, only to be surprised that all of memory is suddenly dirty. At that point it's too late, the VM is already stopped, the predicted short downtime takes far longer than expected. This is exactly why we made the kernel interface mark pinned pages persistently dirty when it was proposed that we only report pinned pages once. Thanks,

Quoted 2 lines

Since there is no way to know if device dirtied pages during iterative phase, QEMU should query pinned pages in stop-and-copy phase.

As above, I don't believe this is true.

Quoted 4 lines

Whenever there will be hardware support or some software mechanism to report pages dirtied by device then we will add a capability bit in migration capability and based on that capability bit qemu/user space app should decide to query dirty pages in iterative phase.

Quoted 8 lines
Quoted 5 lines

Yes, we could advertise support for fine granularity dirty page tracking, but I completely disagree that we should consider pinned pages clean until suddenly exposing them as dirty once the VM is stopped. Thanks,

Should QEMU copy dirtied pages twice, during iterative phase and then when VM is stopped?

Quoted 7 lines

I don't understand why this is controversial. We cannot decide within the vfio device to only expose device dirtied pages in the final stage of migration. It's not our job to minimize the number of pages copied beyond the hardware granularity. If core QEMU migration code asks for dirty pages, we provide them, regardless of how many times we report a page as dirty. So yes, if that migration code asks for dirty pages in the iterative stage and the stopped stage, we provide them both times.

Isn't that would increase total migration time?

Quoted 4 lines

If someone wants to skip the iterative phase altogether, I imagine there are migration parameters that allow it, but we should not be determining that policy at the device level. Thanks,

What is that parameter? should that be documented here?

Thanks,
Kirti

#12 lore raw
Quoted 137 lines · On Fri, 6 Nov 2020 02:22:11 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 128 lines · Alex Williamson
Quoted 116 lines · On Fri, 6 Nov 2020 00:29:36 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 115 lines · Alex Williamson
Quoted 81 lines · On Wed, 4 Nov 2020 13:25:40 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 73 lines · Alex Williamson
Quoted 63 lines · On Wed, 4 Nov 2020 01:18:12 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 3 lines · Alex Williamson

On Thu, 29 Oct 2020 23:11:16 +0530
Kirti Wankhede <[email protected]> wrote:

<snip>

Quoted 47 lines
Quoted 42 lines
Quoted 34 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages  

s/is added to mark/marks those/

Quoted 1 line

+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

Quoted 1 line

+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

Review

Quoted 3 lines
(quoted diff) +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device  

s/by/by the/ (x2)

Review

Quoted 7 lines
(quoted diff) +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.  

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

Quoted 3 lines

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Quoted 8 lines

Now I'm confused, I thought we had abandoned the idea that we can only report pinned pages during stop-and-copy. Doesn't the device needs to expose its dirty memory footprint during the iterative phase regardless of whether that causes repeat copies? If QEMU iterates and sees that all memory is still dirty, it may have transferred more data, but it can actually predict if it can achieve its downtime tolerances. Which is more important, less data transfer or predictability? Thanks,

Quoted 6 lines

Even if QEMU copies and transfers content of all sys mem pages during pre-copy (worst case with IOMMU backed mdev device when its vendor driver is not smart to pin pages explicitly and all sys mem pages are marked dirty), then also its prediction about downtime tolerance will not be correct, because during stop-and-copy again all pages need to be copied as device can write to any of those pinned pages.

Quoted 6 lines

I think you're only reiterating my point. If QEMU copies all of guest memory during the iterative phase and each time it sees that all memory is dirty, such as if CPUs or devices (including assigned devices) are dirtying pages as fast as it copies them (or continuously marks them dirty), then QEMU can predict that downtime will require copying all pages.

Quoted 2 lines

But as of now there is no way to know if device has dirtied pages during iterative phase.

This claim doesn't make any sense, pinned pages are considered persistently dirtied, during the iterative phase and while stopped.

Quoted 10 lines
Quoted 9 lines

If instead devices don't mark dirty pages until the VM is stopped, then QEMU might iterate through memory copy and predict a short downtime because not much memory is dirty, only to be surprised that all of memory is suddenly dirty. At that point it's too late, the VM is already stopped, the predicted short downtime takes far longer than expected. This is exactly why we made the kernel interface mark pinned pages persistently dirty when it was proposed that we only report pinned pages once. Thanks,

Quoted 2 lines

Since there is no way to know if device dirtied pages during iterative phase, QEMU should query pinned pages in stop-and-copy phase.

As above, I don't believe this is true.

Quoted 4 lines

Whenever there will be hardware support or some software mechanism to report pages dirtied by device then we will add a capability bit in migration capability and based on that capability bit qemu/user space app should decide to query dirty pages in iterative phase.

Quoted 8 lines
Quoted 5 lines

Yes, we could advertise support for fine granularity dirty page tracking, but I completely disagree that we should consider pinned pages clean until suddenly exposing them as dirty once the VM is stopped. Thanks,

Should QEMU copy dirtied pages twice, during iterative phase and then when VM is stopped?

Quoted 7 lines

I don't understand why this is controversial. We cannot decide within the vfio device to only expose device dirtied pages in the final stage of migration. It's not our job to minimize the number of pages copied beyond the hardware granularity. If core QEMU migration code asks for dirty pages, we provide them, regardless of how many times we report a page as dirty. So yes, if that migration code asks for dirty pages in the iterative stage and the stopped stage, we provide them both times.

Quoted 1 line

Isn't that would increase total migration time?

As I explained, that's not a policy decision that we as a device within the VM should be making. We do not have the visibility to determine how the footprint of our device will affect the migration and by preventing QEMU migration code from understanding the device footprint, we're creating a scenario where QEMU absolutely cannot predict the downtime.

Quoted 6 lines
Quoted 4 lines

If someone wants to skip the iterative phase altogether, I imagine there are migration parameters that allow it, but we should not be determining that policy at the device level. Thanks,

What is that parameter? should that be documented here?

Dunno, but clearly we could pause the VM, migration, and resume on the target. I imagine there are migration tuning parameters that might do essentially that automatically. Thanks,

Alex

#13 lore raw
Quoted 140 lines · Alex Williamson
Quoted 137 lines · On Fri, 6 Nov 2020 02:22:11 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 135 lines · Alex Williamson
Quoted 116 lines · On Fri, 6 Nov 2020 00:29:36 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 115 lines · Alex Williamson
Quoted 81 lines · On Wed, 4 Nov 2020 13:25:40 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 73 lines · Alex Williamson
Quoted 63 lines · On Wed, 4 Nov 2020 01:18:12 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 3 lines · Alex Williamson

On Thu, 29 Oct 2020 23:11:16 +0530
Kirti Wankhede <[email protected]> wrote:

<snip>

Quoted 47 lines
Quoted 42 lines
Quoted 34 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages

s/is added to mark/marks those/

Quoted 1 line

+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

Quoted 1 line

+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

Review

Quoted 3 lines
(quoted diff) +3 -0
+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device

s/by/by the/ (x2)

Review

Quoted 7 lines
(quoted diff) +7 -0
+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

Quoted 3 lines

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Quoted 8 lines

Now I'm confused, I thought we had abandoned the idea that we can only report pinned pages during stop-and-copy. Doesn't the device needs to expose its dirty memory footprint during the iterative phase regardless of whether that causes repeat copies? If QEMU iterates and sees that all memory is still dirty, it may have transferred more data, but it can actually predict if it can achieve its downtime tolerances. Which is more important, less data transfer or predictability? Thanks,

Quoted 6 lines

Even if QEMU copies and transfers content of all sys mem pages during pre-copy (worst case with IOMMU backed mdev device when its vendor driver is not smart to pin pages explicitly and all sys mem pages are marked dirty), then also its prediction about downtime tolerance will not be correct, because during stop-and-copy again all pages need to be copied as device can write to any of those pinned pages.

Quoted 6 lines

I think you're only reiterating my point. If QEMU copies all of guest memory during the iterative phase and each time it sees that all memory is dirty, such as if CPUs or devices (including assigned devices) are dirtying pages as fast as it copies them (or continuously marks them dirty), then QEMU can predict that downtime will require copying all pages.

Quoted 2 lines

But as of now there is no way to know if device has dirtied pages during iterative phase.

This claim doesn't make any sense, pinned pages are considered persistently dirtied, during the iterative phase and while stopped.

Quoted 10 lines
Quoted 9 lines

If instead devices don't mark dirty pages until the VM is stopped, then QEMU might iterate through memory copy and predict a short downtime because not much memory is dirty, only to be surprised that all of memory is suddenly dirty. At that point it's too late, the VM is already stopped, the predicted short downtime takes far longer than expected. This is exactly why we made the kernel interface mark pinned pages persistently dirty when it was proposed that we only report pinned pages once. Thanks,

Quoted 2 lines

Since there is no way to know if device dirtied pages during iterative phase, QEMU should query pinned pages in stop-and-copy phase.

As above, I don't believe this is true.

Quoted 4 lines

Whenever there will be hardware support or some software mechanism to report pages dirtied by device then we will add a capability bit in migration capability and based on that capability bit qemu/user space app should decide to query dirty pages in iterative phase.

Quoted 8 lines
Quoted 5 lines

Yes, we could advertise support for fine granularity dirty page tracking, but I completely disagree that we should consider pinned pages clean until suddenly exposing them as dirty once the VM is stopped. Thanks,

Should QEMU copy dirtied pages twice, during iterative phase and then when VM is stopped?

I don't understand why this is controversial. We cannot decide within the vfio device to only expose device dirtied pages in the final stage of migration. It's not our job to minimize the number of pages copied beyond the hardware granularity. If core QEMU migration code asks for dirty pages, we provide them, regardless of how many times we report a page as dirty. So yes, if that migration code asks for dirty pages in the iterative stage and the stopped stage, we provide them both times.

Quoted 9 lines
Quoted 1 line

Isn't that would increase total migration time?

As I explained, that's not a policy decision that we as a device within the VM should be making. We do not have the visibility to determine how the footprint of our device will affect the migration and by preventing QEMU migration code from understanding the device footprint, we're creating a scenario where QEMU absolutely cannot predict the downtime.

If VFIO device dirtied lot of memory and each time VFIO device reports all those as dirty during iterative phase, then migration would never converge.

If there is a way to track dirty pages during iterative phase and there is a way to know that pages weren't dirtied again, then it makes sense for them to reported during the iterative phase.

Since there is no hardware support to track pages dirtied by device, for now best way would be to report those during stop-and-copy.

Thanks,
Kirti

Quoted 13 lines
Quoted 6 lines
Quoted 4 lines

If someone wants to skip the iterative phase altogether, I imagine there are migration parameters that allow it, but we should not be determining that policy at the device level. Thanks,

What is that parameter? should that be documented here?

Dunno, but clearly we could pause the VM, migration, and resume on the target. I imagine there are migration tuning parameters that might do essentially that automatically. Thanks,

Alex

#14 lore raw
Quoted 155 lines · On Sat, 7 Nov 2020 00:27:46 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 140 lines · Alex Williamson
Quoted 137 lines · On Fri, 6 Nov 2020 02:22:11 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 135 lines · Alex Williamson
Quoted 116 lines · On Fri, 6 Nov 2020 00:29:36 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 115 lines · Alex Williamson
Quoted 81 lines · On Wed, 4 Nov 2020 13:25:40 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 73 lines · Alex Williamson
Quoted 63 lines · On Wed, 4 Nov 2020 01:18:12 +0530 Kirti Wankhede <[email protected]> wrote:
Quoted 3 lines · Alex Williamson

On Thu, 29 Oct 2020 23:11:16 +0530
Kirti Wankhede <[email protected]> wrote:

<snip>

Quoted 47 lines
Quoted 42 lines
Quoted 34 lines
Quoted 4 lines

+System memory dirty pages tracking
+----------------------------------
+
+A ``log_sync`` memory listener callback is added to mark system memory pages

s/is added to mark/marks those/

Quoted 1 line

+as dirty which are used for DMA by VFIO device. Dirty pages bitmap is queried

s/by/by the/
s/Dirty/The dirty/

Quoted 1 line

+per container. All pages pinned by vendor driver through vfio_pin_pages()

s/by/by the/

Quoted 3 lines

+external API have to be marked as dirty during migration. When there are CPU
+writes, CPU dirty page tracking can identify dirtied pages, but any page pinned
+by vendor driver can also be written by device. There is currently no device

s/by/by the/ (x2)

Quoted 7 lines

+which has hardware support for dirty page tracking. So all pages which are
+pinned by vendor driver are considered as dirty.
+Dirty pages are tracked when device is in stop-and-copy phase because if pages
+are marked dirty during pre-copy phase and content is transfered from source to
+destination, there is no way to know newly dirtied pages from the point they
+were copied earlier until device stops. To avoid repeated copy of same content,
+pinned pages are marked dirty only during stop-and-copy phase.

Let me take a quick stab at rewriting this paragraph (not sure if I understood it correctly):

Quoted 7 lines

"Dirty pages are tracked when the device is in the stop-and-copy phase. During the pre-copy phase, it is not possible to distinguish a dirty page that has been transferred from the source to the destination from newly dirtied pages, which would lead to repeated copying of the same content. Therefore, pinned pages are only marked dirty during the stop-and-copy phase." ?

Quoted 3 lines

I think above rephrase only talks about repeated copying in pre-copy phase. Used "copied earlier until device stops" to indicate both pre-copy and stop-and-copy till device stops.

Quoted 8 lines

Now I'm confused, I thought we had abandoned the idea that we can only report pinned pages during stop-and-copy. Doesn't the device needs to expose its dirty memory footprint during the iterative phase regardless of whether that causes repeat copies? If QEMU iterates and sees that all memory is still dirty, it may have transferred more data, but it can actually predict if it can achieve its downtime tolerances. Which is more important, less data transfer or predictability? Thanks,

Quoted 6 lines

Even if QEMU copies and transfers content of all sys mem pages during pre-copy (worst case with IOMMU backed mdev device when its vendor driver is not smart to pin pages explicitly and all sys mem pages are marked dirty), then also its prediction about downtime tolerance will not be correct, because during stop-and-copy again all pages need to be copied as device can write to any of those pinned pages.

Quoted 6 lines

I think you're only reiterating my point. If QEMU copies all of guest memory during the iterative phase and each time it sees that all memory is dirty, such as if CPUs or devices (including assigned devices) are dirtying pages as fast as it copies them (or continuously marks them dirty), then QEMU can predict that downtime will require copying all pages.

Quoted 2 lines

But as of now there is no way to know if device has dirtied pages during iterative phase.

This claim doesn't make any sense, pinned pages are considered persistently dirtied, during the iterative phase and while stopped.

Quoted 10 lines
Quoted 9 lines

If instead devices don't mark dirty pages until the VM is stopped, then QEMU might iterate through memory copy and predict a short downtime because not much memory is dirty, only to be surprised that all of memory is suddenly dirty. At that point it's too late, the VM is already stopped, the predicted short downtime takes far longer than expected. This is exactly why we made the kernel interface mark pinned pages persistently dirty when it was proposed that we only report pinned pages once. Thanks,

Quoted 2 lines

Since there is no way to know if device dirtied pages during iterative phase, QEMU should query pinned pages in stop-and-copy phase.

As above, I don't believe this is true.

Quoted 4 lines

Whenever there will be hardware support or some software mechanism to report pages dirtied by device then we will add a capability bit in migration capability and based on that capability bit qemu/user space app should decide to query dirty pages in iterative phase.

Quoted 8 lines
Quoted 5 lines

Yes, we could advertise support for fine granularity dirty page tracking, but I completely disagree that we should consider pinned pages clean until suddenly exposing them as dirty once the VM is stopped. Thanks,

Should QEMU copy dirtied pages twice, during iterative phase and then when VM is stopped?

I don't understand why this is controversial. We cannot decide within the vfio device to only expose device dirtied pages in the final stage of migration. It's not our job to minimize the number of pages copied beyond the hardware granularity. If core QEMU migration code asks for dirty pages, we provide them, regardless of how many times we report a page as dirty. So yes, if that migration code asks for dirty pages in the iterative stage and the stopped stage, we provide them both times.

Quoted 9 lines
Quoted 1 line

Isn't that would increase total migration time?

As I explained, that's not a policy decision that we as a device within the VM should be making. We do not have the visibility to determine how the footprint of our device will affect the migration and by preventing QEMU migration code from understanding the device footprint, we're creating a scenario where QEMU absolutely cannot predict the downtime.

If VFIO device dirtied lot of memory and each time VFIO device reports all those as dirty during iterative phase, then migration would never converge.

Quoted 6 lines

If there is a way to track dirty pages during iterative phase and there is a way to know that pages weren't dirtied again, then it makes sense for them to reported during the iterative phase.

Since there is no hardware support to track pages dirtied by device, for now best way would be to report those during stop-and-copy.

You're only re-iterating what we already know and what I've already disagreed with you about. If QEMU assumes that migration is converging because we're not reporting dirty pages during the iterative phase, that's a FALSE conclusion and exactly the crux of my argument that we must provide dirty page data as accurately as possible to QEMU. QEMU needs to know if the migration is not converging and the acceptable downtime needs to be adjusted at the management layers to account for that. Thanks,

Alex