vfio/migration: Make VFIO migration non-experimental
cover vfio/migration: Make VFIO migration non-experimental
Hello,
The major parts of VFIO migration are supported today in QEMU. This includes basic VFIO migration, device dirty page tracking and precopy support. Thus, at this point in time, it seems appropriate to make VFIO migration non-experimental.
This short series (which is based on the precopy series [1]) does that and also adds a few improvements:
- Patch #1 moves the transition from STOP_COPY to STOP state to vfio_save_cleanup(). Testing with a ConnectX-7 VFIO device showed that this can reduce downtime by 6% with loaded devices.
- Patch #2 resets bytes_transferred counter properly.
- Patch #3 cleans up the VFIO migration realize flow and makes VFIO migration non-experimental.
Note that Zhenzhong's series [2] fixes additional bugs and further cleans the VFIO migration realize flow.
Thanks.
[1]
https://lore.kernel.org/qemu-devel/[email protected]/
[2]
https://lore.kernel.org/qemu-devel/[email protected]/
Avihai Horon (3):
vfio/migration: Move from STOP_COPY to STOP in vfio_save_cleanup() vfio/migration: Reset bytes_transferred properly vfio/migration: Make VFIO migration non-experimental
include/hw/vfio/vfio-common.h | 3 +- migration/migration.h | 1 + hw/vfio/migration.c | 54 ++++++++++++++++++++++------------- hw/vfio/pci.c | 4 +-- migration/migration.c | 1 + migration/savevm.c | 1 + migration/target.c | 17 +++++++++-- 7 files changed, 56 insertions(+), 25 deletions(-)
Signature · 1 line
2.26.3
PATCH 1/3 vfio/migration: Move from STOP_COPY to STOP in vfio_save_cleanup()
Changing the device state from STOP_COPY to STOP can take time as the device may need to free resources and do other operations as part of the transition. Currently, this is done in vfio_save_complete_precopy() and therefore it is counted in the migration downtime.
To avoid this, change the device state from STOP_COPY to STOP in vfio_save_cleanup(), which is called after migration has completed and thus is not part of migration downtime.
--- hw/vfio/migration.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
hw/vfio/migration.c +13 -6
@@ -383,6 +383,19 @@ static void vfio_save_cleanup(void *opaque)VFIODevice *vbasedev = opaque;VFIOMigration *migration = vbasedev->migration; +/* +* Changing device state from STOP_COPY to STOP can take time. Do it here, +* after migration has completed, so it won't increase downtime. +*/ +if (migration->device_state ==VFIO_DEVICE_STATE_STOP_COPY ) { +/* +* If setting the device in STOP state fails, the device should be +* reset. To do so, use ERROR state as a recover state. +*/ +vfio_migration_set_state (vbasedev,VFIO_DEVICE_STATE_STOP , +VFIO_DEVICE_STATE_ERROR ); + } +g_free (migration->data_buffer); migration->data_buffer =NULL ; migration->precopy_init_size =0 ;
@@ -508,12 +521,6 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)return ret; } -/* -* If setting the device in STOP state fails, the device should be reset. -* To do so, use ERROR state as a recover state. -*/ - ret =vfio_migration_set_state (vbasedev,VFIO_DEVICE_STATE_STOP , -VFIO_DEVICE_STATE_ERROR );trace_vfio_save_complete_precopy (vbasedev->name, ret);return ret;
Signature · 1 line
2.26.3
Re: [PATCH 1/3] vfio/migration: Move from STOP_COPY to STOP in vfio_save_cleanup()
Quoted 8 lines · Avihai Horon
Changing the device state from
STOP_COPYto STOP can take time as the device may need to free resources and do other operations as part of the transition. Currently, this is done invfio_save_complete_precopy()and therefore it is counted in the migration downtime.To avoid this, change the device state from
STOP_COPYto STOP invfio_save_cleanup(), which is called after migration has completed and thus is not part of migration downtime.
That's optimization and
Quoted 30 lines
Signed-off-by Avihai Horon <[email protected]>--- hw/vfio/migration.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)hw/vfio/migration.c +13
-0@@ -383,6 +383,19 @@ static void vfio_save_cleanup(void *opaque)VFIODevice *vbasedev = opaque;VFIOMigration *migration = vbasedev->migration; +/* +* Changing device state from STOP_COPY to STOP can take time. Do it here, +* after migration has completed, so it won't increase downtime. +*/ +if (migration->device_state ==VFIO_DEVICE_STATE_STOP_COPY ) { +/* +* If setting the device in STOP state fails, the device should be +* reset. To do so, use ERROR state as a recover state. +*/ +vfio_migration_set_state (vbasedev,VFIO_DEVICE_STATE_STOP , +VFIO_DEVICE_STATE_ERROR ); + } +g_free (migration->data_buffer); migration->data_buffer =NULL ; migration->precopy_init_size =0 ;
hw/vfio/migration.c static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) · line 516
Quoted 9 lines
hw/vfio/migration.c +0
-6@@ -508,12 +521,6 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)return ret; } -/* -* If setting the device in STOP state fails, the device should be reset. -* To do so, use ERROR state as a recover state. -*/ - ret =vfio_migration_set_state (vbasedev,VFIO_DEVICE_STATE_STOP , -VFIO_DEVICE_STATE_ERROR );
we loose the possible returned error. Let's keep that change for later.
Thanks,
C.
Quoted 3 lines
hw/vfio/migration.c +0
-0trace_vfio_save_complete_precopy (vbasedev->name, ret);return ret;
Re: [PATCH 1/3] vfio/migration: Move from STOP_COPY to STOP in vfio_save_cleanup()
Quoted 51 lines · Cédric Le Goater
External email: Use caution opening links or attachments
Quoted 8 lines · Avihai Horon
Changing the device state from
STOP_COPYto STOP can take time as the device may need to free resources and do other operations as part of the transition. Currently, this is done invfio_save_complete_precopy()and therefore it is counted in the migration downtime.To avoid this, change the device state from
STOP_COPYto STOP invfio_save_cleanup(), which is called after migration has completed and thus is not part of migration downtime.That's optimization and
Quoted 36 lines
Signed-off-by Avihai Horon <[email protected]>---
hw/vfio/migration.c| 19 +++++++++++++------1 file changed, 13 insertions(+), 6 deletions(-)hw/vfio/migration.c +7
-0@@ -383,6 +383,19 @@ static void vfio_save_cleanup(void *opaque) VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; + /* + * Changing device state from STOP_COPY to STOP can take time. Do it here, + * after migration has completed, so it won't increase downtime. + */ + if (migration->device_state == VFIO_DEVICE_STATE_STOP_COPY) { + /* +* If setting the device in STOP state fails, the device should be
+ * reset. To do so, use ERROR state as a recover state. + */
+ vfio_migration_set_state(vbasedev,VFIO_DEVICE_STATE_STOP, +VFIO_DEVICE_STATE_ERROR); + }
+
g_free(migration->data_buffer);migration->data_buffer= NULL;migration->precopy_init_size= 0;hw/vfio/migration.c +0
-0@@ -508,12 +521,6 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) return ret; }
Quoted 9 lines
Quoted 7 lines
hw/vfio/migration.c +0
-2- /* -* If setting the device in STOP state fails, the device should be reset.
- * To do so, use ERROR state as a recover state.
- */
- ret = vfio_migration_set_state(vbasedev,VFIO_DEVICE_STATE_STOP, -VFIO_DEVICE_STATE_ERROR);we loose the possible returned error.
Right, but vfio_save_cleanup() is called as part of migration cleanup flow after migration has finished, so I don't think returning an error is useful here and the error is still reported in vfio_migration_set_state() so we don't really loose it.
Quoted 1 line
Let's keep that change for later.
Sure.
Thanks!
Quoted 6 lines
Quoted 5 lines
trace_vfio_save_complete_precopy(vbasedev->name, ret);return ret;
PATCH 2/3 vfio/migration: Reset bytes_transferred properly
In reply to Avihai HoronCurrently, VFIO bytes_transferred is not reset properly:
1. bytes_transferred is not reset after a VM snapshot (so a migration following a snapshot will report incorrect value).
2. bytes_transferred is a single counter for all VFIO devices, however upon migration failure it is reset multiple times, by each VFIO device.
Fix it by introducing a new function vfio_reset_bytes_transferred() and calling it during migration and snapshot start.
Remove existing bytes_transferred reset in VFIO migration state notifier, which is not needed anymore.
--- include/hw/vfio/vfio-common.h | 1 + migration/migration.h | 1 + hw/vfio/migration.c | 6 +++++- migration/migration.c | 1 + migration/savevm.c | 1 + migration/target.c | 17 +++++++++++++++-- 6 files changed, 24 insertions(+), 3 deletions(-)
include/hw/vfio/vfio-common.h +1 -0
@@ -228,6 +228,7 @@ int vfio_block_multiple_devices_migration(Error **errp);void vfio_unblock_multiple_devices_migration (void );int vfio_block_giommu_migration (Error **errp);int64_t vfio_mig_bytes_transferred (void ); +void vfio_reset_bytes_transferred (void );#ifdef CONFIG_LINUX int vfio_get_region_info(VFIODevice *vbasedev, int index,
migration/migration.h +1 -0
@@ -514,6 +514,7 @@ bool migration_rate_limit(void);void migration_cancel (const Error *error);void populate_vfio_info (MigrationInfo *info); +void reset_vfio_bytes_transferred (void );void postcopy_temp_page_reset (PostcopyTmpPage *tmp_page);#endif
hw/vfio/migration.c +5 -1
@@ -704,7 +704,6 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data)case MIGRATION_STATUS_CANCELLING :case MIGRATION_STATUS_CANCELLED :case MIGRATION_STATUS_FAILED : - bytes_transferred =0 ;/* * If setting the device in RUNNING state fails, the device should * be reset. To do so, use ERROR state as a recover state.
@@ -825,6 +824,11 @@ int64_t vfio_mig_bytes_transferred(void)return bytes_transferred; } +void vfio_reset_bytes_transferred (void ) +{ + bytes_transferred =0 ; +} +int vfio_migration_realize (VFIODevice *vbasedev,Error **errp) {int ret = -ENOTSUP ;
migration/migration.c +1 -0
@@ -1628,6 +1628,7 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,*/ memset (&mig_stats,0 ,sizeof (mig_stats));memset (&compression_counters,0 ,sizeof (compression_counters)); +reset_vfio_bytes_transferred ();return true; }
migration/savevm.c +1 -0
@@ -1622,6 +1622,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)migrate_init (ms);memset (&mig_stats,0 ,sizeof (mig_stats));memset (&compression_counters,0 ,sizeof (compression_counters)); +reset_vfio_bytes_transferred (); ms->to_dst_file = f;qemu_mutex_unlock_iothread ();
migration/target.c +15 -2
@@ -14,12 +14,25 @@#include "hw/vfio/vfio-common.h" #endif +#ifdef CONFIG_VFIO void populate_vfio_info (MigrationInfo *info) { -#ifdef CONFIG_VFIO if (vfio_mig_active ()) { info->vfio =g_malloc0 (sizeof (*info->vfio)); info->vfio->transferred =vfio_mig_bytes_transferred (); } -#endif } + +void reset_vfio_bytes_transferred (void ) +{ +vfio_reset_bytes_transferred (); +} +#else +void populate_vfio_info (MigrationInfo *info) +{ +} + +void reset_vfio_bytes_transferred (void ) +{ +} +#endif
Signature · 1 line
2.26.3
Re: [PATCH 2/3] vfio/migration: Reset bytes_transferred properly
Hello Avihai,
Quoted 7 lines · Avihai Horon
Currently, VFIO
bytes_transferredis not reset properly:
1.bytes_transferredis not reset after a VM snapshot (so a migration following a snapshot will report incorrect value).
2.bytes_transferredis a single counter for all VFIO devices, however upon migration failure it is reset multiple times, by each VFIO device.
Quoted 5 lines
Fix it by introducing a new function
vfio_reset_bytes_transferred()and calling it during migration and snapshot start.Remove existing
bytes_transferredreset in VFIO migration state notifier, which is not needed anymore.
a Fixes: tag would be useful.
Quoted 104 lines
Signed-off-by Avihai Horon <[email protected]>--- include/hw/vfio/vfio-common.h | 1 + migration/migration.h | 1 + hw/vfio/migration.c | 6 +++++- migration/migration.c | 1 + migration/savevm.c | 1 + migration/target.c | 17 +++++++++++++++-- 6 files changed, 24 insertions(+), 3 deletions(-)include/hw/vfio/vfio-common.h +1
-0@@ -228,6 +228,7 @@ int vfio_block_multiple_devices_migration(Error **errp);void vfio_unblock_multiple_devices_migration (void );int vfio_block_giommu_migration (Error **errp);int64_t vfio_mig_bytes_transferred (void ); +void vfio_reset_bytes_transferred (void );#ifdef CONFIG_LINUX int vfio_get_region_info(VFIODevice *vbasedev, int index,migration/migration.h +1
-0@@ -514,6 +514,7 @@ bool migration_rate_limit(void);void migration_cancel (const Error *error);void populate_vfio_info (MigrationInfo *info); +void reset_vfio_bytes_transferred (void );void postcopy_temp_page_reset (PostcopyTmpPage *tmp_page);#endif hw/vfio/migration.c +5
-1@@ -704,7 +704,6 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data)case MIGRATION_STATUS_CANCELLING :case MIGRATION_STATUS_CANCELLED :case MIGRATION_STATUS_FAILED : - bytes_transferred =0 ;/* * If setting the device in RUNNING state fails, the device should * be reset. To do so, use ERROR state as a recover state. @@ -825,6 +824,11 @@ int64_t vfio_mig_bytes_transferred(void)return bytes_transferred; } +void vfio_reset_bytes_transferred (void ) +{ + bytes_transferred =0 ; +} +int vfio_migration_realize (VFIODevice *vbasedev,Error **errp) {int ret = -ENOTSUP ;migration/migration.c +1
-0@@ -1628,6 +1628,7 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,*/ memset (&mig_stats,0 ,sizeof (mig_stats));memset (&compression_counters,0 ,sizeof (compression_counters)); +reset_vfio_bytes_transferred ();return true; }migration/savevm.c +1
-0@@ -1622,6 +1622,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)migrate_init (ms);memset (&mig_stats,0 ,sizeof (mig_stats));memset (&compression_counters,0 ,sizeof (compression_counters)); +reset_vfio_bytes_transferred (); ms->to_dst_file = f;qemu_mutex_unlock_iothread ();migration/target.c +5
-2@@ -14,12 +14,25 @@#include "hw/vfio/vfio-common.h" #endif +#ifdef CONFIG_VFIO void populate_vfio_info (MigrationInfo *info) { -#ifdef CONFIG_VFIO if (vfio_mig_active ()) { info->vfio =g_malloc0 (sizeof (*info->vfio)); info->vfio->transferred =vfio_mig_bytes_transferred (); } -#endif } + +void reset_vfio_bytes_transferred (void ) +{ +vfio_reset_bytes_transferred ();
migration/target.c line 38
Quoted 10 lines
migration/target.c +10
-0+} +#else +void populate_vfio_info (MigrationInfo *info) +{ +} + +void reset_vfio_bytes_transferred (void ) +{ +} +#endif
I would simply use static inline in migration.h if !CONFIG_VFIO.
Thanks,
C.
Re: [PATCH 2/3] vfio/migration: Reset bytes_transferred properly
Quoted 13 lines · Cédric Le Goater
External email: Use caution opening links or attachments
Hello Avihai,
Quoted 7 lines · Avihai Horon
Currently, VFIO
bytes_transferredis not reset properly:
1.bytes_transferredis not reset after a VM snapshot (so a migration following a snapshot will report incorrect value).
2.bytes_transferredis a single counter for all VFIO devices, however upon migration failure it is reset multiple times, by each VFIO device.
Quoted 7 lines
Quoted 5 lines
Fix it by introducing a new function
vfio_reset_bytes_transferred()and calling it during migration and snapshot start.Remove existing
bytes_transferredreset in VFIO migration state notifier, which is not needed anymore.a Fixes: tag would be useful.
Sure, I will add.
Quoted 114 lines
Quoted 114 lines
Signed-off-by Avihai Horon <[email protected]>---
include/hw/vfio/vfio-common.h| 1 + migration/migration.h | 1 + hw/vfio/migration.c | 6 +++++- migration/migration.c | 1 + migration/savevm.c | 1 + migration/target.c | 17 +++++++++++++++--6 files changed, 24 insertions(+), 3 deletions(-)include/hw/vfio/vfio-common.h +1
-0@@ -228,6 +228,7 @@ int vfio_block_multiple_devices_migration(Error **errp); void vfio_unblock_multiple_devices_migration(void); int vfio_block_giommu_migration(Error **errp); int64_t vfio_mig_bytes_transferred(void); +void vfio_reset_bytes_transferred (void ); #ifdef CONFIG_LINUXint vfio_get_region_info(VFIODevice *vbasedev, int index,
migration/migration.h +1
-0@@ -514,6 +514,7 @@ bool migration_rate_limit(void); void migration_cancel(const Error *error); void populate_vfio_info(MigrationInfo *info); +void reset_vfio_bytes_transferred (void ); void postcopy_temp_page_reset(PostcopyTmpPage *tmp_page); #endifhw/vfio/migration.c +0
-0@@ -704,7 +704,6 @@ static voidvfio_migration_state_notifier (Notifier *notifier, void *data)case
MIGRATION_STATUS_CANCELLING:
caseMIGRATION_STATUS_CANCELLED:
caseMIGRATION_STATUS_FAILED:
-bytes_transferred= 0;
/*
* If setting the device in RUNNING state fails, the device should
* be reset. To do so, use ERROR state as a recover state.hw/vfio/migration.c +5
-0@@ -825,6 +824,11 @@ int64_t vfio_mig_bytes_transferred(void) return bytes_transferred; } +void vfio_reset_bytes_transferred (void ) +{ + bytes_transferred = 0; +} + int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) { int ret = -ENOTSUP;migration/migration.c +1
-0@@ -1628,6 +1628,7 @@ static bool migrate_prepare(MigrationState *s,bool blk, bool blk_inc, */ memset(&mig_stats, 0, sizeof(mig_stats)); memset(&compression_counters, 0, sizeof(compression_counters)); + reset_vfio_bytes_transferred(); return true;}
migration/savevm.c +1
-0@@ -1622,6 +1622,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) migrate_init(ms); memset(&mig_stats, 0, sizeof(mig_stats)); memset(&compression_counters, 0, sizeof(compression_counters)); + reset_vfio_bytes_transferred(); ms->to_dst_file = f;
qemu_mutex_unlock_iothread();migration/target.c +9
-2@@ -14,12 +14,25 @@ #include "hw/vfio/vfio-common.h" #endif +#ifdef CONFIG_VFIO void populate_vfio_info(MigrationInfo *info) { -#ifdef CONFIG_VFIO if (vfio_mig_active()) { info->vfio = g_malloc0(sizeof(*info->vfio)); info->vfio->transferred = vfio_mig_bytes_transferred(); } -#endif } + +void reset_vfio_bytes_transferred (void ) +{ + vfio_reset_bytes_transferred(); +} +#else +void populate_vfio_info (MigrationInfo *info) +{
Quoted 8 lines
migration/target.c line 38
Quoted 6 lines
migration/target.c +6
-0+} + +void reset_vfio_bytes_transferred (void ) +{ +} +#endif I would simply use static inline in
migration.hif !CONFIG_VFIO.
I'm not sure it's possible to use CONFIG_VFIO in migration.h. Got this during compilation:
migration/migration.h:517:8: error: attempt to use poisoned "CONFIG_VFIO"
Plus, see 43bd0bf30fce ("migration: Move populate_vfio_info() into a separate file") commit message:
The CONFIG_VFIO switch only works in target specific code. Since migration/migration.c is common code, the #ifdef does not have the intended behavior here. Move the related code to a separate file now which gets compiled via specific_ss instead.
Or did I misunderstand you?
Thanks.
Re: [PATCH 2/3] vfio/migration: Reset bytes_transferred properly
Quoted 18 lines · Avihai Horon
Quoted 16 lines · Cédric Le Goater
External email: Use caution opening links or attachments
Hello Avihai,
Quoted 10 lines · Avihai Horon
Currently, VFIO
bytes_transferredis not reset properly:
1.bytes_transferredis not reset after a VM snapshot (so a migration following a snapshot will report incorrect value).
2.bytes_transferredis a single counter for all VFIO devices, however upon migration failure it is reset multiple times, by each VFIO device.Fix it by introducing a new function
vfio_reset_bytes_transferred()and calling it during migration and snapshot start.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Remove existing
bytes_transferredreset in VFIO migration state notifier, which is not needed anymore.a Fixes: tag would be useful.
Sure, I will add.
You could reply with the appropriate tag and the tooling will automatically collect it. No need to resend for that.
Quoted 123 lines
Quoted 111 lines
Quoted 111 lines
Signed-off-by Avihai Horon <[email protected]>---
include/hw/vfio/vfio-common.h| 1 + migration/migration.h | 1 + hw/vfio/migration.c | 6 +++++- migration/migration.c | 1 + migration/savevm.c | 1 + migration/target.c | 17 +++++++++++++++--6 files changed, 24 insertions(+), 3 deletions(-)include/hw/vfio/vfio-common.h +1
-0@@ -228,6 +228,7 @@ int vfio_block_multiple_devices_migration(Error **errp); void vfio_unblock_multiple_devices_migration(void); int vfio_block_giommu_migration(Error **errp); int64_t vfio_mig_bytes_transferred(void); +void vfio_reset_bytes_transferred (void ); #ifdef CONFIG_LINUX int vfio_get_region_info(VFIODevice *vbasedev, int index,migration/migration.h +1
-0@@ -514,6 +514,7 @@ bool migration_rate_limit(void); void migration_cancel(const Error *error); void populate_vfio_info(MigrationInfo *info); +void reset_vfio_bytes_transferred (void ); void postcopy_temp_page_reset(PostcopyTmpPage *tmp_page); #endifhw/vfio/migration.c +0
-0@@ -704,7 +704,6 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data)case
MIGRATION_STATUS_CANCELLING:
caseMIGRATION_STATUS_CANCELLED:
caseMIGRATION_STATUS_FAILED:
-bytes_transferred= 0;
/*
* If setting the device in RUNNING state fails, the device should
* be reset. To do so, use ERROR state as a recover state.hw/vfio/migration.c +5
-0@@ -825,6 +824,11 @@ int64_t vfio_mig_bytes_transferred(void) return bytes_transferred; } +void vfio_reset_bytes_transferred (void ) +{ + bytes_transferred = 0; +} + int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) { int ret = -ENOTSUP;migration/migration.c +1
-0@@ -1628,6 +1628,7 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,*/ memset(&mig_stats, 0, sizeof(mig_stats)); memset(&compression_counters, 0, sizeof(compression_counters)); + reset_vfio_bytes_transferred(); return true; }migration/savevm.c +1
-0@@ -1622,6 +1622,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) migrate_init(ms); memset(&mig_stats, 0, sizeof(mig_stats)); memset(&compression_counters, 0, sizeof(compression_counters)); + reset_vfio_bytes_transferred(); ms->to_dst_file = f; qemu_mutex_unlock_iothread();migration/target.c +9
-2@@ -14,12 +14,25 @@ #include "hw/vfio/vfio-common.h" #endif +#ifdef CONFIG_VFIO void populate_vfio_info(MigrationInfo *info) { -#ifdef CONFIG_VFIO if (vfio_mig_active()) { info->vfio = g_malloc0(sizeof(*info->vfio)); info->vfio->transferred = vfio_mig_bytes_transferred(); } -#endif } + +void reset_vfio_bytes_transferred (void ) +{ + vfio_reset_bytes_transferred(); +} +#else +void populate_vfio_info (MigrationInfo *info) +{Quoted 8 lines
migration/target.c line 38
Quoted 6 lines
migration/target.c +6
-0+} + +void reset_vfio_bytes_transferred (void ) +{ +} +#endif I would simply use static inline in
migration.hif !CONFIG_VFIO.I'm not sure it's possible to use
CONFIG_VFIOinmigration.h. Got this during compilation:
migration/migration.h:517:8: error: attempt to use poisoned "CONFIG_VFIO"
Quoted 8 lines
Plus, see
43bd0bf30fce("migration: Movepopulate_vfio_info()into a separate file") commit message:The
CONFIG_VFIOswitch only works in target specific code. Sincemigration/migration.cis common code, the #ifdef does not have the intended behavior here. Move the related code to a separate file now which gets compiled viaspecific_ssinstead.Or did I misunderstand you?
I didn't realize that was not possible. Leave it that way then.
Thanks,
C.
Re: [PATCH 2/3] vfio/migration: Reset bytes_transferred properly
In reply to Avihai HoronQuoted 7 lines · Avihai Horon
Currently, VFIO
bytes_transferredis not reset properly:
1.bytes_transferredis not reset after a VM snapshot (so a migration following a snapshot will report incorrect value).
2.bytes_transferredis a single counter for all VFIO devices, however upon migration failure it is reset multiple times, by each VFIO device.
Quoted 5 lines
Fix it by introducing a new function
vfio_reset_bytes_transferred()and calling it during migration and snapshot start.Remove existing
bytes_transferredreset in VFIO migration state notifier, which is not needed anymore.
Forgot fixes tag:
Quoted 114 lines
Signed-off-by Avihai Horon <[email protected]>--- include/hw/vfio/vfio-common.h | 1 + migration/migration.h | 1 + hw/vfio/migration.c | 6 +++++- migration/migration.c | 1 + migration/savevm.c | 1 + migration/target.c | 17 +++++++++++++++-- 6 files changed, 24 insertions(+), 3 deletions(-)include/hw/vfio/vfio-common.h +1
-0@@ -228,6 +228,7 @@ int vfio_block_multiple_devices_migration(Error **errp);void vfio_unblock_multiple_devices_migration (void );int vfio_block_giommu_migration (Error **errp);int64_t vfio_mig_bytes_transferred (void ); +void vfio_reset_bytes_transferred (void );#ifdef CONFIG_LINUX int vfio_get_region_info(VFIODevice *vbasedev, int index,migration/migration.h +1
-0@@ -514,6 +514,7 @@ bool migration_rate_limit(void);void migration_cancel (const Error *error);void populate_vfio_info (MigrationInfo *info); +void reset_vfio_bytes_transferred (void );void postcopy_temp_page_reset (PostcopyTmpPage *tmp_page);#endif hw/vfio/migration.c +5
-1@@ -704,7 +704,6 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data)case MIGRATION_STATUS_CANCELLING :case MIGRATION_STATUS_CANCELLED :case MIGRATION_STATUS_FAILED : - bytes_transferred =0 ;/* * If setting the device in RUNNING state fails, the device should * be reset. To do so, use ERROR state as a recover state. @@ -825,6 +824,11 @@ int64_t vfio_mig_bytes_transferred(void)return bytes_transferred; } +void vfio_reset_bytes_transferred (void ) +{ + bytes_transferred =0 ; +} +int vfio_migration_realize (VFIODevice *vbasedev,Error **errp) {int ret = -ENOTSUP ;migration/migration.c +1
-0@@ -1628,6 +1628,7 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,*/ memset (&mig_stats,0 ,sizeof (mig_stats));memset (&compression_counters,0 ,sizeof (compression_counters)); +reset_vfio_bytes_transferred ();return true; }migration/savevm.c +1
-0@@ -1622,6 +1622,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)migrate_init (ms);memset (&mig_stats,0 ,sizeof (mig_stats));memset (&compression_counters,0 ,sizeof (compression_counters)); +reset_vfio_bytes_transferred (); ms->to_dst_file = f;qemu_mutex_unlock_iothread ();migration/target.c +15
-2@@ -14,12 +14,25 @@#include "hw/vfio/vfio-common.h" #endif +#ifdef CONFIG_VFIO void populate_vfio_info (MigrationInfo *info) { -#ifdef CONFIG_VFIO if (vfio_mig_active ()) { info->vfio =g_malloc0 (sizeof (*info->vfio)); info->vfio->transferred =vfio_mig_bytes_transferred (); } -#endif } + +void reset_vfio_bytes_transferred (void ) +{ +vfio_reset_bytes_transferred (); +} +#else +void populate_vfio_info (MigrationInfo *info) +{ +} + +void reset_vfio_bytes_transferred (void ) +{ +} +#endif
PATCH 3/3 vfio/migration: Make VFIO migration non-experimental
In reply to Avihai HoronThe major parts of VFIO migration are supported today in QEMU. This includes basic VFIO migration, device dirty page tracking and precopy support.
Thus, at this point in time, it seems appropriate to make VFIO migration
non-experimental: remove the x prefix from enable_migration property, change it to ON_OFF_AUTO and let the default value be AUTO.
In addition, make the following adjustments:
1. Require device dirty tracking support when enable_migration is AUTO (i.e., not explicitly enabled). This is because device dirty tracking is currently the only method to do dirty page tracking, which is essential for migrating in a reasonable downtime. Setting enable_migration to ON will not require device dirty tracking.
2. Make migration blocker messages more elaborate.
3. Remove error prints in vfio_migration_query_flags().
4. Remove a redundant assignment in vfio_migration_realize().
--- include/hw/vfio/vfio-common.h | 2 +- hw/vfio/migration.c | 29 ++++++++++++++++------------- hw/vfio/pci.c | 4 ++-- 3 files changed, 19 insertions(+), 16 deletions(-)
include/hw/vfio/vfio-common.h +1 -1
@@ -139,7 +139,7 @@ typedef struct VFIODevice {bool needs_reset;bool no_mmap;bool ram_block_discard_allowed; -bool enable_migration; +OnOffAuto enable_migration;VFIODeviceOps *ops;unsigned int num_irqs;unsigned int num_regions;
hw/vfio/migration.c +16 -13
@@ -731,14 +731,6 @@ static int vfio_migration_query_flags(VFIODevice *vbasedev, uint64_t *mig_flags) feature->argsz =sizeof (buf); feature->flags =VFIO_DEVICE_FEATURE_GET |VFIO_DEVICE_FEATURE_MIGRATION ;if (ioctl (vbasedev->fd,VFIO_DEVICE_FEATURE , feature)) { -if (errno ==ENOTTY ) { -error_report ("%s: VFIO migration is not supported in kernel" , - vbasedev->name); - }else { -error_report ("%s: Failed to query VFIO migration support, err: %s" , - vbasedev->name,strerror (errno)); - } -return -errno; }
@@ -831,14 +823,28 @@ void vfio_reset_bytes_transferred(void)int vfio_migration_realize (VFIODevice *vbasedev,Error **errp) { -int ret = -ENOTSUP ; +int ret; -if (!vbasedev->enable_migration) { +if (vbasedev->enable_migration ==ON_OFF_AUTO_OFF ) { +error_setg (&vbasedev->migration_blocker, +"%s: Migration is disabled for VFIO device" , vbasedev->name); goto add_blocker; } ret =vfio_migration_init (vbasedev);if (ret) { +error_setg (&vbasedev->migration_blocker, +"%s: Migration couldn't be initialized for VFIO device, " +"err: %d (%s)" , + vbasedev->name, ret,strerror (-ret)); + goto add_blocker; + } + +if (vbasedev->enable_migration ==ON_OFF_AUTO_AUTO && + !vbasedev->dirty_pages_supported) { +error_setg (&vbasedev->migration_blocker, +"%s: VFIO device doesn't support device dirty tracking" , + vbasedev->name); goto add_blocker; }
@@ -856,9 +862,6 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp)return 0 ; add_blocker: -error_setg (&vbasedev->migration_blocker, -"VFIO device doesn't support migration" ); - ret =migrate_add_blocker (vbasedev->migration_blocker, errp);if (ret <0 ) {error_free (vbasedev->migration_blocker);
hw/vfio/pci.c +2 -2
@@ -3347,8 +3347,8 @@ static Property vfio_pci_dev_properties[] = { VFIO_FEATURE_ENABLE_REQ_BIT, true),DEFINE_PROP_BIT ("x-igd-opregion" , VFIOPCIDevice, features,VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT , false), -DEFINE_PROP_BOOL ("x-enable-migration" , VFIOPCIDevice, - vbasedev.enable_migration, false), +DEFINE_PROP_ON_OFF_AUTO ("enable-migration" , VFIOPCIDevice, + vbasedev.enable_migration,ON_OFF_AUTO_AUTO ),DEFINE_PROP_BOOL ("x-no-mmap" , VFIOPCIDevice, vbasedev.no_mmap, false),DEFINE_PROP_BOOL ("x-balloon-allowed" , VFIOPCIDevice, vbasedev.ram_block_discard_allowed, false),
Signature · 1 line
2.26.3
Hello Avihai,
Quoted 8 lines · Avihai Horon
The major parts of VFIO migration are supported today in QEMU. This includes basic VFIO migration, device dirty page tracking and precopy support.
Thus, at this point in time, it seems appropriate to make VFIO migration
non-experimental: remove the x prefix fromenable_migrationproperty, change it toON_OFF_AUTOand let the default value be AUTO.
Quoted 5 lines
In addition, make the following adjustments:
1. Require device dirty tracking support whenenable_migrationis AUTO (i.e., not explicitly enabled). This is because device dirty tracking is currently the only method to do dirty page tracking, which is essential for migrating in a reasonable downtime.
hmm, I don't think QEMU should decide to disable a feature for all devices supposedly because it could be slow for some. That's too restrictive. What about devices with have small states ? for which the downtime would be reasonable even without device dirty tracking support.
Quoted 51 lines
Setting
enable_migrationto ON will not require device dirty tracking.
2. Make migration blocker messages more elaborate.
3. Remove error prints invfio_migration_query_flags().
4. Remove a redundant assignment invfio_migration_realize().Signed-off-by Avihai Horon <[email protected]>--- include/hw/vfio/vfio-common.h | 2 +- hw/vfio/migration.c | 29 ++++++++++++++++------------- hw/vfio/pci.c | 4 ++-- 3 files changed, 19 insertions(+), 16 deletions(-)include/hw/vfio/vfio-common.h +1
-1@@ -139,7 +139,7 @@ typedef struct VFIODevice {bool needs_reset;bool no_mmap;bool ram_block_discard_allowed; -bool enable_migration; +OnOffAuto enable_migration;VFIODeviceOps *ops;unsigned int num_irqs;unsigned int num_regions;hw/vfio/migration.c +1
-9@@ -731,14 +731,6 @@ static int vfio_migration_query_flags(VFIODevice *vbasedev, uint64_t *mig_flags) feature->argsz =sizeof (buf); feature->flags =VFIO_DEVICE_FEATURE_GET |VFIO_DEVICE_FEATURE_MIGRATION ;if (ioctl (vbasedev->fd,VFIO_DEVICE_FEATURE , feature)) { -if (errno ==ENOTTY ) { -error_report ("%s: VFIO migration is not supported in kernel" , - vbasedev->name); - }else { -error_report ("%s: Failed to query VFIO migration support, err: %s" , - vbasedev->name,strerror (errno)); - } -return -errno; }@@ -831,14 +823,28 @@ void vfio_reset_bytes_transferred(void)int vfio_migration_realize (VFIODevice *vbasedev,Error **errp) { -int ret = -ENOTSUP ; +int ret;
hw/vfio/migration.c void vfio_reset_bytes_transferred(void) · line 835
Quoted 10 lines
hw/vfio/migration.c +3
-1-if (!vbasedev->enable_migration) { +if (vbasedev->enable_migration ==ON_OFF_AUTO_OFF ) { +error_setg (&vbasedev->migration_blocker, +"%s: Migration is disabled for VFIO device" , vbasedev->name); goto add_blocker; } ret =vfio_migration_init (vbasedev);if (ret) {
It would be good to keep the message for 'errno == ENOTTY' as it was in vfio_migration_query_flags(). When migration fails, it is an important information to know that it is because the VFIO PCI host device driver doesn't support the feature. The root cause could be deep below in FW or how the VF was set up.
hw/vfio/migration.c void vfio_reset_bytes_transferred(void) · line 844
Quoted 9 lines
hw/vfio/migration.c +9
-0+error_setg (&vbasedev->migration_blocker, +"%s: Migration couldn't be initialized for VFIO device, " +"err: %d (%s)" , + vbasedev->name, ret,strerror (-ret)); + goto add_blocker; + } + +if (vbasedev->enable_migration ==ON_OFF_AUTO_AUTO && + !vbasedev->dirty_pages_supported) {
I don't agree with this test.
hw/vfio/migration.c void vfio_reset_bytes_transferred(void) · line 849
Quoted 5 lines
hw/vfio/migration.c +3
-0+error_setg (&vbasedev->migration_blocker, +"%s: VFIO device doesn't support device dirty tracking" , + vbasedev->name); goto add_blocker; }
I agree that with ON_OFF_AUTO_AUTO, errors at realize time should be recorded in a migration blocker. What about the ON_OFF_AUTO_ON case ? If migration was explicitly requested for the device and the conditions on the host are not met, I think realize should fail and the machine abort.
Thanks,
C.
Quoted 25 lines
hw/vfio/migration.c +0
-3@@ -856,9 +862,6 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp)return 0 ; add_blocker: -error_setg (&vbasedev->migration_blocker, -"VFIO device doesn't support migration" ); - ret =migrate_add_blocker (vbasedev->migration_blocker, errp);if (ret <0 ) {error_free (vbasedev->migration_blocker);hw/vfio/pci.c +2
-2@@ -3347,8 +3347,8 @@ static Property vfio_pci_dev_properties[] = { VFIO_FEATURE_ENABLE_REQ_BIT, true),DEFINE_PROP_BIT ("x-igd-opregion" , VFIOPCIDevice, features,VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT , false), -DEFINE_PROP_BOOL ("x-enable-migration" , VFIOPCIDevice, - vbasedev.enable_migration, false), +DEFINE_PROP_ON_OFF_AUTO ("enable-migration" , VFIOPCIDevice, + vbasedev.enable_migration,ON_OFF_AUTO_AUTO ),DEFINE_PROP_BOOL ("x-no-mmap" , VFIOPCIDevice, vbasedev.no_mmap, false),DEFINE_PROP_BOOL ("x-balloon-allowed" , VFIOPCIDevice, vbasedev.ram_block_discard_allowed, false),
Quoted 17 lines · Cédric Le Goater
Hello Avihai,
Quoted 8 lines · Avihai Horon
The major parts of VFIO migration are supported today in QEMU. This includes basic VFIO migration, device dirty page tracking and precopy support.
Thus, at this point in time, it seems appropriate to make VFIO migration
non-experimental: remove the x prefix fromenable_migrationproperty, change it toON_OFF_AUTOand let the default value be AUTO.Quoted 5 lines
In addition, make the following adjustments:
1. Require device dirty tracking support whenenable_migrationis AUTO (i.e., not explicitly enabled). This is because device dirty tracking is currently the only method to do dirty page tracking, which is essential for migrating in a reasonable downtime.
Quoted 6 lines
hmm, I don't think QEMU should decide to disable a feature for all devices supposedly because it could be slow for some. That's too restrictive. What about devices with have small states ? for which the downtime would be reasonable even without device dirty tracking support.
device dirty tracking refers to the ability to tracking dirty IOVA used by the device which will DMA into RAM. It is required because the consequence/alternative is to transfer all RAM in stop copy phase. Device state size at that point is the least of our problems downtime wise.
I can imagine that allowing without dirty tracking is useful for developer testing of the suspend/device-state flows, but as real default (auto) is very questionable to let it go through without dirty tracking. When we have IOMMUFD dirty tracking that's when we can relieve this restriction as a default.
But then note that (...)
Quoted 3 lines
Quoted 3 lines
Setting
enable_migrationto ON will not require device dirty tracking.
(...) this lets it ignore dirty tracking as you would like.
Quoted 54 lines
Quoted 54 lines
2. Make migration blocker messages more elaborate.
3. Remove error prints invfio_migration_query_flags().
4. Remove a redundant assignment invfio_migration_realize().Signed-off-by Avihai Horon <[email protected]>---
include/hw/vfio/vfio-common.h| 2 +- hw/vfio/migration.c | 29 ++++++++++++++++------------- hw/vfio/pci.c | 4 ++--3 files changed, 19 insertions(+), 16 deletions(-)include/hw/vfio/vfio-common.h +1
-1@@ -139,7 +139,7 @@ typedef struct VFIODevice { bool needs_reset; bool no_mmap; bool ram_block_discard_allowed; - bool enable_migration; + OnOffAuto enable_migration; VFIODeviceOps *ops; unsigned int num_irqs; unsigned int num_regions;hw/vfio/migration.c +0
-8@@ -731,14 +731,6 @@ static int vfio_migration_query_flags(VFIODevice *vbasedev, uint64_t *mig_flags) feature->argsz = sizeof(buf); feature->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_MIGRATION; if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) { - if (errno == ENOTTY) { - error_report("%s: VFIO migration is not supported in kernel", - vbasedev->name); - } else { - error_report("%s: Failed to query VFIO migration support, err: %s", - vbasedev->name, strerror(errno)); - } - return -errno; }@@ -831,14 +823,28 @@ void
vfio_reset_bytes_transferred(void)
intvfio_migration_realize(VFIODevice *vbasedev, Error **errp)
{
- int ret =-ENOTSUP;
+ int ret;
- if (!vbasedev->enable_migration) {
+ if (vbasedev->enable_migration==ON_OFF_AUTO_OFF) {
+ error_setg(&vbasedev->migration_blocker,
+ "%s: Migration is disabled for VFIO device",vbasedev->name); gotoadd_blocker;
}
Quoted 9 lines
Quoted 2 lines
ret =
vfio_migration_init(vbasedev);
if (ret) {It would be good to keep the message for 'errno == ENOTTY' as it was in
vfio_migration_query_flags(). When migration fails, it is an important information to know that it is because the VFIO PCI host device driver doesn't support the feature. The root cause could be deep below in FW or how the VF was set up.
+1 As I have been in this rabbit hole
Quoted 10 lines
Quoted 9 lines
+ error_setg(&vbasedev->migration_blocker,
+ "%s: Migration couldn't be initialized for VFIO device, " + "err: %d (%s)",
+vbasedev->name, ret, strerror(-ret));
+ gotoadd_blocker;
+ }
+
+ if (vbasedev->enable_migration==ON_OFF_AUTO_AUTO&&
+ !vbasedev->dirty_pages_supported) {
Quoted 2 lines
I don't agree with this test.
The alternative right now is perceptual dirty tracking. How is that OK as a default? It's not like we have even an option :(
Maybe perhaps you refer to avoid strongly enforcing *always* it to allow testing of the non dirty tracking parts? Maybe when you 'force' enabling with enable-migration=on is when you ignore the dirty tracking which is what this is doing.
Quoted 10 lines
Quoted 5 lines
+ error_setg(&vbasedev->migration_blocker,
+ "%s: VFIO device doesn't support device dirty tracking", +vbasedev->name);
gotoadd_blocker;
}I agree that with
ON_OFF_AUTO_AUTO, errors at realize time should be recorded in a migration blocker. What about theON_OFF_AUTO_ONcase ? If migration was explicitly requested for the device and the conditions on the host are not met, I think realize should fail and the machine abort.
+1 Good point
Quoted 32 lines
Thanks,
C.
Quoted 25 lines
hw/vfio/migration.c +0
-0@@ -856,9 +862,6 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) return 0;
add_blocker:
- error_setg(&vbasedev->migration_blocker,
- "VFIO device doesn't support migration"); -
ret =migrate_add_blocker(vbasedev->migration_blocker, errp); if (ret < 0) {error_free(vbasedev->migration_blocker);hw/vfio/pci.c +2
-2@@ -3347,8 +3347,8 @@ static Property vfio_pci_dev_properties[] = { VFIO_FEATURE_ENABLE_REQ_BIT, true), DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features, VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false), - DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice, - vbasedev.enable_migration, false), + DEFINE_PROP_ON_OFF_AUTO("enable-migration", VFIOPCIDevice, + vbasedev.enable_migration, ON_OFF_AUTO_AUTO), DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false), DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice, vbasedev.ram_block_discard_allowed, false),
Quoted 25 lines · Joao Martins
Quoted 17 lines · Cédric Le Goater
Hello Avihai,
Quoted 8 lines · Avihai Horon
The major parts of VFIO migration are supported today in QEMU. This includes basic VFIO migration, device dirty page tracking and precopy support.
Thus, at this point in time, it seems appropriate to make VFIO migration
non-experimental: remove the x prefix fromenable_migrationproperty, change it toON_OFF_AUTOand let the default value be AUTO.Quoted 5 lines
In addition, make the following adjustments:
1. Require device dirty tracking support whenenable_migrationis AUTO (i.e., not explicitly enabled). This is because device dirty tracking is currently the only method to do dirty page tracking, which is essential for migrating in a reasonable downtime.Quoted 6 lines
hmm, I don't think QEMU should decide to disable a feature for all devices supposedly because it could be slow for some. That's too restrictive. What about devices with have small states ? for which the downtime would be reasonable even without device dirty tracking support.
Quoted 4 lines
device dirty tracking refers to the ability to tracking dirty IOVA used by the device which will DMA into RAM. It is required because the consequence/alternative is to transfer all RAM in stop copy phase. Device state size at that point is the least of our problems downtime wise.
Arg. thanks for reminding me. I tend to take this for granted ...
Quoted 92 lines
I can imagine that allowing without dirty tracking is useful for developer testing of the suspend/device-state flows, but as real default (auto) is very questionable to let it go through without dirty tracking. When we have IOMMUFD dirty tracking that's when we can relieve this restriction as a default.
But then note that (...)
Quoted 3 lines
Quoted 3 lines
Setting
enable_migrationto ON will not require device dirty tracking.(...) this lets it ignore dirty tracking as you would like.
Quoted 54 lines
Quoted 54 lines
2. Make migration blocker messages more elaborate.
3. Remove error prints invfio_migration_query_flags().
4. Remove a redundant assignment invfio_migration_realize().Signed-off-by Avihai Horon <[email protected]>--- include/hw/vfio/vfio-common.h | 2 +- hw/vfio/migration.c | 29 ++++++++++++++++------------- hw/vfio/pci.c | 4 ++-- 3 files changed, 19 insertions(+), 16 deletions(-)include/hw/vfio/vfio-common.h +1
-1@@ -139,7 +139,7 @@ typedef struct VFIODevice { bool needs_reset; bool no_mmap; bool ram_block_discard_allowed; - bool enable_migration; + OnOffAuto enable_migration; VFIODeviceOps *ops; unsigned int num_irqs; unsigned int num_regions;hw/vfio/migration.c +0
-8@@ -731,14 +731,6 @@ static int vfio_migration_query_flags(VFIODevice *vbasedev, uint64_t *mig_flags) feature->argsz = sizeof(buf); feature->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_MIGRATION; if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) { - if (errno == ENOTTY) { - error_report("%s: VFIO migration is not supported in kernel", - vbasedev->name); - } else { - error_report("%s: Failed to query VFIO migration support, err: %s", - vbasedev->name, strerror(errno)); - } - return -errno; }@@ -831,14 +823,28 @@ void
vfio_reset_bytes_transferred(void)
intvfio_migration_realize(VFIODevice *vbasedev, Error **errp){
- int ret =-ENOTSUP;
+ int ret;
- if (!vbasedev->enable_migration) {
+ if (vbasedev->enable_migration==ON_OFF_AUTO_OFF) {
+ error_setg(&vbasedev->migration_blocker,
+ "%s: Migration is disabled for VFIO device",vbasedev->name);
gotoadd_blocker;
}Quoted 9 lines
Quoted 2 lines
ret =
vfio_migration_init(vbasedev);
if (ret) {It would be good to keep the message for 'errno == ENOTTY' as it was in
vfio_migration_query_flags(). When migration fails, it is an important information to know that it is because the VFIO PCI host device driver doesn't support the feature. The root cause could be deep below in FW or how the VF was set up.+1 As I have been in this rabbit hole
Quoted 10 lines
Quoted 9 lines
+ error_setg(&vbasedev->migration_blocker,
+ "%s: Migration couldn't be initialized for VFIO device, " + "err: %d (%s)",
+vbasedev->name, ret, strerror(-ret));
+ gotoadd_blocker;
+ }
+
+ if (vbasedev->enable_migration==ON_OFF_AUTO_AUTO&&
+ !vbasedev->dirty_pages_supported) {Quoted 2 lines
I don't agree with this test.
Quoted 7 lines
The alternative right now is perceptual dirty tracking. How is that OK as a default? It's not like we have even an option :(
Maybe perhaps you refer to avoid strongly enforcing *always* it to allow testing of the non dirty tracking parts? Maybe when you 'force' enabling with enable-migration=on is when you ignore the dirty tracking which is what this is doing.
I see ON_OFF_AUTO_ON as a way to abort the machine startup while ON_OFF_AUTO_AUTO would let it run but block migration. We would need an extra property to relax the checks, else we are hijacking some pre-existing option to fit our need.
Since dirty tracking is a must-have to implement migration support for any existing and future VFIO PCI variant driver, anything else would be experimental code and we are trying to remove the flag ! Please correct me if I am wrong.
So, the case !vbasedev->dirty_pages_supported is just an extra information to report for why migration is not supported. Does that sound reasonable ?
Thanks,
C.
Quoted 45 lines
Quoted 11 lines
Quoted 6 lines
+ error_setg(&vbasedev->migration_blocker,
+ "%s: VFIO device doesn't support device dirty tracking", +vbasedev->name);
gotoadd_blocker;
}I agree that with
ON_OFF_AUTO_AUTO, errors at realize time should be recorded in a migration blocker. What about theON_OFF_AUTO_ONcase ? If migration was explicitly requested for the device and the conditions on the host are not met, I think realize should fail and the machine abort.+1 Good point
Quoted 32 lines
Thanks,
C.
Quoted 25 lines
hw/vfio/migration.c +0
-3@@ -856,9 +862,6 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) return 0; add_blocker: - error_setg(&vbasedev->migration_blocker, - "VFIO device doesn't support migration"); - ret = migrate_add_blocker(vbasedev->migration_blocker, errp); if (ret < 0) { error_free(vbasedev->migration_blocker);hw/vfio/pci.c +2
-2@@ -3347,8 +3347,8 @@ static Property vfio_pci_dev_properties[] = { VFIO_FEATURE_ENABLE_REQ_BIT, true), DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features, VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false), - DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice, - vbasedev.enable_migration, false), + DEFINE_PROP_ON_OFF_AUTO("enable-migration", VFIOPCIDevice, + vbasedev.enable_migration, ON_OFF_AUTO_AUTO), DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false), DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice, vbasedev.ram_block_discard_allowed, false),
Quoted 4 lines · Cédric Le Goater
Since dirty tracking is a must-have to implement migration support for any existing and future VFIO PCI variant driver, anything else would be experimental code and we are trying to remove the flag ! Please correct me if I am wrong.
I don't think we should call it experimental - it is like other qemu options where we can choose to run qemu in a suboptimal way.
Jason
Quoted 9 lines · Jason Gunthorpe
Quoted 4 lines · Cédric Le Goater
Since dirty tracking is a must-have to implement migration support for any existing and future VFIO PCI variant driver, anything else would be experimental code and we are trying to remove the flag ! Please correct me if I am wrong.
I don't think we should call it experimental - it is like other qemu options where we can choose to run qemu in a suboptimal way.
In that case, whether the driver implements dirty tracking or not is not a condition to enable migration, only the host kernel driver support is.
C.
Quoted 25 lines · Cédric Le Goater
Quoted 15 lines · Joao Martins
Quoted 11 lines · Cédric Le Goater
Quoted 9 lines · Avihai Horon
+ error_setg(&vbasedev->migration_blocker,
+ "%s: Migration couldn't be initialized for VFIO device, " + "err: %d (%s)",
+vbasedev->name, ret, strerror(-ret));
+ gotoadd_blocker;
+ }
+
+ if (vbasedev->enable_migration==ON_OFF_AUTO_AUTO&&
+ !vbasedev->dirty_pages_supported) {Quoted 2 lines
I don't agree with this test.
Quoted 7 lines
The alternative right now is perceptual dirty tracking. How is that OK as a default? It's not like we have even an option :(
Maybe perhaps you refer to avoid strongly enforcing *always* it to allow testing of the non dirty tracking parts? Maybe when you 'force' enabling with enable-migration=on is when you ignore the dirty tracking which is what this is doing.
Quoted 5 lines
I see
ON_OFF_AUTO_ONas a way to abort the machine startup whileON_OFF_AUTO_AUTOwould let it run but block migration. We would need an extra property to relax the checks, else we are hijacking some pre-existing option to fit our need.
OK
Quoted 5 lines
Since dirty tracking is a must-have to implement migration support for any existing and future VFIO PCI variant driver, anything else would be experimental code and we are trying to remove the flag ! Please correct me if I am wrong.
My thinking was mainly requiring the default migration case without any relaxing from the user to require dirty tracking by default. hisilicon driver is the current vfio driver upstream that doesn't support dirty tracking (neither P2P).
Quoted 4 lines
So, the case !vbasedev->dirty_pages_supported is just an extra information to report for why migration is not supported. Does that sound reasonable ?
We can always piggy back on the x-pre-copy-dirty-page-tracking (which already exists too) as means to relax this option, and that should cover this other case. My comment was more targetted at your suggestion of making it optional *by default* to not use the dirty tracking.
Joao
Quoted 26 lines · On Mon, 26 Jun 2023 17:26:42 +0200 Cédric Le Goater <[email protected]> wrote:
Quoted 25 lines · Joao Martins
Quoted 23 lines · Cédric Le Goater
Hello Avihai,
Quoted 8 lines · Avihai Horon
The major parts of VFIO migration are supported today in QEMU. This includes basic VFIO migration, device dirty page tracking and precopy support.
Thus, at this point in time, it seems appropriate to make VFIO migration
non-experimental: remove the x prefix fromenable_migrationproperty, change it toON_OFF_AUTOand let the default value be AUTO.Quoted 5 lines
In addition, make the following adjustments:
1. Require device dirty tracking support whenenable_migrationis AUTO (i.e., not explicitly enabled). This is because device dirty tracking is currently the only method to do dirty page tracking, which is essential for migrating in a reasonable downtime.hmm, I don't think QEMU should decide to disable a feature for all devices supposedly because it could be slow for some. That's too restrictive. What about devices with have small states ? for which the downtime would be reasonable even without device dirty tracking support.
Quoted 6 lines
Quoted 4 lines
device dirty tracking refers to the ability to tracking dirty IOVA used by the device which will DMA into RAM. It is required because the consequence/alternative is to transfer all RAM in stop copy phase. Device state size at that point is the least of our problems downtime wise.
Arg. thanks for reminding me. I tend to take this for granted ...
My initial reaction was the same, for instance we could have a non-DMA device (ex. PCI serial card) that supports migration w/o dirty tracking, but QEMU cannot assume the device doesn't do DMA, nor is it worth our time to monitor whether bus-master is ever enabled for the device, so QEMU needs to assume all memory that's DMA accessible for the device is perpetually dirty. Also, if such a corner case existed for a non-DMA migratable device, the easier path is simply to require dirty tracking stubs in the variant driver to report no memory dirtied.
Quoted 95 lines
Quoted 95 lines
I can imagine that allowing without dirty tracking is useful for developer testing of the suspend/device-state flows, but as real default (auto) is very questionable to let it go through without dirty tracking. When we have IOMMUFD dirty tracking that's when we can relieve this restriction as a default.
But then note that (...)
Quoted 3 lines
Quoted 3 lines
Setting
enable_migrationto ON will not require device dirty tracking.(...) this lets it ignore dirty tracking as you would like.
Quoted 54 lines
Quoted 54 lines
2. Make migration blocker messages more elaborate.
3. Remove error prints invfio_migration_query_flags().
4. Remove a redundant assignment invfio_migration_realize().Signed-off-by Avihai Horon <[email protected]>--- include/hw/vfio/vfio-common.h | 2 +- hw/vfio/migration.c | 29 ++++++++++++++++------------- hw/vfio/pci.c | 4 ++-- 3 files changed, 19 insertions(+), 16 deletions(-)include/hw/vfio/vfio-common.h +1
-1@@ -139,7 +139,7 @@ typedef struct VFIODevice { bool needs_reset; bool no_mmap; bool ram_block_discard_allowed; - bool enable_migration; + OnOffAuto enable_migration; VFIODeviceOps *ops; unsigned int num_irqs; unsigned int num_regions;hw/vfio/migration.c +0
-8@@ -731,14 +731,6 @@ static int vfio_migration_query_flags(VFIODevice *vbasedev, uint64_t *mig_flags) feature->argsz = sizeof(buf); feature->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_MIGRATION; if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) { - if (errno == ENOTTY) { - error_report("%s: VFIO migration is not supported in kernel", - vbasedev->name); - } else { - error_report("%s: Failed to query VFIO migration support, err: %s", - vbasedev->name, strerror(errno)); - } - return -errno; }@@ -831,14 +823,28 @@ void
vfio_reset_bytes_transferred(void)
intvfio_migration_realize(VFIODevice *vbasedev, Error **errp){
- int ret =-ENOTSUP;
+ int ret;
- if (!vbasedev->enable_migration) {
+ if (vbasedev->enable_migration==ON_OFF_AUTO_OFF) {
+ error_setg(&vbasedev->migration_blocker,
+ "%s: Migration is disabled for VFIO device",vbasedev->name);
gotoadd_blocker;
}Quoted 9 lines
Quoted 2 lines
ret =
vfio_migration_init(vbasedev);
if (ret) {It would be good to keep the message for 'errno == ENOTTY' as it was in
vfio_migration_query_flags(). When migration fails, it is an important information to know that it is because the VFIO PCI host device driver doesn't support the feature. The root cause could be deep below in FW or how the VF was set up.+1 As I have been in this rabbit hole
Quoted 10 lines
Quoted 9 lines
+ error_setg(&vbasedev->migration_blocker,
+ "%s: Migration couldn't be initialized for VFIO device, " + "err: %d (%s)",
+vbasedev->name, ret, strerror(-ret));
+ gotoadd_blocker;
+ }
+
+ if (vbasedev->enable_migration==ON_OFF_AUTO_AUTO&&
+ !vbasedev->dirty_pages_supported) {Quoted 2 lines
I don't agree with this test.
The alternative right now is perceptual dirty tracking. How is that OK as a default? It's not like we have even an option :(
Quoted 8 lines
Quoted 4 lines
Maybe perhaps you refer to avoid strongly enforcing *always* it to allow testing of the non dirty tracking parts? Maybe when you 'force' enabling with enable-migration=on is when you ignore the dirty tracking which is what this is doing.
I see
ON_OFF_AUTO_ONas a way to abort the machine startup whileON_OFF_AUTO_AUTOwould let it run but block migration.
Agreed. There's a little bit of redundancy between the device-level "enable-migration=on" option and the global "-only-migratable" option relative to preventing machine startup, but it also doesn't make sense to me if the device-level option let realize complete successfully if the device doesn't support or fails migration setup. So I think we'd generally rely on using the -only-migratable option with the default ON_OFF_AUTO_AUTO value, allow the ON_OFF_AUTO_ON value to enable dis-recommended support, and live with the redundancy that it should also cause the device realize to fail if migration is not supported. Thanks,
Alex
Quoted 65 lines
We would
need an extra property to relax the checks, else we are hijacking some pre-existing option to fit our need.Since dirty tracking is a must-have to implement migration support for any existing and future VFIO PCI variant driver, anything else would be experimental code and we are trying to remove the flag ! Please correct me if I am wrong.
So, the case !vbasedev->dirty_pages_supported is just an extra information to report for why migration is not supported. Does that sound reasonable ?
Thanks,
C.
Quoted 45 lines
Quoted 11 lines
Quoted 6 lines
+ error_setg(&vbasedev->migration_blocker,
+ "%s: VFIO device doesn't support device dirty tracking", +vbasedev->name);
gotoadd_blocker;
}I agree that with
ON_OFF_AUTO_AUTO, errors at realize time should be recorded in a migration blocker. What about theON_OFF_AUTO_ONcase ? If migration was explicitly requested for the device and the conditions on the host are not met, I think realize should fail and the machine abort.+1 Good point
Quoted 32 lines
Thanks,
C.
Quoted 25 lines
hw/vfio/migration.c +0
-3@@ -856,9 +862,6 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) return 0; add_blocker: - error_setg(&vbasedev->migration_blocker, - "VFIO device doesn't support migration"); - ret = migrate_add_blocker(vbasedev->migration_blocker, errp); if (ret < 0) { error_free(vbasedev->migration_blocker);hw/vfio/pci.c +2
-2@@ -3347,8 +3347,8 @@ static Property vfio_pci_dev_properties[] = { VFIO_FEATURE_ENABLE_REQ_BIT, true), DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features, VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false), - DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice, - vbasedev.enable_migration, false), + DEFINE_PROP_ON_OFF_AUTO("enable-migration", VFIOPCIDevice, + vbasedev.enable_migration, ON_OFF_AUTO_AUTO), DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false), DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice, vbasedev.ram_block_discard_allowed, false),
Quoted 35 lines · Alex Williamson
External email: Use caution opening links or attachments
Quoted 24 lines · On Mon, 26 Jun 2023 17:26:42 +0200 Cédric Le Goater <[email protected]> wrote:
Quoted 23 lines · Joao Martins
Quoted 22 lines · Cédric Le Goater
Hello Avihai,
Quoted 8 lines · Avihai Horon
The major parts of VFIO migration are supported today in QEMU. This includes basic VFIO migration, device dirty page tracking and precopy support.
Thus, at this point in time, it seems appropriate to make VFIO migration
non-experimental: remove the x prefix fromenable_migrationproperty, change it toON_OFF_AUTOand let the default value be AUTO.Quoted 5 lines
In addition, make the following adjustments:
1. Require device dirty tracking support whenenable_migrationis AUTO (i.e., not explicitly enabled). This is because device dirty tracking is currently the only method to do dirty page tracking, which is essential for migrating in a reasonable downtime.hmm, I don't think QEMU should decide to disable a feature for all devices supposedly because it could be slow for some. That's too restrictive. What about devices with have small states ? for which the downtime would be reasonable even without device dirty tracking support.
Quoted 5 lines
Quoted 4 lines
device dirty tracking refers to the ability to tracking dirty IOVA used by the device which will DMA into RAM. It is required because the consequence/alternative is to transfer all RAM in stop copy phase. Device state size at that point is the least of our problems downtime wise.
Arg. thanks for reminding me. I tend to take this for granted ...
Quoted 84 lines
My initial reaction was the same, for instance we could have a non-DMA device (ex. PCI serial card) that supports migration w/o dirty tracking, but QEMU cannot assume the device doesn't do DMA, nor is it worth our time to monitor whether bus-master is ever enabled for the device, so QEMU needs to assume all memory that's DMA accessible for the device is perpetually dirty. Also, if such a corner case existed for a non-DMA migratable device, the easier path is simply to require dirty tracking stubs in the variant driver to report no memory dirtied.
Quoted 75 lines
Quoted 75 lines
I can imagine that allowing without dirty tracking is useful for developer testing of the suspend/device-state flows, but as real default (auto) is very questionable to let it go through without dirty tracking. When we have IOMMUFD dirty tracking that's when we can relieve this restriction as a default.
But then note that (...)
Quoted 2 lines
Quoted 2 lines
Setting
enable_migrationto ON will not require device dirty tracking.(...) this lets it ignore dirty tracking as you would like.
Quoted 53 lines
Quoted 53 lines
2. Make migration blocker messages more elaborate.
3. Remove error prints invfio_migration_query_flags().
4. Remove a redundant assignment invfio_migration_realize().Signed-off-by Avihai Horon <[email protected]>--- include/hw/vfio/vfio-common.h | 2 +- hw/vfio/migration.c | 29 ++++++++++++++++------------- hw/vfio/pci.c | 4 ++-- 3 files changed, 19 insertions(+), 16 deletions(-)include/hw/vfio/vfio-common.h +1
-1@@ -139,7 +139,7 @@ typedef struct VFIODevice {bool needs_reset;bool no_mmap;bool ram_block_discard_allowed; -bool enable_migration; +OnOffAuto enable_migration;VFIODeviceOps *ops;unsigned int num_irqs;unsigned int num_regions;hw/vfio/migration.c +0
-8@@ -731,14 +731,6 @@ static int vfio_migration_query_flags(VFIODevice *vbasedev, uint64_t *mig_flags) feature->argsz =sizeof (buf); feature->flags =VFIO_DEVICE_FEATURE_GET |VFIO_DEVICE_FEATURE_MIGRATION ;if (ioctl (vbasedev->fd,VFIO_DEVICE_FEATURE , feature)) { -if (errno ==ENOTTY ) { -error_report ("%s: VFIO migration is not supported in kernel" , - vbasedev->name); - }else { -error_report ("%s: Failed to query VFIO migration support, err: %s" , - vbasedev->name,strerror (errno)); - } -return -errno; }@@ -831,14 +823,28 @@ void vfio_reset_bytes_transferred(void) int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) {hw/vfio/migration.c +4
-1-int ret = -ENOTSUP ; +int ret; - if (!vbasedev->enable_migration) { +if (vbasedev->enable_migration ==ON_OFF_AUTO_OFF ) { +error_setg (&vbasedev->migration_blocker, +"%s: Migration is disabled for VFIO device" , vbasedev->name); goto add_blocker;Quoted 9 lines
Quoted 3 lines
} ret = vfio_migration_init(vbasedev); if (ret) {It would be good to keep the message for 'errno == ENOTTY' as it was in
vfio_migration_query_flags(). When migration fails, it is an important information to know that it is because the VFIO PCI host device driver doesn't support the feature. The root cause could be deep below in FW or how the VF was set up.+1 As I have been in this rabbit hole
Sure, I will add it.
Quoted 33 lines
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
hw/vfio/migration.c void vfio_reset_bytes_transferred(void) · line 844
Quoted 10 lines
hw/vfio/migration.c +9
-0+error_setg (&vbasedev->migration_blocker, +"%s: Migration couldn't be initialized for VFIO device, " +"err: %d (%s)" , + vbasedev->name, ret,strerror (-ret)); + goto add_blocker; + } + +if (vbasedev->enable_migration ==ON_OFF_AUTO_AUTO && + !vbasedev->dirty_pages_supported) {Quoted 2 lines
I don't agree with this test.
The alternative right now is perceptual dirty tracking. How is that OK as a default? It's not like we have even an option :(
Quoted 7 lines
Quoted 4 lines
Maybe perhaps you refer to avoid strongly enforcing *always* it to allow testing of the non dirty tracking parts? Maybe when you 'force' enabling with enable-migration=on is when you ignore the dirty tracking which is what this is doing.
I see
ON_OFF_AUTO_ONas a way to abort the machine startup whileON_OFF_AUTO_AUTOwould let it run but block migration.Agreed. There's a little bit of redundancy between the device-level "enable-migration=on" option and the global "-only-migratable" option relative to preventing machine startup, but it also doesn't make sense to me if the device-level option let realize complete successfully if the device doesn't support or fails migration setup. So I think we'd generally rely on using the -only-migratable option with the default
ON_OFF_AUTO_AUTOvalue, allow theON_OFF_AUTO_ONvalue to enable dis-recommended support, and live with the redundancy that it should also cause the device realize to fail if migration is not supported. Thanks,
Quoted 1 line
Alex
OK.
When enable_migration=AUTO we allow blockers.
When enable_migration=ON we don't allow blockers and instead prevent realization of VFIO device.
Regarding device dirty tracking, we keep current behavior, right?
That is:
When enable_migration=AUTO we block migration if device dirty tracking is not supported.
When enable_migration=ON we allow migration even if device dirty tracking is not supported (in which case DMA-able memory will be perpetually dirtied).
Thanks.
Quoted 63 lines
Quoted 63 lines
We would
need an extra property to relax the checks, else we are hijacking some pre-existing option to fit our need.Since dirty tracking is a must-have to implement migration support for any existing and future VFIO PCI variant driver, anything else would be experimental code and we are trying to remove the flag ! Please correct me if I am wrong.
So, the case !vbasedev->dirty_pages_supported is just an extra information to report for why migration is not supported. Does that sound reasonable ?
Thanks,
C.
Quoted 43 lines
Quoted 10 lines
hw/vfio/migration.c void vfio_reset_bytes_transferred(void) · line 849
Quoted 5 lines
hw/vfio/migration.c +3
-0+error_setg (&vbasedev->migration_blocker, +"%s: VFIO device doesn't support device dirty tracking" , + vbasedev->name); goto add_blocker; }I agree that with
ON_OFF_AUTO_AUTO, errors at realize time should be recorded in a migration blocker. What about theON_OFF_AUTO_ONcase ? If migration was explicitly requested for the device and the conditions on the host are not met, I think realize should fail and the machine abort.+1 Good point
Quoted 31 lines
Thanks,
C.
Quoted 25 lines
hw/vfio/migration.c +0
-3@@ -856,9 +862,6 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp)return 0 ; add_blocker: -error_setg (&vbasedev->migration_blocker, -"VFIO device doesn't support migration" ); - ret =migrate_add_blocker (vbasedev->migration_blocker, errp);if (ret <0 ) {error_free (vbasedev->migration_blocker);hw/vfio/pci.c +2
-2@@ -3347,8 +3347,8 @@ static Property vfio_pci_dev_properties[] = { VFIO_FEATURE_ENABLE_REQ_BIT, true),DEFINE_PROP_BIT ("x-igd-opregion" , VFIOPCIDevice, features,VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT , false), -DEFINE_PROP_BOOL ("x-enable-migration" , VFIOPCIDevice, - vbasedev.enable_migration, false), +DEFINE_PROP_ON_OFF_AUTO ("enable-migration" , VFIOPCIDevice, + vbasedev.enable_migration,ON_OFF_AUTO_AUTO ),DEFINE_PROP_BOOL ("x-no-mmap" , VFIOPCIDevice, vbasedev.no_mmap, false),DEFINE_PROP_BOOL ("x-balloon-allowed" , VFIOPCIDevice, vbasedev.ram_block_discard_allowed, false),
[ Cc: Shameer ]
Quoted 18 lines
Quoted 14 lines
Quoted 2 lines
I see
ON_OFF_AUTO_ONas a way to abort the machine startup whileON_OFF_AUTO_AUTOwould let it run but block migration.Agreed. There's a little bit of redundancy between the device-level "enable-migration=on" option and the global "-only-migratable" option relative to preventing machine startup, but it also doesn't make sense to me if the device-level option let realize complete successfully if the device doesn't support or fails migration setup. So I think we'd generally rely on using the -only-migratable option with the default
ON_OFF_AUTO_AUTOvalue, allow theON_OFF_AUTO_ONvalue to enable dis-recommended support, and live with the redundancy that it should also cause the device realize to fail if migration is not supported. Thanks,Quoted 1 line
Alex
OK.
Quoted 7 lines
When enable_migration=AUTO we allow blockers.
When enable_migration=ON we don't allow blockers and instead prevent realization of VFIO device.Regarding device dirty tracking, we keep current behavior, right?
That is:
When enable_migration=AUTO we block migration if device dirty tracking is not supported.
When enable_migration=ON we allow migration even if device dirty tracking is not supported > (in which case DMA-able memory will be perpetually dirtied).
Yes. That's how I understand it. This is what you initially proposed.
The default behavior is to allow migration only if the host kernel driver supports dirty tracking.
We have a way to run and migrate a machine with a device not supporting dirty tracking. Only Hisilicon is in that case today. May be there are plans to add dirty tracking support ?
Thanks,
C.
Quoted 3 lines · Cédric Le Goater
We have a way to run and migrate a machine with a device not supporting dirty tracking. Only Hisilicon is in that case today. May be there are plans to add dirty tracking support ?
Hisilicon will eventually use Joao's work for IOMMU based tracking, this is what their HW was designed to do.
Jason
Quoted 17 lines
-----Original Message-----
From: Jason Gunthorpe [mailto:[email protected]]
Sent: 27 June 2023 13:30
To: Cédric Le Goater <[email protected]>
Cc: Avihai Horon <[email protected]>; Alex Williamson <[email protected]>; Joao Martins <[email protected]>; Juan Quintela <[email protected]>; Peter Xu <[email protected]>; Leonardo Bras <[email protected]>; Zhenzhong Duan <[email protected]>; Yishai Hadas <[email protected]>; Maor Gottlieb <[email protected]>; Kirti Wankhede <[email protected]>; Tarun Gupta <[email protected]>; [email protected]; Shameerali Kolothum Thodi <[email protected]>
Subject: Re: [PATCH 3/3] vfio/migration: Make VFIO migration non-experimental
Quoted 8 lines
Quoted 3 lines · Cédric Le Goater
We have a way to run and migrate a machine with a device not supporting dirty tracking. Only Hisilicon is in that case today. May be there are plans to add dirty tracking support ?
Hisilicon will eventually use Joao's work for IOMMU based tracking, this is what their HW was designed to do.
Yes. The plan is to make use of SMMUv3 HTTU feature for dirty tracking based on Joao's work here,
https://lore.kernel.org/kvm/[email protected]/
Thanks,
Shameer
Quoted 164 lines · On Tue, 27 Jun 2023 11:00:46 +0300 Avihai Horon <[email protected]> wrote:
Quoted 35 lines · Alex Williamson
External email: Use caution opening links or attachments
Quoted 24 lines · On Mon, 26 Jun 2023 17:26:42 +0200 Cédric Le Goater <[email protected]> wrote:
Quoted 23 lines · Joao Martins
Quoted 22 lines · Cédric Le Goater
Hello Avihai,
Quoted 8 lines · Avihai Horon
The major parts of VFIO migration are supported today in QEMU. This includes basic VFIO migration, device dirty page tracking and precopy support.
Thus, at this point in time, it seems appropriate to make VFIO migration
non-experimental: remove the x prefix fromenable_migrationproperty, change it toON_OFF_AUTOand let the default value be AUTO.Quoted 5 lines
In addition, make the following adjustments:
1. Require device dirty tracking support whenenable_migrationis AUTO (i.e., not explicitly enabled). This is because device dirty tracking is currently the only method to do dirty page tracking, which is essential for migrating in a reasonable downtime.hmm, I don't think QEMU should decide to disable a feature for all devices supposedly because it could be slow for some. That's too restrictive. What about devices with have small states ? for which the downtime would be reasonable even without device dirty tracking support.
Quoted 5 lines
Quoted 4 lines
device dirty tracking refers to the ability to tracking dirty IOVA used by the device which will DMA into RAM. It is required because the consequence/alternative is to transfer all RAM in stop copy phase. Device state size at that point is the least of our problems downtime wise.
Arg. thanks for reminding me. I tend to take this for granted ...
Quoted 84 lines
My initial reaction was the same, for instance we could have a non-DMA device (ex. PCI serial card) that supports migration w/o dirty tracking, but QEMU cannot assume the device doesn't do DMA, nor is it worth our time to monitor whether bus-master is ever enabled for the device, so QEMU needs to assume all memory that's DMA accessible for the device is perpetually dirty. Also, if such a corner case existed for a non-DMA migratable device, the easier path is simply to require dirty tracking stubs in the variant driver to report no memory dirtied.
Quoted 75 lines
Quoted 75 lines
I can imagine that allowing without dirty tracking is useful for developer testing of the suspend/device-state flows, but as real default (auto) is very questionable to let it go through without dirty tracking. When we have IOMMUFD dirty tracking that's when we can relieve this restriction as a default.
But then note that (...)
Quoted 2 lines
Quoted 2 lines
Setting
enable_migrationto ON will not require device dirty tracking.(...) this lets it ignore dirty tracking as you would like.
Quoted 53 lines
Quoted 53 lines
2. Make migration blocker messages more elaborate.
3. Remove error prints invfio_migration_query_flags().
4. Remove a redundant assignment invfio_migration_realize().Signed-off-by Avihai Horon <[email protected]>--- include/hw/vfio/vfio-common.h | 2 +- hw/vfio/migration.c | 29 ++++++++++++++++------------- hw/vfio/pci.c | 4 ++-- 3 files changed, 19 insertions(+), 16 deletions(-)include/hw/vfio/vfio-common.h +1
-1@@ -139,7 +139,7 @@ typedef struct VFIODevice {bool needs_reset;bool no_mmap;bool ram_block_discard_allowed; -bool enable_migration; +OnOffAuto enable_migration;VFIODeviceOps *ops;unsigned int num_irqs;unsigned int num_regions;hw/vfio/migration.c +0
-8@@ -731,14 +731,6 @@ static int vfio_migration_query_flags(VFIODevice *vbasedev, uint64_t *mig_flags) feature->argsz =sizeof (buf); feature->flags =VFIO_DEVICE_FEATURE_GET |VFIO_DEVICE_FEATURE_MIGRATION ;if (ioctl (vbasedev->fd,VFIO_DEVICE_FEATURE , feature)) { -if (errno ==ENOTTY ) { -error_report ("%s: VFIO migration is not supported in kernel" , - vbasedev->name); - }else { -error_report ("%s: Failed to query VFIO migration support, err: %s" , - vbasedev->name,strerror (errno)); - } -return -errno; }@@ -831,14 +823,28 @@ void vfio_reset_bytes_transferred(void) int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) {hw/vfio/migration.c +4
-1-int ret = -ENOTSUP ; +int ret; - if (!vbasedev->enable_migration) { +if (vbasedev->enable_migration ==ON_OFF_AUTO_OFF ) { +error_setg (&vbasedev->migration_blocker, +"%s: Migration is disabled for VFIO device" , vbasedev->name); goto add_blocker;Quoted 9 lines
Quoted 3 lines
} ret = vfio_migration_init(vbasedev); if (ret) {It would be good to keep the message for 'errno == ENOTTY' as it was in
vfio_migration_query_flags(). When migration fails, it is an important information to know that it is because the VFIO PCI host device driver doesn't support the feature. The root cause could be deep below in FW or how the VF was set up.+1 As I have been in this rabbit hole
Sure, I will add it.
Quoted 33 lines
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
hw/vfio/migration.c void vfio_reset_bytes_transferred(void) · line 844
Quoted 10 lines
hw/vfio/migration.c +9
-0+error_setg (&vbasedev->migration_blocker, +"%s: Migration couldn't be initialized for VFIO device, " +"err: %d (%s)" , + vbasedev->name, ret,strerror (-ret)); + goto add_blocker; + } + +if (vbasedev->enable_migration ==ON_OFF_AUTO_AUTO && + !vbasedev->dirty_pages_supported) {Quoted 2 lines
I don't agree with this test.
The alternative right now is perceptual dirty tracking. How is that OK as a default? It's not like we have even an option :(
Quoted 7 lines
Quoted 4 lines
Maybe perhaps you refer to avoid strongly enforcing *always* it to allow testing of the non dirty tracking parts? Maybe when you 'force' enabling with enable-migration=on is when you ignore the dirty tracking which is what this is doing.
I see
ON_OFF_AUTO_ONas a way to abort the machine startup whileON_OFF_AUTO_AUTOwould let it run but block migration.Agreed. There's a little bit of redundancy between the device-level "enable-migration=on" option and the global "-only-migratable" option relative to preventing machine startup, but it also doesn't make sense to me if the device-level option let realize complete successfully if the device doesn't support or fails migration setup. So I think we'd generally rely on using the -only-migratable option with the default
ON_OFF_AUTO_AUTOvalue, allow theON_OFF_AUTO_ONvalue to enable dis-recommended support, and live with the redundancy that it should also cause the device realize to fail if migration is not supported. Thanks,Quoted 1 line
Alex
OK.
When enable_migration=AUTO we allow blockers.
When enable_migration=ON we don't allow blockers and instead prevent realization of VFIO device.
Quoted 7 lines
Regarding device dirty tracking, we keep current behavior, right? That is:
When enable_migration=AUTO we block migration if device dirty tracking is not supported.
When enable_migration=ON we allow migration even if device dirty tracking is not supported (in which case DMA-able memory will be perpetually dirtied).
Yes, and I think we'd be justified in logging a warning when migration is enabled without any dirty page tracking support. Thanks,
Alex