migration: Fix multifd cancel test
cover migration: Fix multifd cancel test
When doing cleanup of the multifd send threads we're calling QLIST_REMOVE concurrently on the migration_threads list. This seems to be the source of the crashes we've seen on the multifd/tcp/plain/cancel tests.
I'm running the test in a loop and after a few dozen iterations I see the crash in dmesg.
QTEST_QEMU_BINARY=./qemu-system-x86_64 \ QEMU_TEST_FLAKY_TESTS=1 \ ./tests/qtest/migration-test -p /x86_64/migration/multifd/tcp/plain/cancel
multifdsend_10[11382]: segfault at 18 ip 0000564b77de1e25 sp 00007fdf767fb610 error 6 in qemu-system-x86_64[564b777b4000+e1c000]
Code: ec 10 48 89 7d f8 48 83 7d f8 00 74 58 48 8b 45 f8 48 8b 40 10 48 85 c0 74 14 48 8b 45 f8 48 8b 40 10 48 8b 55 f8 48 8b 52 18 <48> 89 50 18 48 8b 45 f8 48 8b 40 18 48 8b 55 f8 48 8b 52 10 48 89
the offending instruction is a mov dereferencing the thread->node.le_next pointer at QLIST_REMOVE in MigrationThreadDel:
void MigrationThreadDel(MigrationThread *thread)
{
if (thread) {
QLIST_REMOVE(thread, node);
g_free(thread);
}
}where:
#define QLIST_REMOVE(elm, field) do { \
if ((elm)->field.le_next != NULL) \
(elm)->field.le_next->field.le_prev = \ <-- HERE
(elm)->field.le_prev; \
*(elm)->field.le_prev = (elm)->field.le_next; \
(elm)->field.le_next = NULL; \
(elm)->field.le_prev = NULL; \} while (/*CONSTCOND*/0)
The MigrationThreadDel function is called from the multifd threads and is not under any lock, so several calls can race when accessing the list.
(I actually hit this first on my fixed-ram branch which changes some synchronization in multifd and makes the issue more frequent)
CI run: https://gitlab.com/farosas/qemu/-/pipelines/891000519
Fabiano Rosas (3):
migration/multifd: Rename threadinfo.c functions migration/multifd: Protect accesses to migration_threads tests/qtest: Re-enable multifd cancel test
migration/migration.c | 7 +++++-- migration/multifd.c | 5 +++-- migration/threadinfo.c | 23 ++++++++++++++++++++--- migration/threadinfo.h | 8 ++++---- tests/qtest/migration-test.c | 10 ++-------- 5 files changed, 34 insertions(+), 19 deletions(-)
Signature · 1 line
2.35.3
PATCH 1/3 migration/multifd: Rename threadinfo.c functions
The code in threadinfo.c is only used for the QMP command query-migrationthreads. Make it explicit that this is something related to QMP.
The current names are also too generic for a piece of code that doesn't affect the migration directly in any way.
--- migration/migration.c | 4 ++-- migration/multifd.c | 4 ++-- migration/threadinfo.c | 4 ++-- migration/threadinfo.h | 5 ++--- 4 files changed, 8 insertions(+), 9 deletions(-)
migration/migration.c +2 -2
@@ -2922,7 +2922,7 @@ static void *migration_thread(void *opaque)MigThrError thr_error;bool urgent = false; - thread =MigrationThreadAdd ("live_migration" ,qemu_get_thread_id ()); + thread =qmp_migration_threads_add ("live_migration" ,qemu_get_thread_id ());rcu_register_thread ();
@@ -3000,7 +3000,7 @@ static void *migration_thread(void *opaque)migration_iteration_finish (s);object_unref (OBJECT (s));rcu_unregister_thread (); -MigrationThreadDel (thread); +qmp_migration_threads_remove (thread);return NULL ; }
migration/multifd.c +2 -2
@@ -651,7 +651,7 @@ static void *multifd_send_thread(void *opaque)int ret =0 ;bool use_zero_copy_send =migrate_zero_copy_send (); - thread =MigrationThreadAdd (p->name,qemu_get_thread_id ()); + thread =qmp_migration_threads_add (p->name,qemu_get_thread_id ());trace_multifd_send_thread_start (p->id);rcu_register_thread ();
@@ -767,7 +767,7 @@ out:qemu_mutex_unlock (&p->mutex);rcu_unregister_thread (); -MigrationThreadDel (thread); +qmp_migration_threads_remove (thread);trace_multifd_send_thread_end (p->id, p->num_packets, p->total_normal_pages);return NULL ;
migration/threadinfo.c +2 -2
@@ -14,7 +14,7 @@ static QLIST_HEAD(, MigrationThread) migration_threads; -MigrationThread *MigrationThreadAdd (const char *name,int thread_id) +MigrationThread *qmp_migration_threads_add (const char *name,int thread_id) {MigrationThread *thread =g_new0 (MigrationThread,1 ); thread->name = name;
@@ -25,7 +25,7 @@ MigrationThread *MigrationThreadAdd(const char *name, int thread_id)return thread; } -void MigrationThreadDel (MigrationThread *thread) +void qmp_migration_threads_remove (MigrationThread *thread) {if (thread) {QLIST_REMOVE (thread, node);
migration/threadinfo.h +2 -3
@@ -23,6 +23,5 @@ struct MigrationThread {QLIST_ENTRY (MigrationThread) node; }; -MigrationThread *MigrationThreadAdd (const char *name,int thread_id); - -void MigrationThreadDel (MigrationThread *info); +MigrationThread *qmp_migration_threads_add (const char *name,int thread_id); +void qmp_migration_threads_remove (MigrationThread *info);
Signature · 1 line
2.35.3
Re: [PATCH 1/3] migration/multifd: Rename threadinfo.c functions
Quoted 8 lines · Fabiano Rosas
The code in
threadinfo.cis only used for the QMP command query-migrationthreads. Make it explicit that this is something related to QMP.The current names are also too generic for a piece of code that doesn't affect the migration directly in any way.
Signed-off-by Fabiano Rosas <[email protected]>
Looks good here, but shall we reserve the qmp_* prefix to mostly qmp stuff only? Dropping "qmp_" in the new names would look better to me..
Quoted 89 lines
--- migration/migration.c | 4 ++-- migration/multifd.c | 4 ++-- migration/threadinfo.c | 4 ++-- migration/threadinfo.h | 5 ++--- 4 files changed, 8 insertions(+), 9 deletions(-)migration/migration.c +2
-2@@ -2922,7 +2922,7 @@ static void *migration_thread(void *opaque)MigThrError thr_error;bool urgent = false; - thread =MigrationThreadAdd ("live_migration" ,qemu_get_thread_id ()); + thread =qmp_migration_threads_add ("live_migration" ,qemu_get_thread_id ());rcu_register_thread ();@@ -3000,7 +3000,7 @@ static void *migration_thread(void *opaque)migration_iteration_finish (s);object_unref (OBJECT (s));rcu_unregister_thread (); -MigrationThreadDel (thread); +qmp_migration_threads_remove (thread);return NULL ; }migration/multifd.c +2
-2@@ -651,7 +651,7 @@ static void *multifd_send_thread(void *opaque)int ret =0 ;bool use_zero_copy_send =migrate_zero_copy_send (); - thread =MigrationThreadAdd (p->name,qemu_get_thread_id ()); + thread =qmp_migration_threads_add (p->name,qemu_get_thread_id ());trace_multifd_send_thread_start (p->id);rcu_register_thread ();@@ -767,7 +767,7 @@ out:qemu_mutex_unlock (&p->mutex);rcu_unregister_thread (); -MigrationThreadDel (thread); +qmp_migration_threads_remove (thread);trace_multifd_send_thread_end (p->id, p->num_packets, p->total_normal_pages);return NULL ;migration/threadinfo.c +2
-2@@ -14,7 +14,7 @@ static QLIST_HEAD(, MigrationThread) migration_threads; -MigrationThread *MigrationThreadAdd (const char *name,int thread_id) +MigrationThread *qmp_migration_threads_add (const char *name,int thread_id) {MigrationThread *thread =g_new0 (MigrationThread,1 ); thread->name = name;@@ -25,7 +25,7 @@ MigrationThread *MigrationThreadAdd(const char *name, int thread_id)return thread; } -void MigrationThreadDel (MigrationThread *thread) +void qmp_migration_threads_remove (MigrationThread *thread) {if (thread) {QLIST_REMOVE (thread, node);migration/threadinfo.h +2
-3@@ -23,6 +23,5 @@ struct MigrationThread {QLIST_ENTRY (MigrationThread) node; }; -MigrationThread *MigrationThreadAdd (const char *name,int thread_id); - -void MigrationThreadDel (MigrationThread *info); +MigrationThread *qmp_migration_threads_add (const char *name,int thread_id); +void qmp_migration_threads_remove (MigrationThread *info);Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
Re: [PATCH 1/3] migration/multifd: Rename threadinfo.c functions
Quoted 5 lines · Peter Xu
Quoted 4 lines · Fabiano Rosas
The code in
threadinfo.cis only used for the QMP command query-migrationthreads. Make it explicit that this is something related to QMP.
Quoted 8 lines
Quoted 4 lines
The current names are also too generic for a piece of code that doesn't affect the migration directly in any way.
Signed-off-by Fabiano Rosas <[email protected]>Looks good here, but shall we reserve the qmp_* prefix to mostly qmp stuff only? Dropping "
qmp_" in the new names would look better to me..
Well, we're just putting the thread name and id on a list so that QMP can use them later. It is nothing "important" enough to have a generic name like migration_thread.
Perhaps:
thread_info_addthread_info_removethread_info_initthread_info_cleanup
Anyway, as long as we drop that camel case I'm ok with just removing the qmp =)
Re: [PATCH 1/3] migration/multifd: Rename threadinfo.c functions
Quoted 22 lines · Fabiano Rosas
Quoted 5 lines · Peter Xu
Quoted 4 lines · Fabiano Rosas
The code in
threadinfo.cis only used for the QMP command query-migrationthreads. Make it explicit that this is something related to QMP.Quoted 8 lines
Quoted 4 lines
The current names are also too generic for a piece of code that doesn't affect the migration directly in any way.
Signed-off-by Fabiano Rosas <[email protected]>Looks good here, but shall we reserve the qmp_* prefix to mostly qmp stuff only? Dropping "
qmp_" in the new names would look better to me..Well, we're just putting the thread name and id on a list so that QMP can use them later. It is nothing "important" enough to have a generic name like
migration_thread.Perhaps:
Quoted 7 lines
thread_info_addthread_info_removethread_info_initthread_info_cleanupAnyway, as long as we drop that camel case I'm ok with just removing the qmp =)
Thanks. To me OTOH it's good as long as "qmp_" dropped. :)
I don't worry on using "migration_thread_" as prefix, that's exactly what the api does to me. Or, migration_thread_info_*(), migration_thr_mgr_*(), etc.
Signature · 1 line
Peter Xu
Re: [PATCH 1/3] migration/multifd: Rename threadinfo.c functions
In reply to Fabiano RosasQuoted 8 lines · Fabiano Rosas
The code in
threadinfo.cis only used for the QMP command query-migrationthreads. Make it explicit that this is something related to QMP.The current names are also too generic for a piece of code that doesn't affect the migration directly in any way.
Signed-off-by Fabiano Rosas <[email protected]>
Don't like the caml case. And don't really care with/without the qmp_ preffix. You got it eitherwise.
Re: [PATCH 1/3] migration/multifd: Rename threadinfo.c functions
In reply to Fabiano RosasQuoted 16 lines · Fabiano Rosas
The code in
threadinfo.cis only used for the QMP command query-migrationthreads. Make it explicit that this is something related to QMP.The current names are also too generic for a piece of code that doesn't affect the migration directly in any way.
Signed-off-by Fabiano Rosas <[email protected]>--- migration/migration.c | 4 ++-- migration/multifd.c | 4 ++-- migration/threadinfo.c | 4 ++-- migration/threadinfo.h | 5 ++--- 4 files changed, 8 insertions(+), 9 deletions(-)
migration/threadinfo.h struct MigrationThread { · line 27
Quoted 5 lines
migration/threadinfo.h +2
-3-MigrationThread *MigrationThreadAdd (const char *name,int thread_id); - -void MigrationThreadDel (MigrationThread *info); +MigrationThread *qmp_migration_threads_add (const char *name,int thread_id); +void qmp_migration_threads_remove (MigrationThread *info);
Dropping 'qmp_' prefix:
PATCH 2/3 migration/multifd: Protect accesses to migration_threads
In reply to Fabiano RosasThis doubly linked list is common for all the multifd and migration threads so we need to avoid concurrent access.
Add a mutex to protect the data from concurrent access. This fixes a crash when removing two MigrationThread objects from the list at the same time during cleanup of multifd threads.
To avoid destroying the mutex before the last element has been removed, move calls to qmp_migration_thread_remove so they run before multifd_save_cleanup joins the threads.
--- migration/migration.c | 5 ++++- migration/multifd.c | 3 ++- migration/threadinfo.c | 19 ++++++++++++++++++- migration/threadinfo.h | 5 +++-- 4 files changed, 27 insertions(+), 5 deletions(-)
migration/migration.c +4 -1
@@ -1146,6 +1146,7 @@ static void migrate_fd_cleanup(MigrationState *s)qemu_mutex_lock_iothread ();multifd_save_cleanup (); +qmp_migration_threads_cleanup ();qemu_mutex_lock (&s->qemu_file_lock); tmp = s->to_dst_file; s->to_dst_file =NULL ;
@@ -1405,6 +1406,8 @@ void migrate_init(MigrationState *s) s->vm_old_state =-1 ; s->iteration_initial_bytes =0 ; s->threshold_size =0 ; + +qmp_migration_threads_init (); }int migrate_add_blocker_internal (Error *reason,Error **errp)
@@ -2997,10 +3000,10 @@ static void *migration_thread(void *opaque) }trace_migration_thread_after_loop (); +qmp_migration_threads_remove (thread);migration_iteration_finish (s);object_unref (OBJECT (s));rcu_unregister_thread (); -qmp_migration_threads_remove (thread);return NULL ; }
migration/multifd.c +2 -1
@@ -762,12 +762,13 @@ out:qemu_sem_post (&multifd_send_state->channels_ready); } +qmp_migration_threads_remove (thread); +qemu_mutex_lock (&p->mutex); p->running = false;qemu_mutex_unlock (&p->mutex);rcu_unregister_thread (); -qmp_migration_threads_remove (thread);trace_multifd_send_thread_end (p->id, p->num_packets, p->total_normal_pages);return NULL ;
migration/threadinfo.c +18 -1
@@ -10,23 +10,40 @@* See the COPYING file in the top-level directory. */ +#include "qemu/osdep.h" +#include "qemu/queue.h" +#include "qemu/lockable.h" #include "threadinfo.h" +QemuMutex migration_threads_lock; static QLIST_HEAD(, MigrationThread) migration_threads; +void qmp_migration_threads_init (void ) +{ +qemu_mutex_init (&migration_threads_lock); +} + +void qmp_migration_threads_cleanup (void ) +{ +qemu_mutex_destroy (&migration_threads_lock); +} +MigrationThread *qmp_migration_threads_add (const char *name,int thread_id) {MigrationThread *thread =g_new0 (MigrationThread,1 ); thread->name = name; thread->thread_id = thread_id; -QLIST_INSERT_HEAD (&migration_threads, thread, node); + WITH_QEMU_LOCK_GUARD(&migration_threads_lock) { +QLIST_INSERT_HEAD (&migration_threads, thread, node); + }return thread; }void qmp_migration_threads_remove (MigrationThread *thread) { +QEMU_LOCK_GUARD (&migration_threads_lock);if (thread) {QLIST_REMOVE (thread, node);g_free (thread);
migration/threadinfo.h +3 -2
@@ -10,8 +10,6 @@* See the COPYING file in the top-level directory. */ -#include "qemu/queue.h" -#include "qemu/osdep.h" #include "qapi/error.h" #include "qapi/qapi-commands-migration.h"
@@ -23,5 +21,8 @@ struct MigrationThread {QLIST_ENTRY (MigrationThread) node; }; +void qmp_migration_threads_init (void ); +void qmp_migration_threads_cleanup (void ); +MigrationThread *qmp_migration_threads_add (const char *name,int thread_id);void qmp_migration_threads_remove (MigrationThread *info);
Signature · 1 line
2.35.3
Re: [PATCH 2/3] migration/multifd: Protect accesses to migration_threads
Quoted 108 lines · Fabiano Rosas
This doubly linked list is common for all the multifd and migration threads so we need to avoid concurrent access.
Add a mutex to protect the data from concurrent access. This fixes a crash when removing two MigrationThread objects from the list at the same time during cleanup of multifd threads.
To avoid destroying the mutex before the last element has been removed, move calls to
qmp_migration_thread_removeso they run beforemultifd_save_cleanupjoins the threads.Fixes 671326201d ("migration: Introduce interface query-migrationthreads")Signed-off-by Fabiano Rosas <[email protected]>--- migration/migration.c | 5 ++++- migration/multifd.c | 3 ++- migration/threadinfo.c | 19 ++++++++++++++++++- migration/threadinfo.h | 5 +++-- 4 files changed, 27 insertions(+), 5 deletions(-)migration/migration.c +4
-1@@ -1146,6 +1146,7 @@ static void migrate_fd_cleanup(MigrationState *s)qemu_mutex_lock_iothread ();multifd_save_cleanup (); +qmp_migration_threads_cleanup ();qemu_mutex_lock (&s->qemu_file_lock); tmp = s->to_dst_file; s->to_dst_file =NULL ;@@ -1405,6 +1406,8 @@ void migrate_init(MigrationState *s) s->vm_old_state =-1 ; s->iteration_initial_bytes =0 ; s->threshold_size =0 ; + +qmp_migration_threads_init (); }int migrate_add_blocker_internal (Error *reason,Error **errp)@@ -2997,10 +3000,10 @@ static void *migration_thread(void *opaque) }trace_migration_thread_after_loop (); +qmp_migration_threads_remove (thread);migration_iteration_finish (s);object_unref (OBJECT (s));rcu_unregister_thread (); -qmp_migration_threads_remove (thread);return NULL ; }migration/multifd.c +2
-1@@ -762,12 +762,13 @@ out:qemu_sem_post (&multifd_send_state->channels_ready); } +qmp_migration_threads_remove (thread); +qemu_mutex_lock (&p->mutex); p->running = false;qemu_mutex_unlock (&p->mutex);rcu_unregister_thread (); -qmp_migration_threads_remove (thread);trace_multifd_send_thread_end (p->id, p->num_packets, p->total_normal_pages);return NULL ;migration/threadinfo.c +17
-1@@ -10,23 +10,40 @@* See the COPYING file in the top-level directory. */ +#include "qemu/osdep.h" +#include "qemu/queue.h" +#include "qemu/lockable.h" #include "threadinfo.h" +QemuMutex migration_threads_lock; static QLIST_HEAD(, MigrationThread) migration_threads; +void qmp_migration_threads_init (void ) +{ +qemu_mutex_init (&migration_threads_lock); +} + +void qmp_migration_threads_cleanup (void ) +{ +qemu_mutex_destroy (&migration_threads_lock); +} +MigrationThread *qmp_migration_threads_add (const char *name,int thread_id) {MigrationThread *thread =g_new0 (MigrationThread,1 ); thread->name = name; thread->thread_id = thread_id; -QLIST_INSERT_HEAD (&migration_threads, thread, node); + WITH_QEMU_LOCK_GUARD(&migration_threads_lock) { +QLIST_INSERT_HEAD (&migration_threads, thread, node); + }
migration/threadinfo.c line 49
Quoted 10 lines
migration/threadinfo.c +1
-0return thread; }void qmp_migration_threads_remove (MigrationThread *thread) { +QEMU_LOCK_GUARD (&migration_threads_lock);if (thread) {QLIST_REMOVE (thread, node);g_free (thread);
qmp_query_migrationthreads() better also have the lock?
Other than that looks good, thanks!
Quoted 25 lines
migration/threadinfo.h +3
-2@@ -10,8 +10,6 @@* See the COPYING file in the top-level directory. */ -#include "qemu/queue.h" -#include "qemu/osdep.h" #include "qapi/error.h" #include "qapi/qapi-commands-migration.h" @@ -23,5 +21,8 @@ struct MigrationThread {QLIST_ENTRY (MigrationThread) node; }; +void qmp_migration_threads_init (void ); +void qmp_migration_threads_cleanup (void ); +MigrationThread *qmp_migration_threads_add (const char *name,int thread_id);void qmp_migration_threads_remove (MigrationThread *info);Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
Re: [PATCH 2/3] migration/multifd: Protect accesses to migration_threads
In reply to Fabiano RosasQuoted 7 lines · Fabiano Rosas
This doubly linked list is common for all the multifd and migration threads so we need to avoid concurrent access.
Add a mutex to protect the data from concurrent access. This fixes a crash when removing two MigrationThread objects from the list at the same time during cleanup of multifd threads.
Quoted 6 lines
To avoid destroying the mutex before the last element has been removed, move calls to
qmp_migration_thread_removeso they run beforemultifd_save_cleanupjoins the threads.Fixes 671326201d ("migration: Introduce interface query-migrationthreads")Signed-off-by Fabiano Rosas <[email protected]>
I agree with Peter here. Why don't you have to protect the walking?
Quoted 12 lines
--- migration/migration.c | 5 ++++- migration/multifd.c | 3 ++- migration/threadinfo.c | 19 ++++++++++++++++++- migration/threadinfo.h | 5 +++-- 4 files changed, 27 insertions(+), 5 deletions(-)migration/migration.c +0
-0
migration/migration.c static void migrate_fd_cleanup(MigrationState *s) · line 1149
Quoted 4 lines
migration/migration.c +1
-0@@ -1146,6 +1146,7 @@ static void migrate_fd_cleanup(MigrationState *s)qemu_mutex_lock_iothread ();multifd_save_cleanup (); +qmp_migration_threads_cleanup ();
I think I will spare this one as the mutex is static, so we are not winning any memory back.
migration/migration.c static void *migration_thread(void *opaque) · line 3004
Quoted 5 lines
migration/migration.c +1
-0}trace_migration_thread_after_loop (); +qmp_migration_threads_remove (thread);migration_iteration_finish (s);
I can understand moving it here, but why before migration_iteration_finish?
Quoted 6 lines
object_unref(OBJECT(s)); rcu_unregister_thread();migration/migration.c +1
-1-qmp_migration_threads_remove (thread);return NULL ; } +qmp_migration_threads_remove (thread);
migration/migration.c
Quoted 10 lines
migration/migration.c +1
-1+qemu_mutex_lock (&p->mutex); p->running = false;qemu_mutex_unlock (&p->mutex);rcu_unregister_thread (); -qmp_migration_threads_remove (thread);trace_multifd_send_thread_end (p->id, p->num_packets, p->total_normal_pages);return NULL ;
Here it looks like the right place.
migration/migration.c
Quoted 4 lines
migration/migration.c +3
-0+#include "qemu/osdep.h" +#include "qemu/queue.h" +#include "qemu/lockable.h" #include "threadinfo.h"
Ouch, it missed Markus cleanup. Thanks.
For the rest it looks good.
Later, Juan.
Re: [PATCH 2/3] migration/multifd: Protect accesses to migration_threads
Quoted 8 lines · Juan Quintela
Quoted 7 lines · Fabiano Rosas
This doubly linked list is common for all the multifd and migration threads so we need to avoid concurrent access.
Add a mutex to protect the data from concurrent access. This fixes a crash when removing two MigrationThread objects from the list at the same time during cleanup of multifd threads.
Quoted 9 lines
Quoted 6 lines
To avoid destroying the mutex before the last element has been removed, move calls to
qmp_migration_thread_removeso they run beforemultifd_save_cleanupjoins the threads.Fixes 671326201d ("migration: Introduce interface query-migrationthreads")Signed-off-by Fabiano Rosas <[email protected]>I agree with Peter here. Why don't you have to protect the walking?
Oversight on my part.
Quoted 14 lines
Quoted 14 lines
--- migration/migration.c | 5 ++++- migration/multifd.c | 3 ++- migration/threadinfo.c | 19 ++++++++++++++++++- migration/threadinfo.h | 5 +++-- 4 files changed, 27 insertions(+), 5 deletions(-)migration/migration.c +0
-0@@ -1146,6 +1146,7 @@ static void migrate_fd_cleanup(MigrationState *s)qemu_mutex_lock_iothread ();
Quoted 6 lines
migration/migration.c static void migrate_fd_cleanup(MigrationState *s) · line 1149
Quoted 2 lines
migration/migration.c +1
-0multifd_save_cleanup (); +qmp_migration_threads_cleanup ();I think I will spare this one as the mutex is static, so we are not winning any memory back.
Ok
Quoted 8 lines
migration/migration.c static void *migration_thread(void *opaque) · line 3004
Quoted 5 lines
migration/migration.c +1
-0}trace_migration_thread_after_loop (); +qmp_migration_threads_remove (thread);migration_iteration_finish (s);I can understand moving it here, but why before
migration_iteration_finish?
Because migration_iteration_finish schedules migrate_fd_cleanup and it calls qmp_migration_threads_cleanup. So I wanted to be sure that the removal happens before destroying the mutex.
Quoted 11 lines
Quoted 11 lines
object_unref(OBJECT(s)); rcu_unregister_thread();migration/migration.c +2
-1-qmp_migration_threads_remove (thread);return NULL ; } +qmp_migration_threads_remove (thread); +qemu_mutex_lock (&p->mutex); p->running = false;qemu_mutex_unlock (&p->mutex);
Quoted 8 lines
Quoted 5 lines
rcu_unregister_thread(); - qmp_migration_threads_remove(thread); trace_multifd_send_thread_end(p->id, p->num_packets, p->total_normal_pages); return NULL;Here it looks like the right place.
Yep, we shouldn't really put any new code after that p->running = false. Because multifd_save_cleanup will happily start cleaning up everything while this thread is still running if it sees p->running == false.
Quoted 11 lines
migration/migration.c
Quoted 5 lines
migration/migration.c +3
-0+#include "qemu/osdep.h" +#include "qemu/queue.h" +#include "qemu/lockable.h" #include "threadinfo.h" Ouch, it missed Markus cleanup. Thanks.
For the rest it looks good.
Later, Juan.
Re: [PATCH 2/3] migration/multifd: Protect accesses to migration_threads
Quoted 7 lines · Fabiano Rosas
Quoted 7 lines
Quoted 7 lines
migration/migration.c +0
-0@@ -1146,6 +1146,7 @@ static void migrate_fd_cleanup(MigrationState *s)qemu_mutex_lock_iothread ();
Quoted 8 lines
Quoted 6 lines
migration/migration.c static void migrate_fd_cleanup(MigrationState *s) · line 1149
Quoted 2 lines
migration/migration.c +1
-0multifd_save_cleanup (); +qmp_migration_threads_cleanup ();I think I will spare this one as the mutex is static, so we are not winning any memory back.
Ok
We could consider __attribute__((__constructor__)) in this case.
Signature · 1 line
Peter Xu
Re: [PATCH 2/3] migration/multifd: Protect accesses to migration_threads
Sounds good.
Quoted 25 lines · On Wed, Jun 7, 2023, 18:28 Peter Xu <[email protected]> wrote:
Quoted 5 lines · Fabiano Rosas
Quoted 5 lines
migration/migration.c static void migrate_fd_cleanup(MigrationState
Quoted 5 lines
migration/migration.c +0
-0@@ -1146,6 +1146,7 @@ static void migrate_fd_cleanup(MigrationState*s)
Quoted 10 lines
Quoted 8 lines
migration/migration.c static void migrate_fd_cleanup(MigrationState *s) · line 1149
Quoted 4 lines
migration/migration.c +1
-0qemu_mutex_lock_iothread ();multifd_save_cleanup (); +qmp_migration_threads_cleanup ();I think I will spare this one as the mutex is static, so we are not winning any memory back.
Ok
We could consider __attribute__((__constructor__)) in this case.
--
Peter Xu
PATCH 3/3 tests/qtest: Re-enable multifd cancel test
In reply to Fabiano RosasWe've found the source of flakiness in this test, so re-enable it.
--- tests/qtest/migration-test.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)
tests/qtest/migration-test.c +2 -8
@@ -2778,14 +2778,8 @@ int main(int argc, char **argv) }qtest_add_func ("/migration/multifd/tcp/plain/none" , test_multifd_tcp_none); -/* -* This test is flaky and sometimes fails in CI and otherwise: -* don't run unless user opts in via environment variable. -*/ -if (getenv ("QEMU_TEST_FLAKY_TESTS" )) { -qtest_add_func ("/migration/multifd/tcp/plain/cancel" , - test_multifd_tcp_cancel); - } +qtest_add_func ("/migration/multifd/tcp/plain/cancel" , + test_multifd_tcp_cancel);qtest_add_func ("/migration/multifd/tcp/plain/zlib" , test_multifd_tcp_zlib);#ifdef CONFIG_ZSTD
Signature · 1 line
2.35.3
Re: [PATCH 3/3] tests/qtest: Re-enable multifd cancel test
Quoted 18 lines · Fabiano Rosas
We've found the source of flakiness in this test, so re-enable it.
Signed-off-by Fabiano Rosas <[email protected]>--- tests/qtest/migration-test.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)tests/qtest/migration-test.c +0
-3@@ -2778,14 +2778,8 @@ int main(int argc, char **argv) }qtest_add_func ("/migration/multifd/tcp/plain/none" , test_multifd_tcp_none); -/* -* This test is flaky and sometimes fails in CI and otherwise: -* don't run unless user opts in via environment variable.
tests/qtest/migration-test.c int main(int argc, char **argv) · line 2785
Quoted 10 lines
tests/qtest/migration-test.c +2
-5-*/ -if (getenv ("QEMU_TEST_FLAKY_TESTS" )) { -qtest_add_func ("/migration/multifd/tcp/plain/cancel" , - test_multifd_tcp_cancel); - } +qtest_add_func ("/migration/multifd/tcp/plain/cancel" , + test_multifd_tcp_cancel);qtest_add_func ("/migration/multifd/tcp/plain/zlib" , test_multifd_tcp_zlib);#ifdef CONFIG_ZSTD
There was another failure with migration test that I will post during the rest of the day. It needs both to get it right.
Later, Juan.
Re: [PATCH 3/3] tests/qtest: Re-enable multifd cancel test
Quoted 33 lines · Juan Quintela
Quoted 18 lines · Fabiano Rosas
We've found the source of flakiness in this test, so re-enable it.
Signed-off-by Fabiano Rosas <[email protected]>--- tests/qtest/migration-test.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)tests/qtest/migration-test.c +0
-3@@ -2778,14 +2778,8 @@ int main(int argc, char **argv) }qtest_add_func ("/migration/multifd/tcp/plain/none" , test_multifd_tcp_none); -/* -* This test is flaky and sometimes fails in CI and otherwise: -* don't run unless user opts in via environment variable. tests/qtest/migration-test.c int main(int argc, char **argv) · line 2785
Quoted 10 lines
tests/qtest/migration-test.c +2
-5-*/ -if (getenv ("QEMU_TEST_FLAKY_TESTS" )) { -qtest_add_func ("/migration/multifd/tcp/plain/cancel" , - test_multifd_tcp_cancel); - } +qtest_add_func ("/migration/multifd/tcp/plain/cancel" , + test_multifd_tcp_cancel);qtest_add_func ("/migration/multifd/tcp/plain/zlib" , test_multifd_tcp_zlib);#ifdef CONFIG_ZSTD Reviewed-by Juan Quintela <[email protected]>
Quoted 2 lines
There was another failure with migration test that I will post during the rest of the day. It needs both to get it right.
This one didn't yet land upstream. I'm not sure, but maybe Juan was saying about this change:
commit d2026ee117147893f8d80f060cede6d872ecbd7f
Author: Juan Quintela <[email protected]>
Date: Wed Apr 26 12:20:36 2023 +0200
multifd: Fix the number of channels ready
Fabiano, did you try to reproduce multifd-cancel with current master? I'm wondering whether this test has already been completely fixed, then maybe we can pick up this patch now.
Signature · 1 line
Peter Xu
Fabiano Rosas
Reviewed-by Marc-André LureauRe: [PATCH 3/3] tests/qtest: Re-enable multifd cancel test
Quoted 37 lines · Peter Xu
Quoted 33 lines · Juan Quintela
Quoted 18 lines · Fabiano Rosas
We've found the source of flakiness in this test, so re-enable it.
Signed-off-by Fabiano Rosas <[email protected]>--- tests/qtest/migration-test.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)tests/qtest/migration-test.c +0
-3@@ -2778,14 +2778,8 @@ int main(int argc, char **argv) }qtest_add_func ("/migration/multifd/tcp/plain/none" , test_multifd_tcp_none); -/* -* This test is flaky and sometimes fails in CI and otherwise: -* don't run unless user opts in via environment variable. tests/qtest/migration-test.c int main(int argc, char **argv) · line 2785
Quoted 10 lines
tests/qtest/migration-test.c +2
-5-*/ -if (getenv ("QEMU_TEST_FLAKY_TESTS" )) { -qtest_add_func ("/migration/multifd/tcp/plain/cancel" , - test_multifd_tcp_cancel); - } +qtest_add_func ("/migration/multifd/tcp/plain/cancel" , + test_multifd_tcp_cancel);qtest_add_func ("/migration/multifd/tcp/plain/zlib" , test_multifd_tcp_zlib);#ifdef CONFIG_ZSTD Reviewed-by Juan Quintela <[email protected]>Quoted 2 lines
There was another failure with migration test that I will post during the rest of the day. It needs both to get it right.
Quoted 8 lines
This one didn't yet land upstream. I'm not sure, but maybe Juan was saying about this change:
commit
d2026ee117147893f8d80f060cede6d872ecbd7f
Author: Juan Quintela <[email protected]>
Date: Wed Apr 26 12:20:36 2023 +0200multifd: Fix the number of channels ready
That's not it. It was something in the test itself around the fact that we use two sets of: from/to. There was supposed to be a situation where we'd start 'to2' while 'to' was still running and that would cause issues (possibly with sockets).
I think what might have happened is that someone merged a fix through another tree and Juan didn't notice. I think this is the one:
commit f2d063e61ee2026700ab44bef967f663e976bec8
Author: Xuzhou Cheng <[email protected]>
Date: Fri Oct 28 12:57:32 2022 +0800
tests/qtest: migration-test: Make sure QEMU process "to" exited after migration is canceled
Make sure QEMU process "to" exited before launching another target for migration in the test_multifd_tcp_cancel case.
Message-Id: <[email protected]>
Quoted 3 lines
Fabiano, did you try to reproduce multifd-cancel with current master? I'm wondering whether this test has already been completely fixed, then maybe we can pick up this patch now.
Yes, let's merge it. I have kept it enabled during testing of all of the recent race conditions we've debugged and haven't seen it fail. Current master also looks fine.
Re: [PATCH 3/3] tests/qtest: Re-enable multifd cancel test
Quoted 62 lines · Fabiano Rosas
Quoted 37 lines · Peter Xu
Quoted 33 lines · Juan Quintela
Quoted 18 lines · Fabiano Rosas
We've found the source of flakiness in this test, so re-enable it.
Signed-off-by Fabiano Rosas <[email protected]>--- tests/qtest/migration-test.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)tests/qtest/migration-test.c +0
-3@@ -2778,14 +2778,8 @@ int main(int argc, char **argv) }qtest_add_func ("/migration/multifd/tcp/plain/none" , test_multifd_tcp_none); -/* -* This test is flaky and sometimes fails in CI and otherwise: -* don't run unless user opts in via environment variable. tests/qtest/migration-test.c int main(int argc, char **argv) · line 2785
Quoted 10 lines
tests/qtest/migration-test.c +2
-5-*/ -if (getenv ("QEMU_TEST_FLAKY_TESTS" )) { -qtest_add_func ("/migration/multifd/tcp/plain/cancel" , - test_multifd_tcp_cancel); - } +qtest_add_func ("/migration/multifd/tcp/plain/cancel" , + test_multifd_tcp_cancel);qtest_add_func ("/migration/multifd/tcp/plain/zlib" , test_multifd_tcp_zlib);#ifdef CONFIG_ZSTD Reviewed-by Juan Quintela <[email protected]>Quoted 2 lines
There was another failure with migration test that I will post during the rest of the day. It needs both to get it right.
Quoted 8 lines
This one didn't yet land upstream. I'm not sure, but maybe Juan was saying about this change:
commit
d2026ee117147893f8d80f060cede6d872ecbd7f
Author: Juan Quintela <[email protected]>
Date: Wed Apr 26 12:20:36 2023 +0200multifd: Fix the number of channels ready
That's not it. It was something in the test itself around the fact that we use two sets of: from/to. There was supposed to be a situation where we'd start 'to2' while 'to' was still running and that would cause issues (possibly with sockets).
I think what might have happened is that someone merged a fix through another tree and Juan didn't notice. I think this is the one:
commit
f2d063e61ee2026700ab44bef967f663e976bec8
Author: Xuzhou Cheng <[email protected]>
Date: Fri Oct 28 12:57:32 2022 +0800tests/qtest: migration-test: Make sure QEMU process "to" exited after migration is canceled
Quoted 8 lines
Make sure QEMU process "to" exited before launching another target for migration in the
test_multifd_tcp_cancelcase.Signed-off-by Xuzhou Cheng <[email protected]>Signed-off-by Bin Meng <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Message-Id: <[email protected]>
Signed-off-by Thomas Huth <[email protected]>
Hmm, i see.
Quoted 8 lines
Quoted 4 lines
Fabiano, did you try to reproduce multifd-cancel with current master? I'm wondering whether this test has already been completely fixed, then maybe we can pick up this patch now.
Yes, let's merge it. I have kept it enabled during testing of all of the recent race conditions we've debugged and haven't seen it fail. Current master also looks fine.
It needs a trivial touchup, but then I queued it.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 3/3] tests/qtest: Re-enable multifd cancel test
Quoted 66 lines · Peter Xu
Quoted 65 lines · Fabiano Rosas
Quoted 37 lines · Peter Xu
Quoted 33 lines · Juan Quintela
Quoted 18 lines · Fabiano Rosas
We've found the source of flakiness in this test, so re-enable it.
Signed-off-by Fabiano Rosas <[email protected]>--- tests/qtest/migration-test.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)tests/qtest/migration-test.c +0
-3@@ -2778,14 +2778,8 @@ int main(int argc, char **argv) }qtest_add_func ("/migration/multifd/tcp/plain/none" , test_multifd_tcp_none); -/* -* This test is flaky and sometimes fails in CI and otherwise: -* don't run unless user opts in via environment variable. tests/qtest/migration-test.c int main(int argc, char **argv) · line 2785
Quoted 10 lines
tests/qtest/migration-test.c +2
-5-*/ -if (getenv ("QEMU_TEST_FLAKY_TESTS" )) { -qtest_add_func ("/migration/multifd/tcp/plain/cancel" , - test_multifd_tcp_cancel); - } +qtest_add_func ("/migration/multifd/tcp/plain/cancel" , + test_multifd_tcp_cancel);qtest_add_func ("/migration/multifd/tcp/plain/zlib" , test_multifd_tcp_zlib);#ifdef CONFIG_ZSTD Reviewed-by Juan Quintela <[email protected]>Quoted 2 lines
There was another failure with migration test that I will post during the rest of the day. It needs both to get it right.
Quoted 8 lines
This one didn't yet land upstream. I'm not sure, but maybe Juan was saying about this change:
commit
d2026ee117147893f8d80f060cede6d872ecbd7f
Author: Juan Quintela <[email protected]>
Date: Wed Apr 26 12:20:36 2023 +0200multifd: Fix the number of channels ready
That's not it. It was something in the test itself around the fact that we use two sets of: from/to. There was supposed to be a situation where we'd start 'to2' while 'to' was still running and that would cause issues (possibly with sockets).
I think what might have happened is that someone merged a fix through another tree and Juan didn't notice. I think this is the one:
commit
f2d063e61ee2026700ab44bef967f663e976bec8
Author: Xuzhou Cheng <[email protected]>
Date: Fri Oct 28 12:57:32 2022 +0800tests/qtest: migration-test: Make sure QEMU process "to" exited after migration is canceled
Make sure QEMU process "to" exited before launching another target for migration in the
test_multifd_tcp_cancelcase.
Quoted 7 lines
Quoted 5 lines
Signed-off-by Xuzhou Cheng <[email protected]>Signed-off-by Bin Meng <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Message-Id: <[email protected]>
Signed-off-by Thomas Huth <[email protected]>Hmm, i see.
Sorry for that :-( Maybe it's better if we remove the migration-test from the qtest section in MAINTAINERS? Since the migration test is very well maintained already, there's IMHO no need for picking up the patches via the qtest tree, so something like this should prevent these problems:
MAINTAINERS +1 -0
@@ -3269,6 +3269,7 @@ F: tests/qtest/ F: docs/devel/qgraph.rst F: docs/devel/qtest.rst X: tests/qtest/bios-tables-test* +X: tests/qtest/migration-* Device Fuzzing M: Alexander Bulekov <[email protected]>
(as you can see, we're doing it in a similar way for the bios tables test already)
If you agree, I can send out a proper patch for this later today.
Thomas
Re: [PATCH 3/3] tests/qtest: Re-enable multifd cancel test
Hi, Thomas,
Quoted 1 line · Thomas Huth
Sorry for that :-(
Not at all! I actually appreciate more people looking after it.
Quoted 17 lines
Maybe it's better if we remove the migration-test from
the qtest section inMAINTAINERS? Since the migration test is very well maintained already, there's IMHO no need for picking up the patches via the qtest tree, so something like this should prevent these problems:MAINTAINERS +1
-0@@ -3269,6 +3269,7 @@ F: tests/qtest/ F: docs/devel/qgraph.rst F: docs/devel/qtest.rst X: tests/qtest/bios-tables-test* +X: tests/qtest/migration-* Device Fuzzing M: Alexander Bulekov <[email protected]>
Quoted 4 lines
(as you can see, we're doing it in a similar way for the bios tables test already)
If you agree, I can send out a proper patch for this later today.
Currently the file is covered by both groups of people, which is the best condition to me:
$ ./scripts/get_maintainer.pl -f tests/qtest/migration-test.c Peter Xu <[email protected]> (maintainer:Migration) Fabiano Rosas <[email protected]> (maintainer:Migration) Thomas Huth <[email protected]> (maintainer:qtest) Laurent Vivier <[email protected]> (maintainer:qtest) Paolo Bonzini <[email protected]> (reviewer:qtest) [email protected] (open list:All patches CC here)
It makes sense to me e.g. when qtest reworks the framework, and we'd like migration-test.c to be covered in that same reworks series and reviewed/pulled together, for example, then those can go via qtest's tree directly.
If patch submitter follows the MAINTAINERS file it means all of us will be in the loop and that's the perfect condition, IMHO. It's just that this patch didn't have any migration people copied, which caused a very slight confusion.
It'll be great in that case if qtest maintainers can help submitters to copy us if the submitters forgot to do so. I think we should do the same when there's major changes for qtest framework for a new migration test. Would that work the best for us?
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 3/3] tests/qtest: Re-enable multifd cancel test
Quoted 50 lines · Peter Xu
Hi, Thomas,
Quoted 1 line · Thomas Huth
Sorry for that :-(
Not at all! I actually appreciate more people looking after it.
Quoted 17 lines
Maybe it's better if we remove the migration-test from
the qtest section inMAINTAINERS? Since the migration test is very well maintained already, there's IMHO no need for picking up the patches via the qtest tree, so something like this should prevent these problems:MAINTAINERS +1
-0@@ -3269,6 +3269,7 @@ F: tests/qtest/ F: docs/devel/qgraph.rst F: docs/devel/qtest.rst X: tests/qtest/bios-tables-test* +X: tests/qtest/migration-* Device Fuzzing M: Alexander Bulekov <[email protected]>Quoted 4 lines
(as you can see, we're doing it in a similar way for the bios tables test already)
If you agree, I can send out a proper patch for this later today.
Currently the file is covered by both groups of people, which is the best condition to me:
$ ./scripts/get_maintainer.pl -f tests/qtest/migration-test.c Peter Xu <[email protected]> (maintainer:Migration) Fabiano Rosas <[email protected]> (maintainer:Migration) Thomas Huth <[email protected]> (maintainer:qtest) Laurent Vivier <[email protected]> (maintainer:qtest) Paolo Bonzini <[email protected]> (reviewer:qtest) [email protected] (open list:All patches CC here)It makes sense to me e.g. when qtest reworks the framework, and we'd like
migration-test.cto be covered in that same reworks series and reviewed/pulled together, for example, then those can go via qtest's tree directly.If patch submitter follows the
MAINTAINERSfile it means all of us will be in the loop and that's the perfect condition, IMHO. It's just that this patch didn't have any migration people copied, which caused a very slight confusion.
Quoted 4 lines
It'll be great in that case if qtest maintainers can help submitters to copy us if the submitters forgot to do so. I think we should do the same when there's major changes for qtest framework for a new migration test. Would that work the best for us?
Ok, makes sense, let's try it that way!
Thomas