migration test tweeks
cover migration test tweeks
From: "Dr. David Alan Gilbert" <[email protected]>
Alex noticed that some of the postcopy tests would occasionally hang; this series adds some checks to make them more likely to assert than hang in some failure cases, and changes the migration bandwidth so that under load it's much more likely to manage to land in postcopy.
Dr. David Alan Gilbert (2):
tests/migration: Fail on unexpected migration states tests/migration/postcopy: trim migration bandwidth
tests/migration-test.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/2 tests/migration: Fail on unexpected migration states
From: "Dr. David Alan Gilbert" <[email protected]>
We've got various places where we wait for a migration to enter a given state; but if we enter an unexpected state we tend to fail in odd ways; add a mechanism for explicitly testing for any state which we shouldn't be in.
--- tests/migration-test.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
tests/migration-test.c +17 -6
@@ -255,15 +255,19 @@ static void read_blocktime(QTestState *who) }static void wait_for_migration_status (QTestState *who, -const char *goal) +const char *goal, +const char **ungoals) {while (true) {bool completed;char *status; +const char **ungoal; status =migrate_query_status (who); completed =strcmp (status, goal) ==0 ; - g_assert_cmpstr(status, !=, "failed"); +for (ungoal = ungoals; *ungoal; ungoal++) { + g_assert_cmpstr(status, !=, *ungoal); + }g_free (status);if (completed) {return ;
@@ -274,7 +278,8 @@ static void wait_for_migration_status(QTestState *who,static void wait_for_migration_complete (QTestState *who) { -wait_for_migration_status (who,"completed" ); +wait_for_migration_status (who,"completed" , + (const char * []) {"failed" ,NULL }); }static void wait_for_migration_pass (QTestState *who)
@@ -809,7 +814,9 @@ static void test_postcopy_recovery(void)* Wait until postcopy is really started; we can only run the * migrate-pause command during a postcopy */ -wait_for_migration_status (from,"postcopy-active" ); +wait_for_migration_status (from,"postcopy-active" , + (const char * []) {"failed" , +"completed" ,NULL });/* * Manually stop the postcopy migration. This emulates a network
@@ -822,7 +829,9 @@ static void test_postcopy_recovery(void)* migrate-recover command can only succeed if destination machine * is in the paused state */ -wait_for_migration_status (to,"postcopy-paused" ); +wait_for_migration_status (to,"postcopy-paused" , + (const char * []) {"failed" ,"active" , +"completed" ,NULL });/* * Create a new socket to emulate a new channel that is different
@@ -836,7 +845,9 @@ static void test_postcopy_recovery(void)* Try to rebuild the migration channel using the resume flag and * the newly created channel */ -wait_for_migration_status (from,"postcopy-paused" ); +wait_for_migration_status (from,"postcopy-paused" , + (const char * []) {"failed" ,"active" , +"completed" ,NULL });migrate (from, uri,"{'resume': true}" );g_free (uri);
Signature · 1 line
2.21.0
Re: [PATCH 1/2] tests/migration: Fail on unexpected migration states
Quoted 85 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
We've got various places where we wait for a migration to enter a given state; but if we enter an unexpected state we tend to fail in odd ways; add a mechanism for explicitly testing for any state which we shouldn't be in.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- tests/migration-test.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)tests/migration-test.c +17
-6@@ -255,15 +255,19 @@ static void read_blocktime(QTestState *who) }static void wait_for_migration_status (QTestState *who, -const char *goal) +const char *goal, +const char **ungoals) {while (true) {bool completed;char *status; +const char **ungoal; status =migrate_query_status (who); completed =strcmp (status, goal) ==0 ; - g_assert_cmpstr(status, !=, "failed"); +for (ungoal = ungoals; *ungoal; ungoal++) { + g_assert_cmpstr(status, !=, *ungoal); + }g_free (status);if (completed) {return ;@@ -274,7 +278,8 @@ static void wait_for_migration_status(QTestState *who,static void wait_for_migration_complete (QTestState *who) { -wait_for_migration_status (who,"completed" ); +wait_for_migration_status (who,"completed" , + (const char * []) {"failed" ,NULL }); }static void wait_for_migration_pass (QTestState *who)@@ -809,7 +814,9 @@ static void test_postcopy_recovery(void)* Wait until postcopy is really started; we can only run the * migrate-pause command during a postcopy */ -wait_for_migration_status (from,"postcopy-active" ); +wait_for_migration_status (from,"postcopy-active" , + (const char * []) {"failed" , +"completed" ,NULL });/* * Manually stop the postcopy migration. This emulates a network @@ -822,7 +829,9 @@ static void test_postcopy_recovery(void)* migrate-recover command can only succeed if destination machine * is in the paused state */ -wait_for_migration_status (to,"postcopy-paused" ); +wait_for_migration_status (to,"postcopy-paused" , + (const char * []) {"failed" ,"active" , +"completed" ,NULL });/* * Create a new socket to emulate a new channel that is different @@ -836,7 +845,9 @@ static void test_postcopy_recovery(void)* Try to rebuild the migration channel using the resume flag and * the newly created channel */ -wait_for_migration_status (from,"postcopy-paused" ); +wait_for_migration_status (from,"postcopy-paused" , + (const char * []) {"failed" ,"active" , +"completed" ,NULL });migrate (from, uri,"{'resume': true}" );g_free (uri);Signature · 1 line
2.21.0
LGTM (tested with x86_64 and aarch64 targets on an x86_64 host).
Re: [PATCH 1/2] tests/migration: Fail on unexpected migration states
In reply to Dr. David Alan Gilbert (git)Quoted 25 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
We've got various places where we wait for a migration to enter a given state; but if we enter an unexpected state we tend to fail in odd ways; add a mechanism for explicitly testing for any state which we shouldn't be in.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- tests/migration-test.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)tests/migration-test.c +2
-1@@ -255,15 +255,19 @@ static void read_blocktime(QTestState *who) }static void wait_for_migration_status (QTestState *who, -const char *goal) +const char *goal, +const char **ungoals) {while (true) {
tests/migration-test.c static void read_blocktime(QTestState *who) · line 270
Quoted 10 lines
tests/migration-test.c +4
-1bool completed;char *status; +const char **ungoal; status =migrate_query_status (who); completed =strcmp (status, goal) ==0 ; - g_assert_cmpstr(status, !=, "failed"); +for (ungoal = ungoals; *ungoal; ungoal++) { + g_assert_cmpstr(status, !=, *ungoal); + }
You could use:
g_assert(!g_strv_contains(ungoals, status))
if you wanted to be more gliby. But anyway:
Quoted 36 lines
tests/migration-test.c +8
-3g_free (status);if (completed) {return ;@@ -274,7 +278,8 @@ static void wait_for_migration_status(QTestState *who,static void wait_for_migration_complete (QTestState *who) { -wait_for_migration_status (who,"completed" ); +wait_for_migration_status (who,"completed" , + (const char * []) {"failed" ,NULL }); }static void wait_for_migration_pass (QTestState *who)@@ -809,7 +814,9 @@ static void test_postcopy_recovery(void)* Wait until postcopy is really started; we can only run the * migrate-pause command during a postcopy */ -wait_for_migration_status (from,"postcopy-active" ); +wait_for_migration_status (from,"postcopy-active" , + (const char * []) {"failed" , +"completed" ,NULL });/* * Manually stop the postcopy migration. This emulates a network @@ -822,7 +829,9 @@ static void test_postcopy_recovery(void)* migrate-recover command can only succeed if destination machine * is in the paused state */ -wait_for_migration_status (to,"postcopy-paused" ); +wait_for_migration_status (to,"postcopy-paused" , + (const char * []) {"failed" ,"active" , +"completed" ,NULL });/* * Create a new socket to emulate a new channel that is different
tests/migration-test.c static void test_postcopy_recovery(void) · line 852
Quoted 9 lines
tests/migration-test.c +3
-1@@ -836,7 +845,9 @@ static void test_postcopy_recovery(void)* Try to rebuild the migration channel using the resume flag and * the newly created channel */ -wait_for_migration_status (from,"postcopy-paused" ); +wait_for_migration_status (from,"postcopy-paused" , + (const char * []) {"failed" ,"active" , +"completed" ,NULL });migrate (from, uri,"{'resume': true}" );g_free (uri);
--
Alex Bennée
Re: [PATCH 1/2] tests/migration: Fail on unexpected migration states
In reply to Dr. David Alan Gilbert (git)Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
We've got various places where we wait for a migration to enter a given state; but if we enter an unexpected state we tend to fail in odd ways; add a mechanism for explicitly testing for any state which we shouldn't be in.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
tests/migration-test.c static void test_postcopy_recovery(void) · line 819
Quoted 5 lines
tests/migration-test.c +3
-1-wait_for_migration_status (from,"postcopy-active" ); +wait_for_migration_status (from,"postcopy-active" , + (const char * []) {"failed" , +"completed" ,NULL });
I really preffer not to use that kind of casts.
const char *ungoals[] = { "failed", "completed", NULL };
but as it is you who are doing the patches, I will not complain O:-)
PATCH 2/2 tests/migration/postcopy: trim migration bandwidth
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
On slow hosts with tcg we were sometimes finding that the migration would complete during precopy and never get into the postcopy test. Trim back the bandwidth a bit to make that much less likely.
--- tests/migration-test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/migration-test.c +1 -1
@@ -753,7 +753,7 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,* quickly, but that it doesn't complete precopy even on a slow * machine, so also set the downtime. */ -migrate_set_parameter_int (from,"max-bandwidth" ,100000000 ); +migrate_set_parameter_int (from,"max-bandwidth" ,30000000 );migrate_set_parameter_int (from,"downtime-limit" ,1 );/* Wait for the first serial output from the source */
Signature · 1 line
2.21.0
Re: [PATCH 2/2] tests/migration/postcopy: trim migration bandwidth
Quoted 28 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
On slow hosts with tcg we were sometimes finding that the migration would complete during precopy and never get into the postcopy test. Trim back the bandwidth a bit to make that much less likely.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- tests/migration-test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/migration-test.c +1
-1@@ -753,7 +753,7 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,* quickly, but that it doesn't complete precopy even on a slow * machine, so also set the downtime. */ -migrate_set_parameter_int (from,"max-bandwidth" ,100000000 ); +migrate_set_parameter_int (from,"max-bandwidth" ,30000000 );migrate_set_parameter_int (from,"downtime-limit" ,1 );/* Wait for the first serial output from the source */ Signature · 1 line
2.21.0
Re: [PATCH 2/2] tests/migration/postcopy: trim migration bandwidth
In reply to Dr. David Alan Gilbert (git)Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
On slow hosts with tcg we were sometimes finding that the migration would complete during precopy and never get into the postcopy test. Trim back the bandwidth a bit to make that much less likely.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Quoted 9 lines
--- tests/migration-test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/migration-test.c +0
-0
tests/migration-test.c static int migrate_postcopy_prepare(QTestState **from_ptr, · line 759
Quoted 8 lines
tests/migration-test.c +1
-1@@ -753,7 +753,7 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,* quickly, but that it doesn't complete precopy even on a slow * machine, so also set the downtime. */ -migrate_set_parameter_int (from,"max-bandwidth" ,100000000 ); +migrate_set_parameter_int (from,"max-bandwidth" ,30000000 );migrate_set_parameter_int (from,"downtime-limit" ,1 );/* Wait for the first serial output from the source */
--
Alex Bennée
Re: [PATCH 2/2] tests/migration/postcopy: trim migration bandwidth
In reply to Dr. David Alan Gilbert (git)Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
On slow hosts with tcg we were sometimes finding that the migration would complete during precopy and never get into the postcopy test. Trim back the bandwidth a bit to make that much less likely.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Alex noticed that some of the postcopy tests would occasionally hang; this series adds some checks to make them more likely to assert than hang in some failure cases, and changes the migration bandwidth so that under load it's much more likely to manage to land in postcopy.
Dr. David Alan Gilbert (2):
tests/migration: Fail on unexpected migration states tests/migration/postcopy: trim migration bandwidth
I might be even more aggresive on patch 2 when turning down the bandwidth. :)
Another thing I thought about on the hang issue is that maybe we can give a timeout for the waits and when the timeout triggers before a directly assert in the test case we send sigabrt to QEMU (just like what kill_qemu does) then we could have a chance to see the cores. Not sure whether that could help, though.
Regards,
Signature · 1 line
Peter Xu
Quoted 15 lines · * Peter Xu ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Alex noticed that some of the postcopy tests would occasionally hang; this series adds some checks to make them more likely to assert than hang in some failure cases, and changes the migration bandwidth so that under load it's much more likely to manage to land in postcopy.
Dr. David Alan Gilbert (2):
tests/migration: Fail on unexpected migration states tests/migration/postcopy: trim migration bandwidthReviewed-by Peter Xu <[email protected]>
Quoted 8 lines
I might be even more aggresive on patch 2 when turning down the bandwidth. :)
Another thing I thought about on the hang issue is that maybe we can give a timeout for the waits and when the timeout triggers before a directly assert in the test case we send sigabrt to QEMU (just like what
kill_qemudoes) then we could have a chance to see the cores. Not sure whether that could help, though.
Yes I might need to add that; I can see there's another hang somewhere even with these two patches.
(I *think* it's at the end while it's waiting for completion after resumption - but I've not figured out what's going on yet).
Dave
Quoted 4 lines
Regards,
Signature · 1 line
Peter Xu
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 7 lines · * Dr. David Alan Gilbert (git) ([email protected]) wrote:
From: "Dr. David Alan Gilbert" <[email protected]>
Alex noticed that some of the postcopy tests would occasionally hang; this series adds some checks to make them more likely to assert than hang in some failure cases, and changes the migration bandwidth so that under load it's much more likely to manage to land in postcopy.
Queued
Quoted 12 lines
Dr. David Alan Gilbert (2):
tests/migration: Fail on unexpected migration states tests/migration/postcopy: trim migration bandwidthtests/migration-test.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-)Signature · 1 line
2.21.0
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK