Skip to contentSkip to conversation

migration test tweeks

PATCH In review Dr. David Alan Gilbert (git) 5 participants 12 messages Sep 23, 2019 – Sep 25, 2019

lore
Share & tools
lore
#0 lore raw

cover migration test tweeks

From: "Dr. David Alan Gilbert" <[email protected]>

Alex noticed that some of the postcopy tests would occasionally hang; this series adds some checks to make them more likely to assert than hang in some failure cases, and changes the migration bandwidth so that under load it's much more likely to manage to land in postcopy.

Dr. David Alan Gilbert (2):
tests/migration: Fail on unexpected migration states tests/migration/postcopy: trim migration bandwidth

 tests/migration-test.c | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

PATCH 1/2 tests/migration: Fail on unexpected migration states

From: "Dr. David Alan Gilbert" <[email protected]>

We've got various places where we wait for a migration to enter a given state; but if we enter an unexpected state we tend to fail in odd ways; add a mechanism for explicitly testing for any state which we shouldn't be in.

---
 tests/migration-test.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
tests/migration-test.c +17 -6
index 258aa064d4..9c62ee5331 100644
@@ -255,15 +255,19 @@ static void read_blocktime(QTestState *who)
 }
 
 static void wait_for_migration_status(QTestState *who,
-                                      const char *goal)
+                                      const char *goal,
+                                      const char **ungoals)
 {
     while (true) {
         bool completed;
         char *status;
+        const char **ungoal;
 
         status = migrate_query_status(who);
         completed = strcmp(status, goal) == 0;
-        g_assert_cmpstr(status, !=,  "failed");
+        for (ungoal = ungoals; *ungoal; ungoal++) {
+            g_assert_cmpstr(status, !=,  *ungoal);
+        }
         g_free(status);
         if (completed) {
             return;
@@ -274,7 +278,8 @@ static void wait_for_migration_status(QTestState *who,
 
 static void wait_for_migration_complete(QTestState *who)
 {
-    wait_for_migration_status(who, "completed");
+    wait_for_migration_status(who, "completed",
+                              (const char * []) { "failed", NULL });
 }
 
 static void wait_for_migration_pass(QTestState *who)
@@ -809,7 +814,9 @@ static void test_postcopy_recovery(void)
      * Wait until postcopy is really started; we can only run the
      * migrate-pause command during a postcopy
      */
-    wait_for_migration_status(from, "postcopy-active");
+    wait_for_migration_status(from, "postcopy-active",
+                              (const char * []) { "failed",
+                                                  "completed", NULL });
 
     /*
      * Manually stop the postcopy migration. This emulates a network
@@ -822,7 +829,9 @@ static void test_postcopy_recovery(void)
      * migrate-recover command can only succeed if destination machine
      * is in the paused state
      */
-    wait_for_migration_status(to, "postcopy-paused");
+    wait_for_migration_status(to, "postcopy-paused",
+                              (const char * []) { "failed", "active",
+                                                  "completed", NULL });
 
     /*
      * Create a new socket to emulate a new channel that is different
@@ -836,7 +845,9 @@ static void test_postcopy_recovery(void)
      * Try to rebuild the migration channel using the resume flag and
      * the newly created channel
      */
-    wait_for_migration_status(from, "postcopy-paused");
+    wait_for_migration_status(from, "postcopy-paused",
+                              (const char * []) { "failed", "active",
+                                                  "completed", NULL });
     migrate(from, uri, "{'resume': true}");
     g_free(uri);
 
Signature · 1 line
2.21.0
Reviewed-by Tested-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] tests/migration: Fail on unexpected migration states

Quoted 85 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

We've got various places where we wait for a migration to enter a given state; but if we enter an unexpected state we tend to fail in odd ways; add a mechanism for explicitly testing for any state which we shouldn't be in.

---
 tests/migration-test.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
tests/migration-test.c +17 -6
index 258aa064d4..9c62ee5331 100644
@@ -255,15 +255,19 @@ static void read_blocktime(QTestState *who)
 }
 
 static void wait_for_migration_status(QTestState *who,
-                                      const char *goal)
+                                      const char *goal,
+                                      const char **ungoals)
 {
     while (true) {
         bool completed;
         char *status;
+        const char **ungoal;
 
         status = migrate_query_status(who);
         completed = strcmp(status, goal) == 0;
-        g_assert_cmpstr(status, !=,  "failed");
+        for (ungoal = ungoals; *ungoal; ungoal++) {
+            g_assert_cmpstr(status, !=,  *ungoal);
+        }
         g_free(status);
         if (completed) {
             return;
@@ -274,7 +278,8 @@ static void wait_for_migration_status(QTestState *who,
 
 static void wait_for_migration_complete(QTestState *who)
 {
-    wait_for_migration_status(who, "completed");
+    wait_for_migration_status(who, "completed",
+                              (const char * []) { "failed", NULL });
 }
 
 static void wait_for_migration_pass(QTestState *who)
@@ -809,7 +814,9 @@ static void test_postcopy_recovery(void)
      * Wait until postcopy is really started; we can only run the
      * migrate-pause command during a postcopy
      */
-    wait_for_migration_status(from, "postcopy-active");
+    wait_for_migration_status(from, "postcopy-active",
+                              (const char * []) { "failed",
+                                                  "completed", NULL });
 
     /*
      * Manually stop the postcopy migration. This emulates a network
@@ -822,7 +829,9 @@ static void test_postcopy_recovery(void)
      * migrate-recover command can only succeed if destination machine
      * is in the paused state
      */
-    wait_for_migration_status(to, "postcopy-paused");
+    wait_for_migration_status(to, "postcopy-paused",
+                              (const char * []) { "failed", "active",
+                                                  "completed", NULL });
 
     /*
      * Create a new socket to emulate a new channel that is different
@@ -836,7 +845,9 @@ static void test_postcopy_recovery(void)
      * Try to rebuild the migration channel using the resume flag and
      * the newly created channel
      */
-    wait_for_migration_status(from, "postcopy-paused");
+    wait_for_migration_status(from, "postcopy-paused",
+                              (const char * []) { "failed", "active",
+                                                  "completed", NULL });
     migrate(from, uri, "{'resume': true}");
     g_free(uri);
 
Signature · 1 line
2.21.0

LGTM (tested with x86_64 and aarch64 targets on an x86_64 host).

Reviewed-by on v1 1/2
#3 lore raw

Re: [PATCH 1/2] tests/migration: Fail on unexpected migration states

In reply to Dr. David Alan Gilbert (git)
Quoted 25 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

We've got various places where we wait for a migration to enter a given state; but if we enter an unexpected state we tend to fail in odd ways; add a mechanism for explicitly testing for any state which we shouldn't be in.

---
 tests/migration-test.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
tests/migration-test.c +2 -1
index 258aa064d4..9c62ee5331 100644
@@ -255,15 +255,19 @@ static void read_blocktime(QTestState *who)
 }
 
 static void wait_for_migration_status(QTestState *who,
-                                      const char *goal)
+                                      const char *goal,
+                                      const char **ungoals)
 {
     while (true) {

tests/migration-test.c static void read_blocktime(QTestState *who) · line 270

Quoted 10 lines
tests/migration-test.c +4 -1
         bool completed;
         char *status;
+        const char **ungoal;
 
         status = migrate_query_status(who);
         completed = strcmp(status, goal) == 0;
-        g_assert_cmpstr(status, !=,  "failed");
+        for (ungoal = ungoals; *ungoal; ungoal++) {
+            g_assert_cmpstr(status, !=,  *ungoal);
+        }

You could use:

g_assert(!g_strv_contains(ungoals, status))

if you wanted to be more gliby. But anyway:

Quoted 36 lines
tests/migration-test.c +8 -3
         g_free(status);
         if (completed) {
             return;
@@ -274,7 +278,8 @@ static void wait_for_migration_status(QTestState *who,
 
 static void wait_for_migration_complete(QTestState *who)
 {
-    wait_for_migration_status(who, "completed");
+    wait_for_migration_status(who, "completed",
+                              (const char * []) { "failed", NULL });
 }
 
 static void wait_for_migration_pass(QTestState *who)
@@ -809,7 +814,9 @@ static void test_postcopy_recovery(void)
      * Wait until postcopy is really started; we can only run the
      * migrate-pause command during a postcopy
      */
-    wait_for_migration_status(from, "postcopy-active");
+    wait_for_migration_status(from, "postcopy-active",
+                              (const char * []) { "failed",
+                                                  "completed", NULL });
 
     /*
      * Manually stop the postcopy migration. This emulates a network
@@ -822,7 +829,9 @@ static void test_postcopy_recovery(void)
      * migrate-recover command can only succeed if destination machine
      * is in the paused state
      */
-    wait_for_migration_status(to, "postcopy-paused");
+    wait_for_migration_status(to, "postcopy-paused",
+                              (const char * []) { "failed", "active",
+                                                  "completed", NULL });
 
     /*
      * Create a new socket to emulate a new channel that is different

tests/migration-test.c static void test_postcopy_recovery(void) · line 852

Quoted 9 lines
tests/migration-test.c +3 -1
@@ -836,7 +845,9 @@ static void test_postcopy_recovery(void)
      * Try to rebuild the migration channel using the resume flag and
      * the newly created channel
      */
-    wait_for_migration_status(from, "postcopy-paused");
+    wait_for_migration_status(from, "postcopy-paused",
+                              (const char * []) { "failed", "active",
+                                                  "completed", NULL });
     migrate(from, uri, "{'resume': true}");
     g_free(uri);

--
Alex Bennée

Reviewed-by on v1 1/2
#4 lore raw

Re: [PATCH 1/2] tests/migration: Fail on unexpected migration states

In reply to Dr. David Alan Gilbert (git)
Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

We've got various places where we wait for a migration to enter a given state; but if we enter an unexpected state we tend to fail in odd ways; add a mechanism for explicitly testing for any state which we shouldn't be in.

tests/migration-test.c static void test_postcopy_recovery(void) · line 819

Quoted 5 lines
tests/migration-test.c +3 -1
-    wait_for_migration_status(from, "postcopy-active");
+    wait_for_migration_status(from, "postcopy-active",
+                              (const char * []) { "failed",
+                                                  "completed", NULL });

I really preffer not to use that kind of casts.

const char *ungoals[] = { "failed", "completed", NULL };

but as it is you who are doing the patches, I will not complain O:-)

#5 lore raw

PATCH 2/2 tests/migration/postcopy: trim migration bandwidth

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

On slow hosts with tcg we were sometimes finding that the migration would complete during precopy and never get into the postcopy test. Trim back the bandwidth a bit to make that much less likely.

---
 tests/migration-test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/migration-test.c +1 -1
index 9c62ee5331..221a33d083 100644
@@ -753,7 +753,7 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
      * quickly, but that it doesn't complete precopy even on a slow
      * machine, so also set the downtime.
      */
-    migrate_set_parameter_int(from, "max-bandwidth", 100000000);
+    migrate_set_parameter_int(from, "max-bandwidth", 30000000);
     migrate_set_parameter_int(from, "downtime-limit", 1);
 
     /* Wait for the first serial output from the source */
Signature · 1 line
2.21.0
Reviewed-by Tested-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] tests/migration/postcopy: trim migration bandwidth

Quoted 28 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

On slow hosts with tcg we were sometimes finding that the migration would complete during precopy and never get into the postcopy test. Trim back the bandwidth a bit to make that much less likely.

---
 tests/migration-test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/migration-test.c +1 -1
index 9c62ee5331..221a33d083 100644
@@ -753,7 +753,7 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
      * quickly, but that it doesn't complete precopy even on a slow
      * machine, so also set the downtime.
      */
-    migrate_set_parameter_int(from, "max-bandwidth", 100000000);
+    migrate_set_parameter_int(from, "max-bandwidth", 30000000);
     migrate_set_parameter_int(from, "downtime-limit", 1);
 
     /* Wait for the first serial output from the source */
Signature · 1 line
2.21.0
Acked-by Tested-by on v1 2/2
#7 lore raw

Re: [PATCH 2/2] tests/migration/postcopy: trim migration bandwidth

In reply to Dr. David Alan Gilbert (git)
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

On slow hosts with tcg we were sometimes finding that the migration would complete during precopy and never get into the postcopy test. Trim back the bandwidth a bit to make that much less likely.

Quoted 9 lines
---
 tests/migration-test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/migration-test.c +0 -0
index 9c62ee5331..221a33d083 100644

tests/migration-test.c static int migrate_postcopy_prepare(QTestState **from_ptr, · line 759

Quoted 8 lines
tests/migration-test.c +1 -1
@@ -753,7 +753,7 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
      * quickly, but that it doesn't complete precopy even on a slow
      * machine, so also set the downtime.
      */
-    migrate_set_parameter_int(from, "max-bandwidth", 100000000);
+    migrate_set_parameter_int(from, "max-bandwidth", 30000000);
     migrate_set_parameter_int(from, "downtime-limit", 1);
 
     /* Wait for the first serial output from the source */

--
Alex Bennée

Reviewed-by on v1 2/2
#8 lore raw

Re: [PATCH 2/2] tests/migration/postcopy: trim migration bandwidth

In reply to Dr. David Alan Gilbert (git)
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

On slow hosts with tcg we were sometimes finding that the migration would complete during precopy and never get into the postcopy test. Trim back the bandwidth a bit to make that much less likely.

Reviewed-by on the v1 series
#9 lore raw
In reply to Dr. David Alan Gilbert (git)
Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Alex noticed that some of the postcopy tests would occasionally hang; this series adds some checks to make them more likely to assert than hang in some failure cases, and changes the migration bandwidth so that under load it's much more likely to manage to land in postcopy.

Dr. David Alan Gilbert (2):
tests/migration: Fail on unexpected migration states tests/migration/postcopy: trim migration bandwidth

I might be even more aggresive on patch 2 when turning down the bandwidth. :)

Another thing I thought about on the hang issue is that maybe we can give a timeout for the waits and when the timeout triggers before a directly assert in the test case we send sigabrt to QEMU (just like what kill_qemu does) then we could have a chance to see the cores. Not sure whether that could help, though.

Regards,

Signature · 1 line
Peter Xu
#10 lore raw
Quoted 15 lines · * Peter Xu ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Alex noticed that some of the postcopy tests would occasionally hang; this series adds some checks to make them more likely to assert than hang in some failure cases, and changes the migration bandwidth so that under load it's much more likely to manage to land in postcopy.

Dr. David Alan Gilbert (2):
tests/migration: Fail on unexpected migration states tests/migration/postcopy: trim migration bandwidth

Quoted 8 lines

I might be even more aggresive on patch 2 when turning down the bandwidth. :)

Another thing I thought about on the hang issue is that maybe we can give a timeout for the waits and when the timeout triggers before a directly assert in the test case we send sigabrt to QEMU (just like what kill_qemu does) then we could have a chance to see the cores. Not sure whether that could help, though.

Yes I might need to add that; I can see there's another hang somewhere even with these two patches.
(I *think* it's at the end while it's waiting for completion after resumption - but I've not figured out what's going on yet).

Dave

Quoted 4 lines

Regards,

Signature · 1 line
Peter Xu

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#11 lore raw
In reply to Dr. David Alan Gilbert (git)
Quoted 7 lines · * Dr. David Alan Gilbert (git) ([email protected]) wrote:

From: "Dr. David Alan Gilbert" <[email protected]>

Alex noticed that some of the postcopy tests would occasionally hang; this series adds some checks to make them more likely to assert than hang in some failure cases, and changes the migration bandwidth so that under load it's much more likely to manage to land in postcopy.

Queued

Quoted 12 lines

Dr. David Alan Gilbert (2):
tests/migration: Fail on unexpected migration states tests/migration/postcopy: trim migration bandwidth

 tests/migration-test.c | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)
Signature · 1 line
2.21.0

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK