Skip to contentSkip to conversation

migration: fixes and multiple migration

PATCH In review Nicholas Piggin 4 participants 13 messages Jul 25, 2023 – Oct 19, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover migration: fixes and multiple migration

Spent too long looking at bash code after posting previous RFC, but got it into shape.

The first 2 seem to be real bugs you can trigger by making a test case fail. Third IMO is pretty solid now but no users yet so I can keep it around and resubmit with a user some time later.

Thanks,
Nick

Nicholas Piggin (3):
migration: Fix test harness hang on fifo
migration: Fix test harness hang if source does not reach migration
point
arch-run: Support multiple migrations

 lib/migrate.c         |  8 +++---
 lib/migrate.h         |  1 +
 scripts/arch-run.bash | 65 ++++++++++++++++++++++++++++++++++++-------
 3 files changed, 60 insertions(+), 14 deletions(-)
Signature · 1 line
2.40.1
#1 lore raw

PATCH 1/3 migration: Fix test harness hang on fifo

If the test fails before migration is complete, the input fifo for the destination machine is not written to and that can leave cat waiting for input.

Clear that condition in the error handler so the harness doesn't hang in this case.

---
 scripts/arch-run.bash | 1 +
 1 file changed, 1 insertion(+)
scripts/arch-run.bash +1 -0
index 5e7e4201..518607f4 100644
@@ -165,6 +165,7 @@ run_migration ()
 		migstatus=`qmp ${qmp1} '"query-migrate"' | grep return`
 		if grep -q '"failed"' <<<"$migstatus" ; then
 			echo "ERROR: Migration failed." >&2
+			echo > ${fifo}
 			qmp ${qmp1} '"quit"'> ${qmpout1} 2>/dev/null
 			qmp ${qmp2} '"quit"'> ${qmpout2} 2>/dev/null
 			return 2
Signature · 1 line
2.40.1
Reviewed-by on v1 1/3
#2 lore raw

Re: [kvm-unit-tests PATCH 1/3] migration: Fix test harness hang on fifo

Quoting Nicholas Piggin (2023-07-25 05:39:35)

Quoted 8 lines

If the test fails before migration is complete, the input fifo for the destination machine is not written to and that can leave cat waiting for input.

Clear that condition in the error handler so the harness doesn't hang in this case.

#3 lore raw

PATCH 2/3 migration: Fix test harness hang if source does not reach migration point

In reply to Nicholas Piggin

After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.

Test that the pid is still alive while polling to fix this hang.

While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.

---
 scripts/arch-run.bash | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
scripts/arch-run.bash +9 -1
index 518607f4..30e535c7 100644
@@ -142,6 +142,7 @@ run_migration ()
 
 	eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \
 		-mon chardev=mon1,mode=control | tee ${migout1} &
+	live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`
 
 	# We have to use cat to open the named FIFO, because named FIFO's, unlike
 	# pipes, will block on open() until the other end is also opened, and that
@@ -152,7 +153,14 @@ run_migration ()
 	incoming_pid=`jobs -l %+ | awk '{print$2}'`
 
 	# The test must prompt the user to migrate, so wait for the "migrate" keyword
-	while ! grep -q -i "migrate" < ${migout1} ; do
+	while ! grep -q -i "Now migrate the VM" < ${migout1} ; do
+		if ! ps -p ${live_pid} > /dev/null ; then
+			echo "ERROR: Test exit before migration point." >&2
+			echo > ${fifo}
+			qmp ${qmp1} '"quit"'> ${qmpout1} 2>/dev/null
+			qmp ${qmp2} '"quit"'> ${qmpout2} 2>/dev/null
+			return 3
+		fi
 		sleep 1
 	done
 
Signature · 1 line
2.40.1
#4 lore raw

Re: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point

Quoting Nicholas Piggin (2023-07-25 05:39:36)

Quoted 10 lines

After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.

Test that the pid is still alive while polling to fix this hang.

While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.

Thanks for attempting to fix this!

scripts/arch-run.bash run_migration () · line 145

Quoted 13 lines
---
 scripts/arch-run.bash | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
scripts/arch-run.bash +1 -0
index 518607f4..30e535c7 100644
@@ -142,6 +142,7 @@ run_migration ()
 
        eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \
                -mon chardev=mon1,mode=control | tee ${migout1} &
+       live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`

Pardon my ignorance, but why would $! not work here?

#5 lore raw

Re: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point

Quoted 24 lines · Nico Boehr

Quoting Nicholas Piggin (2023-07-25 05:39:36)

Quoted 10 lines

After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.

Test that the pid is still alive while polling to fix this hang.

While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.

Thanks for attempting to fix this!

Quoted 10 lines
---
 scripts/arch-run.bash | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
scripts/arch-run.bash +0 -0
index 518607f4..30e535c7 100644
@@ -142,6 +142,7 @@ run_migration ()
 
Quoted 5 lines

scripts/arch-run.bash run_migration () · line 145

Quoted 3 lines
scripts/arch-run.bash +1 -0
        eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \
                -mon chardev=mon1,mode=control | tee ${migout1} &
+       live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`

Pardon my ignorance, but why would $! not work here?

My mastery of bash is poor, I copied the incoming_pid line. It seems to work, but if you think $! is better I can try it.

Thanks,
Nick

Reviewed-by
#6 lore raw

Re: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point

Quoting Nicholas Piggin (2023-07-30 12:03:36)

Quoted 25 lines · Nico Boehr
Quoted 24 lines

Quoting Nicholas Piggin (2023-07-25 05:39:36)

Quoted 10 lines

After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.

Test that the pid is still alive while polling to fix this hang.

While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.

Thanks for attempting to fix this!

Quoted 10 lines
---
 scripts/arch-run.bash | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
scripts/arch-run.bash +0 -0
index 518607f4..30e535c7 100644
@@ -142,6 +142,7 @@ run_migration ()
 
Quoted 8 lines
Quoted 5 lines

scripts/arch-run.bash run_migration () · line 145

Quoted 3 lines
scripts/arch-run.bash +1 -0
        eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \
                -mon chardev=mon1,mode=control | tee ${migout1} &
+       live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`

Pardon my ignorance, but why would $! not work here?

My mastery of bash is poor, I copied the incoming_pid line. It seems to work, but if you think $! is better I can try it.

Sorry, this fell off of my radar after going to summer holiday...

#7 lore raw

Re: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point

Quoted 32 lines · Nico Boehr

Quoting Nicholas Piggin (2023-07-30 12:03:36)

Quoted 31 lines · Nico Boehr
Quoted 29 lines

Quoting Nicholas Piggin (2023-07-25 05:39:36)

Quoted 10 lines

After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.

Test that the pid is still alive while polling to fix this hang.

While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.

Thanks for attempting to fix this!

scripts/arch-run.bash run_migration () · line 145

Quoted 13 lines
---
  scripts/arch-run.bash | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
scripts/arch-run.bash +1 -0
index 518607f4..30e535c7 100644
@@ -142,6 +142,7 @@ run_migration ()
 
         eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \
                 -mon chardev=mon1,mode=control | tee ${migout1} &
+       live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`

Pardon my ignorance, but why would $! not work here?

Quoted 6 lines
Quoted 2 lines

My mastery of bash is poor, I copied the incoming_pid line. It seems to work, but if you think $! is better I can try it.

Sorry, this fell off of my radar after going to summer holiday...

Hi Nicholas & Nico,

do you want me to pick up this patch as is, or do you want to respin with $! instead?

Thomas

#8 lore raw

Re: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point

Quoting Thomas Huth (2023-10-16 20:32:47)

Quoted 36 lines · Nico Boehr
Quoted 35 lines

Quoting Nicholas Piggin (2023-07-30 12:03:36)

Quoted 33 lines · Nico Boehr
Quoted 24 lines

Quoting Nicholas Piggin (2023-07-25 05:39:36)

Quoted 10 lines

After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.

Test that the pid is still alive while polling to fix this hang.

While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.

Thanks for attempting to fix this!

Quoted 10 lines
---
  scripts/arch-run.bash | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
scripts/arch-run.bash +0 -0
index 518607f4..30e535c7 100644
@@ -142,6 +142,7 @@ run_migration ()
 
Quoted 5 lines

scripts/arch-run.bash run_migration () · line 145

Quoted 3 lines
scripts/arch-run.bash +1 -0
         eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \
                 -mon chardev=mon1,mode=control | tee ${migout1} &
+       live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`

Pardon my ignorance, but why would $! not work here?

My mastery of bash is poor, I copied the incoming_pid line. It seems to work, but if you think $! is better I can try it.

Quoted 8 lines
Quoted 3 lines

Sorry, this fell off of my radar after going to summer holiday...

Hi Nicholas & Nico,

do you want me to pick up this patch as is, or do you want to respin with $! instead?

Let's not discuss too much and get this fixed, I am fine with this as-is. Thanks.

#9 lore raw

Re: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point

Quoted 41 lines · Nico Boehr

Quoting Thomas Huth (2023-10-16 20:32:47)

Quoted 40 lines · Nico Boehr
Quoted 38 lines

Quoting Nicholas Piggin (2023-07-30 12:03:36)

Quoted 25 lines · Nico Boehr
Quoted 24 lines

Quoting Nicholas Piggin (2023-07-25 05:39:36)

Quoted 10 lines

After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.

Test that the pid is still alive while polling to fix this hang.

While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.

Thanks for attempting to fix this!

Quoted 10 lines
---
   scripts/arch-run.bash | 10 +++++++++-
   1 file changed, 9 insertions(+), 1 deletion(-)
scripts/arch-run.bash +0 -0
index 518607f4..30e535c7 100644
@@ -142,6 +142,7 @@ run_migration ()
 
Quoted 8 lines
Quoted 5 lines

scripts/arch-run.bash run_migration () · line 145

Quoted 3 lines
scripts/arch-run.bash +1 -0
          eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \
                  -mon chardev=mon1,mode=control | tee ${migout1} &
+       live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`

Pardon my ignorance, but why would $! not work here?

My mastery of bash is poor, I copied the incoming_pid line. It seems to work, but if you think $! is better I can try it.

Sorry, this fell off of my radar after going to summer holiday...

Quoted 7 lines
Quoted 4 lines

Hi Nicholas & Nico,

do you want me to pick up this patch as is, or do you want to respin with $! instead?

Let's not discuss too much and get this fixed, I am fine with this as-is. Thanks.

Ok, pushed it now.

Thomas

#10 lore raw

PATCH 3/3 arch-run: Support multiple migrations

In reply to Nicholas Piggin

Support multiple migrations by flipping dest file/socket variables to source after the migration is complete, ready to start again. A new destination is created if the test outputs the migrate line again. Test cases may now switch to calling migrate() one or more times.

---
 lib/migrate.c         |  8 +++----
 lib/migrate.h         |  1 +
 scripts/arch-run.bash | 54 +++++++++++++++++++++++++++++++++++--------
 3 files changed, 50 insertions(+), 13 deletions(-)
lib/migrate.c +4 -4
index 527e63ae..b7721659 100644
@@ -8,8 +8,10 @@
 #include <libcflat.h>
 #include "migrate.h"
 
-/* static for now since we only support migrating exactly once per test. */
-static void migrate(void)
+/*
+ * Initiate migration and wait for it to complete.
+ */
+void migrate(void)
 {
 	puts("Now migrate the VM, then press a key to continue...\n");
 	(void)getchar();
@@ -19,8 +21,6 @@ static void migrate(void)
 /*
  * Initiate migration and wait for it to complete.
  * If this function is called more than once, it is a no-op.
- * Since migrate_cmd can only migrate exactly once this function can
- * simplify the control flow, especially when skipping tests.
  */
 void migrate_once(void)
 {
lib/migrate.h +1 -0
index 3c94e6af..2af06a72 100644
@@ -6,4 +6,5 @@
  * Author: Nico Boehr <[email protected]>
  */
 
+void migrate(void);
 void migrate_once(void);
scripts/arch-run.bash +45 -9

83 lines not on this page · raw message

Signature · 1 line
2.40.1
#11 lore raw
In reply to Nicholas Piggin

Hi Nicholas,

With the patch3 arch-run: Support multiple migrations.

The run_test.sh will fail on the migration test on arm64 platform. But the patch1 and patch2 looks good, only apply these two patches the migration test works good.

I haven't had time to investigate it. But first let you know that.

Thanks,
Shaoqin

Quoted 21 lines · Nicholas Piggin

Spent too long looking at bash code after posting previous RFC, but got it into shape.

The first 2 seem to be real bugs you can trigger by making a test case fail. Third IMO is pretty solid now but no users yet so I can keep it around and resubmit with a user some time later.

Thanks,
Nick

Nicholas Piggin (3):
migration: Fix test harness hang on fifo
migration: Fix test harness hang if source does not reach migration
point
arch-run: Support multiple migrations

  lib/migrate.c         |  8 +++---
  lib/migrate.h         |  1 +
  scripts/arch-run.bash | 65 ++++++++++++++++++++++++++++++++++++-------
  3 files changed, 60 insertions(+), 14 deletions(-)
Signature · 1 line
Shaoqin
#12 lore raw
Quoted 9 lines · Shaoqin Huang

Hi Nicholas,

With the patch3 arch-run: Support multiple migrations.

The run_test.sh will fail on the migration test on arm64 platform. But the patch1 and patch2 looks good, only apply these two patches the migration test works good.

I haven't had time to investigate it. But first let you know that.

Hey, thanks for testing. I noticed some problem with it too. Let's leave patch 3 out for now, I'll post an update with some users added and hopefully bugs fixed.

Thanks,
Nick