migration: fixes and multiple migration
Subsystems (2)
cover migration: fixes and multiple migration
Spent too long looking at bash code after posting previous RFC, but got it into shape.
The first 2 seem to be real bugs you can trigger by making a test case fail. Third IMO is pretty solid now but no users yet so I can keep it around and resubmit with a user some time later.
Thanks,
Nick
Nicholas Piggin (3):
migration: Fix test harness hang on fifo
migration: Fix test harness hang if source does not reach migration
point
arch-run: Support multiple migrations
lib/migrate.c | 8 +++--- lib/migrate.h | 1 + scripts/arch-run.bash | 65 ++++++++++++++++++++++++++++++++++++------- 3 files changed, 60 insertions(+), 14 deletions(-)
Signature · 1 line
2.40.1
PATCH 1/3 migration: Fix test harness hang on fifo
If the test fails before migration is complete, the input fifo for the destination machine is not written to and that can leave cat waiting for input.
Clear that condition in the error handler so the harness doesn't hang in this case.
--- scripts/arch-run.bash | 1 + 1 file changed, 1 insertion(+)
scripts/arch-run.bash +1 -0
@@ -165,6 +165,7 @@ run_migration () migstatus=`qmp ${qmp1} '"query-migrate"' | grep return` if grep -q '"failed"' <<<"$migstatus" ; then echo "ERROR: Migration failed." >&2 + echo > ${fifo} qmp ${qmp1} '"quit"'> ${qmpout1} 2>/dev/null qmp ${qmp2} '"quit"'> ${qmpout2} 2>/dev/null return 2
Signature · 1 line
2.40.1
PATCH 2/3 migration: Fix test harness hang if source does not reach migration point
In reply to Nicholas PigginAfter starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.
Test that the pid is still alive while polling to fix this hang.
While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.
--- scripts/arch-run.bash | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
scripts/arch-run.bash +9 -1
@@ -142,6 +142,7 @@ run_migration () eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \ -mon chardev=mon1,mode=control | tee ${migout1} & + live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'` # We have to use cat to open the named FIFO, because named FIFO's, unlike # pipes, will block on open() until the other end is also opened, and that
@@ -152,7 +153,14 @@ run_migration () incoming_pid=`jobs -l %+ | awk '{print$2}'` # The test must prompt the user to migrate, so wait for the "migrate" keyword - while ! grep -q -i "migrate" < ${migout1} ; do + while ! grep -q -i "Now migrate the VM" < ${migout1} ; do + if ! ps -p ${live_pid} > /dev/null ; then + echo "ERROR: Test exit before migration point." >&2 + echo > ${fifo} + qmp ${qmp1} '"quit"'> ${qmpout1} 2>/dev/null + qmp ${qmp2} '"quit"'> ${qmpout2} 2>/dev/null + return 3 + fi sleep 1 done
Signature · 1 line
2.40.1
PATCH 3/3 arch-run: Support multiple migrations
In reply to Nicholas PigginSupport multiple migrations by flipping dest file/socket variables to source after the migration is complete, ready to start again. A new destination is created if the test outputs the migrate line again. Test cases may now switch to calling migrate() one or more times.
--- lib/migrate.c | 8 +++---- lib/migrate.h | 1 + scripts/arch-run.bash | 54 +++++++++++++++++++++++++++++++++++-------- 3 files changed, 50 insertions(+), 13 deletions(-)
lib/migrate.c +4 -4
@@ -8,8 +8,10 @@#include <libcflat.h> #include "migrate.h" -/* static for now since we only support migrating exactly once per test. */ -static void migrate (void ) +/* +* Initiate migration and wait for it to complete. +*/ +void migrate (void ) {puts ("Now migrate the VM, then press a key to continue...\n" ); (void )getchar ();
@@ -19,8 +21,6 @@ static void migrate(void)/* * Initiate migration and wait for it to complete. * If this function is called more than once, it is a no-op. -* Since migrate_cmd can only migrate exactly once this function can -* simplify the control flow, especially when skipping tests. */ void migrate_once (void ) {
lib/migrate.h +1 -0
@@ -6,4 +6,5 @@* Author: Nico Boehr <[email protected]> */ +void migrate (void );void migrate_once (void );
scripts/arch-run.bash +45 -9
83 lines not on this page · raw message
Signature · 1 line
2.40.1
Re: [kvm-unit-tests PATCH 1/3] migration: Fix test harness hang on fifo
In reply to Nicholas PigginQuoting Nicholas Piggin (2023-07-25 05:39:35)
Quoted 8 lines
If the test fails before migration is complete, the input fifo for the destination machine is not written to and that can leave cat waiting for input.
Clear that condition in the error handler so the harness doesn't hang in this case.
Signed-off-by Nicholas Piggin <[email protected]>
Re: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point
In reply to Nicholas PigginQuoting Nicholas Piggin (2023-07-25 05:39:36)
Quoted 10 lines
After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.
Test that the pid is still alive while polling to fix this hang.
While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.
Signed-off-by Nicholas Piggin <[email protected]>
Thanks for attempting to fix this!
scripts/arch-run.bash run_migration () · line 145
Quoted 13 lines
--- scripts/arch-run.bash | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)scripts/arch-run.bash +1
-0@@ -142,6 +142,7 @@ run_migration () eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \ -mon chardev=mon1,mode=control | tee ${migout1} & + live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`
Pardon my ignorance, but why would $! not work here?
Hi Nicholas,
With the patch3 arch-run: Support multiple migrations.
The run_test.sh will fail on the migration test on arm64 platform. But the patch1 and patch2 looks good, only apply these two patches the migration test works good.
I haven't had time to investigate it. But first let you know that.
Thanks,
Shaoqin
Quoted 21 lines · Nicholas Piggin
Spent too long looking at bash code after posting previous RFC, but got it into shape.
The first 2 seem to be real bugs you can trigger by making a test case fail. Third IMO is pretty solid now but no users yet so I can keep it around and resubmit with a user some time later.
Thanks,
NickNicholas Piggin (3):
migration: Fix test harness hang on fifo
migration: Fix test harness hang if source does not reach migration
point
arch-run: Support multiple migrationslib/migrate.c | 8 +++--- lib/migrate.h | 1 + scripts/arch-run.bash | 65 ++++++++++++++++++++++++++++++++++++------- 3 files changed, 60 insertions(+), 14 deletions(-)
Signature · 1 line
Shaoqin
Re: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point
In reply to Nico BoehrQuoted 24 lines · Nico Boehr
Quoting Nicholas Piggin (2023-07-25 05:39:36)
Quoted 10 lines
After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.
Test that the pid is still alive while polling to fix this hang.
While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.
Signed-off-by Nicholas Piggin <[email protected]>Thanks for attempting to fix this!
Quoted 10 lines
--- scripts/arch-run.bash | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)scripts/arch-run.bash +0
-0@@ -142,6 +142,7 @@ run_migration ()
Quoted 5 lines
scripts/arch-run.bash run_migration () · line 145
Quoted 3 lines
scripts/arch-run.bash +1
-0eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \ -mon chardev=mon1,mode=control | tee ${migout1} & + live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`Pardon my ignorance, but why would $! not work here?
My mastery of bash is poor, I copied the incoming_pid line. It seems to work, but if you think $! is better I can try it.
Thanks,
Nick
Quoted 9 lines · Shaoqin Huang
Hi Nicholas,
With the patch3 arch-run: Support multiple migrations.
The
run_test.shwill fail on the migration test on arm64 platform. But the patch1 and patch2 looks good, only apply these two patches the migration test works good.I haven't had time to investigate it. But first let you know that.
Hey, thanks for testing. I noticed some problem with it too. Let's leave patch 3 out for now, I'll post an update with some users added and hopefully bugs fixed.
Thanks,
Nick
Nico Boehr
Reviewed-byRe: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point
In reply to Nicholas PigginQuoting Nicholas Piggin (2023-07-30 12:03:36)
Quoted 25 lines · Nico Boehr
Quoted 24 lines
Quoting Nicholas Piggin (2023-07-25 05:39:36)
Quoted 10 lines
After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.
Test that the pid is still alive while polling to fix this hang.
While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.
Signed-off-by Nicholas Piggin <[email protected]>Thanks for attempting to fix this!
Quoted 10 lines
--- scripts/arch-run.bash | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)scripts/arch-run.bash +0
-0@@ -142,6 +142,7 @@ run_migration ()
Quoted 8 lines
Quoted 5 lines
scripts/arch-run.bash run_migration () · line 145
Quoted 3 lines
scripts/arch-run.bash +1
-0eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \ -mon chardev=mon1,mode=control | tee ${migout1} & + live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`Pardon my ignorance, but why would $! not work here?
My mastery of bash is poor, I copied the
incoming_pidline. It seems to work, but if you think $! is better I can try it.
Sorry, this fell off of my radar after going to summer holiday...
Re: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point
Quoted 32 lines · Nico Boehr
Quoting Nicholas Piggin (2023-07-30 12:03:36)
Quoted 31 lines · Nico Boehr
Quoted 29 lines
Quoting Nicholas Piggin (2023-07-25 05:39:36)
Quoted 10 lines
After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.
Test that the pid is still alive while polling to fix this hang.
While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.
Signed-off-by Nicholas Piggin <[email protected]>Thanks for attempting to fix this!
scripts/arch-run.bash run_migration () · line 145
Quoted 13 lines
--- scripts/arch-run.bash | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)scripts/arch-run.bash +1
-0@@ -142,6 +142,7 @@ run_migration () eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \ -mon chardev=mon1,mode=control | tee ${migout1} & + live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`Pardon my ignorance, but why would $! not work here?
Quoted 6 lines
Quoted 2 lines
My mastery of bash is poor, I copied the
incoming_pidline. It seems to work, but if you think $! is better I can try it.Sorry, this fell off of my radar after going to summer holiday...
Reviewed-by Nico Boehr <[email protected]>
Hi Nicholas & Nico,
do you want me to pick up this patch as is, or do you want to respin with $! instead?
Thomas
Re: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point
Quoting Thomas Huth (2023-10-16 20:32:47)
Quoted 36 lines · Nico Boehr
Quoted 35 lines
Quoting Nicholas Piggin (2023-07-30 12:03:36)
Quoted 33 lines · Nico Boehr
Quoted 24 lines
Quoting Nicholas Piggin (2023-07-25 05:39:36)
Quoted 10 lines
After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.
Test that the pid is still alive while polling to fix this hang.
While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.
Signed-off-by Nicholas Piggin <[email protected]>Thanks for attempting to fix this!
Quoted 10 lines
--- scripts/arch-run.bash | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)scripts/arch-run.bash +0
-0@@ -142,6 +142,7 @@ run_migration ()Quoted 5 lines
scripts/arch-run.bash run_migration () · line 145
Quoted 3 lines
scripts/arch-run.bash +1
-0eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \ -mon chardev=mon1,mode=control | tee ${migout1} & + live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`Pardon my ignorance, but why would $! not work here?
My mastery of bash is poor, I copied the
incoming_pidline. It seems to work, but if you think $! is better I can try it.
Quoted 8 lines
Quoted 3 lines
Sorry, this fell off of my radar after going to summer holiday...
Reviewed-by Nico Boehr <[email protected]>Hi Nicholas & Nico,
do you want me to pick up this patch as is, or do you want to respin with $! instead?
Let's not discuss too much and get this fixed, I am fine with this as-is. Thanks.
Re: [kvm-unit-tests PATCH 2/3] migration: Fix test harness hang if source does not reach migration point
Quoted 41 lines · Nico Boehr
Quoting Thomas Huth (2023-10-16 20:32:47)
Quoted 40 lines · Nico Boehr
Quoted 38 lines
Quoting Nicholas Piggin (2023-07-30 12:03:36)
Quoted 25 lines · Nico Boehr
Quoted 24 lines
Quoting Nicholas Piggin (2023-07-25 05:39:36)
Quoted 10 lines
After starting the test, the harness waits polling for "migrate" in the output. If the test does not print for some reason, the harness hangs.
Test that the pid is still alive while polling to fix this hang.
While here, wait for the full string "Now migrate the VM", which I think makes it more obvious to read and could avoid an unfortunate collision with some debugging output in a test case.
Signed-off-by Nicholas Piggin <[email protected]>Thanks for attempting to fix this!
Quoted 10 lines
--- scripts/arch-run.bash | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)scripts/arch-run.bash +0
-0@@ -142,6 +142,7 @@ run_migration ()Quoted 8 lines
Quoted 5 lines
scripts/arch-run.bash run_migration () · line 145
Quoted 3 lines
scripts/arch-run.bash +1
-0eval "$@" -chardev socket,id=mon1,path=${qmp1},server=on,wait=off \ -mon chardev=mon1,mode=control | tee ${migout1} & + live_pid=`jobs -l %+ | grep "eval" | awk '{print$2}'`Pardon my ignorance, but why would $! not work here?
My mastery of bash is poor, I copied the
incoming_pidline. It seems to work, but if you think $! is better I can try it.Sorry, this fell off of my radar after going to summer holiday...
Reviewed-by Nico Boehr <[email protected]>
Quoted 7 lines
Quoted 4 lines
Hi Nicholas & Nico,
do you want me to pick up this patch as is, or do you want to respin with $! instead?
Let's not discuss too much and get this fixed, I am fine with this as-is. Thanks.
Ok, pushed it now.
Thomas