lib/arm/io: Fix calling getchar() multiple times
Subsystems (3)
PATCH 1/1 lib/arm/io: Fix calling getchar() multiple times
getchar() can currently only be called once on arm since the implementation is a little bit too naïve: After the first character has arrived, the data register never gets set to zero again. To properly check whether a byte is available, we need to check the "RX fifo empty" on the pl011 UART or the "RX data ready" bit on the ns16550a UART instead.
With this proper check in place, we can finally also get rid of the ugly assert(count < 16) statement here.
--- lib/arm/io.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-)
lib/arm/io.c +14 -20
@@ -28,6 +28,7 @@ static struct spinlock uart_lock;*/ #define UART_EARLY_BASE (u8 *)(unsigned long)CONFIG_UART_EARLY_BASEstatic volatile u8 *uart0_base =UART_EARLY_BASE ; +bool is_pl011_uart;static void uart0_init_fdt (void ) {
@@ -59,7 +60,10 @@ static void uart0_init_fdt(void)abort (); } + is_pl011_uart = (i ==0 ); }else { + is_pl011_uart = !fdt_node_check_compatible (dt_fdt (), ret, +"arm,pl011" ); ret =dt_pbus_translate_node (ret,0 , &base);assert (ret ==0 ); }
@@ -111,31 +115,21 @@ void puts(const char *s)spin_unlock (&uart_lock); } -static int do_getchar (void ) +int __getchar (void ) { -int c; +int c =-1 ;spin_lock (&uart_lock); - c =readb (uart0_base); -spin_unlock (&uart_lock); - -return c ?:-1 ; -} - -/* -* Minimalist implementation for migration completion detection. -* Without FIFOs enabled on the QEMU UART device we just read -* the data register: we cannot read more than 16 characters. -*/ -int __getchar (void ) -{ -int c =do_getchar (); -static int count; -if (c !=-1 ) - ++count; +if (is_pl011_uart) { +if (!(readb (uart0_base +6 *4 ) &0x10 ))/* RX not empty? */ + c =readb (uart0_base); + }else { +if (readb (uart0_base +5 ) &0x01 )/* RX data ready? */ + c =readb (uart0_base); + } -assert (count <16 ); +spin_unlock (&uart_lock);return c; }
Signature · 1 line
2.43.0
Quoted 10 lines · Thomas Huth
getchar()can currently only be called once on arm since the implementation is a little bit too naïve: After the first character has arrived, the data register never gets set to zero again. To properly check whether a byte is available, we need to check the "RX fifo empty" on the pl011 UART or the "RX data ready" bit on the ns16550a UART instead.With this proper check in place, we can finally also get rid of the ugly assert(count < 16) statement here.
Signed-off-by Thomas Huth <[email protected]>
Nice, thanks for fixing this up.
I see what you mean about multi-migration not waiting. It seems to be an arm issue, ppc works properly. This patch changed things so it works a bit better (or at least differently) now, but still has some bugs. Maybe buggy uart migration?
Thanks,
Nick
Quoted 10 lines · Nicholas Piggin
Quoted 9 lines · Thomas Huth
getchar()can currently only be called once on arm since the implementation is a little bit too naïve: After the first character has arrived, the data register never gets set to zero again. To properly check whether a byte is available, we need to check the "RX fifo empty" on the pl011 UART or the "RX data ready" bit on the ns16550a UART instead.With this proper check in place, we can finally also get rid of the ugly assert(count < 16) statement here.
Quoted 6 lines
Quoted 1 line
Signed-off-by Thomas Huth <[email protected]>Nice, thanks for fixing this up.
I see what you mean about multi-migration not waiting. It seems to be an arm issue, ppc works properly.
Yes, it's an arm issue. s390x also works fine.
Quoted 3 lines
This patch changed things
so it works a bit better (or at least differently) now, but still has some bugs. Maybe buggy uart migration?
I'm also seeing hangs when running the arm migration-test multiple times, but also without my UART patch here - so I assume the problem is not really related to the UART?
Thomas
Quoted 20 lines · Thomas Huth
Quoted 10 lines · Nicholas Piggin
Quoted 9 lines · Thomas Huth
getchar()can currently only be called once on arm since the implementation is a little bit too naïve: After the first character has arrived, the data register never gets set to zero again. To properly check whether a byte is available, we need to check the "RX fifo empty" on the pl011 UART or the "RX data ready" bit on the ns16550a UART instead.With this proper check in place, we can finally also get rid of the ugly assert(count < 16) statement here.
Quoted 6 lines
Quoted 1 line
Signed-off-by Thomas Huth <[email protected]>Nice, thanks for fixing this up.
I see what you mean about multi-migration not waiting. It seems to be an arm issue, ppc works properly.
Yes, it's an arm issue. s390x also works fine.
Quoted 7 lines
Quoted 3 lines
This patch changed things
so it works a bit better (or at least differently) now, but still has some bugs. Maybe buggy uart migration?I'm also seeing hangs when running the arm migration-test multiple times, but also without my UART patch here - so I assume the problem is not really related to the UART?
Yeah, I ended up figuring it out. A 11 year old TCG migration memory corruption bug!
https://lists.gnu.org/archive/html/qemu-devel/2024-02/msg03486.html
All the weirdness was just symptoms of that. The hang that arm usually got was target machine trying to lock the uart spinlock that is already locked (because the unlock store got lost in migration).
powerpc and s390x were just luckier in avoiding the race, maybe the way their translation blocks around getchar code were constructed made the problem not show up easily or at all. I did end up causing problems for them by rearranging the code (test case is linked in that msg).
Thanks,
Nick
Quoted 25 lines · Nicholas Piggin
Quoted 24 lines · Thomas Huth
Quoted 10 lines · Nicholas Piggin
Quoted 9 lines · Thomas Huth
getchar()can currently only be called once on arm since the implementation is a little bit too naïve: After the first character has arrived, the data register never gets set to zero again. To properly check whether a byte is available, we need to check the "RX fifo empty" on the pl011 UART or the "RX data ready" bit on the ns16550a UART instead.With this proper check in place, we can finally also get rid of the ugly assert(count < 16) statement here.
Quoted 6 lines
Quoted 1 line
Signed-off-by Thomas Huth <[email protected]>Nice, thanks for fixing this up.
I see what you mean about multi-migration not waiting. It seems to be an arm issue, ppc works properly.
Yes, it's an arm issue. s390x also works fine.
Quoted 3 lines
This patch changed things
so it works a bit better (or at least differently) now, but still has some bugs. Maybe buggy uart migration?
Quoted 8 lines
Quoted 3 lines
I'm also seeing hangs when running the arm migration-test multiple times, but also without my UART patch here - so I assume the problem is not really related to the UART?
Yeah, I ended up figuring it out. A 11 year old TCG migration memory corruption bug!
https://lists.gnu.org/archive/html/qemu-devel/2024-02/msg03486.html
Nice! And thanks for bringing this multi-migration test support to kvm-unit-tests!
drew
Quoted 12 lines
All the weirdness was just symptoms of that. The hang that arm usually got was target machine trying to lock the uart spinlock that is already locked (because the unlock store got lost in migration).
powerpc and s390x were just luckier in avoiding the race, maybe the way their translation blocks around getchar code were constructed made the problem not show up easily or at all. I did end up causing problems for them by rearranging the code (test case is linked in that msg).
Thanks,
Nick
Quoted 9 lines · Thomas Huth
getchar()can currently only be called once on arm since the implementation is a little bit too naïve: After the first character has arrived, the data register never gets set to zero again. To properly check whether a byte is available, we need to check the "RX fifo empty" on the pl011 UART or the "RX data ready" bit on the ns16550a UART instead.With this proper check in place, we can finally also get rid of the ugly assert(count < 16) statement here.
Thanks, this seems to work well. But arm64 still behaves strangely with QEMU migration. It seems some odd corruption around __getchar, but this turns out not to have anything to do with the MMIO read, but it's just because the test case generally is spinning there when a migration happens.
I modified __getchar() to the following, which removes the MMIO entirely and just waits a number of times called before returning a getchar, chosen to be enough time for QEMU migration to complete on my system...
int __getchar(void)
{
static int i;
static volatile int locked;
int ret = -1;
assert(!locked);
locked = 1;
assert(locked);
cpu_relax();
assert(locked);
i++;
if (i >= 3000000) {
i = 0;
ret = 1;
}
assert(locked);
locked = 0;
assert(!locked);
return ret;
}I get asserts on the line right after cpu_relax() on arm64 after a few migrations on migration selftest.
Without the cpu_relax() it takes longer to hit an assert and it's usually the first one. I expect this is related to TCG translation blocks and where it exits back to be migrated. Something is very strange, I suspect it's QEMU migration bug.
Thanks,
Nick
Quoted 63 lines · Thomas Huth
getchar()can currently only be called once on arm since the implementation is a little bit too naïve: After the first character has arrived, the data register never gets set to zero again. To properly check whether a byte is available, we need to check the "RX fifo empty" on the pl011 UART or the "RX data ready" bit on the ns16550a UART instead.With this proper check in place, we can finally also get rid of the ugly assert(count < 16) statement here.
Signed-off-by Thomas Huth <[email protected]>--- lib/arm/io.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-)lib/arm/io.c +6
-17@@ -28,6 +28,7 @@ static struct spinlock uart_lock;*/ #define UART_EARLY_BASE (u8 *)(unsigned long)CONFIG_UART_EARLY_BASEstatic volatile u8 *uart0_base =UART_EARLY_BASE ; +bool is_pl011_uart;static void uart0_init_fdt (void ) {@@ -59,7 +60,10 @@ static void uart0_init_fdt(void)abort (); } + is_pl011_uart = (i ==0 ); }else { + is_pl011_uart = !fdt_node_check_compatible (dt_fdt (), ret, +"arm,pl011" ); ret =dt_pbus_translate_node (ret,0 , &base);assert (ret ==0 ); }@@ -111,31 +115,21 @@ void puts(const char *s)spin_unlock (&uart_lock); } -static int do_getchar (void ) +int __getchar (void ) { -int c; +int c =-1 ;spin_lock (&uart_lock); - c =readb (uart0_base); -spin_unlock (&uart_lock); - -return c ?:-1 ; -} - -/* -* Minimalist implementation for migration completion detection. -* Without FIFOs enabled on the QEMU UART device we just read -* the data register: we cannot read more than 16 characters. -*/ -int __getchar (void ) -{ -int c =do_getchar (); -static int count;
lib/arm/io.c void puts(const char *s) · line 130
Quoted 10 lines
lib/arm/io.c +7
-2-if (c !=-1 ) - ++count; +if (is_pl011_uart) { +if (!(readb (uart0_base +6 *4 ) &0x10 ))/* RX not empty? */ + c =readb (uart0_base); + }else { +if (readb (uart0_base +5 ) &0x01 )/* RX data ready? */ + c =readb (uart0_base); + }
I think I'd eventually prefer encapsulating these separate implementations into a console device, but that might be a bit heavy handed for a fix that we want to pull into the multi-migration series.
Thanks,
drew
Hi,
Thanks for writing this. I've tested it with kvmtool, which emulates a 8250 UART:
This fixes a longstanding bug with kvmtool, where migrate_once() would read the last character that was sent, and then think that migration was completed even though it was never performed.
While we are on the subject of migration:
SKIP: gicv3: its-migration: Test requires at least 4 vcpus Now migrate the VM, then press a key to continue...
INFO: gicv3: its-migration: Migration complete
SUMMARY: 1 tests, 1 skipped
That's extremely confusing. Why is migrate_once() executed after the test_its_pending() function call without checking if the test was skipped?
Nitpicks below.
Quoted 58 lines · Thomas Huth
getchar()can currently only be called once on arm since the implementation is a little bit too naïve: After the first character has arrived, the data register never gets set to zero again. To properly check whether a byte is available, we need to check the "RX fifo empty" on the pl011 UART or the "RX data ready" bit on the ns16550a UART instead.With this proper check in place, we can finally also get rid of the ugly assert(count < 16) statement here.
Signed-off-by Thomas Huth <[email protected]>--- lib/arm/io.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-)lib/arm/io.c +6
-12@@ -28,6 +28,7 @@ static struct spinlock uart_lock;*/ #define UART_EARLY_BASE (u8 *)(unsigned long)CONFIG_UART_EARLY_BASEstatic volatile u8 *uart0_base =UART_EARLY_BASE ; +bool is_pl011_uart;static void uart0_init_fdt (void ) {@@ -59,7 +60,10 @@ static void uart0_init_fdt(void)abort (); } + is_pl011_uart = (i ==0 ); }else { + is_pl011_uart = !fdt_node_check_compatible (dt_fdt (), ret, +"arm,pl011" ); ret =dt_pbus_translate_node (ret,0 , &base);assert (ret ==0 ); }@@ -111,31 +115,21 @@ void puts(const char *s)spin_unlock (&uart_lock); } -static int do_getchar (void ) +int __getchar (void ) { -int c; +int c =-1 ;spin_lock (&uart_lock); - c =readb (uart0_base); -spin_unlock (&uart_lock); - -return c ?:-1 ; -} - -/* -* Minimalist implementation for migration completion detection. -* Without FIFOs enabled on the QEMU UART device we just read -* the data register: we cannot read more than 16 characters.
lib/arm/io.c void puts(const char *s) · line 125
Quoted 10 lines
lib/arm/io.c +2
-7-*/ -int __getchar (void ) -{ -int c =do_getchar (); -static int count; -if (c !=-1 ) - ++count; +if (is_pl011_uart) { +if (!(readb (uart0_base +6 *4 ) &0x10 ))/* RX not empty? */
I think it would be useful if the magic numbers were replaced by something less opaque, something like:
if (!(readb(uart0_base + PL011_UARTFR) & PL011_UARTFR_RXFE))
lib/arm/io.c void puts(const char *s) · line 128
Quoted 3 lines
lib/arm/io.c +3
-0+ c =readb (uart0_base); + }else { +if (readb (uart0_base +5 ) &0x01 )/* RX data ready? */
Same as above, perhaps:
if (readb(uart0_base + UART16550_LSR) & UART16550_LSR_DR)
Naming of course being subject to taste.
Thanks,
Alex
Quoted 11 lines
lib/arm/io.c +3
-1+ c =readb (uart0_base); + } -assert (count <16 ); +spin_unlock (&uart_lock);return c; }Signature · 1 line
2.43.0
Quoted 13 lines · Alexandru Elisei
Hi,
Thanks for writing this. I've tested it with kvmtool, which emulates a 8250 UART:
Tested-by Alexandru Elisei <[email protected]>This fixes a longstanding bug with kvmtool, where
migrate_once()would read the last character that was sent, and then think that migration was completed even though it was never performed.While we are on the subject of migration:
Quoted 7 lines
SKIP: gicv3: its-migration: Test requires at least 4 vcpus Now migrate the VM, then press a key to continue...
INFO: gicv3: its-migration: Migration complete
SUMMARY: 1 tests, 1 skippedThat's extremely confusing. Why is
migrate_once()executed after thetest_its_pending()function call without checking if the test was skipped?
Looks like it was done so the test is skipped without killing the harness due to expected migration point not being reached.
After the multi-migration series, you could just put a migrate_quiet() there as a quick fix.
But I'm thinking we could just remove the requirement for the harness to see at least 1 migration point. The test itself knows how many migrations it should perform, by how many times it calls migrate*(). It is somewhat a sanity test against test being invoked the wrong way and not doing the migration, but how much is that worth...? Actually we could have a new sideband message like "VM migration is skipped" that doesn't do anything except tell the test harness to not fail due to missing migration point. That gets the best of both worlds, just needs tests to be updated.
Thanks,
Nick
Quoted 102 lines
Nitpicks below.
Quoted 58 lines · Thomas Huth
getchar()can currently only be called once on arm since the implementation is a little bit too naïve: After the first character has arrived, the data register never gets set to zero again. To properly check whether a byte is available, we need to check the "RX fifo empty" on the pl011 UART or the "RX data ready" bit on the ns16550a UART instead.With this proper check in place, we can finally also get rid of the ugly assert(count < 16) statement here.
Signed-off-by Thomas Huth <[email protected]>--- lib/arm/io.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-)lib/arm/io.c +6
-12@@ -28,6 +28,7 @@ static struct spinlock uart_lock;*/ #define UART_EARLY_BASE (u8 *)(unsigned long)CONFIG_UART_EARLY_BASEstatic volatile u8 *uart0_base =UART_EARLY_BASE ; +bool is_pl011_uart;static void uart0_init_fdt (void ) {@@ -59,7 +60,10 @@ static void uart0_init_fdt(void)abort (); } + is_pl011_uart = (i ==0 ); }else { + is_pl011_uart = !fdt_node_check_compatible (dt_fdt (), ret, +"arm,pl011" ); ret =dt_pbus_translate_node (ret,0 , &base);assert (ret ==0 ); }@@ -111,31 +115,21 @@ void puts(const char *s)spin_unlock (&uart_lock); } -static int do_getchar (void ) +int __getchar (void ) { -int c; +int c =-1 ;spin_lock (&uart_lock); - c =readb (uart0_base); -spin_unlock (&uart_lock); - -return c ?:-1 ; -} - -/* -* Minimalist implementation for migration completion detection. -* Without FIFOs enabled on the QEMU UART device we just read -* the data register: we cannot read more than 16 characters. lib/arm/io.c void puts(const char *s) · line 125
Quoted 10 lines
lib/arm/io.c +2
-7-*/ -int __getchar (void ) -{ -int c =do_getchar (); -static int count; -if (c !=-1 ) - ++count; +if (is_pl011_uart) { +if (!(readb (uart0_base +6 *4 ) &0x10 ))/* RX not empty? */ I think it would be useful if the magic numbers were replaced by something less opaque, something like:
if (!(readb(uart0_base + PL011_UARTFR) & PL011_UARTFR_RXFE))lib/arm/io.c void puts(const char *s) · line 128
Quoted 3 lines
lib/arm/io.c +3
-0+ c =readb (uart0_base); + }else { +if (readb (uart0_base +5 ) &0x01 )/* RX data ready? */ Same as above, perhaps:
if (readb(uart0_base + UART16550_LSR) & UART16550_LSR_DR)Naming of course being subject to taste.
Thanks,
AlexQuoted 11 lines
lib/arm/io.c +3
-1+ c =readb (uart0_base); + } -assert (count <16 ); +spin_unlock (&uart_lock);return c; }Signature · 1 line
2.43.0
Quoted 13 lines · Alexandru Elisei
Hi,
Thanks for writing this. I've tested it with kvmtool, which emulates a 8250 UART:
Tested-by Alexandru Elisei <[email protected]>This fixes a longstanding bug with kvmtool, where
migrate_once()would read the last character that was sent, and then think that migration was completed even though it was never performed.While we are on the subject of migration:
Quoted 7 lines
SKIP: gicv3: its-migration: Test requires at least 4 vcpus Now migrate the VM, then press a key to continue...
INFO: gicv3: its-migration: Migration complete
SUMMARY: 1 tests, 1 skippedThat's extremely confusing. Why is
migrate_once()executed after thetest_its_pending()function call without checking if the test was skipped?
Seems not too hard, incremental patch on top of multi migration series below. After this series is merged I can try that (s390 could benefit with some changes too).
Thanks,
Nick
---
arm/gic.c +12 -9
@@ -782,13 +782,15 @@ static void test_its_migration(void)struct its_device *dev2, *dev7;cpumask_t mask; -if (its_setup1 ()) +if (its_setup1 ()) { +migrate_skip ();return ; + } dev2 =its_get_device (2 ); dev7 =its_get_device (7 ); -migrate_once (); +migrate ();stats_reset ();cpumask_clear (&mask);
@@ -819,8 +821,10 @@ static void test_migrate_unmapped_collection(void)int pe0 =0 ;u8 config; -if (its_setup1 ()) +if (its_setup1 ()) { +migrate_skip ();return ; + }if (!errata (ERRATA_UNMAPPED_COLLECTIONS )) { report_skip("Skipping test, as this test hangs without the fix. "
@@ -836,7 +840,7 @@ static void test_migrate_unmapped_collection(void)its_send_mapti (dev2,8192 ,0 , col);gicv3_lpi_set_config (8192 ,LPI_PROP_DEFAULT ); -migrate_once (); +migrate ();/* on the destination, map the collection */ its_send_mapc (col, true);
@@ -875,8 +879,10 @@ static void test_its_pending_migration(void)void *ptr;int i; -if (its_prerequisites (4 )) +if (its_prerequisites (4 )) { +migrate_skip ();return ; + } dev =its_create_device (2 /* dev id */ ,8 /* nb_ites */ );its_send_mapd (dev, true);
@@ -923,7 +929,7 @@ static void test_its_pending_migration(void)gicv3_lpi_rdist_enable (pe0);gicv3_lpi_rdist_enable (pe1); -migrate_once (); +migrate ();/* let's wait for the 256 LPIs to be handled */ mdelay (1000 );
@@ -970,17 +976,14 @@ int main(int argc, char **argv) }else if (!strcmp (argv[1 ],"its-migration" )) {report_prefix_push (argv[1 ]);test_its_migration (); -migrate_once ();report_prefix_pop (); }else if (!strcmp (argv[1 ],"its-pending-migration" )) {report_prefix_push (argv[1 ]);test_its_pending_migration (); -migrate_once ();report_prefix_pop (); }else if (!strcmp (argv[1 ],"its-migrate-unmapped-collection" )) {report_prefix_push (argv[1 ]);test_migrate_unmapped_collection (); -migrate_once ();report_prefix_pop (); }else if (strcmp (argv[1 ],"its-introspection" ) ==0 ) {report_prefix_push (argv[1 ]);
lib/migrate.c +10 -0
@@ -43,3 +43,13 @@ void migrate_once(void) migrated = true;migrate (); } + +/* +* When the test has been started in migration mode, but the test case is +* skipped and no migration point is reached, this can be used to tell the +* harness not to mark it as a failure to migrate. +*/ +void migrate_skip (void ) +{ +puts ("Skipped VM migration (quiet)\n" ); +}
lib/migrate.h +2 -0
@@ -9,3 +9,5 @@void migrate (void );void migrate_quiet (void );void migrate_once (void ); + +void migrate_skip (void );
scripts/arch-run.bash +12 -4
@@ -152,7 +152,9 @@ run_migration () -chardev socket,id=mon,path=${src_qmp},server=on,wait=off \ -mon chardev=mon,mode=control > ${src_outfifo} & live_pid=$! - cat ${src_outfifo} | tee ${src_out} | grep -v "Now migrate the VM (quiet)" & + cat ${src_outfifo} | tee ${src_out} | \ + grep -v "Now migrate the VM (quiet)" | \ + grep -v "Skipped VM migration (quiet)" & # Start the first destination QEMU machine in advance of the test # reaching the migration point, since we expect at least one migration.
@@ -190,16 +192,22 @@ do_migration () -mon chardev=mon,mode=control -incoming unix:${dst_incoming} \ < <(cat ${dst_infifo}) > ${dst_outfifo} & incoming_pid=$! - cat ${dst_outfifo} | tee ${dst_out} | grep -v "Now migrate the VM (quiet)" & + cat ${dst_outfifo} | tee ${dst_out} | \ + grep -v "Now migrate the VM (quiet)" | \ + grep -v "Skipped VM migration (quiet)" & # The test must prompt the user to migrate, so wait for the # "Now migrate VM" console message. while ! grep -q -i "Now migrate the VM" < ${src_out} ; do if ! ps -p ${live_pid} > /dev/null ; then - echo "ERROR: Test exit before migration point." >&2 echo > ${dst_infifo} - qmp ${src_qmp} '"quit"'> ${src_qmpout} 2>/dev/null qmp ${dst_qmp} '"quit"'> ${dst_qmpout} 2>/dev/null + if grep -q -i "Skipped VM migration" < ${src_out} ; then + wait ${live_pid} + return $? + fi + echo "ERROR: Test exit before migration point." >&2 + qmp ${src_qmp} '"quit"'> ${src_qmpout} 2>/dev/null return 3 fi sleep 0.1
Hi,
Quoted 19 lines · Nicholas Piggin
Quoted 18 lines · Alexandru Elisei
Hi,
Thanks for writing this. I've tested it with kvmtool, which emulates a 8250 UART:
Tested-by Alexandru Elisei <[email protected]>This fixes a longstanding bug with kvmtool, where
migrate_once()would read the last character that was sent, and then think that migration was completed even though it was never performed.While we are on the subject of migration:
SKIP: gicv3: its-migration: Test requires at least 4 vcpus Now migrate the VM, then press a key to continue...
INFO: gicv3: its-migration: Migration complete
SUMMARY: 1 tests, 1 skipped
Quoted 6 lines
Quoted 2 lines
That's extremely confusing. Why is
migrate_once()executed after thetest_its_pending()function call without checking if the test was skipped?Seems not too hard, incremental patch on top of multi migration series below. After this series is merged I can try that (s390 could benefit with some changes too).
Thank you for having a look at this so quickly. The changes to the gic test look good to me.
As an alternative, have you considered modifying the test harness to parse the SKIP message, and if the test is in the migration group to not mark it as a migration failure? That would require that the test name printed by a test matches the test name from unittests.cfg (should probably be the case already), but any new migration tests will just work, without having to put migrate_skip() at each failure point.
Thanks,
Alex
Quoted 156 lines
Thanks,
Nick---
arm/gic.c +12
-9@@ -782,13 +782,15 @@ static void test_its_migration(void)struct its_device *dev2, *dev7;cpumask_t mask; -if (its_setup1 ()) +if (its_setup1 ()) { +migrate_skip ();return ; + } dev2 =its_get_device (2 ); dev7 =its_get_device (7 ); -migrate_once (); +migrate ();stats_reset ();cpumask_clear (&mask);@@ -819,8 +821,10 @@ static void test_migrate_unmapped_collection(void)int pe0 =0 ;u8 config; -if (its_setup1 ()) +if (its_setup1 ()) { +migrate_skip ();return ; + }if (!errata (ERRATA_UNMAPPED_COLLECTIONS )) { report_skip("Skipping test, as this test hangs without the fix. "@@ -836,7 +840,7 @@ static void test_migrate_unmapped_collection(void)its_send_mapti (dev2,8192 ,0 , col);gicv3_lpi_set_config (8192 ,LPI_PROP_DEFAULT ); -migrate_once (); +migrate ();/* on the destination, map the collection */ its_send_mapc (col, true);@@ -875,8 +879,10 @@ static void test_its_pending_migration(void)void *ptr;int i; -if (its_prerequisites (4 )) +if (its_prerequisites (4 )) { +migrate_skip ();return ; + } dev =its_create_device (2 /* dev id */ ,8 /* nb_ites */ );its_send_mapd (dev, true);@@ -923,7 +929,7 @@ static void test_its_pending_migration(void)gicv3_lpi_rdist_enable (pe0);gicv3_lpi_rdist_enable (pe1); -migrate_once (); +migrate ();/* let's wait for the 256 LPIs to be handled */ mdelay (1000 );@@ -970,17 +976,14 @@ int main(int argc, char **argv) }else if (!strcmp (argv[1 ],"its-migration" )) {report_prefix_push (argv[1 ]);test_its_migration (); -migrate_once ();report_prefix_pop (); }else if (!strcmp (argv[1 ],"its-pending-migration" )) {report_prefix_push (argv[1 ]);test_its_pending_migration (); -migrate_once ();report_prefix_pop (); }else if (!strcmp (argv[1 ],"its-migrate-unmapped-collection" )) {report_prefix_push (argv[1 ]);test_migrate_unmapped_collection (); -migrate_once ();report_prefix_pop (); }else if (strcmp (argv[1 ],"its-introspection" ) ==0 ) {report_prefix_push (argv[1 ]);lib/migrate.c +10
-0@@ -43,3 +43,13 @@ void migrate_once(void) migrated = true;migrate (); } + +/* +* When the test has been started in migration mode, but the test case is +* skipped and no migration point is reached, this can be used to tell the +* harness not to mark it as a failure to migrate. +*/ +void migrate_skip (void ) +{ +puts ("Skipped VM migration (quiet)\n" ); +}lib/migrate.h +2
-0@@ -9,3 +9,5 @@void migrate (void );void migrate_quiet (void );void migrate_once (void ); + +void migrate_skip (void );scripts/arch-run.bash +12
-4@@ -152,7 +152,9 @@ run_migration () -chardev socket,id=mon,path=${src_qmp},server=on,wait=off \ -mon chardev=mon,mode=control > ${src_outfifo} & live_pid=$! - cat ${src_outfifo} | tee ${src_out} | grep -v "Now migrate the VM (quiet)" & + cat ${src_outfifo} | tee ${src_out} | \ + grep -v "Now migrate the VM (quiet)" | \ + grep -v "Skipped VM migration (quiet)" & # Start the first destination QEMU machine in advance of the test # reaching the migration point, since we expect at least one migration.@@ -190,16 +192,22 @@ do_migration () -mon chardev=mon,mode=control -incoming unix:${dst_incoming} \ < <(cat ${dst_infifo}) > ${dst_outfifo} & incoming_pid=$! - cat ${dst_outfifo} | tee ${dst_out} | grep -v "Now migrate the VM (quiet)" & + cat ${dst_outfifo} | tee ${dst_out} | \ + grep -v "Now migrate the VM (quiet)" | \ + grep -v "Skipped VM migration (quiet)" & # The test must prompt the user to migrate, so wait for the # "Now migrate VM" console message. while ! grep -q -i "Now migrate the VM" < ${src_out} ; do if ! ps -p ${live_pid} > /dev/null ; then - echo "ERROR: Test exit before migration point." >&2 echo > ${dst_infifo} - qmp ${src_qmp} '"quit"'> ${src_qmpout} 2>/dev/null qmp ${dst_qmp} '"quit"'> ${dst_qmpout} 2>/dev/null + if grep -q -i "Skipped VM migration" < ${src_out} ; then + wait ${live_pid} + return $? + fi + echo "ERROR: Test exit before migration point." >&2 + qmp ${src_qmp} '"quit"'> ${src_qmpout} 2>/dev/null return 3 fi sleep 0.1
Quoted 10 lines · Thomas Huth
getchar()can currently only be called once on arm since the implementation is a little bit too naïve: After the first character has arrived, the data register never gets set to zero again. To properly check whether a byte is available, we need to check the "RX fifo empty" on the pl011 UART or the "RX data ready" bit on the ns16550a UART instead.With this proper check in place, we can finally also get rid of the ugly assert(count < 16) statement here.
Signed-off-by Thomas Huth <[email protected]>
With the QEMU bug fix, the multi migration patches work well on arm64 with this patch for me (on TCG).
Thanks,
Nick
Quoted 69 lines
--- lib/arm/io.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-)lib/arm/io.c +14
-20@@ -28,6 +28,7 @@ static struct spinlock uart_lock;*/ #define UART_EARLY_BASE (u8 *)(unsigned long)CONFIG_UART_EARLY_BASEstatic volatile u8 *uart0_base =UART_EARLY_BASE ; +bool is_pl011_uart;static void uart0_init_fdt (void ) {@@ -59,7 +60,10 @@ static void uart0_init_fdt(void)abort (); } + is_pl011_uart = (i ==0 ); }else { + is_pl011_uart = !fdt_node_check_compatible (dt_fdt (), ret, +"arm,pl011" ); ret =dt_pbus_translate_node (ret,0 , &base);assert (ret ==0 ); }@@ -111,31 +115,21 @@ void puts(const char *s)spin_unlock (&uart_lock); } -static int do_getchar (void ) +int __getchar (void ) { -int c; +int c =-1 ;spin_lock (&uart_lock); - c =readb (uart0_base); -spin_unlock (&uart_lock); - -return c ?:-1 ; -} - -/* -* Minimalist implementation for migration completion detection. -* Without FIFOs enabled on the QEMU UART device we just read -* the data register: we cannot read more than 16 characters. -*/ -int __getchar (void ) -{ -int c =do_getchar (); -static int count; -if (c !=-1 ) - ++count; +if (is_pl011_uart) { +if (!(readb (uart0_base +6 *4 ) &0x10 ))/* RX not empty? */ + c =readb (uart0_base); + }else { +if (readb (uart0_base +5 ) &0x01 )/* RX data ready? */ + c =readb (uart0_base); + } -assert (count <16 ); +spin_unlock (&uart_lock);return c; }
Quoted 6 lines · Thomas Huth
getchar()can currently only be called once on arm since the implementation is a little bit too naïve: After the first character has arrived, the data register never gets set to zero again. To properly check whether a byte is available, we need to check the "RX fifo empty" on the pl011 UART or the "RX data ready" bit on the ns16550a UART instead.
Quoted 8 lines
With this proper check in place, we can finally also get rid of the ugly assert(count < 16) statement here.
Signed-off-by Thomas Huth <[email protected]>--- lib/arm/io.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-)