9pfs: test suite fixes
v3 cover 9pfs: test suite fixes
Fixes test failures with the 9pfs 'local' tests as discussed with latest 9P PR. See the discussion of that PR v2 (Fri, Oct 30th) for details.
In conjunction with Peter Xu's two migration patches (fixing occasional lockups of migration tests) overall situation appears to be smooth now:
https://lore.kernel.org/qemu-devel/20201030135350.GA588069@xz-x1/
v2->v3:
- Make the two functions for creating and removing the 9pfs test directory public [NEW patch 1].
- Place the constructor and destructor functions in virtio-9p-test.c, not in virtio-9p.c, because the latter location would cause the constructor to be executed whenever libqos is loaded, which would break other, completely unrelated tests suites that just link to libqos [patch 2].
- Previous patch 2 (coverity fix) is already queued, no changes, hence omitted in this v3.
v1->v2:
- Added Greg's tested-by tag [patch 1].
- Log an info-level message if mkdir() failed [patch 2].
- Update commit log message about coverity being the reporter and details of the coverity report [patch 2].
Christian Schoenebeck (2):
tests/9pfs: make create/remove test dir public
tests/9pfs: fix test dir for parallel tests
tests/qtest/libqos/virtio-9p.c | 20 ++++++++++---------- tests/qtest/libqos/virtio-9p.h | 10 ++++++++++ tests/qtest/virtio-9p-test.c | 12 ++++++++++++ 3 files changed, 32 insertions(+), 10 deletions(-)
Signature · 1 line
2.20.1
PATCH v3 1/2 tests/9pfs: make create/remove test dir public
Make functions create_local_test_dir() and remove_local_test_dir() public. They're going to be used in the next patch.
--- tests/qtest/libqos/virtio-9p.c | 10 ++++------ tests/qtest/libqos/virtio-9p.h | 10 ++++++++++ 2 files changed, 14 insertions(+), 6 deletions(-)
tests/qtest/libqos/virtio-9p.c +4 -6
@@ -39,8 +39,7 @@ static void init_local_test_path(void)g_free (pwd); } -/* Creates the directory for the 9pfs 'local' filesystem driver to access. */ -static void create_local_test_dir (void ) +void virtio_9p_create_local_test_dir (void ) {struct stat st;
@@ -53,8 +52,7 @@ static void create_local_test_dir(void)g_assert ((st.st_mode &S_IFMT ) ==S_IFDIR ); } -/* Deletes directory previously created by create_local_test_dir(). */ -static void remove_local_test_dir (void ) +void virtio_9p_remove_local_test_dir (void ) {g_assert (local_test_path !=NULL );char *cmd =g_strdup_printf ("rm -r '%s'\n" , local_test_path);
@@ -248,8 +246,8 @@ static void virtio_9p_register_nodes(void)/* make sure test dir for the 'local' tests exists and is clean */ init_local_test_path (); -remove_local_test_dir (); -create_local_test_dir (); +virtio_9p_remove_local_test_dir (); +virtio_9p_create_local_test_dir ();QPCIAddress addr = { .devfn =QPCI_DEVFN (4 ,0 ),
tests/qtest/libqos/virtio-9p.h +10 -0
@@ -44,6 +44,16 @@ struct QVirtio9PDevice {QVirtio9P v9p; }; +/** +* Creates the directory for the 9pfs 'local' filesystem driver to access. +*/ +void virtio_9p_create_local_test_dir (void ); + +/** +* Deletes directory previously created by virtio_9p_create_local_test_dir(). +*/ +void virtio_9p_remove_local_test_dir (void ); +/** * Prepares QEMU command line for 9pfs tests using the 'local' fs driver. */
Signature · 1 line
2.20.1
Re: [PATCH v3 1/2] tests/9pfs: make create/remove test dir public
Quoted 5 lines · On Sun, 1 Nov 2020 15:25:14 +0100 Christian Schoenebeck…
Make functions
create_local_test_dir()andremove_local_test_dir()public. They're going to be used in the next patch.Signed-off-by Christian Schoenebeck <[email protected]>---
Quoted 60 lines
tests/qtest/libqos/virtio-9p.c | 10 ++++------ tests/qtest/libqos/virtio-9p.h | 10 ++++++++++ 2 files changed, 14 insertions(+), 6 deletions(-)tests/qtest/libqos/virtio-9p.c +4
-6@@ -39,8 +39,7 @@ static void init_local_test_path(void)g_free (pwd); } -/* Creates the directory for the 9pfs 'local' filesystem driver to access. */ -static void create_local_test_dir (void ) +void virtio_9p_create_local_test_dir (void ) {struct stat st;@@ -53,8 +52,7 @@ static void create_local_test_dir(void)g_assert ((st.st_mode &S_IFMT ) ==S_IFDIR ); } -/* Deletes directory previously created by create_local_test_dir(). */ -static void remove_local_test_dir (void ) +void virtio_9p_remove_local_test_dir (void ) {g_assert (local_test_path !=NULL );char *cmd =g_strdup_printf ("rm -r '%s'\n" , local_test_path);@@ -248,8 +246,8 @@ static void virtio_9p_register_nodes(void)/* make sure test dir for the 'local' tests exists and is clean */ init_local_test_path (); -remove_local_test_dir (); -create_local_test_dir (); +virtio_9p_remove_local_test_dir (); +virtio_9p_create_local_test_dir ();QPCIAddress addr = { .devfn =QPCI_DEVFN (4 ,0 ),tests/qtest/libqos/virtio-9p.h +10
-0@@ -44,6 +44,16 @@ struct QVirtio9PDevice {QVirtio9P v9p; }; +/** +* Creates the directory for the 9pfs 'local' filesystem driver to access. +*/ +void virtio_9p_create_local_test_dir (void ); + +/** +* Deletes directory previously created by virtio_9p_create_local_test_dir(). +*/ +void virtio_9p_remove_local_test_dir (void ); +/** * Prepares QEMU command line for 9pfs tests using the 'local' fs driver. */
PATCH v3 2/2 tests/9pfs: fix test dir for parallel tests
In reply to Christian SchoenebeckUse mkdtemp() to generate a unique directory for the 9p 'local' tests.
This fixes occasional 9p test failures when running 'make check -jN' if QEMU was compiled for multiple target architectures, because the individual architecture's test suites would run in parallel and interfere with each other's data as the test directory was previously hard coded and hence the same directory was used by all of them simultaniously.
This also requires a change how the test directory is created and deleted:
As the test path is now randomized and virtio_9p_register_nodes() being called in a somewhat undeterministic way, that's no longer an appropriate place to create and remove the test directory. Use a constructor and destructor function for creating and removing the test directory instead. Unfortunately libqos currently does not support setup/teardown callbacks to handle this more cleanly.
The constructor functions needs to be in virtio-9p-test.c, not in virtio-9p.c, because in the latter location it would cause all apps that link to libqos (i.e. entirely unrelated test suites) to create a 9pfs test directory as well, which would even break other test suites.
--- tests/qtest/libqos/virtio-9p.c | 14 ++++++++------ tests/qtest/virtio-9p-test.c | 12 ++++++++++++ 2 files changed, 20 insertions(+), 6 deletions(-)
tests/qtest/libqos/virtio-9p.c +8 -6
@@ -35,7 +35,12 @@ static char *concat_path(const char* a, const char* b)static void init_local_test_path (void ) {char *pwd =g_get_current_dir (); - local_test_path =concat_path (pwd,"qtest-9p-local" ); +char *template =concat_path (pwd,"qtest-9p-local-XXXXXX" ); + local_test_path =mkdtemp (template); +if (!local_test_path) { +g_test_message ("mkdtemp('%s') failed: %s" , template,strerror (errno)); + } +g_assert (local_test_path);g_free (pwd); }
@@ -43,6 +48,8 @@ void virtio_9p_create_local_test_dir(void) {struct stat st; +init_local_test_path (); +g_assert (local_test_path !=NULL );mkdir (local_test_path,0777 );
@@ -244,11 +251,6 @@ static void virtio_9p_register_nodes(void)const char *str_simple ="fsdev=fsdev0,mount_tag=" MOUNT_TAG ;const char *str_addr ="fsdev=fsdev0,addr=04.0,mount_tag=" MOUNT_TAG ; -/* make sure test dir for the 'local' tests exists and is clean */ -init_local_test_path (); -virtio_9p_remove_local_test_dir (); -virtio_9p_create_local_test_dir (); -QPCIAddress addr = { .devfn =QPCI_DEVFN (4 ,0 ), };
tests/qtest/virtio-9p-test.c +12 -0
@@ -1076,3 +1076,15 @@ static void register_virtio_9p_test(void) }libqos_init (register_virtio_9p_test); + +static void __attribute__((constructor))construct_9p_test (void ) +{ +/* make sure test dir for the 'local' tests exists */ +virtio_9p_create_local_test_dir (); +} + +static void __attribute__((destructor))destruct_9p_test (void ) +{ +/* remove previously created test dir when test suite completed */ +virtio_9p_remove_local_test_dir (); +}
Signature · 1 line
2.20.1
Re: [PATCH v3 2/2] tests/9pfs: fix test dir for parallel tests
Quoted 16 lines · On Sun, 1 Nov 2020 15:37:12 +0100 Christian Schoenebeck…
Use
mkdtemp()to generate a unique directory for the 9p 'local' tests.This fixes occasional 9p test failures when running 'make check -jN' if QEMU was compiled for multiple target architectures, because the individual architecture's test suites would run in parallel and interfere with each other's data as the test directory was previously hard coded and hence the same directory was used by all of them simultaniously.
This also requires a change how the test directory is created and deleted:
As the test path is now randomized andvirtio_9p_register_nodes()being called in a somewhat undeterministic way, that's no longer an appropriate place to create and remove the test directory. Use a constructor and destructor function for creating and removing the test directory instead. Unfortunately libqos currently does not support setup/teardown callbacks to handle this more cleanly.
Quoted 7 lines
The constructor functions needs to be in
virtio-9p-test.c, not invirtio-9p.c, because in the latter location it would cause all apps that link to libqos (i.e. entirely unrelated test suites) to create a 9pfs test directory as well, which would even break other test suites.Signed-off-by Christian Schoenebeck <[email protected]>---
I could run 'make check -j' with 4 archs (ppc64, x86_64, aarch64, s390x) on a POWER9 system with 128 cpus, for ~1 hour without seeing any failure.
Quoted 63 lines
tests/qtest/libqos/virtio-9p.c | 14 ++++++++------ tests/qtest/virtio-9p-test.c | 12 ++++++++++++ 2 files changed, 20 insertions(+), 6 deletions(-)tests/qtest/libqos/virtio-9p.c +8
-6@@ -35,7 +35,12 @@ static char *concat_path(const char* a, const char* b)static void init_local_test_path (void ) {char *pwd =g_get_current_dir (); - local_test_path =concat_path (pwd,"qtest-9p-local" ); +char *template =concat_path (pwd,"qtest-9p-local-XXXXXX" ); + local_test_path =mkdtemp (template); +if (!local_test_path) { +g_test_message ("mkdtemp('%s') failed: %s" , template,strerror (errno)); + } +g_assert (local_test_path);g_free (pwd); }@@ -43,6 +48,8 @@ void virtio_9p_create_local_test_dir(void) {struct stat st; +init_local_test_path (); +g_assert (local_test_path !=NULL );mkdir (local_test_path,0777 );@@ -244,11 +251,6 @@ static void virtio_9p_register_nodes(void)const char *str_simple ="fsdev=fsdev0,mount_tag=" MOUNT_TAG ;const char *str_addr ="fsdev=fsdev0,addr=04.0,mount_tag=" MOUNT_TAG ; -/* make sure test dir for the 'local' tests exists and is clean */ -init_local_test_path (); -virtio_9p_remove_local_test_dir (); -virtio_9p_create_local_test_dir (); -QPCIAddress addr = { .devfn =QPCI_DEVFN (4 ,0 ), };tests/qtest/virtio-9p-test.c +12
-0@@ -1076,3 +1076,15 @@ static void register_virtio_9p_test(void) }libqos_init (register_virtio_9p_test); + +static void __attribute__((constructor))construct_9p_test (void ) +{ +/* make sure test dir for the 'local' tests exists */ +virtio_9p_create_local_test_dir (); +} + +static void __attribute__((destructor))destruct_9p_test (void ) +{ +/* remove previously created test dir when test suite completed */ +virtio_9p_remove_local_test_dir (); +}
Re: [PATCH v3 2/2] tests/9pfs: fix test dir for parallel tests
Quoted 25 lines · On Sonntag, 1. November 2020 18:44:44 CET Greg Kurz wrote:
On Sun, 1 Nov 2020 15:37:12 +0100
Quoted 22 lines · Christian Schoenebeck
Use
mkdtemp()to generate a unique directory for the 9p 'local' tests.This fixes occasional 9p test failures when running 'make check -jN' if QEMU was compiled for multiple target architectures, because the individual
architecture's test suites would run in parallel and interfere with each other's data as the test directory was previously hard coded and hence the same directory was used by all of them simultaniously.This also requires a change how the test directory is created and deleted:
As the test path is now randomized andvirtio_9p_register_nodes()being called in a somewhat undeterministic way, that's no longer an appropriate place to create and remove the test directory. Use a constructor and destructor function for creating and removing the test directory instead. Unfortunately libqos currently does not support setup/teardown callbacks to handle this more cleanly.The constructor functions needs to be in
virtio-9p-test.c, not invirtio-9p.c, because in the latter location it would cause all apps that link to libqos (i.e. entirely unrelated test suites) to create a 9pfs test directory as well, which would even break other test suites.
Quoted 4 lines
Quoted 2 lines
Signed-off-by Christian Schoenebeck <[email protected]>---
Reviewed-by Greg Kurz <[email protected]>
Thanks for the overtime, on a Sunday!
Queued on 9p.next:
https://github.com/cschoenebeck/qemu/commits/9p.next
And this one with Peter Xu's patches on top, just for testing:
https://github.com/cschoenebeck/qemu/commits/9p.experimental.2
Quoted 4 lines
I could run 'make check -j' with 4 archs (ppc64,
x86_64, aarch64, s390x) on a POWER9 system with 128 cpus, for ~1 hour without seeing any failure.Tested-by Greg Kurz <[email protected]>
OO Sounds like having advantages working for IBM. Respect. I start to get envy as these beasts are running towards PCIe 6, while we regular x86 users would already be glad having PCIe 4.
I give it some more spinning hours this time, just to be sure, before sending the PR tomorrow morning. But I think it's all right now.
Thanks!
Best regards,
Christian Schoenebeck
Re: [PATCH v3 2/2] tests/9pfs: fix test dir for parallel tests
Quoted 39 lines · On Sun, 01 Nov 2020 20:14:16 +0100 Christian Schoenebeck…
Quoted 25 lines · On Sonntag, 1. November 2020 18:44:44 CET Greg Kurz wrote:
On Sun, 1 Nov 2020 15:37:12 +0100
Quoted 22 lines · Christian Schoenebeck
Use
mkdtemp()to generate a unique directory for the 9p 'local' tests.This fixes occasional 9p test failures when running 'make check -jN' if QEMU was compiled for multiple target architectures, because the individual
architecture's test suites would run in parallel and interfere with each other's data as the test directory was previously hard coded and hence the same directory was used by all of them simultaniously.This also requires a change how the test directory is created and deleted:
As the test path is now randomized andvirtio_9p_register_nodes()being called in a somewhat undeterministic way, that's no longer an appropriate place to create and remove the test directory. Use a constructor and destructor function for creating and removing the test directory instead. Unfortunately libqos currently does not support setup/teardown callbacks to handle this more cleanly.The constructor functions needs to be in
virtio-9p-test.c, not invirtio-9p.c, because in the latter location it would cause all apps that link to libqos (i.e. entirely unrelated test suites) to create a 9pfs test directory as well, which would even break other test suites.Quoted 4 lines
Quoted 2 lines
Signed-off-by Christian Schoenebeck <[email protected]>---
Reviewed-by Greg Kurz <[email protected]>Thanks for the overtime, on a Sunday!
Queued on 9p.next:
https://github.com/cschoenebeck/qemu/commits/9p.nextAnd this one with Peter Xu's patches on top, just for testing:
https://github.com/cschoenebeck/qemu/commits/9p.experimental.2
Quoted 9 lines
Quoted 4 lines
I could run 'make check -j' with 4 archs (ppc64,
x86_64, aarch64, s390x) on a POWER9 system with 128 cpus, for ~1 hour without seeing any failure.Tested-by Greg Kurz <[email protected]>OO Sounds like having advantages working for IBM. Respect. I start to get envy as these beasts are running towards PCIe 6, while we regular x86 users would already be glad having PCIe 4.
I work for Red Hat now but yes, this allows easier access to bigger systems.
Quoted 3 lines
I give it some more spinning hours this time, just to be sure, before sending the PR tomorrow morning. But I think it's all right now.
Cool ! :)
Quoted 6 lines
Thanks!
Best regards,
Christian Schoenebeck
Quoted 29 lines · Christian Schoenebeck
Fixes test failures with the 9pfs 'local' tests as discussed with latest 9P PR. See the discussion of that PR v2 (Fri, Oct 30th) for details.
In conjunction with Peter Xu's two migration patches (fixing occasional lockups of migration tests) overall situation appears to be smooth now:
https://lore.kernel.org/qemu-devel/20201030135350.GA588069@xz-x1/
v2->v3:- Make the two functions for creating and removing the 9pfs test directory public [NEW patch 1].
- Place the constructor and destructor functions in
virtio-9p-test.c, not invirtio-9p.c, because the latter location would cause the constructor to be executed whenever libqos is loaded, which would break other, completely unrelated tests suites that just link to libqos [patch 2].- Previous patch 2 (coverity fix) is already queued, no changes, hence omitted in this v3.
v1->v2:- Added Greg's tested-by tag [patch 1].
- Log an info-level message if
mkdir()failed [patch 2].- Update commit log message about coverity being the reporter and details of the coverity report [patch 2].
Quoted 8 lines
Christian Schoenebeck (2):
tests/9pfs: make create/remove test dir public
tests/9pfs: fix test dir for parallel teststests/qtest/libqos/virtio-9p.c | 20 ++++++++++---------- tests/qtest/libqos/virtio-9p.h | 10 ++++++++++ tests/qtest/virtio-9p-test.c | 12 ++++++++++++ 3 files changed, 32 insertions(+), 10 deletions(-)
FWIW one thing I've noticed recently is that my builds for qemu-system-sparc64 have started giving this warning about a missing "qtest-9p-local" directory during make check:
...
...
Running test QAPI schema regression tests
Running test qtest-sparc64/endianness-test
Running test qtest-sparc64/prom-env-test
Running test qtest-sparc64/boot-serial-test
Running test qtest-sparc64/cdrom-test
Running test qtest-sparc64/device-introspect-test
Running test qtest-sparc64/machine-none-test
Running test qtest-sparc64/qmp-test
Running test qtest-sparc64/qmp-cmd-test
Running test qtest-sparc64/qom-test
Running test qtest-sparc64/test-hmp
Running test qtest-sparc64/qos-test
rm: cannot remove '/home/build/src/qemu/git/qemu/build/qtest-9p-local': No such file or directory
TEST iotest-qcow2: 001 TEST iotest-qcow2: 002 TEST iotest-qcow2: 003 TEST iotest-qcow2: 004 TEST iotest-qcow2: 005
...
...
Would this get resolved by the changes to the test directory in this patchset? The build is a simple configure run with "--target-list=sparc64-softmmu".
ATB,
Mark.
Quoted 72 lines · On Sonntag, 1. November 2020 19:02:28 CET Mark Cave-Ayland wrote:
Quoted 34 lines · Christian Schoenebeck
Fixes test failures with the 9pfs 'local' tests as discussed with latest 9P PR. See the discussion of that PR v2 (Fri, Oct 30th) for details.
In conjunction with Peter Xu's two migration patches (fixing occasional lockups of migration tests) overall situation appears to be smooth now:
https://lore.kernel.org/qemu-devel/20201030135350.GA588069@xz-x1/
v2->v3:
- Make the two functions for creating and removing the 9pfs test directorypublic [NEW patch 1].
- Place the constructor and destructor functions in
virtio-9p-test.c, notin
virtio-9p.c, because the latter location would cause the constructor
to be executed whenever libqos is loaded, which would break other, completely unrelated tests suites that just link to libqos [patch 2].- Previous patch 2 (coverity fix) is already queued, no changes, hence
omitted in this v3.
v1->v2:
- Added Greg's tested-by tag [patch 1].- Log an info-level message if
mkdir()failed [patch 2].- Update commit log message about coverity being the reporter and
details of the coverity report [patch 2].
Quoted 8 lines
Christian Schoenebeck (2):
tests/9pfs: make create/remove test dir public
tests/9pfs: fix test dir for parallel teststests/qtest/libqos/virtio-9p.c | 20 ++++++++++---------- tests/qtest/libqos/virtio-9p.h | 10 ++++++++++ tests/qtest/virtio-9p-test.c | 12 ++++++++++++ 3 files changed, 32 insertions(+), 10 deletions(-)FWIW one thing I've noticed recently is that my builds for qemu-system-sparc64 have started giving this warning about a missing "qtest-9p-local" directory during make check:
...
...
Running test QAPI schema regression tests
Running test qtest-sparc64/endianness-test
Running test qtest-sparc64/prom-env-test
Running test qtest-sparc64/boot-serial-test
Running test qtest-sparc64/cdrom-test
Running test qtest-sparc64/device-introspect-test
Running test qtest-sparc64/machine-none-test
Running test qtest-sparc64/qmp-test
Running test qtest-sparc64/qmp-cmd-test
Running test qtest-sparc64/qom-test
Running test qtest-sparc64/test-hmp
Running test qtest-sparc64/qos-test
rm: cannot remove '/home/build/src/qemu/git/qemu/build/qtest-9p-local': No such file or directoryTEST iotest-qcow2: 001 TEST iotest-qcow2: 002 TEST iotest-qcow2: 003 TEST iotest-qcow2: 004 TEST iotest-qcow2: 005...
...
Quoted 8 lines
Would this get resolved by the changes to the test directory in this patchset? The build is a simple configure run with "--target-list=sparc64-softmmu".
ATB,
Mark.
Yes, that should be resolved with the next 9p PR as well, additionally with the following patch that is:
https://github.com/cschoenebeck/qemu/commit/603cc76a6069
Thanks for the feedback!
Best regards,
Christian Schoenebeck
Quoted 5 lines · Christian Schoenebeck
Yes, that should be resolved with the next 9p PR as well, additionally with the following patch that is:
https://github.com/cschoenebeck/qemu/commit/603cc76a6069Thanks for the feedback!
Fantastic - thanks a lot :)
ATB,
Mark.