Address OS X Travis failures

RFC PATCH In review Cleber Rosa 3 participants 7 messages Nov 9, 2018

Share & tools

Cleber Rosa

author

2018-11-09 15:30 UTC #0 lore raw

cover Address OS X Travis failures

The OS X jobs on Travis fail for a number of reasons. This is an attempt to decide its faith, either as something that will be kept on the configuration (and properly maintained), removed, or maybe active but with failures waived (Travis allows for that).

The result of this can be seen in these jobs part of a recent build:

- https://travis-ci.org/clebergnu/qemu/jobs/452517040
- https://travis-ci.org/clebergnu/qemu/jobs/452517041
- https://travis-ci.org/clebergnu/qemu/jobs/452517042
- https://travis-ci.org/clebergnu/qemu/jobs/452517043
- https://travis-ci.org/clebergnu/qemu/jobs/452517044
- https://travis-ci.org/clebergnu/qemu/jobs/452517045
- https://travis-ci.org/clebergnu/qemu/jobs/452517046

Cleber Rosa (2):
tests/test-qht-par: test gets stuck intermittently on OSX Travis CI: break down OSX+clang jobs

 .travis.yml            | 29 ++++++++++++++++++++++++++++-
 tests/Makefile.include |  3 +++
 2 files changed, 31 insertions(+), 1 deletion(-)

Signature · 1 line

2.19.1

Cleber Rosa

author

2018-11-09 15:30 UTC #1 lore raw

RFC PATCH 1/2 tests/test-qht-par: test gets stuck intermittently on OSX

To be fully honest, this may not be a OSX (alone) condition, but may be a situation that only happens with OSX on Travis-CI, were resources are quite limited.

I have personal experience with tests that exercise parallelism or depend on timing to fail on Travis. Because I'm not 100% certain that this is a situation that only happens with OSX on Travis-CI, and because I'm not certain that we should be skipping tests because they're running on Travis-CI, let's disable them on OSX as a whole.

A small note: this type of change makes me believe that there should be a list of testing related caveats or TODO list tracked on the documentation.

Signed-off-by Cleber Rosa <[email protected]>

---
 tests/Makefile.include | 3 +++
 1 file changed, 3 insertions(+)

tests/Makefile.include +3 -0

index 074eece558..c821b01467 100644
@@ -87,7 +87,10 @@ check-unit-y += tests/test-rcu-simpleq$(EXESUF)
 check-unit-y += tests/test-rcu-tailq$(EXESUF)
 check-unit-y += tests/test-qdist$(EXESUF)
 check-unit-y += tests/test-qht$(EXESUF)
+# test-qht-par gets stuck quite often on OSX
+ifneq ($(CONFIG_DARWIN),y)
 check-unit-y += tests/test-qht-par$(EXESUF)
+endif
 check-unit-y += tests/test-bitops$(EXESUF)
 check-unit-y += tests/test-bitcnt$(EXESUF)
 check-unit-y += tests/test-qdev-global-props$(EXESUF)

Signature · 1 line

2.19.1

Peter Maydell

2018-11-09 15:35 UTC #2 lore raw

Re: [Qemu-devel] [RFC PATCH 1/2] tests/test-qht-par: test gets stuck intermittently on OSX

Quoted 9 lines · Cleber Rosa

To be fully honest, this may not be a OSX (alone) condition, but may be a situation that only happens with OSX on Travis-CI, were resources are quite limited.
I have personal experience with tests that exercise parallelism or depend on timing to fail on Travis. Because I'm not 100% certain that this is a situation that only happens with OSX on Travis-CI, and because I'm not certain that we should be skipping tests because they're running on Travis-CI, let's disable them on OSX as a whole.

Does this test still get stuck in builds after a458774ad711bceabefb ? That fixed an OSX-specific problem that caused hangs (notably in test-bdrv-drain, but it could I guess also affect other tests).

thanks
-- PMM

Cleber Rosa

author

2018-11-09 17:07 UTC #3 lore raw

Re: [Qemu-devel] [RFC PATCH 1/2] tests/test-qht-par: test gets stuck intermittently on OSX

Quoted 11 lines · Peter Maydell

Quoted 9 lines · Cleber Rosa
To be fully honest, this may not be a OSX (alone) condition, but may be a situation that only happens with OSX on Travis-CI, were resources are quite limited.
I have personal experience with tests that exercise parallelism or depend on timing to fail on Travis. Because I'm not 100% certain that this is a situation that only happens with OSX on Travis-CI, and because I'm not certain that we should be skipping tests because they're running on Travis-CI, let's disable them on OSX as a whole.

Quoted 4 lines

Does this test still get stuck in builds after a458774ad711bceabefb ? That fixed an OSX-specific problem that caused hangs (notably in test-bdrv-drain, but it could I guess also affect other tests).

It didn't get stuck here:

https://travis-ci.org/clebergnu/qemu/jobs/452663111

So I believe that was the common issue indeed. Thanks for pointing that out. Also, the other patch in this series also seems to be unnecessary now.

- Cleber.

Quoted 3 lines

thanks
-- PMM

Emilio G. Cota

2018-11-09 15:56 UTC #4 lore raw

Re: [Qemu-devel] [RFC PATCH 1/2] tests/test-qht-par: test gets stuck intermittently on OSX

In reply to Cleber Rosa

Quoted 10 lines · Cleber Rosa

To be fully honest, this may not be a OSX (alone) condition, but may be a situation that only happens with OSX on Travis-CI, were resources are quite limited.
I have personal experience with tests that exercise parallelism or depend on timing to fail on Travis. Because I'm not 100% certain that this is a situation that only happens with OSX on Travis-CI, and because I'm not certain that we should be skipping tests because they're running on Travis-CI, let's disable them on OSX as a whole.

Quoted 5 lines

A small note: this type of change makes me believe that there should be a list of testing related caveats or TODO list tracked on the documentation.
Signed-off-by Cleber Rosa <[email protected]>

Is there a way to know where it hangs?

Also, are these the g_test_quick tests, or the other ones? It it's just a matter of the program taking too long to run, we could just reduce the length of the !quick tests (currently at 5s vs. 1s for the quick ones).

Thanks,

Emilio

Cleber Rosa

author

2018-11-09 17:09 UTC #5 lore raw

Re: [Qemu-devel] [RFC PATCH 1/2] tests/test-qht-par: test gets stuck intermittently on OSX

Quoted 19 lines · Emilio G. Cota

Quoted 10 lines · Cleber Rosa
To be fully honest, this may not be a OSX (alone) condition, but may be a situation that only happens with OSX on Travis-CI, were resources are quite limited.
I have personal experience with tests that exercise parallelism or depend on timing to fail on Travis. Because I'm not 100% certain that this is a situation that only happens with OSX on Travis-CI, and because I'm not certain that we should be skipping tests because they're running on Travis-CI, let's disable them on OSX as a whole.
Quoted 5 lines
A small note: this type of change makes me believe that there should be a list of testing related caveats or TODO list tracked on the documentation.
Signed-off-by Cleber Rosa <[email protected]>
Is there a way to know where it hangs?

Quoted 9 lines

Also, are these the g_test_quick tests, or the other ones? It it's just a matter of the program taking too long to run, we could just reduce the length of the !quick tests (currently at 5s vs. 1s for the quick ones).
Thanks,
Emilio

Looks like the common cause was fixed a458774ad711bceabefb, as pointed out by Peter.

- Cleber.

Cleber Rosa

author

2018-11-09 15:30 UTC #6 lore raw

RFC PATCH 2/2 Travis CI: break down OSX+clang jobs

In reply to Cleber Rosa

The OSX jobs were timing out on Travis, due to the long time they need to run tests with all targets. Let's break them down to avoid hitting the time limit, and at the same time, give faster results.

Additionally the qtest based tests were hanging intermittently. The first debugging attempt involved making their execution verbose. That alone seemed to produce a positive effect and no hangs were observed after that. Given that Travis checks for output to decide if a test is hung or not, it makes sense that a verbose execution minimizes false positives.

Signed-off-by Cleber Rosa <[email protected]>

---
 .travis.yml | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

.travis.yml +28 -1

index aa49c7b114..505561aae6 100644
@@ -107,7 +107,34 @@ matrix:
     - env: CONFIG="--disable-tcg"
            TEST_CMD=""
       compiler: gcc
-    - env: CONFIG=""
+    # osx+clang jobs are broken in a number of target specific sets to
+    # allow jobs to finish before timing out
+    - env: CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,cris-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=i386-softmmu,lm32-softmmu,m68k-softmmu,moxie-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,riscv32-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=s390x-softmmu,sh4-softmmu,sh4eb-softmmu,sparc-softmmu,sparc64-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=unicore32-softmmu,x86_64-softmmu,xtensa-softmmu,xtensaeb-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=hppa-softmmu,microblaze-softmmu,microblazeel-softmmu,tricore-softmmu,riscv64-softmmu"
+           TEST_CMD="make V=1 check"
       os: osx
       compiler: clang
     # Python builds

Signature · 1 line

2.19.1