Skip to contentSkip to conversation

.gitlab-ci.d/windows: Work-around timeout and OpenGL problems of the MSYS2 jobs

PATCH In review Thomas Huth 6 participants 9 messages Jan 4, 2023 – Jan 7, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 .gitlab-ci.d/windows: Work-around timeout and OpenGL problems of the MSYS2 jobs

The windows jobs (especially the 32-bit job) recently started to hit the timeout limit. Bump it a little bit to ease the situation (80 minutes is quite long already - OTOH, these jobs do not have to wait for a job from the container stage to finish, so this should still be OK).

Additionally, some update on the container side recently enabled OpenGL in these jobs - but the corresponding code fails to compile. Thus disable OpenGL here for the time being until someone figured out the proper fix in the shader code for this.

---
Now that the timeout and OpenGL problems are gone, the 64-bit is working fine for me again. However, I'm still seeing random issues with the 32-bit job ... not sure whether it's a problem on the QEMU side or whether the builders are currently instable, since the issues do not reproduce reliably...

 .gitlab-ci.d/windows.yml | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
.gitlab-ci.d/windows.yml +4 -3
index 9b5c4bcd8a..22f794e537 100644
@@ -10,7 +10,7 @@
       - ${CI_PROJECT_DIR}/msys64/var/cache
   needs: []
   stage: build
-  timeout: 70m
+  timeout: 80m
   before_script:
   - If ( !(Test-Path -Path msys64\var\cache ) ) {
       mkdir msys64\var\cache
@@ -71,7 +71,7 @@ msys2-64bit:
   # for the msys2 64-bit job, due to the build could not complete within
   # the project timeout.
   - ..\msys64\usr\bin\bash -lc '../configure --target-list=x86_64-softmmu
-      --without-default-devices'
+      --without-default-devices --disable-opengl'
   - ..\msys64\usr\bin\bash -lc 'make'
   # qTests don't run successfully with "--without-default-devices",
   # so let's exclude the qtests from CI for now.
@@ -113,6 +113,7 @@ msys2-32bit:
   - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
   - mkdir output
   - cd output
-  - ..\msys64\usr\bin\bash -lc '../configure --target-list=ppc64-softmmu'
+  - ..\msys64\usr\bin\bash -lc '../configure --target-list=ppc64-softmmu
+        --disable-opengl'
   - ..\msys64\usr\bin\bash -lc 'make'
   - ..\msys64\usr\bin\bash -lc 'make check || { cat meson-logs/testlog.txt; exit 1; } ;'
Signature · 1 line
2.31.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 19 lines · Thomas Huth

The windows jobs (especially the 32-bit job) recently started to hit the timeout limit. Bump it a little bit to ease the situation (80 minutes is quite long already - OTOH, these jobs do not have to wait for a job from the container stage to finish, so this should still be OK).

Additionally, some update on the container side recently enabled OpenGL in these jobs - but the corresponding code fails to compile. Thus disable OpenGL here for the time being until someone figured out the proper fix in the shader code for this.

---
Now that the timeout and OpenGL problems are gone, the 64-bit is working fine for me again. However, I'm still seeing random issues with the 32-bit job ... not sure whether it's a problem on the QEMU side or whether the builders are currently instable, since the issues do not reproduce reliably...

Quoted 2 lines
  .gitlab-ci.d/windows.yml | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)
Reviewed-by on v1 1/1
#2 lore raw
In reply to Thomas Huth

Hi

Quoted 11 lines · Thomas Huth

The windows jobs (especially the 32-bit job) recently started to hit the timeout limit. Bump it a little bit to ease the situation (80 minutes is quite long already - OTOH, these jobs do not have to wait for a job from the container stage to finish, so this should still be OK).

Additionally, some update on the container side recently enabled OpenGL in these jobs - but the corresponding code fails to compile. Thus disable OpenGL here for the time being until someone figured out the proper fix in the shader code for this.

It seems msys2 recently enabled egl support, but qemu egl code has not been tested on win32 yet.

I'll take a look. I am adding egl support in fedora mingw as well:
https://src.fedoraproject.org/rpms/mingw-libepoxy/pull-request/3

Quoted 44 lines

---
Now that the timeout and OpenGL problems are gone, the 64-bit is working fine for me again. However, I'm still seeing random issues with the 32-bit job ... not sure whether it's a problem on the QEMU side or whether the builders are currently instable, since the issues do not reproduce reliably...

 .gitlab-ci.d/windows.yml | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
.gitlab-ci.d/windows.yml +4 -4
index 9b5c4bcd8a..22f794e537 100644
@@ -10,7 +10,7 @@
       - ${CI_PROJECT_DIR}/msys64/var/cache
   needs: []
   stage: build
-  timeout: 70m
+  timeout: 80m
   before_script:
   - If ( !(Test-Path -Path msys64\var\cache ) ) {
       mkdir msys64\var\cache
@@ -71,7 +71,7 @@ msys2-64bit:
   # for the msys2 64-bit job, due to the build could not complete within
   # the project timeout.
   - ..\msys64\usr\bin\bash -lc '../configure --target-list=x86_64-softmmu
-      --without-default-devices'
+      --without-default-devices --disable-opengl'
   - ..\msys64\usr\bin\bash -lc 'make'
   # qTests don't run successfully with "--without-default-devices",
   # so let's exclude the qtests from CI for now.
@@ -113,6 +113,7 @@ msys2-32bit:
   - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
   - mkdir output
   - cd output
-  - ..\msys64\usr\bin\bash -lc '../configure --target-list=ppc64-softmmu'
+  - ..\msys64\usr\bin\bash -lc '../configure --target-list=ppc64-softmmu
+        --disable-opengl'
   - ..\msys64\usr\bin\bash -lc 'make'
   - ..\msys64\usr\bin\bash -lc 'make check || { cat meson-logs/testlog.txt; exit 1; } ;'
--
Quoted 3 lines

2.31.1

Signature · 1 line
Marc-André Lureau
#3 lore raw
In reply to Thomas Huth
Quoted 20 lines · Thomas Huth

The windows jobs (especially the 32-bit job) recently started to hit the timeout limit. Bump it a little bit to ease the situation (80 minutes is quite long already - OTOH, these jobs do not have to wait for a job from the container stage to finish, so this should still be OK).

Additionally, some update on the container side recently enabled OpenGL in these jobs - but the corresponding code fails to compile. Thus disable OpenGL here for the time being until someone figured out the proper fix in the shader code for this.

---
Now that the timeout and OpenGL problems are gone, the 64-bit is working fine for me again. However, I'm still seeing random issues with the 32-bit job ... not sure whether it's a problem on the QEMU side or whether the builders are currently instable, since the issues do not reproduce reliably...

Quoted 2 lines
 .gitlab-ci.d/windows.yml | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

Thanks; applied to master on the assumption it will improve the CI situation. I found that the msys2-32bit job still timed out at 1h20, though:

https://gitlab.com/qemu-project/qemu/-/jobs/3555245586

-- PMM

#4 lore raw
Quoted 21 lines · Peter Maydell
Quoted 20 lines · Thomas Huth

The windows jobs (especially the 32-bit job) recently started to hit the timeout limit. Bump it a little bit to ease the situation (80 minutes is quite long already - OTOH, these jobs do not have to wait for a job from the container stage to finish, so this should still be OK).

Additionally, some update on the container side recently enabled OpenGL in these jobs - but the corresponding code fails to compile. Thus disable OpenGL here for the time being until someone figured out the proper fix in the shader code for this.

---
Now that the timeout and OpenGL problems are gone, the 64-bit is working fine for me again. However, I'm still seeing random issues with the 32-bit job ... not sure whether it's a problem on the QEMU side or whether the builders are currently instable, since the issues do not reproduce reliably...

Quoted 8 lines
Quoted 2 lines
  .gitlab-ci.d/windows.yml | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)

Thanks; applied to master on the assumption it will improve the CI situation. I found that the msys2-32bit job still timed out at 1h20, though:

https://gitlab.com/qemu-project/qemu/-/jobs/3555245586

I just gave it a try again, too, and for me, it finished within 65 minutes:

https://gitlab.com/thuth/qemu/-/jobs/3557600268

... let's keep looking for a while, maybe it's ok in most cases now, but if not, we have to consider something else.

Thomas

#5 lore raw
Quoted 29 lines · Thomas Huth
Quoted 28 lines · Peter Maydell
Quoted 20 lines · Thomas Huth

The windows jobs (especially the 32-bit job) recently started to hit the timeout limit. Bump it a little bit to ease the situation (80 minutes is quite long already - OTOH, these jobs do not have to wait for a job from the container stage to finish, so this should still be OK).

Additionally, some update on the container side recently enabled OpenGL in these jobs - but the corresponding code fails to compile. Thus disable OpenGL here for the time being until someone figured out the proper fix in the shader code for this.

---
  Now that the timeout and OpenGL problems are gone, the 64-bit is working fine for me again. However, I'm still seeing random issues with the 32-bit job ... not sure whether it's a problem on the QEMU side or whether the builders are currently instable, since the issues do not reproduce reliably...

Quoted 2 lines

  .gitlab-ci.d/windows.yml | 7 ++++---

  1 file changed, 4 insertions(+), 3 deletions(-)

Thanks; applied to master on the assumption it will improve the CI situation. I found that the msys2-32bit job still timed out at 1h20, though:

Quoted 7 lines
Quoted 1 line

https://gitlab.com/qemu-project/qemu/-/jobs/3555245586

I just gave it a try again, too, and for me, it finished within 65 minutes:

 https://gitlab.com/thuth/qemu/-/jobs/3557600268

... let's keep looking for a while, maybe it's ok in most cases now, but if not, we have to consider something else.

Ok, so after I've been struggling with a failing msys2-32bit job for my new upcoming pull request the last two days (I thought I had a bad patch in there), where I had some problems with the test-hmp and qom-test qtests, I've come up with a new theory after looking at this CI run from the qemu-project staging branch and seeing that these tests are also failing there:

https://gitlab.com/qemu-project/qemu/-/jobs/3558798544
https://gitlab.com/qemu-project/qemu/-/jobs/3560870904

That might also explain the timed-out job that you have seen earlier, Peter, it was likely a hanging qom-test since that seems to be the first test to be executed during the "make check" there.

So the qtests for Windows are definitely not ready for the CI yet (after we've enabled them just in December). I think it's best to disable them there again completely until the issues are understood and fixed.

Thomas

#6 lore raw
Quoted 49 lines · Thomas Huth
Quoted 29 lines · Thomas Huth
Quoted 28 lines · Peter Maydell
Quoted 20 lines · Thomas Huth

The windows jobs (especially the 32-bit job) recently started to hit the timeout limit. Bump it a little bit to ease the situation (80 minutes is quite long already - OTOH, these jobs do not have to wait for a job from the container stage to finish, so this should still be OK).

Additionally, some update on the container side recently enabled OpenGL in these jobs - but the corresponding code fails to compile. Thus disable OpenGL here for the time being until someone figured out the proper fix in the shader code for this.

---
Now that the timeout and OpenGL problems are gone, the 64-bit is working fine for me again. However, I'm still seeing random issues with the 32-bit job ... not sure whether it's a problem on the QEMU side or whether the builders are currently instable, since the issues do not reproduce reliably...

Quoted 2 lines
  .gitlab-ci.d/windows.yml | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)

Thanks; applied to master on the assumption it will improve the CI situation. I found that the msys2-32bit job still timed out at 1h20, though:

Quoted 8 lines
Quoted 1 line

https://gitlab.com/qemu-project/qemu/-/jobs/3555245586

I just gave it a try again, too, and for me, it finished within 65 minutes:

https://gitlab.com/thuth/qemu/-/jobs/3557600268

... let's keep looking for a while, maybe it's ok in most cases now, but if not, we have to consider something else.

Ok, so after I've been struggling with a failing msys2-32bit job for my new upcoming pull request the last two days (I thought I had a bad patch in there), where I had some problems with the test-hmp and qom-test qtests, I've come up with a new theory after looking at this CI run from the qemu-project staging branch and seeing that these tests are also failing there:

https://gitlab.com/qemu-project/qemu/-/jobs/3558798544
https://gitlab.com/qemu-project/qemu/-/jobs/3560870904

Quoted 8 lines

That might also explain the timed-out job that you have seen earlier, Peter, it was likely a hanging qom-test since that seems to be the first test to be executed during the "make check" there.

So the qtests for Windows are definitely not ready for the CI yet (after we've enabled them just in December). I think it's best to disable them there again completely until the issues are understood and fixed.

I cannot reproduce the test failures of both tests (test-hmp and qom-test) with w32 executables. Neither did the w64 executables.

My testing repo is at commit d1852caab131ea898134fdcea8c14bc2ee75fbe9.

Regards,
Bin

#7 lore raw
Quoted 59 lines · Bin Meng
Quoted 49 lines · Thomas Huth
Quoted 29 lines · Thomas Huth
Quoted 28 lines · Peter Maydell
Quoted 20 lines · Thomas Huth

The windows jobs (especially the 32-bit job) recently started to hit the timeout limit. Bump it a little bit to ease the situation (80 minutes is quite long already - OTOH, these jobs do not have to wait for a job from the container stage to finish, so this should still be OK).

Additionally, some update on the container side recently enabled OpenGL in these jobs - but the corresponding code fails to compile. Thus disable OpenGL here for the time being until someone figured out the proper fix in the shader code for this.

---
Now that the timeout and OpenGL problems are gone, the 64-bit is working fine for me again. However, I'm still seeing random issues with the 32-bit job ... not sure whether it's a problem on the QEMU side or whether the builders are currently instable, since the issues do not reproduce reliably...

Quoted 2 lines
   .gitlab-ci.d/windows.yml | 7 ++++---
   1 file changed, 4 insertions(+), 3 deletions(-)

Thanks; applied to master on the assumption it will improve the CI situation. I found that the msys2-32bit job still timed out at 1h20, though:

Quoted 8 lines
Quoted 1 line

https://gitlab.com/qemu-project/qemu/-/jobs/3555245586

I just gave it a try again, too, and for me, it finished within 65 minutes:

   https://gitlab.com/thuth/qemu/-/jobs/3557600268

... let's keep looking for a while, maybe it's ok in most cases now, but if not, we have to consider something else.

Ok, so after I've been struggling with a failing msys2-32bit job for my new upcoming pull request the last two days (I thought I had a bad patch in there), where I had some problems with the test-hmp and qom-test qtests, I've come up with a new theory after looking at this CI run from the qemu-project staging branch and seeing that these tests are also failing there:

   https://gitlab.com/qemu-project/qemu/-/jobs/3558798544
   https://gitlab.com/qemu-project/qemu/-/jobs/3560870904
Quoted 8 lines

That might also explain the timed-out job that you have seen earlier, Peter, it was likely a hanging qom-test since that seems to be the first test to be executed during the "make check" there.

So the qtests for Windows are definitely not ready for the CI yet (after we've enabled them just in December). I think it's best to disable them there again completely until the issues are understood and fixed.

Quoted 4 lines

I cannot reproduce the test failures of both tests (test-hmp and qom-test) with w32 executables. Neither did the w64 executables.

My testing repo is at commit d1852caab131ea898134fdcea8c14bc2ee75fbe9.

Can you at least reproduce it in the Gitlab-CI? ... it also does not always occur, sometimes the jobs are working fine. I suspect it's some kind of race or memory problem ... is there something similar to "Valgrind" on Windows? If so, could you try to run those qtests there with such tooling enabled?

Thomas

#8 lore raw
In reply to Thomas Huth

Am 04.01.23 um 13:35 schrieb Thomas Huth:

Quoted 10 lines

The windows jobs (especially the 32-bit job) recently started to hit the timeout limit. Bump it a little bit to ease the situation (80 minutes is quite long already - OTOH, these jobs do not have to wait for a job from the container stage to finish, so this should still be OK).

Additionally, some update on the container side recently enabled OpenGL in these jobs - but the corresponding code fails to compile. Thus disable OpenGL here for the time being until someone figured out the proper fix in the shader code for this.

This is strange. On my Windows msys2 system, I didn't even notice the OpenGL code was silently enabled. The code compiles without issues. Today I enabled the GtkGLArea code initialization in ui/gtk.c to test OpenGL acceleration on Windows.

Quoted 5 lines
ui/gtk.c +0 -0
@@ -2435,6 +2435,12 @@ static void early_gtk_display_init(DisplayOptions *opts)
              gtk_use_gl_area = true;
              gtk_gl_area_init();

ui/gtk.c static void early_gtk_display_init(DisplayOptions *opts) · line 2446

Quoted 10 lines
ui/gtk.c +6 -0
          } else
+#endif
+#if defined(GDK_WINDOWING_WIN32)
+        if (GDK_IS_WIN32_DISPLAY(gdk_display_get_default())) {
+            gtk_use_gl_area = true;
+            gtk_gl_area_init();
+        } else
 #endif
          {
 #ifdef CONFIG_X11

Well, it's a start. On a Linux guest system the WebGL Aquarium frame rate increased from 6fps to 14fps while the host processor load went down from 100% to 65%.

QEMU was started with:
./qemu-system-x86_64.exe -accel whpx \
-machine pc,usb=off,vmport=off,kernel-irqchip=off \
-cpu Skylake-Client-v4,tsc-deadline=off,x2apic=off \
-smp 4,sockets=1,cores=4,threads=1 \
-device virtio-vga-gl,xres=1280,yres=768,bus=pci.0 \
-display gtk,zoom-to-fit=off,gl=on \
-trace "gd_gl_area_*_context" \
...

This is the start of the QEMU log file:
Windows Hypervisor Platform accelerator is operational
qemu: GtkGLArea console lacks DMABUF support.
Realize gdk gl context failed: Unable to create a GL context Realize gdk gl context failed: Unable to create a GL context gd_gl_area_create_context ctx=000002934703bc10, major=4, minor=4 gl_version 44 - core profile enabled
gd_gl_area_destroy_context ctx=000002934703bc10, current_ctx=000002934703bc10 gd_gl_area_create_context ctx=000002934703bc10, major=4, minor=4 gd_gl_area_create_context ctx=000002934703bba0, major=4, minor=4 gd_gl_area_create_context ctx=000002934703bb30, major=4, minor=4 GLSL feature level 440
gd_gl_area_create_context ctx=000002934703b890, major=4, minor=4 gd_gl_area_create_context ctx=000002934703bc80, major=4, minor=4 gd_gl_area_create_context ctx=000002934703bcf0, major=4, minor=4 gd_gl_area_destroy_context ctx=000002934703bcf0, current_ctx=000002934703bcf0
...

With best regards,
Volker

Quoted 2 lines

---